Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

I/O Customizaton Rules: add support for changing the input type. #17347

Merged
merged 30 commits into from
Jan 19, 2025
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
30 commits
Select commit Hold shift + click to select a range
8682080
meta: Correct RemoveStreamerInfo for slot != current
pcanal Dec 6, 2024
3ab3d22
io: Fix in-memory class of cached members.
pcanal Dec 6, 2024
25b7f40
meta: Add clarifying alias TSources::GetTypeName
pcanal Dec 12, 2024
6c2a653
meta: TClass::GetRealData better search for array data member.
pcanal Dec 12, 2024
29298f6
meta: slight optimization in InsertArtificialElements
pcanal Dec 12, 2024
58cc31b
TTreeReader: TBranchProxy handle cases of split objects.
pcanal Dec 16, 2024
8588ce8
meta: GetDataMemberOffset return kMissing for errors.
pcanal Dec 12, 2024
3d37332
io: Enable implicit conversion from A* to B*
pcanal Dec 12, 2024
75ace06
io: Enable implicit conversion from A[] to B[]
pcanal Dec 12, 2024
aec62b1
meta: Improve TSources interfaces.
pcanal Dec 12, 2024
fe0e774
Revert "Don't warn when we skip a rule due to missing source member."
pcanal Dec 12, 2024
9211b5b
io: update element with rule input types.
pcanal Dec 12, 2024
445b919
io: Correct the code skipping array of objects.
pcanal Dec 17, 2024
57a90db
io: Update StreamerElement as requested by rule's input
pcanal Jan 2, 2025
f844d92
io: error out for rule with conflicting type for sources
pcanal Dec 17, 2024
09f6518
io: When rule change in-memory type, update size
pcanal Jan 2, 2025
37c860f
io-rule: don't complain if input is in base class
pcanal Jan 3, 2025
e54b0ea
io: improve dim error message in UpdateFromRule
pcanal Jan 3, 2025
bcd9230
[NFC] Fix indentation
pcanal Jan 19, 2025
982dada
io: Base sizeof TStreamerElement on the in-memory type.
pcanal Jan 2, 2025
efc25b3
io: Allocated/Destroy in StreamerInfo give the in-memory type
pcanal Jan 2, 2025
60a2a1a
io: Announce attribute for I/O customization rules: `CanIgnore`
pcanal Dec 27, 2024
f9cc521
[NFC] io: remove spurrious cast
pcanal Dec 27, 2024
56fdd13
io: Correct validation test in add rule to set
pcanal Jan 9, 2025
963e17b
io: Actual check in StreamerInfo for rule validation
pcanal Jan 9, 2025
8fae3ce
io-meta: allow registration of rules to classes from another dict.
pcanal Jan 11, 2025
915cf10
io: TStreamerInfo::kObject don't use virtual function if class changed
pcanal Jan 13, 2025
10efc69
io: ReadClassEmulated actually use ConversionStreamerInfo
pcanal Jan 13, 2025
197d3ba
io: for rule input don't use virtual function if class changed
pcanal Jan 13, 2025
1965fb4
[NFC] clang-format white space
pcanal Jan 19, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions README/ReleaseNotes/v636/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -64,6 +64,7 @@ The following people have contributed to this new version:
[RNTupleMergeOptions](https://root.cern/doc/v634/structROOT_1_1Experimental_1_1Internal_1_1RNTupleMergeOptions.html));
* "rntuple.ErrBehavior=(Abort|Skip)": RNTuple-specific option that specifies the behavior of the RNTupleMerger on error (see link above);
* "rntuple.ExtraVerbose": RNTuple-specific option that tells the RNTupleMerger to emit more information during the merge process.
* New attribute for I/O customization rules: `CanIgnore`. When using this attribute the rule will be ignored if the input is missing from the schema/class-layout they apply to instead of issue a `Warning`

## RDataFrame

Expand Down
8 changes: 8 additions & 0 deletions core/clingutils/res/TClingUtils.h
Original file line number Diff line number Diff line change
Expand Up @@ -567,6 +567,14 @@ void WriteClassInit(std::ostream& finalString,
const RConstructorTypes& ctorTypes,
bool& needCollectionProxy);

//______________________________________________________________________________
void WriteStandaloneReadRules(std::ostream &finalString, bool rawrules, std::vector<std::string> &standaloneTargets,
const cling::Interpreter &interp);

//______________________________________________________________________________
void WriteRulesRegistration(std::ostream &finalString, const std::string &dictName,
const std::vector<std::string> &standaloneTargets);

//______________________________________________________________________________
bool HasCustomStreamerMemberFunction(const AnnotatedRecordDecl &cl,
const clang::CXXRecordDecl* clxx,
Expand Down
127 changes: 119 additions & 8 deletions core/clingutils/src/TClingUtils.cxx
Original file line number Diff line number Diff line change
Expand Up @@ -1822,8 +1822,8 @@ void ROOT::TMetaUtils::WriteClassInit(std::ostream& finalString,
if( rulesIt1 != ROOT::gReadRules.end() ) {
int i = 0;
finalString << "\n // Schema evolution read functions\n";
std::list<ROOT::SchemaRuleMap_t>::iterator rIt = rulesIt1->second.begin();
while( rIt != rulesIt1->second.end() ) {
std::list<ROOT::SchemaRuleMap_t>::iterator rIt = rulesIt1->second.fRules.begin();
while (rIt != rulesIt1->second.fRules.end()) {

//--------------------------------------------------------------------
// Check if the rules refer to valid data members
Expand All @@ -1832,7 +1832,7 @@ void ROOT::TMetaUtils::WriteClassInit(std::ostream& finalString,
std::string error_string;
if( !HasValidDataMembers( *rIt, nameTypeMap, error_string ) ) {
Warning(nullptr, "%s", error_string.c_str());
rIt = rulesIt1->second.erase(rIt);
rIt = rulesIt1->second.fRules.erase(rIt);
continue;
}

Expand All @@ -1857,8 +1857,8 @@ void ROOT::TMetaUtils::WriteClassInit(std::ostream& finalString,
if( rulesIt2 != ROOT::gReadRawRules.end() ) {
int i = 0;
finalString << "\n // Schema evolution read raw functions\n";
std::list<ROOT::SchemaRuleMap_t>::iterator rIt = rulesIt2->second.begin();
while( rIt != rulesIt2->second.end() ) {
std::list<ROOT::SchemaRuleMap_t>::iterator rIt = rulesIt2->second.fRules.begin();
while (rIt != rulesIt2->second.fRules.end()) {

//--------------------------------------------------------------------
// Check if the rules refer to valid data members
Expand All @@ -1867,7 +1867,7 @@ void ROOT::TMetaUtils::WriteClassInit(std::ostream& finalString,
std::string error_string;
if( !HasValidDataMembers( *rIt, nameTypeMap, error_string ) ) {
Warning(nullptr, "%s", error_string.c_str());
rIt = rulesIt2->second.erase(rIt);
rIt = rulesIt2->second.fRules.erase(rIt);
continue;
}

Expand Down Expand Up @@ -2052,14 +2052,16 @@ void ROOT::TMetaUtils::WriteClassInit(std::ostream& finalString,

if( rulesIt1 != ROOT::gReadRules.end() ) {
finalString << "\n" << " // the io read rules" << "\n" << " std::vector<::ROOT::Internal::TSchemaHelper> readrules(" << rulesIt1->second.size() << ");" << "\n";
ROOT::WriteSchemaList( rulesIt1->second, "readrules", finalString );
ROOT::WriteSchemaList(rulesIt1->second.fRules, "readrules", finalString);
finalString << " instance.SetReadRules( readrules );" << "\n";
rulesIt1->second.fGenerated = true;
}

if( rulesIt2 != ROOT::gReadRawRules.end() ) {
finalString << "\n" << " // the io read raw rules" << "\n" << " std::vector<::ROOT::Internal::TSchemaHelper> readrawrules(" << rulesIt2->second.size() << ");" << "\n";
ROOT::WriteSchemaList( rulesIt2->second, "readrawrules", finalString );
ROOT::WriteSchemaList(rulesIt2->second.fRules, "readrawrules", finalString);
finalString << " instance.SetReadRawRules( readrawrules );" << "\n";
rulesIt2->second.fGenerated = true;
}

finalString << " return &instance;" << "\n" << " }" << "\n";
Expand Down Expand Up @@ -2189,6 +2191,115 @@ void ROOT::TMetaUtils::WriteClassInit(std::ostream& finalString,
finalString << "} // end of namespace ROOT" << "\n" << "\n";
}

void ROOT::TMetaUtils::WriteStandaloneReadRules(std::ostream &finalString, bool rawrules,
std::vector<std::string> &standaloneTargets,
const cling::Interpreter &interp)
{
for (auto &rulesIt1 : rawrules ? ROOT::gReadRawRules : ROOT::gReadRules) {
if (!rulesIt1.second.fGenerated) {
const clang::Type *typeptr = nullptr;
const clang::CXXRecordDecl *target =
ROOT::TMetaUtils::ScopeSearch(rulesIt1.first.c_str(), interp, true /*diag*/, &typeptr);

if (!target && !rulesIt1.second.fTargetDecl) {
ROOT::TMetaUtils::Warning(nullptr, "%d Rule(s) for target class %s was not used!\n", rulesIt1.second.size(),
rulesIt1.first.c_str());
continue;
}

ROOT::MembersTypeMap_t nameTypeMap;
CreateNameTypeMap(*target, nameTypeMap);

std::string name;
TClassEdit::GetNormalizedName(name, rulesIt1.first);

std::string mappedname;
ROOT::TMetaUtils::GetCppName(mappedname, name.c_str());

finalString << "namespace ROOT {" << "\n";
// Also TClingUtils.cxx:1823
int i = 0;
finalString << "\n // Schema evolution read functions\n";
std::list<ROOT::SchemaRuleMap_t>::iterator rIt = rulesIt1.second.fRules.begin();
while (rIt != rulesIt1.second.fRules.end()) {

//--------------------------------------------------------------------
// Check if the rules refer to valid data members
///////////////////////////////////////////////////////////////////////

std::string error_string;
if (!HasValidDataMembers(*rIt, nameTypeMap, error_string)) {
ROOT::TMetaUtils::Warning(nullptr, "%s", error_string.c_str());
rIt = rulesIt1.second.fRules.erase(rIt);
continue;
}

//---------------------------------------------------------------------
// Write the conversion function if necessary
///////////////////////////////////////////////////////////////////////

if (rIt->find("code") != rIt->end()) {
if (rawrules)
WriteReadRawRuleFunc(*rIt, i++, mappedname, nameTypeMap, finalString);
else
WriteReadRuleFunc(*rIt, i++, mappedname, nameTypeMap, finalString);
}
++rIt;
}
finalString << "} // namespace ROOT" << "\n";

standaloneTargets.push_back(rulesIt1.first);
rulesIt1.second.fGenerated = true;
}
}
}

void ROOT::TMetaUtils::WriteRulesRegistration(std::ostream &finalString, const std::string &dictName,
const std::vector<std::string> &standaloneTargets)
{
std::string functionname("RecordReadRules_");
functionname += dictName;

finalString << "namespace ROOT {" << "\n";
finalString << " // Registration Schema evolution read functions\n";
finalString << " int " << functionname << "() {" << "\n";
if (!standaloneTargets.empty())
finalString << "\n"
<< " ::ROOT::Internal::TSchemaHelper* rule;" << "\n";
for (const auto &target : standaloneTargets) {
std::string name;
TClassEdit::GetNormalizedName(name, target);

ROOT::SchemaRuleClassMap_t::iterator rulesIt1 = ROOT::gReadRules.find(target.c_str());
finalString << " {\n";
if (rulesIt1 != ROOT::gReadRules.end()) {
finalString << " // the io read rules for " << target << "\n";
finalString << " std::vector<::ROOT::Internal::TSchemaHelper> readrules(" << rulesIt1->second.size()
<< ");" << "\n";
ROOT::WriteSchemaList(rulesIt1->second.fRules, "readrules", finalString);
finalString << " TClass::RegisterReadRules(TSchemaRule::kReadRule, \"" << name
<< "\", std::move(readrules));\n";
rulesIt1->second.fGenerated = true;
}
ROOT::SchemaRuleClassMap_t::iterator rulesIt2 = ROOT::gReadRawRules.find(target.c_str());
if (rulesIt2 != ROOT::gReadRawRules.end()) {
finalString << "\n // the io read raw rules for " << target << "\n";
finalString << " std::vector<::ROOT::Internal::TSchemaHelper> readrawrules(" << rulesIt2->second.size()
<< ");" << "\n";
ROOT::WriteSchemaList(rulesIt2->second.fRules, "readrawrules", finalString);
finalString << " TClass::RegisterReadRules(TSchemaRule::kReadRawRule, \"" << name
<< "\", std::move(readrawrules));\n";
rulesIt2->second.fGenerated = true;
}
finalString << " }\n";
}
finalString << " return 0;\n";
finalString << " }\n";
finalString << " static int _R__UNIQUE_DICT_(ReadRules_" << dictName << ") = " << functionname << "();";
finalString << "R__UseDummy(_R__UNIQUE_DICT_(ReadRules_" << dictName << "));" << "\n";
finalString << "} // namespace ROOT" << "\n";
}

////////////////////////////////////////////////////////////////////////////////
/// Return true if one of the class' enclosing scope is a namespace and
/// set fullname to the fully qualified name,
Expand Down
26 changes: 10 additions & 16 deletions core/dictgen/src/rootcling_impl.cxx
Original file line number Diff line number Diff line change
Expand Up @@ -2649,14 +2649,9 @@ int FinalizeStreamerInfoWriting(cling::Interpreter &interp, bool writeEmptyRootP

////////////////////////////////////////////////////////////////////////////////

int GenerateFullDict(std::ostream &dictStream,
cling::Interpreter &interp,
RScanner &scan,
const ROOT::TMetaUtils::RConstructorTypes &ctorTypes,
bool isSplit,
bool isGenreflex,
bool isSelXML,
bool writeEmptyRootPCM)
int GenerateFullDict(std::ostream &dictStream, std::string dictName, cling::Interpreter &interp, RScanner &scan,
const ROOT::TMetaUtils::RConstructorTypes &ctorTypes, bool isSplit, bool isGenreflex,
bool isSelXML, bool writeEmptyRootPCM)
{
ROOT::TMetaUtils::TNormalizedCtxt normCtxt(interp.getLookupHelper());

Expand Down Expand Up @@ -2795,6 +2790,11 @@ int GenerateFullDict(std::ostream &dictStream,
EmitStreamerInfo);
}

std::vector<std::string> standaloneTargets;
ROOT::TMetaUtils::WriteStandaloneReadRules(dictStream, false, standaloneTargets, interp);
ROOT::TMetaUtils::WriteStandaloneReadRules(dictStream, true, standaloneTargets, interp);
ROOT::TMetaUtils::WriteRulesRegistration(dictStream, dictName, standaloneTargets);

if (!gDriverConfig->fBuildingROOTStage1) {
EmitTypedefs(scan.fSelectedTypedefs);
EmitEnums(scan.fSelectedEnums);
Expand Down Expand Up @@ -4961,14 +4961,8 @@ int RootClingMain(int argc,
rootclingRetCode += FinalizeStreamerInfoWriting(interp);
}
} else {
rootclingRetCode += GenerateFullDict(*splitDictStream,
interp,
scan,
constructorTypes,
gOptSplit,
isGenreflex,
isSelXML,
gOptWriteEmptyRootPCM);
rootclingRetCode += GenerateFullDict(*splitDictStream, modGen.GetDictionaryName(), interp, scan, constructorTypes,
gOptSplit, isGenreflex, isSelXML, gOptWriteEmptyRootPCM);
}

if (rootclingRetCode != 0) {
Expand Down
13 changes: 12 additions & 1 deletion core/foundation/res/RConversionRuleParser.h
Original file line number Diff line number Diff line change
Expand Up @@ -14,13 +14,24 @@
#include "TSchemaType.h"
#include "DllImport.h"

namespace clang {
class CXXRecordDecl;
}

namespace ROOT
{
//---------------------------------------------------------------------------
// Global variables
//---------------------------------------------------------------------------
typedef std::map<std::string, std::string> SchemaRuleMap_t;
typedef std::map<std::string, std::list<SchemaRuleMap_t> > SchemaRuleClassMap_t;
struct RRulesList {
bool fGenerated = false;
std::list<SchemaRuleMap_t> fRules;
const clang::CXXRecordDecl *fTargetDecl;

size_t size() const { return fRules.size(); }
};
typedef std::map<std::string, RRulesList> SchemaRuleClassMap_t;
R__EXTERN SchemaRuleClassMap_t gReadRules;
R__EXTERN SchemaRuleClassMap_t gReadRawRules;

Expand Down
12 changes: 6 additions & 6 deletions core/foundation/src/RConversionRuleParser.cxx
Original file line number Diff line number Diff line change
Expand Up @@ -864,7 +864,7 @@ namespace ROOT
//////////////////////////////////////////////////////////////////////////

for( it = gReadRules.begin(); it != gReadRules.end(); ++it ) {
for( rule = it->second.begin(); rule != it->second.end(); ++rule ) {
for (rule = it->second.fRules.begin(); rule != it->second.fRules.end(); ++rule) {
attr = rule->find( "include" );
if( attr == rule->end() ) continue;
TSchemaRuleProcessor::SplitList( attr->second, tmp );
Expand All @@ -877,7 +877,7 @@ namespace ROOT
//////////////////////////////////////////////////////////////////////////

for( it = gReadRawRules.begin(); it != gReadRawRules.end(); ++it ) {
for( rule = it->second.begin(); rule != it->second.end(); ++rule ) {
for (rule = it->second.fRules.begin(); rule != it->second.fRules.end(); ++rule) {
attr = rule->find( "include" );
if( attr == rule->end() ) continue;
TSchemaRuleProcessor::SplitList( attr->second, tmp );
Expand Down Expand Up @@ -923,10 +923,10 @@ namespace ROOT
if( it == gReadRules.end() ) {
std::list<SchemaRuleMap_t> lst;
lst.push_back( rule );
gReadRules[normalizedTargetName] = lst;
gReadRules[normalizedTargetName].fRules = lst;
}
else
it->second.push_back( rule );
it->second.fRules.push_back(rule);
}

/////////////////////////////////////////////////////////////////////////////
Expand Down Expand Up @@ -958,10 +958,10 @@ namespace ROOT
if( it == gReadRawRules.end() ) {
std::list<SchemaRuleMap_t> lst;
lst.push_back( rule );
gReadRawRules[normalizedTargetName] = lst;
gReadRawRules[normalizedTargetName].fRules = lst;
}
else
it->second.push_back( rule );
it->second.fRules.push_back(rule);
}


Expand Down
8 changes: 8 additions & 0 deletions core/meta/inc/TClass.h
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,7 @@

#include "TDictionary.h"
#include "TString.h"
#include "TSchemaRule.h"

#ifdef R__LESS_INCLUDES
class TObjArray;
Expand All @@ -36,6 +37,7 @@ class TObjArray;
#include <cstddef>
#include <map>
#include <string>
#include <unordered_map>
#include <unordered_set>
#include <vector>
#include <atomic>
Expand Down Expand Up @@ -72,6 +74,7 @@ namespace ROOT {
}
namespace Internal {
class TCheckHashRecursiveRemoveConsistency;
struct TSchemaHelper;
}
}

Expand Down Expand Up @@ -361,6 +364,9 @@ friend class TStreamerInfo;
void GetMissingDictionariesWithRecursionCheck(TCollection &result, TCollection &visited, bool recurse);
void GetMissingDictionariesForPairElements(TCollection &result, TCollection &visited, bool recurse);

using SchemaHelperMap_t = std::unordered_map<std::string, std::vector<ROOT::Internal::TSchemaHelper>>;
static SchemaHelperMap_t &GetReadRulesRegistry(ROOT::TSchemaRule::RuleType_t type);

public:
TClass();
TClass(const char *name, Bool_t silent = kFALSE);
Expand Down Expand Up @@ -539,6 +545,8 @@ friend class TStreamerInfo;
Long_t Property() const override;
Int_t ReadBuffer(TBuffer &b, void *pointer, Int_t version, UInt_t start, UInt_t count);
Int_t ReadBuffer(TBuffer &b, void *pointer);
static void RegisterReadRules(ROOT::TSchemaRule::RuleType_t, const char *classname,
std::vector<::ROOT::Internal::TSchemaHelper> &&rules);
void RegisterStreamerInfo(TVirtualStreamerInfo *info);
void RemoveStreamerInfo(Int_t slot);
void ReplaceWith(TClass *newcl) const;
Expand Down
Loading
Loading