HPCC-30781 Introduce format registry for pluggable file formats #18334
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
https://track.hpccsystems.com/browse/HPCC-30781 |
jackdelv
approved these changes
Feb 26, 2024
ghalliday
reviewed
Feb 27, 2024
system/jlib/jfile.hpp
Outdated
| @@ -788,4 +788,9 @@ extern jlib_decl IPropertyTreeIterator * getPlanesIterator(const char * category | |||
| extern jlib_decl IFileIO *createBlockedIO(IFileIO *base, size32_t blockSize); | |||
| extern jlib_decl size32_t getBlockedFileIOSize(const char *planeName, size32_t defaultSize=0); | |||
|
|
|||
| //---- Pluggable file type related functions ---------------------------------------------- | |||
|
|
|||
| void addAvailableGenericFileTypeName(const char * name); | |||
There was a problem hiding this comment.
Needs jlib_decl to ensure the symbol is visible outside the dll/so. (extern also generally added for clarify - see above).
There was a problem hiding this comment.
Declarations added.
common/thorhelper/thorread.cpp
Outdated
|
|
||
| MODULE_EXIT() | ||
| { | ||
| genericFileTypeMap.clear(); |
There was a problem hiding this comment.
trivial - not needed, will be destroyed when dll is unloaded
| // should be defined here; the key is the lowecase name of the format, | ||
| // as will be used in ECL, and the value should be a lambda | ||
| // that creates the appropriate disk row reader object | ||
| genericFileTypeMap.emplace("flat", [](IDiskReadMapping * _mapping) { return new BinaryDiskRowReader(_mapping); }); |
There was a problem hiding this comment.
In the future I suspect this will change to registering factory class instances that have createReader()/writer methods, but this is good for now.
|
@ghalliday I made some small changes based on your feedback. Please give it a once-over. Thanks! |
ghalliday
approved these changes
Feb 27, 2024
dcamper
force-pushed
the
hpcc-30781-filetype-format-registry
branch
from
9014551 to
d343b9e
Compare
|
@ghalliday Squashed and ready to merge. Thanks! |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
The goal of this PR is to define an easy-to-maintain interface between code that supports new file format types and the rest of the platform, at both compile and execution time.
Type of change:
Checklist:
Smoketest:
Testing:
The compiler successfully generates local a.out binaries (hthor format). It does crash when attempting to read Parquet files, as that code does not yet correctly extract the file name/path.