Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feature/gribjump plugin #28

Merged
merged 5 commits into from
Jul 17, 2024
Merged

Feature/gribjump plugin #28

merged 5 commits into from
Jul 17, 2024

Conversation

ChrisspyB
Copy link
Member

@ChrisspyB ChrisspyB commented Jul 15, 2024

Changes:

  • Modify ArchiveCallback signature to also take void* data and length
  • Add registration of FlushCallbacks.
  • Add initPlugins method which is called by FDB constructor, which calls checks if gribjump plugin is enabled and passes this to gribjump if so.

Questions:

  • Should gribjump explicitly check data passed to callback is GRIB (by reading first four bytes?). I think yes.

@ChrisspyB
Copy link
Member Author

NB In case I forget: requires new setup method in eckit::system::plugin to be merged into eckit.

All tests will fail to build until then.

src/fdb5/api/FDB.h Outdated Show resolved Hide resolved
@@ -155,6 +158,7 @@ class FDB {

FDBStats stats_;

FlushCallback flushCallback_ = CALLBACK_FLUSH_NOOP;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In constructor:

flushCallback_ = LibFDB::instance().defaultFlushCallback();

}

void FDB::initPlugins(const Config& config){
bool enableGribjump = eckit::Resource<bool>("fdbEnableGribjump;$FDB_ENABLE_GRIBJUMP", false);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I do not want to see the words gribJump in the FDB (or really even the word GRIB)

src/fdb5/api/FDB.h Outdated Show resolved Hide resolved
src/fdb5/api/helpers/ArchiveCallback.h Show resolved Hide resolved
@@ -20,8 +20,10 @@

namespace fdb5 {

using ArchiveCallback = std::function<void(const Key&, const FieldLocation&)>;
using ArchiveCallback = std::function<void(const Key& key, const void* data, size_t length, const FieldLocation&)>;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pass the data and a promise
Note in comment that the callback must return before the FieldLocation will be satisfied (so promise.get() will need to be called in a thread/async/something)

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Passing a future rather than the promise (promise allows you to set the value, future allows you to get the value. Callback should only get the value).

callback must return before the FieldLocation will be satisfied

Personally, I think it would be better if the callback did not have to implemented its own thread (though we do for gribjump): the important thing is that promse.set_value(location) is called before calling the callback (when using a single-threaded fdb backend), or the thread which sets the value has been started (when using remote fdb) before calling the callback.

@codecov-commenter
Copy link

Codecov Report

Attention: Patch coverage is 90.32258% with 3 lines in your changes missing coverage. Please review.

Project coverage is 63.52%. Comparing base (e4743c2) to head (74b1591).
Report is 2 commits behind head on develop.

Files Patch % Lines
src/fdb5/LibFdb5.cc 40.00% 3 Missing ⚠️
Additional details and impacted files
@@             Coverage Diff             @@
##           develop      #28      +/-   ##
===========================================
+ Coverage    63.49%   63.52%   +0.02%     
===========================================
  Files          236      236              
  Lines        13391    13410      +19     
  Branches      1291     1291              
===========================================
+ Hits          8503     8519      +16     
- Misses        4888     4891       +3     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@simondsmart
Copy link
Contributor

Looks great. Let's do it.

@simondsmart simondsmart marked this pull request as ready for review July 17, 2024 10:23
@simondsmart simondsmart merged commit c2fbb55 into develop Jul 17, 2024
109 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants