[ENH] Add derivatives command and `pipeline-catalog` submodule #349

alyssadai · 2024-09-27T20:04:59Z

Closes Investigate schema and examples for the Nipoppy imaging bagel #344
Closes Add new sub-command for bagel-CLI derivatives: #345
See also [ENH] Update JSONLD examples with @context and outputs from bagel derivatives neurobagel_examples#34
- should be merged first, currently causing test failures due to examples in this repo being outdated (old @context)

Changes proposed in this pull request:

Add nipoppy/pipeline-catalog as submodule
Create new bagel derivatives command
- Add basic input checks of processing status TSV: no missing subject IDs, no new (unique) subjects, only recognized pipeline names + vers
- Add command logic to extract completed pipelines per subject-session based on step statuses
- Aggregate valid pipeline info for missing sessions for a subject under a new custom ses-nb01 ImagingSession
Move I/O utils shared between commands into a new module file_utils.py to avoid circular imports
Add new derivative_utils.py to hold helper funcs for bagel derivatives
Add new example processing status TSVs (7 total)
Switch to re-generating the @context in each command (previously, we were stripping it from an input JSONLD and re-adding it at the end. This led to sometimes ending up with an incomplete @context without all the terms used/added to the dataset by the command that was just run)

Checklist

This section is for the PR reviewer

PR has an interpretable title with a prefix ([ENH], [FIX], [REF], [TST], [CI], [MNT], [INF], [MODEL], [DOC]) (see our Contributing Guidelines for more info)
PR has a label for the release changelog or skip-release (to be applied by maintainers only)
PR links to GitHub issue with mention Closes #XXXX
Tests pass
Checks pass

For new features:

Tests have been added

For bug fixes:

There is at least one test that would fail under the original bug conditions.

…dict

- participant_id col of existing examples also updated to match Nipoppy

- prevents circular import errors since some utils require models.py, which in turn requires IO utils

codecov · 2024-09-27T20:07:36Z

Codecov Report

Attention: Patch coverage is 99.24242% with 2 lines in your changes missing coverage. Please review.

Project coverage is 98.34%. Comparing base (5d15125) to head (5af6a6d).

Files with missing lines	Patch %	Lines
bagel/utility.py	94.73%	2 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #349      +/-   ##
==========================================
+ Coverage   98.08%   98.34%   +0.26%     
==========================================
  Files          13       16       +3     
  Lines         782      966     +184     
==========================================
+ Hits          767      950     +183     
- Misses         15       16       +1

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

…el/bagelbids into add-derivatives-command

surchs

Thanks @alyssadai for this very big PR and for setting up a brand new subcommand!
I really like the extensive testing of the new functionalities.

That said, I have a couple of comments on the code itself that I think make sense to address before merging. Everything runs, so I'll let you take a look at them and tell me when you're ready to merge before approving.

As I mentioned today, I don't know yet how to better break down or review such a big feature, but we should find a more efficient way - this takes a lot of uninterrupted concentration to review well and thus a lot of time.

bagel/cli.py

bagel/tests/test_utility.py

bagel/tests/test_cli_derivatives.py

surchs

Thanks very much @alyssadai, looks great.

🧑‍🍳

alyssadai · 2024-10-03T18:31:57Z

Keeping this PR open for now to merge branch for #346 back into this one

alyssadai added 18 commits September 26, 2024 14:24

add examples of nipoppy proc status files

9d6bbbc

use actual allowed pipelines + vers in example TSVs

b723c57

add skeleton derivatives command

714755b

add basic smoke test of derivatives cmd

e4aa01a

load TSV and check for missing IDs in derivatives cmd

2dc72d8

generalize util to load a tabular file

2f274a4

Test added for load_tabular

41b5d58

Added nipoppy pipeline catalogue as submodule

3d762a6

Add loading of pipeline names and versions

666be05

validate pipeline names & versions and store expected col names in a …

4d552e3

…dict

update help text & docstrings

783b57f

refactor unique subject check to generic util

5f09328

refactor pipeline name / version validation

611cd58

add example proc status file w/ subjects not in the synthetic dataset

826ee2a

- participant_id col of existing examples also updated to match Nipoppy

check that proc status subs are in pheno-containing JSONLD

1699a6b

refactor out jsonld validation & move IO utils into new module

42b96e7

- prevents circular import errors since some utils require models.py, which in turn requires IO utils

switch to typer echo statement for model validation error

0b7f8c5

factor out context extraction

58b38e9

alyssadai added 11 commits September 27, 2024 18:51

remove space from pipeline URIs

166aaf1

use fixture for output path in tests

7c3faed

add logic to add completed pipelines to existing or new imaging sessions

25f9ba5

create utility for extracting imaging sessions from a JSONLD

78f1082

create util for creating completed pipelines

e694e8f

update example proc status files for readability

f400d62

handle missing BIDS sessions

f0d50fd

refine smoke test and add test using pheno-bids JSONLD

26bdf13

update test data README

71932ab

refactor out custom session ID

aae4b39

switch to using 3.9 syntax

8e95027

alyssadai marked this pull request as ready for review October 1, 2024 16:31

alyssadai added 6 commits October 1, 2024 12:31

Merge branch 'main' into add-derivatives-command

0e30b90

create list of namespaces & update tests to catch outdated @context

fd80223

regenerate context in each cmd to ensure they are up-to-date

092a934

add short option for overwrite to error msg

dd632c8

Merge branch 'main' into add-derivatives-command

52f0c56

Merge branch 'add-derivatives-command' of https://github.com/neurobag…

6abd911

…el/bagelbids into add-derivatives-command

alyssadai mentioned this pull request Oct 2, 2024

[ENH] Update JSONLD examples with @context and outputs from bagel derivatives neurobagel/neurobagel_examples#34

Merged

6 tasks

alyssadai requested a review from surchs October 2, 2024 16:01

surchs reviewed Oct 2, 2024

View reviewed changes

alyssadai added 2 commits October 2, 2024 16:21

update comments

6d18c54

update test data README

b239cc9

alyssadai mentioned this pull request Oct 2, 2024

update bagel bids subcommand to play nice with new derivatives #346

Open

4 tasks

alyssadai added 2 commits October 2, 2024 17:03

store known pipelines/versions in local var

e33e7db

rename funcs and vars for clarity

a0fe97d

alyssadai mentioned this pull request Oct 3, 2024

Refactor utility modules and test_utility.py #350

Open

6 tasks

alyssadai added 9 commits October 2, 2024 23:40

handle jsonld loading tgt with dataset parsing

d0ddefc

update test names and docstrings

6b65f22

make tests more explicit

46ae71a

refactor test

e603e56

create global vars for known pipelines + vers

0c94522

handle error for mismatched subs in separate func

be5847e

update print statement

d82b199

make test docstring clearer

fea6415

Merge branch 'main' into add-derivatives-command

4f89e44

alyssadai requested a review from surchs October 3, 2024 15:35

fix NP namespace url to ensure proper expansion in graph

9f6280a

surchs approved these changes Oct 3, 2024

View reviewed changes

Merge branch 'main' into add-derivatives-command

5af6a6d

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[ENH] Add derivatives command and `pipeline-catalog` submodule #349

[ENH] Add derivatives command and `pipeline-catalog` submodule #349

alyssadai commented Sep 27, 2024 •

edited by surchs

Loading

codecov bot commented Sep 27, 2024 •

edited

Loading

surchs left a comment

surchs left a comment

alyssadai commented Oct 3, 2024

[ENH] Add derivatives command and pipeline-catalog submodule #349

Are you sure you want to change the base?

[ENH] Add derivatives command and pipeline-catalog submodule #349

Conversation

alyssadai commented Sep 27, 2024 • edited by surchs Loading

Checklist

codecov bot commented Sep 27, 2024 • edited Loading

Codecov Report

surchs left a comment

Choose a reason for hiding this comment

surchs left a comment

Choose a reason for hiding this comment

alyssadai commented Oct 3, 2024

[ENH] Add derivatives command and `pipeline-catalog` submodule #349

[ENH] Add derivatives command and `pipeline-catalog` submodule #349

alyssadai commented Sep 27, 2024 •

edited by surchs

Loading

codecov bot commented Sep 27, 2024 •

edited

Loading