visualizer.py fails on chained MEG processing pipeline [update: solved] #120
Comments
|
Note that the processing looks something like this
where "preproc", "avg" and "combined" are processing steps, and all others are data. The "avg" (computing the ERFs) is repeated 3 times for different selection of trials (corresponding to conditions), and consequently the "combined" step is also executed three times (for the three conditions). |
|
@robertoostenveld: Thanks for sharing those! I think the problem is probably related to recent update to the library to better reflect key names defined in the spec document (mainly changes in case):
In case I missed some, all the keys are defined in this context file (now harmonized with what is in the BIDS-Prov spec). A revised version of the minimal example we had worked on in Copenhagen is here: https://github.com/bids-standard/BEP028_BIDSprov/blob/master/examples/simple_example/simple_example.prov.jsonld Does this help? |
|
Also the example does not result in a figure with |
|
[Edit: redirect is now in place] Thanks for spotting that and sorry! There was an issue in the PURL redirection https://purl.org/nidash/bidsprov/context.json that I've just fixed. Now the PURL context link should redirect to https://raw.githubusercontent.com/bids-standard/BEP028_BIDSprov/master/context.json and solve the issue. On my machine
|
|
Updating your example to (update key names and paths in bids URIs): The vizualize function gives us: Compared to your example there are 3 "combine" activities. |
|
Removing duplicate activities (this is closer to what we have above in your diagram but maybe not what we want as provenance):
|
|
Following up on our discussion. Is the id, for example, "@id":"bb67ad88-e56c-4c37-ba0c-b6b9e47daff1", supposed to be common across BIDS datasets? Would it be possible to have simpler IDs? How are the IDs generated? For example, could we have "preproc-fieldtrip-0962f5dce." It would make the text file much easier to read by the common mortal. Also, are the parameters to the functions called timelockFIC defined somewhere (even if it is not in a standard format). |
|
Hi @arnodelorme! Yes you can have simpler ids (but they still have to be URIs), I'm pasting here the BIDS-Prov spec: @id: REQUIRED. A unique identifier like a UUID that will be used to associate activities with this software (e.g., urn:1264-1233-11231-12312, "urn:bet-o1ef4rt" So what you are proposing as an id could work, possibly prepending a prefix For the parameters I would recommend adding a Pingging: @satra in case he'd like to chime in! |
|
@arnodelorme - in addition to what camille said, the |
|
Hi @arnodelorme, the example I constructed and shared above was using random identifiers generated in MATLAB with What I understand from the FSL example, for example when searching for After iterating a few more times, and also realizing that I want each activity (= function call) to be represented separately, I now have this: which renders like this
The most relevant changes are that I give the activities the FieldTrip function as their label, and the entities get the MATLAB variable name as the label. So the MATLAB data structures that I have in memory after running this for a single subject are What I don't yet have figured out is how to add "Command" details, which in FieldTrip would be the input What I also have not yet figured out is how to deal with steps that are not represented on disk. In the case above that is the output of ft_megplanar, which is followed by ft_combineplanar. In the case of ICA cleaning it could also be ft_componentanalysis, followed by ft_rejectcomponent. The back-projected clean raw data could be written to disk according to BEP021, but the unmixed data (component topos and timeseries) not. I think this also shows where (i)EEG and MEG diverge from (f)MRI analyses, which mostly takes nifti files as input and spits out other nifti files. The data structures of (i)EEG/MEG analyses are not so consistent. I'll keep working on it for a bit longer, to see whether I can make further progress in line with this BEP028 and BEP021. |
|
Thank you Robert. Yes, this is a step in the right direction. For the IDs of the activities, could you use something like |
|
Hi Arno, I understand that IDs have to be unique (i.e., separate nodes in the graph), so two calls to ft_timelockanalysis should have different IDs. It would be possible to do something like this but I don't see an advantage of this over having fully random UUIDs. I think it is better to fully ignore the IDs when you look at it, unless you use them to link things together (as with the lines in the graph). Oh, and note that the ID also does not have to have a reference to the software package, as that is something you can do with the PS I noticed that |
robertoostenveld
changed the title
|
Thanks Robert
…On Sat, Sep 23, 2023 at 5:32 AM Robert Oostenveld ***@***.***> wrote:
Hi Arno, I understand that IDs have to be unique (i.e., separate nodes in
the graph), so two calls to ft_timelockanalysis should have different IDs.
It would be possible to do something like this
{
***@***.***":"https://raw.githubusercontent.com/bids-standard/BEP028_BIDSprov/master/context.json",
"BIDSProvVersion":"dev",
"Records":{
"Software":[
{
"Id":"683ef168-914c-4c2e-a957-885c15cf2636",
"RRID":"RRID:SCR_004849",
"Label":"FieldTrip",
"Type":"Software",
"Version":"0aa4735f6"
}
],
"Activities":[
{
"Id":"ft_timelockanalysis-7e50855c",
"Label":"ft_timelockanalysis",
"Used":"sub-01/meg/sub-01_task-language_desc-preproc_meg.ds",
"AssociatedWith":"683ef168-914c-4c2e-a957-885c15cf2636"
},
{
"Id":"ft_timelockanalysis-d1da5e67",
"Label":"ft_timelockanalysis",
"Used":"sub-01/meg/sub-01_task-language_desc-preproc_meg.ds",
"AssociatedWith":"683ef168-914c-4c2e-a957-885c15cf2636"
},
...
but I don't see an advantage of this over having fully random UUIDs. I
think it is better to fully ignore the IDs when you look at it, unless you
use them to link things together (as with the lines in the graph). Oh, and
note that the ID also does not have to have a reference to the software
package, as that is something you can do with the AssociatedWith
reference.
PS I noticed that @id is not needed, it can also be Id. That makes
writing the json easier. Have a look at my code_v3 directory, which has a
writeprov.m function, and two functions that help in constructing the
provenance as MATLAB structure.
—
Reply to this email directly, view it on GitHub
<#120 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AAOJGQNUGZVRLSSUT4XWZ33X33JFHANCNFSM6AAAAAA4X6IP5Q>
.
You are receiving this because you were mentioned.Message ID:
***@***.***>
|
|
Besides the BIDS specific example script whose results are being discussed at bids-standard/bep021#8, I have now also added support for the I think there is nothing any more to be done on this issue. |
Following bids-standard/bep021#8 (comment) I am trying to construct a
.prov.jsonldfile for an example chained analysis pipeline of MEG data.I cannot get the visualizer.py to work on my example, neither on the one that we hand-crafted in Kopenhagen. Both return an empty png figure. @cmaumet can you spot what is wrong?
The text was updated successfully, but these errors were encountered: