DWI bids compliant but MRIQC cannot process #1302

dkp · 2024-06-01T20:34:43Z

What happened?

I'm trying MRIQC on 50 subjects on the HPC using apptainer. Most subjects run fine for all three modalities at once: T1w, bold, dwi. However, some crashed.
To investigate, I ran a subject with -m T1w bold This ran fine and produced reports for those 2 modalities.
Then I ran with -m dwi and this one crashed. See log below.

What command did you use?

apptainer run --cleanenv --bind ${DATA}:/data:ro --bind ${APP_DERIV_DIR}:/outputs ${APP} \
/data /outputs participant --participant_label ${Subject} -m dwi -w ${WORK_DIR} --verbose-reports

What version of the software are you running?

Running MRIQC version 24.1.0.dev0+g3fe90466.d20240417

How are you running this software?

Singularity

Is your data BIDS valid?

Yes

Are you reusing any previously computed results?

No

Please copy and paste any relevant log output.

$ sbatch --export sub=083 runmriqc_dev.sh
Submitted batch job 1968242
$ cat slurm-1968242.out 
/xdisk/eplante/dkp/plante/data should contain dataset_description.json!
------------------------------------------------------------------
  Running MRIQC version 24.1.0.dev0+g3fe90466.d20240417
  ----------------------------------------------------------------
  * BIDS dataset path: /data.
  * Output folder: /outputs.
  * Analysis levels: ['participant'].
------------------------------------------------------------------

2024-06-01 13:19:07 | IMPORTANT | mriqc            | Building MRIQC's workflows...
2024-06-01 13:19:17 | IMPORTANT | mriqc            | Workflow building finished (exit code 0).
2024-06-01 13:19:25 | WARNING  | mriqc            | IMPORTANT: Anonymized quality metrics (IQMs) will be submitted to MRIQC's metrics repository. Submission of IQMs can be disabled using the ``--no-sub`` argument. Please visit https://mriqc.readthedocs.io/en/latest/dsa.html to revise MRIQC's Data Sharing Agreement.
2024-06-01 13:19:56 | WARNING  | nipype.workflow  | Storing result file without outputs
2024-06-01 13:19:56 | WARNING  | nipype.workflow  | [Node] Error on "mriqc_wf.dwiMRIQC.ComputeIQMs.datasink" (/xdisk/eplante/dkp/plante/mriqc_work_dev/mriqc_wf/dwiMRIQC/ComputeIQMs/_in_file_..data..sub-083..dwi..sub-083_acq-AP_dwi.nii.gz/datasink)
Traceback (most recent call last):
  File "/opt/conda/lib/python3.11/site-packages/mriqc/engine/plugin.py", line 579, in _send_procs_to_workers
    self.procs[jobid].run(updatehash=updatehash)
  File "/opt/conda/lib/python3.11/site-packages/nipype/pipeline/engine/nodes.py", line 527, in run
    result = self._run_interface(execute=True)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.11/site-packages/nipype/pipeline/engine/nodes.py", line 645, in _run_interface
    return self._run_command(execute)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.11/site-packages/nipype/pipeline/engine/nodes.py", line 771, in _run_command
    raise NodeExecutionError(msg)
nipype.pipeline.engine.nodes.NodeExecutionError: Exception raised while executing Node datasink.

Traceback:
        Traceback (most recent call last):
          File "/opt/conda/lib/python3.11/site-packages/nipype/interfaces/base/core.py", line 397, in run
            runtime = self._run_interface(runtime)
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
          File "/opt/conda/lib/python3.11/site-packages/mriqc/interfaces/bids.py", line 188, in _run_interface
            json.dumps(
          File "/opt/conda/lib/python3.11/site-packages/simplejson/__init__.py", line 395, in dumps
            **kw).encode(obj)
                  ^^^^^^^^^^^
          File "/opt/conda/lib/python3.11/site-packages/simplejson/encoder.py", line 300, in encode
            chunks = list(chunks)
                     ^^^^^^^^^^^^
          File "/opt/conda/lib/python3.11/site-packages/simplejson/encoder.py", line 714, in _iterencode
            for chunk in _iterencode_dict(o, _current_indent_level):
          File "/opt/conda/lib/python3.11/site-packages/simplejson/encoder.py", line 645, in _iterencode_dict
            yield _floatstr(value)
                  ^^^^^^^^^^^^^^^^
          File "/opt/conda/lib/python3.11/site-packages/simplejson/encoder.py", line 351, in floatstr
            raise ValueError(
        ValueError: Out of range float values are not JSON compliant: nan


During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/conda/bin/mriqc", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/opt/conda/lib/python3.11/site-packages/mriqc/cli/run.py", line 178, in main
    mriqc_wf.run(**_plugin)
  File "/opt/conda/lib/python3.11/site-packages/nipype/pipeline/engine/workflows.py", line 638, in run
    runner.run(execgraph, updatehash=updatehash, config=self.config)
  File "/opt/conda/lib/python3.11/site-packages/mriqc/engine/plugin.py", line 214, in run
    self._send_procs_to_workers(updatehash=updatehash, graph=graph)
  File "/opt/conda/lib/python3.11/site-packages/mriqc/engine/plugin.py", line 582, in _send_procs_to_workers
    self._clean_queue(
  File "/opt/conda/lib/python3.11/site-packages/mriqc/engine/plugin.py", line 259, in _clean_queue
    raise RuntimeError(''.join(result['traceback']))
RuntimeError: Traceback (most recent call last):
  File "/opt/conda/lib/python3.11/site-packages/mriqc/engine/plugin.py", line 579, in _send_procs_to_workers
    self.procs[jobid].run(updatehash=updatehash)
  File "/opt/conda/lib/python3.11/site-packages/nipype/pipeline/engine/nodes.py", line 527, in run
    result = self._run_interface(execute=True)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.11/site-packages/nipype/pipeline/engine/nodes.py", line 645, in _run_interface
    return self._run_command(execute)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.11/site-packages/nipype/pipeline/engine/nodes.py", line 771, in _run_command
    raise NodeExecutionError(msg)
nipype.pipeline.engine.nodes.NodeExecutionError: Exception raised while executing Node datasink.

Traceback:
        Traceback (most recent call last):
          File "/opt/conda/lib/python3.11/site-packages/nipype/interfaces/base/core.py", line 397, in run
            runtime = self._run_interface(runtime)
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
          File "/opt/conda/lib/python3.11/site-packages/mriqc/interfaces/bids.py", line 188, in _run_interface
            json.dumps(
          File "/opt/conda/lib/python3.11/site-packages/simplejson/__init__.py", line 395, in dumps
            **kw).encode(obj)
                  ^^^^^^^^^^^
          File "/opt/conda/lib/python3.11/site-packages/simplejson/encoder.py", line 300, in encode
            chunks = list(chunks)
                     ^^^^^^^^^^^^
          File "/opt/conda/lib/python3.11/site-packages/simplejson/encoder.py", line 714, in _iterencode
            for chunk in _iterencode_dict(o, _current_indent_level):
          File "/opt/conda/lib/python3.11/site-packages/simplejson/encoder.py", line 645, in _iterencode_dict
            yield _floatstr(value)
                  ^^^^^^^^^^^^^^^^
          File "/opt/conda/lib/python3.11/site-packages/simplejson/encoder.py", line 351, in floatstr
            raise ValueError(
        ValueError: Out of range float values are not JSON compliant: nan

Additional information / screenshots

No response

The text was updated successfully, but these errors were encountered:

araikes · 2024-06-02T06:31:21Z

@dkp
Can you post the JSON or put it somewhere on the HPC I can look at it?

dkp · 2024-06-02T20:46:23Z

@araikes I sent you a separate email sharing the dataset.

araikes · 2024-06-03T18:29:44Z

@oesteban and @dkp,

The offending component (at least for the dataset shared) is the computation of EFC on the b=0 shell. It encounters an invalid value in the computation:

2024-06-03 10:17:37 | DEBUG    | indexed_gzip.indexed_gzip | _IndexedGzipFile.close()
/opt/conda/lib/python3.11/site-packages/mriqc/qc/anatomical.py:391: RuntimeWarning: invalid value encountered in sqrt
  b_max = np.sqrt((img[framemask == 0] ** 2).sum()

which results in a nan for that shell (truncating the whole line for visibility):

2024-06-03 10:17:43 | DEBUG    | nipype.workflow  | [Node] datasink - setting input root = {... 'efc_shell01': nan, ...}

And then the JSON writer can't figure out how to deal with the NaN when expecting a double.

I haven't figured out what the offending value in the image is. There don't appear to be any unusual or invalid values as far as I can tell and I can manually compute the EFC for each of the two b=0 images independently (in their raw, unprocessed state... just loading the data, creating the framemask, and computing as per the code in mriqc/qc/anatomical.py). Notably, something seems to be off (for lack of a better term) about those b=0s because FBER is computed as -4527 which doesn't feel like a plausible value either.

dkp · 2024-06-03T18:36:44Z

This is older data, if that helps at all. Thanks for looking into it.

s-madhavan · 2024-06-04T21:30:56Z

Hi,

I'm running into the same issue as above with my dwi data.

Error:

Traceback:
Traceback (most recent call last):
File "/opt/conda/lib/python3.11/site-packages/nipype/interfaces/base/core.py", line 397, in run
runtime = self._run_interface(runtime)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/conda/lib/python3.11/site-packages/mriqc/interfaces/bids.py", line 188, in _run_interface
json.dumps(
File "/opt/conda/lib/python3.11/site-packages/simplejson/init.py", line 395, in dumps
**kw).encode(obj)
^^^^^^^^^^^
File "/opt/conda/lib/python3.11/site-packages/simplejson/encoder.py", line 300, in encode
chunks = list(chunks)
^^^^^^^^^^^^
File "/opt/conda/lib/python3.11/site-packages/simplejson/encoder.py", line 714, in _iterencode
for chunk in _iterencode_dict(o, _current_indent_level):
File "/opt/conda/lib/python3.11/site-packages/simplejson/encoder.py", line 645, in _iterencode_dict
yield _floatstr(value)
^^^^^^^^^^^^^^^^
File "/opt/conda/lib/python3.11/site-packages/simplejson/encoder.py", line 351, in floatstr
raise ValueError(
ValueError: Out of range float values are not JSON compliant: nan

Finished subject SOARTravel01 20240524CCH15

My data is new, and BIDS compliant. Going to try using an older version of MRIQC for now without DWI processing. DWI doesn't seem to be working for now.

dkp · 2024-06-04T21:37:46Z

You can just set the -m (modality flag): -m T1w bold
so you can still use the new version, it just won't try to run dwi.

s-madhavan · 2024-06-04T21:46:57Z

Hi,

Noted. Will do that for anat and func files. Thanks.

--Subbi M.

costantinoai · 2024-06-07T14:25:18Z

I am having a similar error (see here for the crash report and error), but I do not have DWI scans.

The problem for me seems to be the 'summary_bg_k': nan entry.

Any idea from where this nan comes from? I am running mriqc on a dataset preprocessed with fMRIprep – could this be the cause?

pradhanhitesh · 2024-06-19T13:42:24Z

Facing the same issue with DWI images. Any update on the issues?

Thanks.

s-madhavan · 2024-06-19T15:42:45Z

@pradhanhitesh I'm also having the same issue. On a different note does the new version of MRIQC take forever to run for some people. I set my SBATCH time limit to be 15 hours and it crashed because the time limit expired. What version is everyone else using for now?

dkp · 2024-06-19T16:55:05Z

I'm now running 24.0.1: It pretty fully implements the dwi report (just some little issues). I've never seen the time issue you mention.

pradhanhitesh · 2024-06-19T17:13:27Z

I have the latest docker image for MRI-QC. It's true that DWI QC Reports are generated but for few DWI images, there seems to be an error (the NaN value and subsequent JSON encoding).

I'm running QC for T1w, bold and DWI for 1600 subjects for my study. When I ran the QC, most of the time it crashed. Therefore, I'm forced to run the QC in a batch of 20. In these batches of 20, there are few DWI files which are producing the NaN errors and therefore I'm not able to generate reports.

dkp · 2024-06-19T17:16:41Z

Interesting, I ran 50 subjects and lost about 25% of them because of the dwi report.

oesteban · 2024-08-20T08:07:11Z

@dkp this may have been solved by #1324 -- could you confirm?

dkp · 2024-08-20T21:09:38Z

I ran the subjects again using 24.0.1 (MRIQC v24.1.0.dev0+g5c3f8b6e.d20240820), but encountered more problems. I am attaching two SLURM logs.

The first (runmriqc_dev_array_subjects-3.txt) is a subject that succeeded with 24.0.0, but failed to generate any output for 24.0.1.
runmriqc_dev_array_subjects-4.txt
runmriqc_dev_array_subjects-3.txt

The second is a subject that failed both times (runmriqc_dev_array_subjects-4.txt), and which reports the "ValueError: Out of range float values are not JSON compliant: nan" error.

Replaces the standard `json` library with `orjson`, which is way faster and more reliable in terms of encoding numpy objects (the main issue we typically hit within *MRIQC*). Incidentally, this PR fixes an import of `simplejson`, which was not listed as a dependency. Related-to: #1302. Closes: #546. Closes: #1089. Closes: #1133.

oesteban · 2024-08-26T08:18:03Z

I just cut 24.0.2, which I believe should address @dkp's new issues.

I'm going to close this thread as the primary problem has been addressed and resolved. @dkp, please open new issues if the two problems you reported above keep happening with the new release.

Thank you all for your contribution!

dkp · 2024-08-28T20:18:41Z

dwi reports for all subjects are now being generated! Thank you

dkp added the bug label Jun 1, 2024

psadil mentioned this issue Aug 8, 2024

Numeric overflow when computing drift on int16 DWI (that has slope) #1315

Closed

oesteban mentioned this issue Aug 23, 2024

ENH: Handle WebAPI timeouts more gently #1336

Merged

oesteban mentioned this issue Aug 23, 2024

ENH: Use orjson to serialize JSON, addressing Numpy serialization issues #1337

Merged

oesteban closed this as completed Aug 26, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

DWI bids compliant but MRIQC cannot process #1302

DWI bids compliant but MRIQC cannot process #1302

dkp commented Jun 1, 2024

araikes commented Jun 2, 2024

dkp commented Jun 2, 2024

araikes commented Jun 3, 2024

dkp commented Jun 3, 2024

s-madhavan commented Jun 4, 2024

dkp commented Jun 4, 2024

s-madhavan commented Jun 4, 2024

costantinoai commented Jun 7, 2024

pradhanhitesh commented Jun 19, 2024

s-madhavan commented Jun 19, 2024

dkp commented Jun 19, 2024 •

edited

Loading

pradhanhitesh commented Jun 19, 2024

dkp commented Jun 19, 2024

oesteban commented Aug 20, 2024

dkp commented Aug 20, 2024

oesteban commented Aug 26, 2024

dkp commented Aug 28, 2024

DWI bids compliant but MRIQC cannot process #1302

DWI bids compliant but MRIQC cannot process #1302

Comments

dkp commented Jun 1, 2024

What happened?

What command did you use?

What version of the software are you running?

How are you running this software?

Is your data BIDS valid?

Are you reusing any previously computed results?

Please copy and paste any relevant log output.

Additional information / screenshots

araikes commented Jun 2, 2024

dkp commented Jun 2, 2024

araikes commented Jun 3, 2024

dkp commented Jun 3, 2024

s-madhavan commented Jun 4, 2024

dkp commented Jun 4, 2024

s-madhavan commented Jun 4, 2024

costantinoai commented Jun 7, 2024

pradhanhitesh commented Jun 19, 2024

s-madhavan commented Jun 19, 2024

dkp commented Jun 19, 2024 • edited Loading

pradhanhitesh commented Jun 19, 2024

dkp commented Jun 19, 2024

oesteban commented Aug 20, 2024

dkp commented Aug 20, 2024

oesteban commented Aug 26, 2024

dkp commented Aug 28, 2024

dkp commented Jun 19, 2024 •

edited

Loading