CPAC 1.8 running - nuisance problem. #9

hahaai · 2021-02-15T16:19:08Z

C-PAC Run Info


Description	Run Default, fmriprep, and XCP (modified from fmriprep option)
Version	1.8
Container	Singularity
System	Uppen Cluster.
Data Size	4 HBN CBIC subjects used in the CPAC regression test. Each subject has several bold images (750, 350 and 250 volumes)
Results	It seems the jobs are hanging there. Nuisance was not done, have some crash files related.

Include:

Run Command:
singularity run -e -B /cbica:/cbica /cbica/projects/RBC/CPACTesting/Pipeline_Timing/cpac_1.8.simg /cbica/projects/RBC/CPACTesting/Pipeline_Timing/Running_problem_testing/AllNew/HBNData /cbica/projects/RBC/CPACTesting/Pipeline_Timing/Running_problem_testing/AllNew/CPAC_out/default_NDARAA504CRN participant --pipeline_file /cbica/projects/RBC/CPACTesting/Pipeline_Timing/Running_problem_testing/AllNew/cpac_pipeline_1.8_default.yml --skip_bids_validator --n_cpus 8 --save_working_dir --mem_gb 15 --participant_label NDARAA504CRN


(the command-line command you used to start the run)

Pipeline Config (can drag file directly into issue)

pipeline_configs.zip

Data Config (can drag file directly into issue)

Developers only:

Default Pipeline Diff:


(print-out from cpac_pipe_diff.py)

Screenshots of brain extraction and registration wireframe overlays from QC pages (if available):


screenshots here

Node timing information:


(print-out from callback_log_time_parse.py)

Extracted time series 1D and nuisance regressors 1D correlations against previous version or some benchmark (can use [corr_two_1D.py](https://github.com/sgiavasis/CPAC_regtest_pack/blob/ma

ster/corr_two_1D.py) but heatmaps or other visualizations are good too):


correlations here

hahaai · 2021-02-15T16:22:56Z

Here are also a few crash files:
nipypecli crash default_NDARAB348EWR/log/crash-20210211-221048-rbc-build_nuisance_regressors.a1-698aaaa1-38dd-4378-b9c0-fe84a0d36356.pklz

ile: /cbica/projects/RBC/CPACTesting/Pipeline_Timing/Running_problem_testing/AllNew/CPAC_out/default_NDARAB348EWR/log/crash-20210211-221048-rbc-build_nuisance_regressors.a1-698aaaa1-38dd-4378-b9c0-fe84a0d36356.pklz
Node: cpac__sub-NDARAB348EWR.nuisance_regressors_default_166.build_nuisance_regressors
Working directory: /cbica/projects/RBC/CPACTesting/Pipeline_Timing/Running_problem_testing/AllNew/CPAC_out/default_NDARAB348EWR/working/cpac__sub-NDARAB348EWR/nuisance_regressors_default_166/_scan_movieTP/build_nuisance_regressors

Node inputs:

acompcor_file_path =
censor_file_path =
csf_summary_file_path =
custom_file_paths =
function_str = CPAC.nuisance.nuisance.gather_nuisance
functional_file_path =
global_summary_file_path =
grey_matter_summary_file_path =
motion_parameters_file_path =
selector = {'Name': 'default', 'Motion': {'include_delayed': True, 'include_squared': True, 'include_delayed_squared': True}, 'aCompCor': {'summary': {'method': 'DetrendPC', 'components': 5}, 'tissues': ['WhiteMatter', 'CerebrospinalFluid'], 'extraction_resolution': 2}, 'CerebrospinalFluid': {'summary': {'filter': 'Mean', 'method': 'Mean'}, 'extraction_resolution': 2, 'erode_mask': True}, 'GlobalSignal': {'summary': {'filter': 'Mean', 'method': 'Mean'}}, 'PolyOrt': {'degree': 2}, 'Bandpass': {'bottom_frequency': 0.01, 'top_frequency': 0.1}}
tcompcor_file_path =
white_matter_summary_file_path =

Traceback:
Traceback (most recent call last):
File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/plugins/multiproc.py", line 67, in run_node
result["result"] = node.run(updatehash=updatehash)
File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/engine/nodes.py", line 516, in run
result = self._run_interface(execute=True)
File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/engine/nodes.py", line 635, in _run_interface
return self._run_command(execute)
File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/engine/nodes.py", line 741, in _run_command
result = self._interface.run(cwd=outdir)
File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/interfaces/base/core.py", line 419, in run
runtime = self._run_interface(runtime)
File "/code/CPAC/utils/interfaces/function.py", line 152, in _run_interface
out = function_handle(**args)
File "/code/CPAC/nuisance/nuisance.py", line 198, in gather_nuisance
regressor_length))
ValueError: Number of time points in /cbica/projects/RBC/CPACTesting/Pipeline_Timing/Running_problem_testing/AllNew/CPAC_out/default_NDARAB348EWR/working/cpac__sub-NDARAB348EWR/nuisance_regressors_default_166/_scan_movieTP/CerebrospinalFluid_mean/denoised_func_data_nonaggr_0_trans_tcat_flirt_roistat.1D (0) is inconsistent with length of functional file /cbica/projects/RBC/CPACTesting/Pipeline_Timing/Running_problem_testing/AllNew/CPAC_out/default_NDARAB348EWR/working/cpac__sub-NDARAB348EWR/ICA-AROMA_ANTs_template_to_bold_159/_scan_movieTP/func_concat_ICA-AROMA_ANTs_template_to_bold_159/denoised_func_data_nonaggr_0_trans_tcat.nii.gz (250)I found some crash files in the log folder:

File: /cbica/projects/RBC/CPACTesting/Pipeline_Timing/Running_problem_testing/AllNew/CPAC_out/default_NDARAB348EWR/log/crash-20210211-221048-rbc-build_nuisance_regressors.a1-698aaaa1-38dd-4378-b9c0-fe84a0d36356.pklz
Node: cpac__sub-NDARAB348EWR.nuisance_regressors_default_166.build_nuisance_regressors
Working directory: /cbica/projects/RBC/CPACTesting/Pipeline_Timing/Running_problem_testing/AllNew/CPAC_out/default_NDARAB348EWR/working/cpac__sub-NDARAB348EWR/nuisance_regressors_default_166/_scan_movieTP/build_nuisance_regressors

Node inputs:

acompcor_file_path =
censor_file_path =
csf_summary_file_path =
custom_file_paths =
function_str = CPAC.nuisance.nuisance.gather_nuisance
functional_file_path =
global_summary_file_path =
grey_matter_summary_file_path =
motion_parameters_file_path =
selector = {'Name': 'default', 'Motion': {'include_delayed': True, 'include_squared': True, 'include_delayed_squared': True}, 'aCompCor': {'summary': {'method': 'DetrendPC', 'components': 5}, 'tissues': ['WhiteMatter', 'CerebrospinalFluid'], 'extraction_resolution': 2}, 'CerebrospinalFluid': {'summary': {'filter': 'Mean', 'method': 'Mean'}, 'extraction_resolution': 2, 'erode_mask': True}, 'GlobalSignal': {'summary': {'filter': 'Mean', 'method': 'Mean'}}, 'PolyOrt': {'degree': 2}, 'Bandpass': {'bottom_frequency': 0.01, 'top_frequency': 0.1}}
tcompcor_file_path =
white_matter_summary_file_path =

Traceback:
Traceback (most recent call last):
File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/plugins/multiproc.py", line 67, in run_node
result["result"] = node.run(updatehash=updatehash)
File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/engine/nodes.py", line 516, in run
result = self._run_interface(execute=True)
File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/engine/nodes.py", line 635, in _run_interface
return self._run_command(execute)
File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/engine/nodes.py", line 741, in _run_command
result = self._interface.run(cwd=outdir)
File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/interfaces/base/core.py", line 419, in run
runtime = self._run_interface(runtime)
File "/code/CPAC/utils/interfaces/function.py", line 152, in _run_interface
out = function_handle(**args)
File "/code/CPAC/nuisance/nuisance.py", line 198, in gather_nuisance
regressor_length))
ValueError: Number of time points in /cbica/projects/RBC/CPACTesting/Pipeline_Timing/Running_problem_testing/AllNew/CPAC_out/default_NDARAB348EWR/working/cpac__sub-NDARAB348EWR/nuisance_regressors_default_166/_scan_movieTP/CerebrospinalFluid_mean/denoised_func_data_nonaggr_0_trans_tcat_flirt_roistat.1D (0) is inconsistent with length of functional file /cbica/projects/RBC/CPACTesting/Pipeline_Timing/Running_problem_testing/AllNew/CPAC_out/default_NDARAB348EWR/working/cpac__sub-NDARAB348EWR/ICA-AROMA_ANTs_template_to_bold_159/_scan_movieTP/func_concat_ICA-AROMA_ANTs_template_to_bold_159/denoised_func_data_nonaggr_0_trans_tcat.nii.gz (250)I found some crash files in the log folder:

File: /cbica/projects/RBC/CPACTesting/Pipeline_Timing/Running_problem_testing/AllNew/CPAC_out/default_NDARAB348EWR/log/crash-20210211-221048-rbc-build_nuisance_regressors.a1-698aaaa1-38dd-4378-b9c0-fe84a0d36356.pklz
Node: cpac__sub-NDARAB348EWR.nuisance_regressors_default_166.build_nuisance_regressors
Working directory: /cbica/projects/RBC/CPACTesting/Pipeline_Timing/Running_problem_testing/AllNew/CPAC_out/default_NDARAB348EWR/working/cpac__sub-NDARAB348EWR/nuisance_regressors_default_166/_scan_movieTP/build_nuisance_regressors

Node inputs:

acompcor_file_path =
censor_file_path =
csf_summary_file_path =
custom_file_paths =
function_str = CPAC.nuisance.nuisance.gather_nuisance
functional_file_path =
global_summary_file_path =
grey_matter_summary_file_path =
motion_parameters_file_path =
selector = {'Name': 'default', 'Motion': {'include_delayed': True, 'include_squared': True, 'include_delayed_squared': True}, 'aCompCor': {'summary': {'method': 'DetrendPC', 'components': 5}, 'tissues': ['WhiteMatter', 'CerebrospinalFluid'], 'extraction_resolution': 2}, 'CerebrospinalFluid': {'summary': {'filter': 'Mean', 'method': 'Mean'}, 'extraction_resolution': 2, 'erode_mask': True}, 'GlobalSignal': {'summary': {'filter': 'Mean', 'method': 'Mean'}}, 'PolyOrt': {'degree': 2}, 'Bandpass': {'bottom_frequency': 0.01, 'top_frequency': 0.1}}
tcompcor_file_path =
white_matter_summary_file_path =

Traceback:
Traceback (most recent call last):
File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/plugins/multiproc.py", line 67, in run_node
result["result"] = node.run(updatehash=updatehash)
File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/engine/nodes.py", line 516, in run
result = self._run_interface(execute=True)
File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/engine/nodes.py", line 635, in _run_interface
return self._run_command(execute)
File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/engine/nodes.py", line 741, in _run_command
result = self._interface.run(cwd=outdir)
File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/interfaces/base/core.py", line 419, in run
runtime = self._run_interface(runtime)
File "/code/CPAC/utils/interfaces/function.py", line 152, in _run_interface
out = function_handle(**args)
File "/code/CPAC/nuisance/nuisance.py", line 198, in gather_nuisance
regressor_length))
ValueError: Number of time points in /cbica/projects/RBC/CPACTesting/Pipeline_Timing/Running_problem_testing/AllNew/CPAC_out/default_NDARAB348EWR/working/cpac__sub-NDARAB348EWR/nuisance_regressors_default_166/_scan_movieTP/CerebrospinalFluid_mean/denoised_func_data_nonaggr_0_trans_tcat_flirt_roistat.1D (0) is inconsistent with length of functional file /cbica/projects/RBC/CPACTesting/Pipeline_Timing/Running_problem_testing/AllNew/CPAC_out/default_NDARAB348EWR/working/cpac__sub-NDARAB348EWR/ICA-AROMA_ANTs_template_to_bold_159/_scan_movieTP/func_concat_ICA-AROMA_ANTs_template_to_bold_159/denoised_func_data_nonaggr_0_trans_tcat.nii.gz (250)I found some crash files in the log folder:

File: /cbica/projects/RBC/CPACTesting/Pipeline_Timing/Running_problem_testing/AllNew/CPAC_out/default_NDARAB348EWR/log/crash-20210211-221048-rbc-build_nuisance_regressors.a1-698aaaa1-38dd-4378-b9c0-fe84a0d36356.pklz
Node: cpac__sub-NDARAB348EWR.nuisance_regressors_default_166.build_nuisance_regressors
Working directory: /cbica/projects/RBC/CPACTesting/Pipeline_Timing/Running_problem_testing/AllNew/CPAC_out/default_NDARAB348EWR/working/cpac__sub-NDARAB348EWR/nuisance_regressors_default_166/_scan_movieTP/build_nuisance_regressors

Node inputs:

acompcor_file_path =
censor_file_path =
csf_summary_file_path =
custom_file_paths =
function_str = CPAC.nuisance.nuisance.gather_nuisance
functional_file_path =
global_summary_file_path =
grey_matter_summary_file_path =
motion_parameters_file_path =
selector = {'Name': 'default', 'Motion': {'include_delayed': True, 'include_squared': True, 'include_delayed_squared': True}, 'aCompCor': {'summary': {'method': 'DetrendPC', 'components': 5}, 'tissues': ['WhiteMatter', 'CerebrospinalFluid'], 'extraction_resolution': 2}, 'CerebrospinalFluid': {'summary': {'filter': 'Mean', 'method': 'Mean'}, 'extraction_resolution': 2, 'erode_mask': True}, 'GlobalSignal': {'summary': {'filter': 'Mean', 'method': 'Mean'}}, 'PolyOrt': {'degree': 2}, 'Bandpass': {'bottom_frequency': 0.01, 'top_frequency': 0.1}}
tcompcor_file_path =
white_matter_summary_file_path =

Traceback:
Traceback (most recent call last):
File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/plugins/multiproc.py", line 67, in run_node
result["result"] = node.run(updatehash=updatehash)
File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/engine/nodes.py", line 516, in run
result = self._run_interface(execute=True)
File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/engine/nodes.py", line 635, in _run_interface
return self._run_command(execute)
File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/engine/nodes.py", line 741, in _run_command
result = self._interface.run(cwd=outdir)
File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/interfaces/base/core.py", line 419, in run
runtime = self._run_interface(runtime)
File "/code/CPAC/utils/interfaces/function.py", line 152, in _run_interface
out = function_handle(**args)
File "/code/CPAC/nuisance/nuisance.py", line 198, in gather_nuisance
regressor_length))
ValueError: Number of time points in /cbica/projects/RBC/CPACTesting/Pipeline_Timing/Running_problem_testing/AllNew/CPAC_out/default_NDARAB348EWR/working/cpac__sub-NDARAB348EWR/nuisance_regressors_default_166/_scan_movieTP/CerebrospinalFluid_mean/denoised_func_data_nonaggr_0_trans_tcat_flirt_roistat.1D (0) is inconsistent with length of functional file /cbica/projects/RBC/CPACTesting/Pipeline_Timing/Running_problem_testing/AllNew/CPAC_out/default_NDARAB348EWR/working/cpac__sub-NDARAB348EWR/ICA-AROMA_ANTs_template_to_bold_159/_scan_movieTP/func_concat_ICA-AROMA_ANTs_template_to_bold_159/denoised_func_data_nonaggr_0_trans_tcat.nii.gz (250)

hahaai · 2021-02-15T16:24:12Z

Another crash file type:

nipypecli crash default_NDARAB348EWR/log/crash-20210211-215342-rbc-chunk_warp_ts_to_T1template_205.a0-8e108e9c-661b-4735-b9f0-7e975af67f2c.pklz

File: /cbica/projects/RBC/CPACTesting/Pipeline_Timing/Running_problem_testing/AllNew/CPAC_out/default_NDARAB348EWR/log/crash-20210211-215342-rbc-chunk_warp_ts_to_T1template_205.a0-8e108e9c-661b-4735-b9f0-7e975af67f2c.pklz
Node: cpac__sub-NDARAB348EWR.warp_ts_to_T1template_205.chunk_warp_ts_to_T1template_205
Working directory: /cbica/projects/RBC/CPACTesting/Pipeline_Timing/Running_problem_testing/AllNew/CPAC_out/default_NDARAB348EWR/working/cpac__sub-NDARAB348EWR/warp_ts_to_T1template_205/_scan_movieDM/chunk_warp_ts_to_T1template_205

Node inputs:

func_file =
function_str = def chunk_ts(func_file, n_cpus):
func_img = nb.load(func_file)
trs = func_img.shape[3]
chunk = trs/n_cpus
TR_ranges = []

for chunk_idx in range(0, n_cpus):
    if chunk_idx == n_cpus - 1:
        TR_ranges.append((int(chunk_idx*chunk), int(trs - 1)))
    else:
        TR_ranges.append((int(chunk_idx*chunk), int((chunk_idx+1)*chunk -1)))
return TR_ranges

n_cpus = 8

Traceback:
Traceback (most recent call last):
File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/plugins/multiproc.py", line 67, in run_node
result["result"] = node.run(updatehash=updatehash)
File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/engine/nodes.py", line 516, in run
result = self._run_interface(execute=True)
File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/engine/nodes.py", line 635, in _run_interface
return self._run_command(execute)
File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/engine/nodes.py", line 741, in _run_command
result = self._interface.run(cwd=outdir)
File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/interfaces/base/core.py", line 419, in run
runtime = self._run_interface(runtime)
File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/interfaces/utility/wrappers.py", line 142, in _run_interface
out = function_handle(**args)
File "", line 3, in chunk_ts
IndexError: tuple index out of range

hahaai · 2021-02-15T16:26:40Z

pipelines used mcflirt (fmriprep and XCP) also has the following crash file:

nipypecli crash fmriprep_NDARAB348EWR/log/crash-20210211-213148-rbc-get_mcflirt_rms_abs_90-c1eee511-cff5-4fc7-b2bc-204c595b2a3f.pklz

File: /cbica/projects/RBC/CPACTesting/Pipeline_Timing/Running_problem_testing/AllNew/CPAC_out/fmriprep_NDARAB348EWR/log/crash-20210211-213148-rbc-get_mcflirt_rms_abs_90-c1eee511-cff5-4fc7-b2bc-204c595b2a3f.pklz
Node: cpac__sub-NDARAB348EWR.get_mcflirt_rms_abs_90
Working directory: /cbica/projects/RBC/CPACTesting/Pipeline_Timing/Running_problem_testing/AllNew/CPAC_out/fmriprep_NDARAB348EWR/working/cpac__sub-NDARAB348EWR/get_mcflirt_rms_abs_90

Node inputs:

function_str = def get_mcflirt_rms_abs(rms_files):
for path in rms_files:
if 'abs.rms' in path:
return path

rms_files =

Traceback:
Traceback (most recent call last):
File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/plugins/multiproc.py", line 67, in run_node
result["result"] = node.run(updatehash=updatehash)
File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/engine/nodes.py", line 516, in run
result = self._run_interface(execute=True)
File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/engine/nodes.py", line 635, in _run_interface
return self._run_command(execute)
File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/engine/nodes.py", line 741, in _run_command
result = self._interface.run(cwd=outdir)
File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/interfaces/base/core.py", line 419, in run
runtime = self._run_interface(runtime)
File "/code/CPAC/utils/interfaces/function.py", line 152, in _run_interface
out = function_handle(**args)
TypeError: get_mcflirt_rms_abs() missing 1 required positional argument: 'rms_files'

shnizzedy · 2021-02-18T19:12:16Z

ValueError: Number of time points in /cbica/projects/RBC/CPACTesting/Pipeline_Timing/Running_problem_testing/AllNew/CPAC_out/default_NDARAB348EWR/working/cpac__sub-NDARAB348EWR/nuisance_regressors_default_166/_scan_movieTP/CerebrospinalFluid_mean/denoised_func_data_nonaggr_0_trans_tcat_flirt_roistat.1D (0) is inconsistent with length of functional file

I wonder if this is a data quality issue or if something isn't mapping correctly. It looks like it's creating 0-length ROI files.

hahaai · 2021-02-19T01:03:55Z

ValueError: Number of time points in /cbica/projects/RBC/CPACTesting/Pipeline_Timing/Running_problem_testing/AllNew/CPAC_out/default_NDARAB348EWR/working/cpac__sub-NDARAB348EWR/nuisance_regressors_default_166/_scan_movieTP/CerebrospinalFluid_mean/denoised_func_data_nonaggr_0_trans_tcat_flirt_roistat.1D (0) is inconsistent with length of functional file

I wonder if this is a data quality issue or if something isn't mapping correctly. It looks like it's creating 0-length ROI files.

Actually this the lateral ventrical problem where the reference has 3 volume in it rather than 1. This has beee fixed.

shnizzedy added debug Pipeline run failed due to an error or crash. dev branch Pipeline run is from a development version of C-PAC. log A summary of a run attempt labels Feb 18, 2021

shnizzedy mentioned this issue Feb 18, 2021

C-PAC v1.8.0.dev - preproc preconfig #14

Open

3 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

CPAC 1.8 running - nuisance problem. #9

CPAC 1.8 running - nuisance problem. #9

hahaai commented Feb 15, 2021 •

edited

Loading

hahaai commented Feb 15, 2021

hahaai commented Feb 15, 2021

hahaai commented Feb 15, 2021

shnizzedy commented Feb 18, 2021 •

edited

Loading

hahaai commented Feb 19, 2021

CPAC 1.8 running - nuisance problem. #9

CPAC 1.8 running - nuisance problem. #9

Comments

hahaai commented Feb 15, 2021 • edited Loading

C-PAC Run Info

hahaai commented Feb 15, 2021

hahaai commented Feb 15, 2021

hahaai commented Feb 15, 2021

shnizzedy commented Feb 18, 2021 • edited Loading

hahaai commented Feb 19, 2021

hahaai commented Feb 15, 2021 •

edited

Loading

shnizzedy commented Feb 18, 2021 •

edited

Loading