Analysis on the cleaned timeseries: only for making FC matrix

[Siyi13]

Hi, I have been trying to perform analysis using this toolbox on data that has gone through preprocessing and confound correction (I'll just call it as cleaned time series). So far I just to want to see how to make a correlation matrix on one dataset, to understand the workflow and outputs. However, I got some errors in the very beginning and I do not really understand how to fix this. Could anyone take a look at the script and log to provide some advice? I just used everything I got as output from the previous steps as input for this analysis step, including the original raw data, the preprocess output and the confound outputs. Thank you in advance!

1. The script for analysis:

#!/bin/bash

set -e


# Get full path for datafolder and append _bids for output folder
bids_folder="./bids_exp1a_anesthesia"

# Delete old folder and make new one
rm -rf ${bids_folder}_analysis
mkdir ${bids_folder}_analysis

# Location of temp files
export APPTAINER_BIND="/scratch/rnga/name"

# Limit ANTS threads
export ITK_GLOBAL_DEFAULT_NUMBER_OF_THREADS=8

# Analysis
singularity run \
-B ${bids_folder}:/input_BIDS:ro \
-B ${bids_folder}_preprocess:/preprocess_outputs \
-B ${bids_folder}_confound:/confound_correction_outputs \
-B ${bids_folder}_analysis:/analysis_outputs \
rabies-0.5.0.sif -p MultiProc --local_threads 8 \
analysis \
/confound_correction_outputs /analysis_outputs \
--ROI_csv "/home/name/SIGMA_Anatomical_Brain_Atlas_rs.nii" \
--ROI_type parcellated \
--FC_matrix \

2. the output log I got:

240419-14:28:26,531 nipype.workflow INFO:
	 Running RABIES - version: 0.5.0
240419-14:28:26,531 nipype.workflow INFO:
	 CLI INPUTS: 
-> rabies_stage = analysis 
-> plugin = MultiProc 
-> local_threads = 8 
-> scale_min_memory = 1.0 
-> min_proc = 1 
-> figure_format = png 
-> verbose = 1 
-> confound_correction_out = /confound_correction_outputs 
-> output_dir = /analysis_outputs 
-> scan_list = ['all'] 
-> prior_maps = /home/rabies/.local/share/rabies/melodic_IC.nii.gz 
-> prior_bold_idx = [5, 12, 19] 
-> prior_confound_idx = [0, 2, 6, 7, 8, 9, 10, 11, 13, 14, 21, 22, 24, 26, 28, 29] 
-> data_diagnosis = False 
-> scan_QC_thresholds = {} 
-> outlier_threshold = 3.5 
-> extended_QC = False 
-> seed_list = [] 
-> seed_prior_list = [] 
-> FC_matrix = True 
-> ROI_type = parcellated 
-> ROI_csv = /scratch/rnga/szeng/template_mask/SIGMA_Anatomical_Brain_Atlas_rs.nii 
-> group_ica = {'apply': False, 'dim': 0, 'random_seed': 1} 
-> DR_ICA = False 
-> NPR_temporal_comp = -1 
-> NPR_spatial_comp = -1 
-> optimize_NPR = {'apply': False, 'window_size': 5, 'min_prior_corr': 0.5, 'diff_thresh': 0.03, 'max_iter': 20, 'compute_max': False} 
-> network_weighting = absolute 

/home/rabies/miniforge/lib/python3.9/site-packages/nilearn/datasets/__init__.py:87: FutureWarning: Fetchers from the nilearn.datasets module will be updated in version 0.9 to return python strings instead of bytes and Pandas dataframes instead of Numpy arrays.
  warn("Fetchers from the nilearn.datasets module will be "
240419-14:28:28,295 nipype.utils WARNING:
	 No metadata was found in the pkl file. Make sure you are currently using the same Nipype version from the generated pkl.
Traceback (most recent call last):
  File "/home/rabies/miniforge/bin/rabies", line 7, in <module>
    exec(compile(f.read(), __file__, 'exec'))
  File "/home/rabies/RABIES/scripts/rabies", line 3, in <module>
    execute_workflow()
  File "/home/rabies/RABIES/rabies/run_main.py", line 48, in execute_workflow
    workflow = analysis(opts, log)
  File "/home/rabies/RABIES/rabies/run_main.py", line 279, in analysis
    workflow = init_main_analysis_wf(preprocess_opts, confound_correction_opts, opts)
  File "/home/rabies/RABIES/rabies/analysis_pkg/main_wf.py", line 18, in init_main_analysis_wf
    split_dict, split_name_list, target_list = read_confound_workflow(conf_output, nativespace=cr_opts.nativespace_analysis)
  File "/home/rabies/RABIES/rabies/analysis_pkg/main_wf.py", line 355, in read_confound_workflow
    fill_split_dict(bold_dict, output_bold, split_name, split_dict, [], node_dict, match_targets)
  File "/home/rabies/RABIES/rabies/utils.py", line 399, in fill_split_dict
    fill_split_dict(d[key], output_bold, split_name, split_dict, keys+[key], node_dict, match_targets)
  File "/home/rabies/RABIES/rabies/utils.py", line 401, in fill_split_dict
    f = d.result.outputs.get()[output_bold]
  File "/home/rabies/miniforge/lib/python3.9/site-packages/nipype/pipeline/engine/nodes.py", line 216, in result
    return _load_resultfile(
  File "/home/rabies/miniforge/lib/python3.9/site-packages/nipype/pipeline/engine/utils.py", line 293, in load_resultfile
    result = loadpkl(results_file)
  File "/home/rabies/miniforge/lib/python3.9/site-packages/nipype/utils/filemanip.py", line 668, in loadpkl
    raise e
  File "/home/rabies/miniforge/lib/python3.9/site-packages/nipype/utils/filemanip.py", line 645, in loadpkl
    unpkl = pickle.loads(pkl_contents)
EOFError: Ran out of input

[Gab-D-G]

Hi, it seems like some outputs are missing. Is it possible that some outputs from the confound correction steps are missing? did you make sure the confound correction step completed without error?

[Siyi13]

yes I believe the confound correction steps are completed without errors (there were no signifcant errors I recognize in the log). However, I do notice in the analysis log there was line saying "no metadata was found in pkl file" and I wonder what are included in the pkl files? (When using a dataset of 8 subjects) For the preprocess, we have a 2kb "rabies_preprocess" and 619kb "rabies_preprocess_workflow". For the confound correction, we have a 1kb "rabies_confound_correction" and 85kb "rabies_confound_correction_workflow".
rabies_confound_correction.log

[Gab-D-G]

I can see from your log that it seems this was run a year ago. Is it possible that files were automatically removed from your scratch directory? Can you inspect the datasink folders and see if the expect outputs are present? Could you also print the outputs in the /confound_correction_outputs folder?

[gdevenyi]

If the processing was done a year ago, is there a version mismatch between preprocessing and this step?