Error during preprocessing and InhoCorrection with RABIES

[geowk]

Ran into issue running preprocessing with Rabies.

Running it locally. It skips the inhocorrection step altogether, but continues to run. I halted the processing, so the log file attached isn't complete but shows the message.

I'm running it locally due to the ITK_GLOBAL_DEFAULT_NUMBER_OF_THREADS issue #303 running it with singularity container.

I cloned the latest version from Github and used the setup file to install in a python virtual environment.

The dependencies are all installed locally: minctools 1.9.18, ANTS 2.5.1, AFNI, and FSL.

Here is were errors begin:

+ modelmask=/tmp/tmp.q0XGLrOAl9/modelmask.mnc
+ ThresholdImage 3 /tmp/tmp.q0XGLrOAl9/modelfile.mnc /tmp/tmp.q0XGLrOAl9/bgmask.h5 1e-12 Inf 1 0
/data/ANTS_05_02_1/build/ITKv5/Modules/ThirdParty/MINC/src/libminc/libsrc2/volume.c:1403 (from MINC): Unable to open file '/tmp/tmp.q0XGLrOAl9/modelfile.mnc'
Exception caught during reference file reading 

itk::ExceptionObject (0x560287733f50)
Location: "unknown" 
File: /data/ANTS_05_02_1/build/ITKv5/Modules/IO/MINC/src/itkMINCImageIO.cxx
Line: 312
Description: ITK ERROR: MINCImageIO(0x56028772cf60): Could not open file "/tmp/tmp.q0XGLrOAl9/modelfile.mnc".

 file /tmp/tmp.q0XGLrOAl9/modelfile.mnc
terminate called after throwing an instance of 'itk::ExceptionObject'
  what():  /data/ANTS_05_02_1/ITKv5/Modules/Core/Common/src/itkProcessObject.cxx:1339:
ITK ERROR: BinaryThresholdImageFilter(0x560287735480): Input Primary is required but not set.
/data/venv/new_env.venv/bin/structural-preprocessing.sh: line 31: 3069634 Aborted                 (core dumped) ThresholdImage 3 ${modelfile} ${tmpdir}/bgmask.h5 1e
-12 Inf 1 0

rabies_preprocess_copy.log

Interestingly enough, I receive the same error with an installment of rabies from PyPi and attempt to preprocess.

Thanks.

-g

[Gab-D-G]

Hi, something is likely wrong with the minc-toolkit package. Does this file exist /tmp/tmp.qd7HezTsWT/modelfile.mnc? Unfortunately we can't guarantee reproducible behavior of RABIES without using a container. You may try debugging your minc-toolkit installation. Have you tried error_check_rabies.py --complete to see if the execution is appropriate first?

[gdevenyi]

This error is from an IO component of ANTs. Note that the containers for RABIES use ANTs/2.5.0 and we have not tested/validated 2.5.1

[gdevenyi]

Following up, this is probably the origin of your issues:
minctools 1.9.18

The dependencies is minc-toolkit-v2 >= 1.9.18, minctools != minc-toolkit-v2 which is a larger metaproject with additional tools.

In particular, the environment configuration MINC_FORCE_V2=1 which forces all tools to save the V2 HDF5 based format. ITK's MINC reader can only read MINCV2 files.

So, fixing this immediate bug would be export MINC_FORCE_V2=1, however I expect you may run into other problems, as you may be missing other minc-based tools since you have the bare-minimum minc tools, not the complete meta-project.

[geowk]

Hi, something is likely wrong with the minc-toolkit package. Does this file exist /tmp/tmp.qd7HezTsWT/modelfile.mnc?

Yes, the file exists.

Additionally, is there way to assign the /tmp folder to another directory? The mnc files are filling up our /tmp partition that doesn't hold much space (~30GB). We'd like to assign the /tmp output to another larger directory where those files are stored during processing.

Have you tried error_check_rabies.py --complete to see if the execution is appropriate first?

Yes, execution works running error_check_rabies. However, it doesn't run the inhocor steps due to the token data.

Following up, this is probably the origin of your issues:
minctools 1.9.18

The dependencies is minc-toolkit-v2 >= 1.9.18, minctools != minc-toolkit-v2 which is a larger metaproject with additional tools.

My apologies. I meant to type minc-toolkit-v2 instead of minctools. I installed the toolkit based on these instructions. https://bic-mni.github.io/

In particular, the environment configuration MINC_FORCE_V2=1 which forces all tools to save the V2 HDF5 based format. ITK's MINC reader can only read MINCV2 files.

So, fixing this immediate bug would be export MINC_FORCE_V2=1, however I expect you may run into other problems, as you may be missing other minc-based tools since you have the bare-minimum minc tools, not the complete meta-project.

Set the MINC_FORCE_V2=1, and some of the steps producing the mnc files are now working.

However, I now receive this message when minc_anlm attempts to run during preprocessing:

+ mv -f /tmp/tmp.Jj1y94ZRNJ/repad_orig.mnc /tmp/tmp.Jj1y94ZRNJ/originput.mnc
+ mv -f /tmp/tmp.Jj1y94ZRNJ/repad.mnc /tmp/tmp.Jj1y94ZRNJ/originput.mnc
+ ThresholdImage 3 /tmp/tmp.Jj1y94ZRNJ/originput.mnc /tmp/tmp.Jj1y94ZRNJ/bgmask.mnc 1e-12 Inf 1 0
+ mincresample -like /tmp/tmp.Jj1y94ZRNJ/originput.mnc -keep -near -labels /tmp/tmp.Jj1y94ZRNJ/weight.mnc /tmp/tmp.Jj1y94ZRNJ/weight2.mnc
Transforming slices:.....................................................................................................................................Done
+ mv -f /tmp/tmp.Jj1y94ZRNJ/weight2.mnc /tmp/tmp.Jj1y94ZRNJ/weight.mnc
++ nproc
+ minc_anlm --clobber --mt 1 /tmp/tmp.Jj1y94ZRNJ/originput.mnc /tmp/tmp.Jj1y94ZRNJ/denoise.mnc
minc_anlm: error while loading shared libraries: libminc_io.so.5.3.0: cannot open shared object file: No such file or directory

-g

[gdevenyi]

However, I now receive this message when minc_anlm attempts to run during preprocessing:

How are you enabling the minc-toolkit-v2 commands in your terminal environment.

[geowk]

How are you enabling the minc-toolkit-v2 commands in your terminal environment.

Okay, I added this line in .bashrc

export MINC_TOOLKIT=/opt/minc/1.9.18

However, the instructions from ----> https://bic-mni.github.io/ describe running the .sh file

/opt/minc/1.9.18/minc-toolkit-config.sh

It contains the following:

#export MINC_FORCE_V2=1 
# minc-toolkit configuration parameters for 1.9.18-20200813
export MINC_TOOLKIT=/opt/minc/1.9.18
export MINC_TOOLKIT_VERSION="1.9.18-20200813"
export PATH=${MINC_TOOLKIT}/bin:${MINC_TOOLKIT}/pipeline:${PATH}
export PERL5LIB=${MINC_TOOLKIT}/perl:${MINC_TOOLKIT}/pipeline${PERL5LIB:+:$PERL5LIB}
export LD_LIBRARY_PATH=${MINC_TOOLKIT}/lib:${MINC_TOOLKIT}/lib/InsightToolkit${LD_LIBRARY_PATH:+:$LD_LIBRARY_PATH}
export MNI_DATAPATH=${MINC_TOOLKIT}/../share:${MINC_TOOLKIT}/share
export MINC_FORCE_V2=1
export MINC_COMPRESS=4
export VOLUME_CACHE_THRESHOLD=-1
export MANPATH=${MINC_TOOLKIT}/man${MANPATH:+:$MANPATH}
# integrated ANTs tools
export ANTSPATH=${MINC_TOOLKIT}/bin

Running it now, i still get crash reports while running InhoCorrection

One of those exports command for ANTS calls old versions of those commands. For antsRegistration:

ANTs Version: 2.2.0.dev133-gc7310
Compiled: Oct  3 2020 09:13:19

At this point, since I found a what seems to be a workaround with #303, no need at this point to running it locally. It runs fine with the singularity container.

Thank you for your help.

-g

[gdevenyi]

However, the instructions from ----> bic-mni.github.io describe running the .sh file
/opt/minc/1.9.18/minc-toolkit-config.sh

This is what I suspected, your MINC environment wasn't completely setup. However, it doesn't say "run" the file. It says do
source /opt/minc/1.9.18/minc-toolkit-config.sh which enables the settings in the current shell (you could put this line in ~/.bashrc)

To use the newer ants, you will also need to prepend your ANTs install to PATH after the MINC line:
export PATH=/path/to/ANTs/bin:${PATH}

[geowk]

Thank for your suggestions.

I put those lines of code in the home .bashrc file and added the 2.5.0 ants in the path.

I still get an error, but I think it is not related to minctoolkit.

Looks like there is an issue with a missing command called parallel from the structural-preprocessing.sh: line 138

We are using Ubuntu 20.04. Is this the same command 'parallel' one could install from the Ubuntu repository?

-g

Number of iterations: 25 
CV of field change: 0.00152041
+ n3input=/tmp/tmp.ZVVCeAuAwZ/corrected_3.91875_2.mnc
+ (( ++i ))
+ ((  i < cycles  ))
++ calc 3.91875/2
++ awk 'BEGIN { print 3.91875/2 }'
+ distance=1.95938
+ (( ++j ))
+ ((  j < levels  ))
+ parallel
+ for file in ${tmpdir}/*imp
/data/example_env.venv/bin/structural-preprocessing.sh: line 138: parallel: command not found
++ basename /tmp/tmp.ZVVCeAuAwZ/corrected_15.675_0.imp .imp
++ basename /tmp/tmp.ZVVCeAuAwZ/corrected_15.675_0.imp .imp
+ echo nu_evaluate -clobber -mapping /tmp/tmp.ZVVCeAuAwZ/corrected_15.675_0.imp -mask /tmp/tmp.ZVVCeAuAwZ/weight.mnc -field /tmp/tmp.ZVVCeAuAwZ/corrected_15.675_0_field.mnc /tmp/tmp.ZVVCeAuA
wZ/originput.mnc /tmp/tmp.ZVVCeAuAwZ/corrected_15.675_0.mnc

[gdevenyi]

Gnu-parallel is a dependency.