Mesmer in MacOS

[josenimo]

Description of the bug

Running nfcore/mcmicro test locally on a macOS (m2) runs into an issue with Mesmer.
Probably due to Docker container incompatibility with amd64 processing.

Command used and terminal output

nextflow run main.nf --input ./assets/samplesheet-test.csv --outdir ./output/ -profile docker

N E X T F L O W  ~  version 23.10.1
Launching `main.nf` [nauseous_cantor] DSL2 - revision: 68b73cf768
WARN: Access to undefined parameter `monochromeLogs` -- Initialise it to a default value eg. `params.monochromeLogs = some_value`


------------------------------------------------------
                                        ,--./,-.
        ___     __   __   __   ___     /,-._.--~'
  |\ | |__  __ /  ` /  \ |__) |__         }  {
  | \| |       \__, \__/ |  \ |___     \`-._,-`-,
                                        `._,._,'
  nf-core/mcmicro v2.0.0
------------------------------------------------------
Core Nextflow options
  runName        : nauseous_cantor
  containerEngine: docker
  launchDir      : /Users/jnimoca/Jose_BI/MCMICRO/mcmicro
  workDir        : /Users/jnimoca/Jose_BI/MCMICRO/mcmicro/work
  projectDir     : /Users/jnimoca/Jose_BI/MCMICRO/mcmicro
  userName       : jnimoca
  profile        : docker
  configFiles    : 

Input/output options
  input          : ./assets/samplesheet-test.csv
  outdir         : ./output/

!! Only displaying parameters that differ from the pipeline defaults !!
------------------------------------------------------
If you use nf-core/mcmicro for your analysis please cite:

* The pipeline
  

* The nf-core framework
  https://doi.org/10.1038/s41587-020-0439-x

* Software dependencies
  https://github.com/nf-core/mcmicro/blob/master/CITATIONS.md
------------------------------------------------------
executor >  local (2)
executor >  local (2)
executor >  local (2)
[56/3e9311] process > NFCORE_MCMICRO:MCMICRO:ASHLAR (TEST1)          [100%] 1 of 1 ✔
[23/f99a61] process > NFCORE_MCMICRO:MCMICRO:DEEPCELL_MESMER (TEST1) [100%] 1 of 1, failed: 1 ✘
[-        ] process > NFCORE_MCMICRO:MCMICRO:MCQUANT                 -
[-        ] process > NFCORE_MCMICRO:MCMICRO:MULTIQC                 -
Execution cancelled -- Finishing pending tasks before exit
-[nf-core/mcmicro] Pipeline completed with errors-
ERROR ~ Error executing process > 'NFCORE_MCMICRO:MCMICRO:DEEPCELL_MESMER (TEST1)'

Caused by:
  Process `NFCORE_MCMICRO:MCMICRO:DEEPCELL_MESMER (TEST1)` terminated with an error exit status (125)

Command executed:

  python /usr/src/app/run_app.py mesmer         --squeeze         --nuclear-image TEST1.ome.tif         --output-directory .         --output-name mask.tif                  --image-mpp=0.215 --nuclear-channel 0 --compartment nuclear
  
  cat <<-END_VERSIONS > versions.yml
  "NFCORE_MCMICRO:MCMICRO:DEEPCELL_MESMER":
      deepcell_mesmer:: 0.4.0
  END_VERSIONS

Command exit status:
  125

Command output:
  (empty)

Command error:
  Unable to find image 'vanvalenlab/deepcell-applications:0.4.1' locally
  0.4.1: Pulling from vanvalenlab/deepcell-applications
  docker: no matching manifest for linux/arm64/v8 in the manifest list entries.
  See 'docker run --help'.

Work dir:
  /Users/jnimoca/Jose_BI/MCMICRO/mcmicro/work/23/f99a614e8a4663dda3090b8e713ba2

Tip: you can try to figure out what's wrong by changing to the process work dir and showing the script file named `.command.sh`

 -- Check '.nextflow.log' file for details

Relevant files

nextflow.log

System information

Nextflow version 23.10.1
Hardware: MacBook Pro Apple M2 Pro 16Gb Memory
Executor: Local
Container Engine: Docker
OS: Sonoma 14.0 (23A344)
nfcore/mcmicro version: https://github.com/jmuhlich/mcmicro/tree/sample_sheet_mod

[josenimo]

Just to add a bit more context.
I tried running it in our HPC, with singularity and got a different error :(
To be fair I don't think this is running the lastest PR changes that Jeremy staged.
I think we should rather wait for those changes, and then it should be all good.
For some reason, I could not run jmuhlich/mcmicro -r sample_sheet_mod, kept running into

(java) nextflow run jmuhlich/mcmicro -r sample_sheet_mod -profile test,singularity --outdir ./testoutput/
N E X T F L O W  ~  version 23.10.1
Can't find git repository config file -- Repository may be corrupted: /home/jnimoca/.nextflow/assets/jmuhlich/mcmicro

Full traceback of MESMER with singularity

(java) nextflow run nf-core/mcmicro -r dev -profile test,singularity --outdir ./testoutput/
N E X T F L O W  ~  version 23.10.1
Launching `https://github.com/nf-core/mcmicro` [lonely_leavitt] DSL2 - revision: 5ab80be22f [dev]
WARN: Access to undefined parameter `monochromeLogs` -- Initialise it to a default value eg. `params.monochromeLogs = some_value`


------------------------------------------------------
                                        ,--./,-.
        ___     __   __   __   ___     /,-._.--~'
  |\ | |__  __ /  ` /  \ |__) |__         }  {
  | \| |       \__, \__/ |  \ |___     \`-._,-`-,
                                        `._,._,'
  nf-core/mcmicro v2.0.0-g5ab80be
------------------------------------------------------
Core Nextflow options
  revision                  : dev
  runName                   : lonely_leavitt
  containerEngine           : singularity
  launchDir                 : /fast/home/j/jnimoca
  workDir                   : /fast/AG_Coscia/Jose/Nextflow/Work
  projectDir                : /home/jnimoca/.nextflow/assets/nf-core/mcmicro
  userName                  : jnimoca
  profile                   : test,singularity
  configFiles               :

Input/output options
  input                     : /home/jnimoca/.nextflow/assets/nf-core/mcmicro/assets/samplesheet-test.csv
  outdir                    : ./testoutput/

Institutional config options
  config_profile_name       : Test profile
  config_profile_description: Minimal test dataset to check pipeline function

Max job request options
  max_cpus                  : 2
  max_memory                : 6.GB
  max_time                  : 6.h

!! Only displaying parameters that differ from the pipeline defaults !!
------------------------------------------------------
If you use nf-core/mcmicro for your analysis please cite:

* The pipeline


* The nf-core framework
  https://doi.org/10.1038/s41587-020-0439-x

* Software dependencies
  https://github.com/nf-core/mcmicro/blob/master/CITATIONS.md
------------------------------------------------------
[-        ] process > NFCORE_MCMICRO:MCMICRO:ASHLAR          -
executor >  local (1)
executor >  local (1)
executor >  local (2)
[28/d83b3b] process > NFCORE_MCMICRO:MCMICRO:ASHLAR (TEST1)          [100%] 1 of 1 ✔
[c7/dda7a9] process > NFCORE_MCMICRO:MCMICRO:DEEPCELL_MESMER (TEST1) [  0%] 0 of 1
[-        ] process > NFCORE_MCMICRO:MCMICRO:MCQUANT                 -
[-        ] process > NFCORE_MCMICRO:MCMICRO:MULTIQC                 -
ERROR ~ Error executing process > 'NFCORE_MCMICRO:MCMICRO:DEEPCELL_MESMER (TEST1)'

Caused by:
  Process `NFCORE_MCMICRO:MCMICRO:DEEPCELL_MESMER (TEST1)` terminated with an error exit status (1)

Command executed:

  python /usr/src/app/run_app.py mesmer         --squeeze         --nuclear-image TEST1.ome.tif         --output-directory .         --output-name mask.tif                  --image-mpp=0.215 --nuclear-channel 0 --compartment nuclear

  cat <<-END_VERSIONS > versions.yml
  "NFCORE_MCMICRO:MCMICRO:DEEPCELL_MESMER":
      deepcell_mesmer:: 0.4.0
  END_VERSIONS

Command exit status:
  1

Command output:
        --no-mount strings              disable one or more 'mount xxx'
                                        options set in apptainer.conf and/or
                                        specify absolute destination path to
                                        disable a bind path entry, or
                                        'bind-paths' to disable all bind
                                        path entries.
        --no-pid                        do not run container in a new PID
                                        namespace
        --no-privs                      drop all privileges from root user
                                        in container)
        --no-umask                      do not propagate umask to the
                                        container, set default 0022 umask
        --nv                            enable Nvidia support
        --nvccli                        use nvidia-container-cli for GPU
                                        setup (experimental)
        --oom-kill-disable              Disable OOM killer
    -o, --overlay strings               use an overlayFS image for
                                        persistent data storage or as
                                        read-only layer of container
        --passphrase                    prompt for an encryption passphrase
        --pem-path string               enter an path to a PEM formatted RSA
                                        key for an encrypted container
    -p, --pid                           run container in a new PID namespace
        --pids-limit int                Limit number of container PIDs, use
                                        -1 for unlimited
        --rocm                          enable experimental Rocm support
    -S, --scratch strings               include a scratch directory within
                                        the container that is linked to a
                                        temporary dir (use -W to force location)
        --security strings              enable security features (SELinux,
                                        Apparmor, Seccomp)
        --sharens                       share the namespace and image with
                                        other containers launched from the
                                        same parent process
        --unsquash                      Convert SIF file to temporary
                                        sandbox before running
    -u, --userns                        run container in a new user namespace
        --uts                           run container in a new UTS namespace
    -W, --workdir string                working directory to be used for
                                        /tmp, /var/tmp and $HOME (if
                                        -c/--contain was also used)
    -w, --writable                      by default all Apptainer containers
                                        are available as read only. This
                                        option makes the file system
executor >  local (2)
[28/d83b3b] process > NFCORE_MCMICRO:MCMICRO:ASHLAR (TEST1)          [100%] 1 of 1 ✔
[c7/dda7a9] process > NFCORE_MCMICRO:MCMICRO:DEEPCELL_MESMER (TEST1) [100%] 1 of 1, failed: 1 ✘
[-        ] process > NFCORE_MCMICRO:MCMICRO:MCQUANT                 -
[-        ] process > NFCORE_MCMICRO:MCMICRO:MULTIQC                 -
Execution cancelled -- Finishing pending tasks before exit
-[nf-core/mcmicro] Pipeline completed with errors-
ERROR ~ Error executing process > 'NFCORE_MCMICRO:MCMICRO:DEEPCELL_MESMER (TEST1)'

Caused by:
  Process `NFCORE_MCMICRO:MCMICRO:DEEPCELL_MESMER (TEST1)` terminated with an error exit status (1)

Command executed:

  python /usr/src/app/run_app.py mesmer         --squeeze         --nuclear-image TEST1.ome.tif         --output-directory .         --output-name mask.tif                  --image-mpp=0.215 --nuclear-channel 0 --compartment nuclear

  cat <<-END_VERSIONS > versions.yml
  "NFCORE_MCMICRO:MCMICRO:DEEPCELL_MESMER":
      deepcell_mesmer:: 0.4.0
  END_VERSIONS

Command exit status:
  1

Command output:
        --no-mount strings              disable one or more 'mount xxx'
                                        options set in apptainer.conf and/or
                                        specify absolute destination path to
                                        disable a bind path entry, or
                                        'bind-paths' to disable all bind
                                        path entries.
        --no-pid                        do not run container in a new PID
                                        namespace
        --no-privs                      drop all privileges from root user
                                        in container)
        --no-umask                      do not propagate umask to the
                                        container, set default 0022 umask
        --nv                            enable Nvidia support
        --nvccli                        use nvidia-container-cli for GPU
                                        setup (experimental)
        --oom-kill-disable              Disable OOM killer
    -o, --overlay strings               use an overlayFS image for
                                        persistent data storage or as
                                        read-only layer of container
        --passphrase                    prompt for an encryption passphrase
        --pem-path string               enter an path to a PEM formatted RSA
                                        key for an encrypted container
    -p, --pid                           run container in a new PID namespace
        --pids-limit int                Limit number of container PIDs, use
                                        -1 for unlimited
        --rocm                          enable experimental Rocm support
    -S, --scratch strings               include a scratch directory within
                                        the container that is linked to a
                                        temporary dir (use -W to force location)
        --security strings              enable security features (SELinux,
                                        Apparmor, Seccomp)
        --sharens                       share the namespace and image with
                                        other containers launched from the
                                        same parent process
        --unsquash                      Convert SIF file to temporary
                                        sandbox before running
    -u, --userns                        run container in a new user namespace
        --uts                           run container in a new UTS namespace
    -W, --workdir string                working directory to be used for
                                        /tmp, /var/tmp and $HOME (if
                                        -c/--contain was also used)
    -w, --writable                      by default all Apptainer containers
                                        are available as read only. This
                                        option makes the file system
executor >  local (2)
[28/d83b3b] process > NFCORE_MCMICRO:MCMICRO:ASHLAR (TEST1)          [100%] 1 of 1 ✔
[c7/dda7a9] process > NFCORE_MCMICRO:MCMICRO:DEEPCELL_MESMER (TEST1) [100%] 1 of 1, failed: 1 ✘
[-        ] process > NFCORE_MCMICRO:MCMICRO:MCQUANT                 -
[-        ] process > NFCORE_MCMICRO:MCMICRO:MULTIQC                 -
Execution cancelled -- Finishing pending tasks before exit
-[nf-core/mcmicro] Pipeline completed with errors-
ERROR ~ Error executing process > 'NFCORE_MCMICRO:MCMICRO:DEEPCELL_MESMER (TEST1)'

Caused by:
  Process `NFCORE_MCMICRO:MCMICRO:DEEPCELL_MESMER (TEST1)` terminated with an error exit status (1)

Command executed:

  python /usr/src/app/run_app.py mesmer         --squeeze         --nuclear-image TEST1.ome.tif         --output-directory .         --output-name mask.tif                  --image-mpp=0.215 --nuclear-channel 0 --compartment nuclear

  cat <<-END_VERSIONS > versions.yml
  "NFCORE_MCMICRO:MCMICRO:DEEPCELL_MESMER":
      deepcell_mesmer:: 0.4.0
  END_VERSIONS

Command exit status:
  1

Command output:
        --no-mount strings              disable one or more 'mount xxx'
                                        options set in apptainer.conf and/or
                                        specify absolute destination path to
                                        disable a bind path entry, or
                                        'bind-paths' to disable all bind
                                        path entries.
        --no-pid                        do not run container in a new PID
                                        namespace
        --no-privs                      drop all privileges from root user
                                        in container)
        --no-umask                      do not propagate umask to the
                                        container, set default 0022 umask
        --nv                            enable Nvidia support
        --nvccli                        use nvidia-container-cli for GPU
                                        setup (experimental)
        --oom-kill-disable              Disable OOM killer
    -o, --overlay strings               use an overlayFS image for
                                        persistent data storage or as
                                        read-only layer of container
        --passphrase                    prompt for an encryption passphrase
        --pem-path string               enter an path to a PEM formatted RSA
                                        key for an encrypted container
    -p, --pid                           run container in a new PID namespace
        --pids-limit int                Limit number of container PIDs, use
                                        -1 for unlimited
        --rocm                          enable experimental Rocm support
    -S, --scratch strings               include a scratch directory within
                                        the container that is linked to a
                                        temporary dir (use -W to force location)
        --security strings              enable security features (SELinux,
                                        Apparmor, Seccomp)
        --sharens                       share the namespace and image with
                                        other containers launched from the
                                        same parent process
        --unsquash                      Convert SIF file to temporary
                                        sandbox before running
    -u, --userns                        run container in a new user namespace
        --uts                           run container in a new UTS namespace
    -W, --workdir string                working directory to be used for
                                        /tmp, /var/tmp and $HOME (if
                                        -c/--contain was also used)
    -w, --writable                      by default all Apptainer containers
                                        are available as read only. This
                                        option makes the file system
                                        accessible as read/write.
        --writable-tmpfs                makes the file system accessible as
                                        read-write with non persistent data
                                        (with overlay support only)

  Run 'apptainer --help' for more detailed usage information.

Command error:
        --no-mount strings              disable one or more 'mount xxx'
                                        options set in apptainer.conf and/or
                                        specify absolute destination path to
                                        disable a bind path entry, or
                                        'bind-paths' to disable all bind
                                        path entries.
        --no-pid                        do not run container in a new PID
                                        namespace
        --no-privs                      drop all privileges from root user
                                        in container)
        --no-umask                      do not propagate umask to the
                                        container, set default 0022 umask
        --nv                            enable Nvidia support
        --nvccli                        use nvidia-container-cli for GPU
                                        setup (experimental)
        --oom-kill-disable              Disable OOM killer
    -o, --overlay strings               use an overlayFS image for
                                        persistent data storage or as
                                        read-only layer of container
        --passphrase                    prompt for an encryption passphrase
        --pem-path string               enter an path to a PEM formatted RSA
                                        key for an encrypted container
    -p, --pid                           run container in a new PID namespace
        --pids-limit int                Limit number of container PIDs, use
                                        -1 for unlimited
        --rocm                          enable experimental Rocm support
    -S, --scratch strings               include a scratch directory within
                                        the container that is linked to a
                                        temporary dir (use -W to force location)
        --security strings              enable security features (SELinux,
                                        Apparmor, Seccomp)
        --sharens                       share the namespace and image with
                                        other containers launched from the
                                        same parent process
        --unsquash                      Convert SIF file to temporary
                                        sandbox before running
    -u, --userns                        run container in a new user namespace
        --uts                           run container in a new UTS namespace
    -W, --workdir string                working directory to be used for
                                        /tmp, /var/tmp and $HOME (if
                                        -c/--contain was also used)
    -w, --writable                      by default all Apptainer containers
                                        are available as read only. This
                                        option makes the file system
                                        accessible as read/write.
        --writable-tmpfs                makes the file system accessible as
                                        read-write with non persistent data
                                        (with overlay support only)

  Run 'apptainer --help' for more detailed usage information.

Work dir:
  /fast/AG_Coscia/Jose/Nextflow/Work/c7/dda7a931e80759de129e1ed29d37b9

Tip: you can try to figure out what's wrong by changing to the process work dir and showing the script file named `.command.sh`

 -- Check '.nextflow.log' file for details

[jmuhlich]

Regarding the inability to run jmuhlich/mcmicro, run rm -rf ~/.nextflow/assets/jmuhlich and try again?

I think the mesmer/singularity problem is caused by the line containerOptions = '--entrypoint ""' in the DEEPCELL_MESMER section of conf/modules.config. I had the same problem on our HPC cluster. Anything with containerOptions will actually cause a problem because docker and singularity (and other containerization tools) take different options so there's no way to use that portably. We should actually not specify containerOptions at all in the pipeline config -- that option should be left open for end users to customize for their own environment (in which they will know whether they are using docker or singularity). This will require us to hunt down the reasons why containerOptions is needed and fix them either in our own config or by submitting fixes to the upstream tools.

[josenimo]

ran rm -rf ~/.nextflow/assets/jmuhlich and then ran
(java) nextflow run jmuhlich/mcmicro -r sample_sheet_mod -profile test,singularity --outdir ./testoutput/

Traceback

N E X T F L O W  ~  version 23.10.1
Pulling jmuhlich/mcmicro ...
 downloaded from https://github.com/jmuhlich/mcmicro.git
Launching `https://github.com/jmuhlich/mcmicro` [hopeful_moriondo] DSL2 - revision: eb69c1ca66 [sample_sheet_mod]
WARN: Access to undefined parameter `monochromeLogs` -- Initialise it to a default value eg. `params.monochromeLogs = some_value`


------------------------------------------------------
                                        ,--./,-.
        ___     __   __   __   ___     /,-._.--~'
  |\ | |__  __ /  ` /  \ |__) |__         }  {
  | \| |       \__, \__/ |  \ |___     \`-._,-`-,
                                        `._,._,'
  nf-core/mcmicro v2.0.0-geb69c1c
------------------------------------------------------
Core Nextflow options
  revision                  : sample_sheet_mod
  runName                   : hopeful_moriondo
  containerEngine           : singularity
  launchDir                 : /fast/home/j/jnimoca
  workDir                   : /fast/AG_Coscia/Jose/Nextflow/Work
  projectDir                : /home/jnimoca/.nextflow/assets/jmuhlich/mcmicro
  userName                  : jnimoca
  profile                   : test,singularity
  configFiles               :

Input/output options
  input_cycle               : /home/jnimoca/.nextflow/assets/jmuhlich/mcmicro/assets/samplesheet-test.csv
  outdir                    : ./testoutput/
  marker_sheet              : /home/jnimoca/.nextflow/assets/jmuhlich/mcmicro/assets/markers-test.csv

Institutional config options
  config_profile_name       : Test profile
  config_profile_description: Minimal test dataset to check pipeline function

Max job request options
  max_cpus                  : 2
  max_memory                : 6.GB
  max_time                  : 6.h

!! Only displaying parameters that differ from the pipeline defaults !!
------------------------------------------------------
If you use nf-core/mcmicro for your analysis please cite:

* The pipeline


* The nf-core framework
  https://doi.org/10.1038/s41587-020-0439-x

* Software dependencies
  https://github.com/nf-core/mcmicro/blob/master/CITATIONS.md
------------------------------------------------------
[-        ] process > NFCORE_MCMICRO:MCMICRO:INPUT_CHECK:SAMPLE_AND_MARKER_SHEET_CHECK -
[-        ] process > NFCORE_MCMICRO:MCMICRO:INPUT_CHECK:SAMPLE_AND_MARKER_SHEET_CHECK -
executor >  local (1)
[-        ] process > NFCORE_MCMICRO:MCMICRO:INPUT_CHECK:SAMPLE_AND_MARKER_SHEET_CHECK -
[-        ] process > NFCORE_MCMICRO:MCMICRO:INPUT_CHECK:MARKER_SHEET_CHECK            -
[55/9d1812] process > NFCORE_MCMICRO:MCMICRO:ASHLAR (TEST1)                            [  0%] 0 of 1
[-        ] process > NFCORE_MCMICRO:MCMICRO:DEEPCELL_MESMER                           -
[-        ] process > NFCORE_MCMICRO:MCMICRO:MCQUANT                                   -
[-        ] process > NFCORE_MCMICRO:MCMICRO:MULTIQC                                   -
Pulling Singularity image https://depot.galaxyproject.org/singularity/python:3.8.3 [cache /fast/AG_Coscia/Jose/Nextflow/Singularity/depot.galaxyproject.org-singularity-python-3.8.3.img]
ERROR ~ Error executing process > 'NFCORE_MCMICRO:MCMICRO:ASHLAR (TEST1)'

Caused by:
  Process `NFCORE_MCMICRO:MCMICRO:ASHLAR (TEST1)` terminated with an error exit status (1)

Command executed:

  ashlar \
      -o TEST1.ome.tif \
      cycif-tonsil-cycle1.ome.tif \
       \
       \


  sed -i -E 's/UUID="urn:uuid:[[:xdigit:]]{8}-[[:xdigit:]]{4}-[[:xdigit:]]{4}-[[:xdigit:]]{4}-[[:xdigit:]]{12}"/                                                    /g' TEST1.ome.tif

  cat <<-END_VERSIONS > versions.yml
  "NFCORE_MCMICRO:MCMICRO:ASHLAR":
      ashlar: $(ashlar --version | sed 's/^.*ashlar //' )
  END_VERSIONS

Command exit status:
  1

Command output:
        --no-mount strings              disable one or more 'mount xxx'
                                        options set in apptainer.conf and/or
                                        specify absolute destination path to
                                        disable a bind path entry, or
                                        'bind-paths' to disable all bind
                                        path entries.
        --no-pid                        do not run container in a new PID
                                        namespace
        --no-privs                      drop all privileges from root user
                                        in container)
        --no-umask                      do not propagate umask to the
                                        container, set default 0022 umask
        --nv                            enable Nvidia support
        --nvccli                        use nvidia-container-cli for GPU
                                        setup (experimental)
        --oom-kill-disable              Disable OOM killer
    -o, --overlay strings               use an overlayFS image for
                                        persistent data storage or as
                                        read-only layer of container
        --passphrase                    prompt for an encryption passphrase
        --pem-path string               enter an path to a PEM formatted RSA
                                        key for an encrypted container
    -p, --pid                           run container in a new PID namespace
        --pids-limit int                Limit number of container PIDs, use
                                        -1 for unlimited
        --rocm                          enable experimental Rocm support
    -S, --scratch strings               include a scratch directory within
                                        the container that is linked to a
                                        temporary dir (use -W to force location)
        --security strings              enable security features (SELinux,
                                        Apparmor, Seccomp)
        --sharens                       share the namespace and image with
                                        other containers launched from the
                                        same parent process
        --unsquash                      Convert SIF file to temporary
                                        sandbox before running
    -u, --userns                        run container in a new user namespace
        --uts                           run container in a new UTS namespace
    -W, --workdir string                working directory to be used for
                                        /tmp, /var/tmp and $HOME (if
                                        -c/--contain was also used)
    -w, --writable                      by default all Apptainer containers
                                        are available as read only. This
                                        option makes the file system
                                        accessible as read/write.
        --writable-tmpfs                makes the file system accessible as
                                        read-write with non persistent data
                                        (with overlay support only)

  Run 'apptainer --help' for more detailed usage information.

Command error:
        --no-mount strings              disable one or more 'mount xxx'
                                        options set in apptainer.conf and/or
                                        specify absolute destination path to
                                        disable a bind path entry, or
                                        'bind-paths' to disable all bind
                                        path entries.
        --no-pid                        do not run container in a new PID
                                        namespace
        --no-privs                      drop all privileges from root user
                                        in container)
        --no-umask                      do not propagate umask to the
                                        container, set default 0022 umask
        --nv                            enable Nvidia support
        --nvccli                        use nvidia-container-cli for GPU
                                        setup (experimental)
        --oom-kill-disable              Disable OOM killer
    -o, --overlay strings               use an overlayFS image for
                                        persistent data storage or as
                                        read-only layer of container
        --passphrase                    prompt for an encryption passphrase
        --pem-path string               enter an path to a PEM formatted RSA
                                        key for an encrypted container
    -p, --pid                           run container in a new PID namespace
        --pids-limit int                Limit number of container PIDs, use
                                        -1 for unlimited
        --rocm                          enable experimental Rocm support
    -S, --scratch strings               include a scratch directory within
                                        the container that is linked to a
                                        temporary dir (use -W to force location)
        --security strings              enable security features (SELinux,
                                        Apparmor, Seccomp)
        --sharens                       share the namespace and image with
                                        other containers launched from the
                                        same parent process
executor >  local (1)
[-        ] process > NFCORE_MCMICRO:MCMICRO:INPUT_CHECK:SAMPLE_AND_MARKER_SHEET_CHECK -
[-        ] process > NFCORE_MCMICRO:MCMICRO:INPUT_CHECK:MARKER_SHEET_CHECK            -
[55/9d1812] process > NFCORE_MCMICRO:MCMICRO:ASHLAR (TEST1)                            [100%] 1 of 1, failed: 1 ✘
[-        ] process > NFCORE_MCMICRO:MCMICRO:DEEPCELL_MESMER                           -
[-        ] process > NFCORE_MCMICRO:MCMICRO:MCQUANT                                   -
[-        ] process > NFCORE_MCMICRO:MCMICRO:MULTIQC                                   -
Pulling Singularity image https://depot.galaxyproject.org/singularity/python:3.8.3 [cache /fast/AG_Coscia/Jose/Nextflow/Singularity/depot.galaxyproject.org-singularity-python-3.8.3.img]
Execution cancelled -- Finishing pending tasks before exit
Pulling Singularity image https://depot.galaxyproject.org/singularity/multiqc:1.21--pyhdfd78af_0 [cache /fast/AG_Coscia/Jose/Nextflow/Singularity/depot.galaxyproject.org-singularity-multiqc-1.21--pyhdfd78af_0.img]
-[nf-core/mcmicro] Pipeline completed with errors-
ERROR ~ Error executing process > 'NFCORE_MCMICRO:MCMICRO:ASHLAR (TEST1)'

Caused by:
  Process `NFCORE_MCMICRO:MCMICRO:ASHLAR (TEST1)` terminated with an error exit status (1)

Command executed:

  ashlar \
      -o TEST1.ome.tif \
      cycif-tonsil-cycle1.ome.tif \
       \
       \


  sed -i -E 's/UUID="urn:uuid:[[:xdigit:]]{8}-[[:xdigit:]]{4}-[[:xdigit:]]{4}-[[:xdigit:]]{4}-[[:xdigit:]]{12}"/                                                    /g' TEST1.ome.tif

  cat <<-END_VERSIONS > versions.yml
  "NFCORE_MCMICRO:MCMICRO:ASHLAR":
      ashlar: $(ashlar --version | sed 's/^.*ashlar //' )
  END_VERSIONS

Command exit status:
  1

Command output:
        --no-mount strings              disable one or more 'mount xxx'
                                        options set in apptainer.conf and/or
                                        specify absolute destination path to
                                        disable a bind path entry, or
                                        'bind-paths' to disable all bind
                                        path entries.
        --no-pid                        do not run container in a new PID
                                        namespace
        --no-privs                      drop all privileges from root user
                                        in container)
        --no-umask                      do not propagate umask to the
                                        container, set default 0022 umask
        --nv                            enable Nvidia support
        --nvccli                        use nvidia-container-cli for GPU
                                        setup (experimental)
        --oom-kill-disable              Disable OOM killer
    -o, --overlay strings               use an overlayFS image for
                                        persistent data storage or as
                                        read-only layer of container
        --passphrase                    prompt for an encryption passphrase
        --pem-path string               enter an path to a PEM formatted RSA
                                        key for an encrypted container
    -p, --pid                           run container in a new PID namespace
        --pids-limit int                Limit number of container PIDs, use
                                        -1 for unlimited
        --rocm                          enable experimental Rocm support
    -S, --scratch strings               include a scratch directory within
                                        the container that is linked to a
                                        temporary dir (use -W to force location)
        --security strings              enable security features (SELinux,
                                        Apparmor, Seccomp)
        --sharens                       share the namespace and image with
                                        other containers launched from the
                                        same parent process
        --unsquash                      Convert SIF file to temporary
                                        sandbox before running
    -u, --userns                        run container in a new user namespace
        --uts                           run container in a new UTS namespace
    -W, --workdir string                working directory to be used for
                                        /tmp, /var/tmp and $HOME (if
                                        -c/--contain was also used)
    -w, --writable                      by default all Apptainer containers
                                        are available as read only. This
                                        option makes the file system
                                        accessible as read/write.
        --writable-tmpfs                makes the file system accessible as
                                        read-write with non persistent data
                                        (with overlay support only)

  Run 'apptainer --help' for more detailed usage information.

Command error:
        --no-mount strings              disable one or more 'mount xxx'
                                        options set in apptainer.conf and/or
                                        specify absolute destination path to
                                        disable a bind path entry, or
                                        'bind-paths' to disable all bind
                                        path entries.
        --no-pid                        do not run container in a new PID
                                        namespace
        --no-privs                      drop all privileges from root user
                                        in container)
        --no-umask                      do not propagate umask to the
                                        container, set default 0022 umask
        --nv                            enable Nvidia support
        --nvccli                        use nvidia-container-cli for GPU
                                        setup (experimental)
        --oom-kill-disable              Disable OOM killer
    -o, --overlay strings               use an overlayFS image for
                                        persistent data storage or as
                                        read-only layer of container
        --passphrase                    prompt for an encryption passphrase
        --pem-path string               enter an path to a PEM formatted RSA
                                        key for an encrypted container
    -p, --pid                           run container in a new PID namespace
        --pids-limit int                Limit number of container PIDs, use
                                        -1 for unlimited
        --rocm                          enable experimental Rocm support
    -S, --scratch strings               include a scratch directory within
                                        the container that is linked to a
                                        temporary dir (use -W to force location)
        --security strings              enable security features (SELinux,
                                        Apparmor, Seccomp)
        --sharens                       share the namespace and image with
                                        other containers launched from the
                                        same parent process
executor >  local (1)
[-        ] process > NFCORE_MCMICRO:MCMICRO:INPUT_CHECK:SAMPLE_AND_MARKER_SHEET_CHECK -
[-        ] process > NFCORE_MCMICRO:MCMICRO:INPUT_CHECK:MARKER_SHEET_CHECK            -
[55/9d1812] process > NFCORE_MCMICRO:MCMICRO:ASHLAR (TEST1)                            [100%] 1 of 1, failed: 1 ✘
[-        ] process > NFCORE_MCMICRO:MCMICRO:DEEPCELL_MESMER                           -
[-        ] process > NFCORE_MCMICRO:MCMICRO:MCQUANT                                   -
[-        ] process > NFCORE_MCMICRO:MCMICRO:MULTIQC                                   -
Pulling Singularity image https://depot.galaxyproject.org/singularity/python:3.8.3 [cache /fast/AG_Coscia/Jose/Nextflow/Singularity/depot.galaxyproject.org-singularity-python-3.8.3.img]
Execution cancelled -- Finishing pending tasks before exit
Pulling Singularity image https://depot.galaxyproject.org/singularity/multiqc:1.21--pyhdfd78af_0 [cache /fast/AG_Coscia/Jose/Nextflow/Singularity/depot.galaxyproject.org-singularity-multiqc-1.21--pyhdfd78af_0.img]
-[nf-core/mcmicro] Pipeline completed with errors-
ERROR ~ Error executing process > 'NFCORE_MCMICRO:MCMICRO:ASHLAR (TEST1)'

Caused by:
  Process `NFCORE_MCMICRO:MCMICRO:ASHLAR (TEST1)` terminated with an error exit status (1)

Command executed:

  ashlar \
      -o TEST1.ome.tif \
      cycif-tonsil-cycle1.ome.tif \
       \
       \


  sed -i -E 's/UUID="urn:uuid:[[:xdigit:]]{8}-[[:xdigit:]]{4}-[[:xdigit:]]{4}-[[:xdigit:]]{4}-[[:xdigit:]]{12}"/                                                    /g' TEST1.ome.tif

  cat <<-END_VERSIONS > versions.yml
  "NFCORE_MCMICRO:MCMICRO:ASHLAR":
      ashlar: $(ashlar --version | sed 's/^.*ashlar //' )
  END_VERSIONS

Command exit status:
  1

Command output:
        --no-mount strings              disable one or more 'mount xxx'
                                        options set in apptainer.conf and/or
                                        specify absolute destination path to
                                        disable a bind path entry, or
                                        'bind-paths' to disable all bind
                                        path entries.
        --no-pid                        do not run container in a new PID
                                        namespace
        --no-privs                      drop all privileges from root user
                                        in container)
        --no-umask                      do not propagate umask to the
                                        container, set default 0022 umask
        --nv                            enable Nvidia support
        --nvccli                        use nvidia-container-cli for GPU
                                        setup (experimental)
        --oom-kill-disable              Disable OOM killer
    -o, --overlay strings               use an overlayFS image for
                                        persistent data storage or as
                                        read-only layer of container
        --passphrase                    prompt for an encryption passphrase
        --pem-path string               enter an path to a PEM formatted RSA
                                        key for an encrypted container
    -p, --pid                           run container in a new PID namespace
        --pids-limit int                Limit number of container PIDs, use
                                        -1 for unlimited
        --rocm                          enable experimental Rocm support
    -S, --scratch strings               include a scratch directory within
                                        the container that is linked to a
                                        temporary dir (use -W to force location)
        --security strings              enable security features (SELinux,
                                        Apparmor, Seccomp)
        --sharens                       share the namespace and image with
                                        other containers launched from the
                                        same parent process
        --unsquash                      Convert SIF file to temporary
                                        sandbox before running
    -u, --userns                        run container in a new user namespace
        --uts                           run container in a new UTS namespace
    -W, --workdir string                working directory to be used for
                                        /tmp, /var/tmp and $HOME (if
                                        -c/--contain was also used)
    -w, --writable                      by default all Apptainer containers
                                        are available as read only. This
                                        option makes the file system
                                        accessible as read/write.
        --writable-tmpfs                makes the file system accessible as
                                        read-write with non persistent data
                                        (with overlay support only)

  Run 'apptainer --help' for more detailed usage information.

Command error:
        --no-mount strings              disable one or more 'mount xxx'
                                        options set in apptainer.conf and/or
                                        specify absolute destination path to
                                        disable a bind path entry, or
                                        'bind-paths' to disable all bind
                                        path entries.
        --no-pid                        do not run container in a new PID
                                        namespace
        --no-privs                      drop all privileges from root user
                                        in container)
        --no-umask                      do not propagate umask to the
                                        container, set default 0022 umask
        --nv                            enable Nvidia support
        --nvccli                        use nvidia-container-cli for GPU
                                        setup (experimental)
        --oom-kill-disable              Disable OOM killer
    -o, --overlay strings               use an overlayFS image for
                                        persistent data storage or as
                                        read-only layer of container
        --passphrase                    prompt for an encryption passphrase
        --pem-path string               enter an path to a PEM formatted RSA
                                        key for an encrypted container
    -p, --pid                           run container in a new PID namespace
        --pids-limit int                Limit number of container PIDs, use
                                        -1 for unlimited
        --rocm                          enable experimental Rocm support
    -S, --scratch strings               include a scratch directory within
                                        the container that is linked to a
                                        temporary dir (use -W to force location)
        --security strings              enable security features (SELinux,
                                        Apparmor, Seccomp)
        --sharens                       share the namespace and image with
                                        other containers launched from the
                                        same parent process
        --unsquash                      Convert SIF file to temporary
                                        sandbox before running
    -u, --userns                        run container in a new user namespace
        --uts                           run container in a new UTS namespace
    -W, --workdir string                working directory to be used for
                                        /tmp, /var/tmp and $HOME (if
                                        -c/--contain was also used)
    -w, --writable                      by default all Apptainer containers
                                        are available as read only. This
                                        option makes the file system
                                        accessible as read/write.
        --writable-tmpfs                makes the file system accessible as
                                        read-write with non persistent data
                                        (with overlay support only)

  Run 'apptainer --help' for more detailed usage information.

Work dir:
  /fast/AG_Coscia/Jose/Nextflow/Work/55/9d18123aaceaf9336cd2ee2b1bc547

Tip: you can replicate the issue by changing to the process work dir and entering the command `bash .command.run`

 -- Check '.nextflow.log' file for details

Seems there is an issue with container and the exec command, I am adding the Work files as a zip here:
Archive.zip

[josenimo]

Regarding the MESMER containerOptions, thank you for explaining, that makes sense.
I remember @FloWuenne dealing with this in the Cellpose module. Hopefully the fix is similar.

[FloWuenne]

Thanks for pinging me @josenimo ! The reason we had to put containerOptions is that the official Mesmer container unfortunately specifies an ENTRYPOINT which will break the nf-core/mesmer module. In the newest version of the mesmer module, we have replaced the official image with one hosted on my dockerhub, where we removed the Entrypoint and the containerOptions can thus safely be removed from the modules config.

I would try nf-core modules update deepcell/mesmer, which should pull the latest version. Hope this fixes the issue!

[jmuhlich]

Yes, the mesmer module has already been updated in #29 to solve this.