moduleBinaries in toplevel module/projectDir not found, (sharp edge)

[feiloo]

Bug report

Expected behavior and actual behavior

I expected module binaries under projectDir/resources/usr/bin to be usable in projectDir/main.nf but they arent found.
It is possible to use projectDir/bin instead but that creates some sharp-edged behavior.
When a project is included instead of run directly, resources/usr/bin would be used instead of bin/.
So the workflows behavior is dependent on it and scripts have to be copied to both directories to get the expected behavior.
Correct me, but this doesnt look like an intended feature.

Steps to reproduce the problem

projectDir/main.nf:

process process1 {
        script:
        """
        hello1.sh
        """
}

workflow {
    main:
        process1()

}

projectDir/resources/usr/bin/hello1.sh:

#!/bin/bash
echo hello1

projectDir/nextflow.config:

nextflow.enable.dsl=2
nextflow.enable.moduleBinaries = true

run with:

nextflow run ../projectDir -c nextflow.config

Program output

.nextflow.log

Sep-05 13:53:44.677 [main] DEBUG nextflow.cli.Launcher - $> nextflow run ../projectDir/ -c nextflow.config
Sep-05 13:53:44.773 [main] DEBUG nextflow.cli.CmdRun - N E X T F L O W  ~  version 24.04.4
Sep-05 13:53:44.790 [main] DEBUG nextflow.plugin.PluginsFacade - Setting up plugin manager > mode=prod; embedded=false; plugins-dir=/home/user/.nextflow/plugins; core-plugins: [email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected]
Sep-05 13:53:44.800 [main] INFO  o.pf4j.DefaultPluginStatusProvider - Enabled plugins: []
Sep-05 13:53:44.801 [main] INFO  o.pf4j.DefaultPluginStatusProvider - Disabled plugins: []
Sep-05 13:53:44.803 [main] INFO  org.pf4j.DefaultPluginManager - PF4J version 3.12.0 in 'deployment' mode
Sep-05 13:53:44.814 [main] INFO  org.pf4j.AbstractPluginManager - No plugins
Sep-05 13:53:45.200 [main] DEBUG nextflow.config.ConfigBuilder - Found config local: /data/user/miscellaneous/issues/projectDir/nextflow.config
Sep-05 13:53:45.202 [main] DEBUG nextflow.config.ConfigBuilder - User config file: /data/user/miscellaneous/issues/projectDir/nextflow.config
Sep-05 13:53:45.204 [main] DEBUG nextflow.config.ConfigBuilder - Parsing config file: /data/user/miscellaneous/issues/projectDir/nextflow.config
Sep-05 13:53:45.204 [main] DEBUG nextflow.config.ConfigBuilder - Parsing config file: /data/user/miscellaneous/issues/projectDir/nextflow.config
Sep-05 13:53:45.221 [main] DEBUG n.secret.LocalSecretsProvider - Secrets store: /home/user/.nextflow/secrets/store.json
Sep-05 13:53:45.224 [main] DEBUG nextflow.secret.SecretsLoader - Discovered secrets providers: [nextflow.secret.LocalSecretsProvider@4649d70a] - activable => nextflow.secret.LocalSecretsProvider@4649d70a
Sep-05 13:53:45.252 [main] DEBUG nextflow.config.ConfigBuilder - Applying config profile: `standard`
Sep-05 13:53:45.275 [main] DEBUG nextflow.config.ConfigBuilder - Applying config profile: `standard`
Sep-05 13:53:45.299 [main] DEBUG nextflow.cli.CmdRun - Applied DSL=2 from config declaration
Sep-05 13:53:45.311 [main] DEBUG nextflow.cli.CmdRun - Launching `../projectDir/main.nf` [cranky_rutherford] DSL2 - revision: ada3cad76b
Sep-05 13:53:45.313 [main] DEBUG nextflow.plugin.PluginsFacade - Plugins default=[]
Sep-05 13:53:45.313 [main] DEBUG nextflow.plugin.PluginsFacade - Plugins resolved requirement=[]
Sep-05 13:53:45.360 [main] DEBUG nextflow.Session - Session UUID: a1abebb7-fa07-451a-a136-9f9bf4f13393
Sep-05 13:53:45.360 [main] DEBUG nextflow.Session - Run name: cranky_rutherford
Sep-05 13:53:45.361 [main] DEBUG nextflow.Session - Executor pool size: 12
Sep-05 13:53:45.368 [main] DEBUG nextflow.file.FilePorter - File porter settings maxRetries=3; maxTransfers=50; pollTimeout=null
Sep-05 13:53:45.373 [main] DEBUG nextflow.util.ThreadPoolBuilder - Creating thread pool 'FileTransfer' minSize=10; maxSize=36; workQueue=LinkedBlockingQueue[10000]; allowCoreThreadTimeout=false
Sep-05 13:53:45.389 [main] DEBUG nextflow.cli.CmdRun -
  Version: 24.04.4 build 5917
  Created: 01-08-2024 07:05 UTC (09:05 CEST)
  System: Linux 5.14.21-150500.55.68-default
  Runtime: Groovy 4.0.21 on OpenJDK 64-Bit Server VM 17.0.12+7-suse-150400.3.45.1-x8664
  Encoding: UTF-8 (UTF-8)
  Process: 81278@ukb2580 [10.14.25.80]
  CPUs: 12 - Mem: 62.7 GB (10.1 GB) - Swap: 2 GB (1.9 GB)
Sep-05 13:53:45.407 [main] DEBUG nextflow.Session - Work-dir: /data/user/miscellaneous/issues/projectDir/work [xfs]
Sep-05 13:53:45.408 [main] DEBUG nextflow.Session - Script base path does not exist or is not a directory: /data/user/miscellaneous/issues/projectDir/bin
Sep-05 13:53:45.416 [main] DEBUG nextflow.executor.ExecutorFactory - Extension executors providers=[]
Sep-05 13:53:45.424 [main] DEBUG nextflow.Session - Observer factory: DefaultObserverFactory
Sep-05 13:53:45.443 [main] DEBUG nextflow.cache.CacheFactory - Using Nextflow cache factory: nextflow.cache.DefaultCacheFactory
Sep-05 13:53:45.451 [main] DEBUG nextflow.util.CustomThreadPool - Creating default thread pool > poolSize: 13; maxThreads: 1000
Sep-05 13:53:45.500 [main] DEBUG nextflow.Session - Session start
Sep-05 13:53:45.637 [main] DEBUG nextflow.script.ScriptRunner - > Launching execution
Sep-05 13:53:45.687 [main] DEBUG nextflow.executor.ExecutorFactory - << taskConfig executor: null
Sep-05 13:53:45.688 [main] DEBUG nextflow.executor.ExecutorFactory - >> processorType: 'local'
Sep-05 13:53:45.693 [main] DEBUG nextflow.executor.Executor - [warm up] executor > local
Sep-05 13:53:45.698 [main] DEBUG n.processor.LocalPollingMonitor - Creating local task monitor for executor 'local' > cpus=12; memory=62.7 GB; capacity=12; pollInterval=100ms; dumpInterval=5m
Sep-05 13:53:45.700 [main] DEBUG n.processor.TaskPollingMonitor - >>> barrier register (monitor: local)
Sep-05 13:53:45.780 [main] DEBUG nextflow.Session - Workflow process names [dsl2]: process1
Sep-05 13:53:45.781 [main] DEBUG nextflow.Session - Igniting dataflow network (1)
Sep-05 13:53:45.781 [main] DEBUG nextflow.processor.TaskProcessor - Starting process > process1
Sep-05 13:53:45.782 [main] DEBUG nextflow.script.ScriptRunner - Parsed script files:
  Script_b6df2e05938cee55: /data/user/miscellaneous/issues/projectDir/main.nf
Sep-05 13:53:45.782 [main] DEBUG nextflow.script.ScriptRunner - > Awaiting termination
Sep-05 13:53:45.782 [main] DEBUG nextflow.Session - Session await
Sep-05 13:53:45.922 [Task submitter] DEBUG n.executor.local.LocalTaskHandler - Launch cmd line: /bin/bash -ue .command.run
Sep-05 13:53:45.925 [Task submitter] INFO  nextflow.Session - [ba/4756a3] Submitted process > process1
Sep-05 13:53:45.957 [Task monitor] DEBUG n.processor.TaskPollingMonitor - Task completed > TaskHandler[id: 1; name: process1; status: COMPLETED; exit: 127; error: -; workDir: /data/user/miscellaneous/issues/projectDir/work/ba/4756a300431b89b0a1c8b6ae7361ae]
Sep-05 13:53:45.958 [Task monitor] DEBUG nextflow.util.ThreadPoolBuilder - Creating thread pool 'TaskFinalizer' minSize=10; maxSize=36; workQueue=LinkedBlockingQueue[10000]; allowCoreThreadTimeout=false
Sep-05 13:53:45.965 [TaskFinalizer-1] DEBUG nextflow.processor.TaskProcessor - Handling unexpected condition for
  task: name=process1; work-dir=/data/user/miscellaneous/issues/projectDir/work/ba/4756a300431b89b0a1c8b6ae7361ae
  error [nextflow.exception.ProcessFailedException]: Process `process1` terminated with an error exit status (127)
Sep-05 13:53:45.985 [TaskFinalizer-1] ERROR nextflow.processor.TaskProcessor - Error executing process > 'process1'

Caused by:
  Process `process1` terminated with an error exit status (127)


Command executed:

  hello1.sh

Command exit status:
  127

Command output:
  (empty)

Command error:
  .command.sh: line 2: hello1.sh: command not found

Work dir:
  /data/user/miscellaneous/issues/projectDir/work/ba/4756a300431b89b0a1c8b6ae7361ae

Tip: you can replicate the issue by changing to the process work dir and entering the command `bash .command.run`
Sep-05 13:53:45.992 [main] DEBUG nextflow.Session - Session await > all processes finished
Sep-05 13:53:45.996 [TaskFinalizer-1] DEBUG nextflow.Session - Session aborted -- Cause: Process `process1` terminated with an error exit status (127)
Sep-05 13:53:46.012 [main] DEBUG nextflow.Session - Session await > all barriers passed
Sep-05 13:53:46.012 [Task monitor] DEBUG n.processor.TaskPollingMonitor - <<< barrier arrives (monitor: local) - terminating tasks monitor poll loop
Sep-05 13:53:46.017 [main] DEBUG n.trace.WorkflowStatsObserver - Workflow completed > WorkflowStats[succeededCount=0; failedCount=1; ignoredCount=0; cachedCount=0; pendingCount=0; submittedCount=0; runningCount=0; retriesCount=0; abortedCount=0; succeedDuration=0ms; failedDuration=21ms; cachedDuration=0ms;loadCpus=0; loadMemory=0; peakRunning=1; peakCpus=1; peakMemory=0; ]
Sep-05 13:53:46.196 [main] DEBUG nextflow.cache.CacheDB - Closing CacheDB done
Sep-05 13:53:46.211 [main] DEBUG nextflow.script.ScriptRunner - > Execution complete -- Goodbye

Environment

• Nextflow version: 24.04.4.5917
• Java version: openjdk version "17.0.12"
• Operating system: Linux
• Bash version: 4.4.23(1)-release

[bentsherman]

An interesting point. The module binaries feature was intended for scripts that are only ever included as modules, so if you have some script that is used both as a module and an entrypoint, it is better to move the "include-able" definitions into a separate module.

On the other hand, I was thinking it would be useful to have an entry workflow in each module that simply wraps the process or subworkflow, as a way to test as well as to show example usage. But I think that would also run into this issue.

[feiloo]

I sometimes run some modules directly instead of from an workflow, so I agree on its usefulness.
I also try to keep down layers and indirections in the code for simplicity.

My understanding is that the main difference between module binaries and projectDir/bin is that projectDir/bin binaries are visible everywhere in the workflows, subworkflows and modules.

Since i dont see a particular downside to it, i would be in favor of having module binaries visible to the entry workflow, with an error or priority for them over projectDir/bin binaries in case the script-names collide.