Unable to run a private github repository with a private submodule

[mark-gerarts]

Bug report

Expected behavior and actual behavior

I am trying to run a pipeline that is defined in a private Github repository. The pipeline has a git submodule that is also private. I run it like this (with my SCM credentials in ~/.nextflow/scm):

nextflow run [email protected]:my-repo/my-main-pipeline.git -r main

Which outputs the following:

 N E X T F L O W   ~  version 24.04.4

Unknown error accessing project `my-repo/my-main-pipeline` -- Repository may be corrupted: /home/me/.nextflow/assets/my-repo/my-main-pipeline

Steps to reproduce the problem

Not possible to provide a repo/script since the repositories are private, unfortunately.

Program output

The .nextflow.log file reveals the following error message:

cat .nextflow.log
Sep-23 13:58:16.617 [main] DEBUG nextflow.cli.Launcher - $> nextflow run 'https://github.com/my-repo/my-main-pipeline' -r main --help
Sep-23 13:58:16.730 [main] DEBUG nextflow.cli.CmdRun - N E X T F L O W  ~  version 24.04.4
Sep-23 13:58:16.757 [main] DEBUG nextflow.plugin.PluginsFacade - Setting up plugin manager > mode=prod; embedded=false; plugins-dir=/home/me/.nextflow/plugins; core-plugins: [email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected]
Sep-23 13:58:16.785 [main] INFO  o.pf4j.DefaultPluginStatusProvider - Enabled plugins: []
Sep-23 13:58:16.787 [main] INFO  o.pf4j.DefaultPluginStatusProvider - Disabled plugins: []
Sep-23 13:58:16.796 [main] INFO  org.pf4j.DefaultPluginManager - PF4J version 3.12.0 in 'deployment' mode
Sep-23 13:58:16.824 [main] INFO  org.pf4j.AbstractPluginManager - No plugins
Sep-23 13:58:16.838 [main] DEBUG nextflow.scm.ProviderConfig - Using SCM config path: /home/me/.nextflow/scm
Sep-23 13:58:17.309 [main] DEBUG nextflow.scm.ProviderConfig - Detected SCM config: [providers:[github:[user:mark-gerarts, password:ghp****]]]
Sep-23 13:58:17.335 [main] DEBUG nextflow.scm.RepositoryFactory - Found Git repository result: [RepositoryFactory]
Sep-23 13:58:17.375 [main] DEBUG nextflow.scm.RepositoryFactory - Git url=nextflow.scm.GitUrl(https, null, github.com, my-repo/my-main-pipeline) -> config=ProviderConfig[name: github, platform: github, server: https://github.com]
Sep-23 13:58:17.376 [main] DEBUG nextflow.scm.AssetManager - Repository URL: https://github.com/my-repo/my-main-pipeline; Project:my-repo/my-main-pipeline; Hub provider: github
Sep-23 13:58:17.399 [main] DEBUG nextflow.scm.AssetManager - Project manifest does not exist: /home/me/.nextflow/assets/my-repo/my-main-pipeline/nextflow.config (No such file or directory)
Sep-23 13:58:17.403 [main] DEBUG nextflow.scm.AssetManager - Git config: /home/me/.nextflow/assets/my-repo/my-main-pipeline/.git/config; branch: master; remote: origin; url: https://github.com/my-repo/my-main-pipeline.git
Sep-23 13:58:17.651 [main] DEBUG nextflow.cli.Launcher - Operation aborted
org.eclipse.jgit.api.errors.TransportException: [email protected]:my-repo/my-submodule.git: remote hung up unexpectedly
        at org.eclipse.jgit.api.FetchCommand.call(FetchCommand.java:249)
        at org.eclipse.jgit.api.CloneCommand.fetch(CloneCommand.java:319)
        at org.eclipse.jgit.api.CloneCommand.call(CloneCommand.java:189)
        at org.eclipse.jgit.api.SubmoduleUpdateCommand.getOrCloneSubmodule(SubmoduleUpdateCommand.java:136)
        at org.eclipse.jgit.api.SubmoduleUpdateCommand.call(SubmoduleUpdateCommand.java:177)
        at nextflow.scm.AssetManager.updateModules(AssetManager.groovy:999)
        at nextflow.cli.CmdRun.getScriptFile0(CmdRun.groovy:588)
        at nextflow.cli.CmdRun.getScriptFile(CmdRun.groovy:513)
        at nextflow.cli.CmdRun.run(CmdRun.groovy:321)
        at nextflow.cli.Launcher.run(Launcher.groovy:503)
        at nextflow.cli.Launcher.main(Launcher.groovy:657)
Caused by: org.eclipse.jgit.errors.TransportException: [email protected]:my-repo/my-submodule.git: remote hung up unexpectedly
        at org.eclipse.jgit.transport.TransportGitSsh$SshFetchConnection.<init>(TransportGitSsh.java:311)
        at org.eclipse.jgit.transport.TransportGitSsh.openFetch(TransportGitSsh.java:152)
        at org.eclipse.jgit.transport.FetchProcess.executeImp(FetchProcess.java:153)
        at org.eclipse.jgit.transport.FetchProcess.execute(FetchProcess.java:105)
        at org.eclipse.jgit.transport.Transport.fetch(Transport.java:1480)
        at org.eclipse.jgit.api.FetchCommand.call(FetchCommand.java:238)
        ... 10 common frames omitted
Caused by: java.lang.NullPointerException: Cannot invoke "org.eclipse.jgit.transport.SshSessionFactory.getSession(org.eclipse.jgit.transport.URIish, org.eclipse.jgit.transport.CredentialsProvider, org.eclipse.jgit.util.FS, int)" because "this.sch" is null
        at org.eclipse.jgit.transport.SshTransport.getSession(SshTransport.java:107)
        at org.eclipse.jgit.transport.TransportGitSsh$SshFetchConnection.<init>(TransportGitSsh.java:279)
        ... 15 common frames omitted

Environment

• Nextflow version: 24.04.4 build 5917
• Java version: openjdk version "19.0.2" 2023-01-17
• Operating system: Ubuntu 22.04.4 LTS
• Bash version: GNU bash, version 5.1.16(1)-release (x86_64-pc-linux-gnu)

[bentsherman]

@pditommaso should this work? I thought SSH transport was not supported but I don't remember why

[pditommaso]

Nope, only https transport is supported. Don't remember the exact reason

[mark-gerarts]

Thanks for the swift response! If it's not supported that's fine, I'll have to switch to https then. This does make local development a bit more cumbersome, since every git submodule update then prompts for username + PAT authentication instead of simply using SSH keys.