add software required for 'mag' Nextflow pipeline to EESSI 2021.12 #202

smoretti · 2022-11-10T12:53:23Z

Nextflow pipeline https://nf-co.re/mag ported in EESSI

ocaisa · 2022-11-10T13:02:39Z

eessi-2021.12.yml

@@ -57,3 +57,7 @@ software:
    toolchains:
      SYSTEM:
        versions: '22.10.1'
+  Nextflow-mag-pipeline:


This is premature here I think, there is no PR (that I can find) for Nextflow-mag-pipeline-2.2.1-foss-2021a.eb

What should be a good name in that case?

I would use Nextflow-pipeline, and use -mag as versionsuffix

But doesn't this have to map to an easyconfig? Is there a PR somewhere (for Nextflow-pipeline-2.2.1-foss-2021a-mag.eb) being worked on that wraps all the other PRs?

The pipeline is actually called nf-core/mag, I'd be inclined to stick close to that so people can actually find it...not sure how we deal with / though

boegel

@smoretti For MetaBAT, the updated EasyBuild hooks from #203 are also required, because the fixed MetaBAT easyconfigs in easybuilders/easybuild-easyconfigs#16624 assume that $EBROOTZLIB is defined, which it is not in the EESSI build environment because we filter out zlib as depedency via --filter-deps (see our script to configure EasyBuild).

boegel · 2022-11-12T16:45:06Z

EESSI-pilot-install-software.sh

+echo ">> Installing requirements for the Nextflow mag 20221110 pipeline..."
+ok_msg="The full Nextflow pipeline is installed, the work must flow..."
+fail_msg="Installation of the Nextflow pipeline failed, that's unexpected..."
+$EB --robot --from-pr 16576,16577,16589,16590,16591,16592,16612,16613,16615,16617


@smoretti This should be trimmed to:

$EB --from-pr 16613 --robot

As long as all PRs required by easybuilders/easybuild-easyconfigs#16613 are merged (most, if not all, already are), then that's sufficient.

boegel · 2022-11-13T09:02:07Z

Just for context, this requires 115 additional installations in total (assuming that both #160 and #195 are already installed), see details below.

115 out of 191 required modules missing:

* HTSlib/1.12-GCC-10.3.0 (HTSlib-1.12-GCC-10.3.0.eb)
* AdapterRemoval/2.3.2-GCC-10.3.0 (AdapterRemoval-2.3.2-GCC-10.3.0.eb)
* GSL/2.7-GCC-10.3.0 (GSL-2.7-GCC-10.3.0.eb)
* BCFtools/1.12-GCC-10.3.0 (BCFtools-1.12-GCC-10.3.0.eb)
* prodigal/2.6.3-GCCcore-10.3.0 (prodigal-2.6.3-GCCcore-10.3.0.eb)
* Ruby/3.0.1-GCCcore-10.3.0 (Ruby-3.0.1-GCCcore-10.3.0.eb)
* FastQC/0.11.9-Java-11 (FastQC-0.11.9-Java-11.eb)
* file/5.40-GCCcore-10.3.0 (file-5.40-GCCcore-10.3.0.eb)
* DIAMOND/2.0.13-GCC-10.3.0 (DIAMOND-2.0.13-GCC-10.3.0.eb)
* NGS/2.10.9-GCCcore-10.3.0 (NGS-2.10.9-GCCcore-10.3.0.eb)
* Bowtie2/2.4.4-GCC-10.3.0 (Bowtie2-2.4.4-GCC-10.3.0.eb)
* CAT-BAT/5.2.3-GCC-10.3.0 (CAT-BAT-5.2.3-GCC-10.3.0.eb)
* Filtlong/0.2.1-GCC-10.3.0 (Filtlong-0.2.1-GCC-10.3.0.eb)
* libdeflate/1.7-GCCcore-10.3.0 (libdeflate-1.7-GCCcore-10.3.0.eb)
* PCRE/8.44-GCCcore-10.3.0 (PCRE-8.44-GCCcore-10.3.0.eb)
* pullseq/1.0.2-GCCcore-10.3.0 (pullseq-1.0.2-GCCcore-10.3.0.eb)
* ISA-L/2.30.0-GCCcore-10.3.0 (ISA-L-2.30.0-GCCcore-10.3.0.eb)
* fastp/0.23.2-GCC-10.3.0 (fastp-0.23.2-GCC-10.3.0.eb)
* Perl/5.32.1-GCCcore-10.3.0-minimal (Perl-5.32.1-GCCcore-10.3.0-minimal.eb)
* SAMtools/1.13-GCC-10.3.0 (SAMtools-1.13-GCC-10.3.0.eb)
* MetaEuk/5-GCC-10.3.0 (MetaEuk-5-GCC-10.3.0.eb)
* nettle/3.7.2-GCCcore-10.3.0 (nettle-3.7.2-GCCcore-10.3.0.eb)
* lpsolve/5.5.2.11-GCC-10.3.0 (lpsolve-5.5.2.11-GCC-10.3.0.eb)
* DendroPy/4.5.2-GCCcore-10.3.0 (DendroPy-4.5.2-GCCcore-10.3.0.eb)
* KronaTools/2.8-GCC-10.3.0 (KronaTools-2.8-GCC-10.3.0.eb)
* BBMap/38.96-GCC-10.3.0 (BBMap-38.96-GCC-10.3.0.eb)
* LMDB/0.9.28-GCCcore-10.3.0 (LMDB-0.9.28-GCCcore-10.3.0.eb)
* parallel/20210622-GCCcore-10.3.0 (parallel-20210622-GCCcore-10.3.0.eb)
* VCFtools/0.1.16-GCC-10.3.0 (VCFtools-0.1.16-GCC-10.3.0.eb)
* BamTools/2.5.2-GCC-10.3.0 (BamTools-2.5.2-GCC-10.3.0.eb)
* pplacer/1.1.alpha19 (pplacer-1.1.alpha19.eb)
* cppy/1.1.0-GCCcore-10.3.0 (cppy-1.1.0-GCCcore-10.3.0.eb)
* libidn2/2.3.0-GCCcore-10.3.0 (libidn2-2.3.0-GCCcore-10.3.0.eb)
* wget/1.21.1-GCCcore-10.3.0 (wget-1.21.1-GCCcore-10.3.0.eb)
* HMMER/3.3.2-gompi-2021a (HMMER-3.3.2-gompi-2021a.eb)
* ncbi-vdb/2.10.9-gompi-2021a (ncbi-vdb-2.10.9-gompi-2021a.eb)
* BLAST+/2.11.0-gompi-2021a (BLAST+-2.11.0-gompi-2021a.eb)
* SEPP/4.5.0-foss-2021a (SEPP-4.5.0-foss-2021a.eb)
* Kraken2/2.1.2-gompi-2021a (Kraken2-2.1.2-gompi-2021a.eb)
* Centrifuge/1.0.4-gompi-2021a (Centrifuge-1.0.4-gompi-2021a.eb)
* Biopython/1.79-foss-2021a (Biopython-1.79-foss-2021a.eb)
* Tk/8.6.11-GCCcore-10.3.0 (Tk-8.6.11-GCCcore-10.3.0.eb)
* FastANI/1.33-GCC-10.3.0 (FastANI-1.33-GCC-10.3.0.eb)
* Tkinter/3.9.5-GCCcore-10.3.0 (Tkinter-3.9.5-GCCcore-10.3.0.eb)
* FragGeneScan/1.31-GCCcore-10.3.0 (FragGeneScan-1.31-GCCcore-10.3.0.eb)
* jbigkit/2.1-GCCcore-10.3.0 (jbigkit-2.1-GCCcore-10.3.0.eb)
* SuiteSparse/5.10.1-foss-2021a-METIS-5.1.0 (SuiteSparse-5.10.1-foss-2021a-METIS-5.1.0.eb)
* MEGAHIT/1.2.9-GCCcore-10.3.0 (MEGAHIT-1.2.9-GCCcore-10.3.0.eb)
* AUGUSTUS/3.4.0-foss-2021a (AUGUSTUS-3.4.0-foss-2021a.eb)
* MetaBAT/2.15-gompi-2021a (MetaBAT-2.15-gompi-2021a.eb)
* NLopt/2.7.0-GCCcore-10.3.0 (NLopt-2.7.0-GCCcore-10.3.0.eb)
* FastTree/2.1.11-GCCcore-10.3.0 (FastTree-2.1.11-GCCcore-10.3.0.eb)
* IDBA-UD/1.1.3-GCC-10.3.0 (IDBA-UD-1.1.3-GCC-10.3.0.eb)
* tabixpp/1.1.0-GCC-10.3.0 (tabixpp-1.1.0-GCC-10.3.0.eb)
* MaxBin/2.2.7-gompi-2021a (MaxBin-2.2.7-gompi-2021a.eb)
* LibTIFF/4.2.0-GCCcore-10.3.0 (LibTIFF-4.2.0-GCCcore-10.3.0.eb)
* Pillow/8.2.0-GCCcore-10.3.0 (Pillow-8.2.0-GCCcore-10.3.0.eb)
* Xvfb/1.20.11-GCCcore-10.3.0 (Xvfb-1.20.11-GCCcore-10.3.0.eb)
* Qhull/2020.2-GCCcore-10.3.0 (Qhull-2020.2-GCCcore-10.3.0.eb)
* intervaltree/0.1-GCCcore-10.3.0 (intervaltree-0.1-GCCcore-10.3.0.eb)
* JsonCpp/1.9.4-GCCcore-10.3.0 (JsonCpp-1.9.4-GCCcore-10.3.0.eb)
* matplotlib/3.4.2-foss-2021a (matplotlib-3.4.2-foss-2021a.eb)
* UDUNITS/2.2.28-GCCcore-10.3.0 (UDUNITS-2.2.28-GCCcore-10.3.0.eb)
* tqdm/4.61.2-GCCcore-10.3.0 (tqdm-4.61.2-GCCcore-10.3.0.eb)
* fastahack/1.0.0-GCCcore-10.3.0 (fastahack-1.0.0-GCCcore-10.3.0.eb)
* BWA/0.7.17-GCC-10.3.0 (BWA-0.7.17-GCC-10.3.0.eb)
* CapnProto/0.9.1-GCCcore-10.3.0 (CapnProto-0.9.1-GCCcore-10.3.0.eb)
* Mash/2.3-GCC-10.3.0 (Mash-2.3-GCC-10.3.0.eb)
* GTDB-Tk/2.0.0-foss-2021a (GTDB-Tk-2.0.0-foss-2021a.eb)
* filevercmp/20191210-GCCcore-10.3.0 (filevercmp-20191210-GCCcore-10.3.0.eb)
* fermi-lite/20190320-GCCcore-10.3.0 (fermi-lite-20190320-GCCcore-10.3.0.eb)
* networkx/2.5.1-foss-2021a (networkx-2.5.1-foss-2021a.eb)
* minimap2/2.20-GCCcore-10.3.0 (minimap2-2.20-GCCcore-10.3.0.eb)
* libyaml/0.2.5-GCCcore-10.3.0 (libyaml-0.2.5-GCCcore-10.3.0.eb)
* libogg/1.3.4-GCCcore-10.3.0 (libogg-1.3.4-GCCcore-10.3.0.eb)
* NanoLyse/1.2.1-foss-2021a (NanoLyse-1.2.1-foss-2021a.eb)
* PyYAML/5.4.1-GCCcore-10.3.0 (PyYAML-5.4.1-GCCcore-10.3.0.eb)
* FLAC/1.3.3-GCCcore-10.3.0 (FLAC-1.3.3-GCCcore-10.3.0.eb)
* libvorbis/1.3.7-GCCcore-10.3.0 (libvorbis-1.3.7-GCCcore-10.3.0.eb)
* MultiQC/1.11-foss-2021a (MultiQC-1.11-foss-2021a.eb)
* libsndfile/1.0.31-GCCcore-10.3.0 (libsndfile-1.0.31-GCCcore-10.3.0.eb)
* GLPK/5.0-GCCcore-10.3.0 (GLPK-5.0-GCCcore-10.3.0.eb)
* fsom/20141119-GCCcore-10.3.0 (fsom-20141119-GCCcore-10.3.0.eb)
* SSW/1.1-GCCcore-10.3.0 (SSW-1.1-GCCcore-10.3.0.eb)
* Ghostscript/9.54.0-GCCcore-10.3.0 (Ghostscript-9.54.0-GCCcore-10.3.0.eb)
* SeqLib/1.2.0-GCC-10.3.0 (SeqLib-1.2.0-GCC-10.3.0.eb)
* QUAST/5.0.2-foss-2021a (QUAST-5.0.2-foss-2021a.eb)
* nodejs/14.17.0-GCCcore-10.3.0 (nodejs-14.17.0-GCCcore-10.3.0.eb)
* multichoose/1.0.3-GCCcore-10.3.0 (multichoose-1.0.3-GCCcore-10.3.0.eb)
* Pysam/0.16.0.1-GCC-10.3.0 (Pysam-0.16.0.1-GCC-10.3.0.eb)
* XML-LibXML/2.0207-GCCcore-10.3.0 (XML-LibXML-2.0207-GCCcore-10.3.0.eb)
* SPAdes/3.13.0-GCC-10.3.0 (SPAdes-3.13.0-GCC-10.3.0.eb)
* smithwaterman/20160702-GCCcore-10.3.0 (smithwaterman-20160702-GCCcore-10.3.0.eb)
* tbl2asn/20220427-linux64 (tbl2asn-20220427-linux64.eb)
* LittleCMS/2.12-GCCcore-10.3.0 (LittleCMS-2.12-GCCcore-10.3.0.eb)
* statsmodels/0.12.2-foss-2021a (statsmodels-0.12.2-foss-2021a.eb)
* DB_File/1.856-GCCcore-10.3.0 (DB_File-1.856-GCCcore-10.3.0.eb)
* ImageMagick/7.0.11-14-GCCcore-10.3.0 (ImageMagick-7.0.11-14-GCCcore-10.3.0.eb)
* PyDamage/0.70-foss-2021a (PyDamage-0.70-foss-2021a.eb)
* BioPerl/1.7.8-GCCcore-10.3.0 (BioPerl-1.7.8-GCCcore-10.3.0.eb)
* Bio-SearchIO-hmmer/1.7.3-GCC-10.3.0 (Bio-SearchIO-hmmer-1.7.3-GCC-10.3.0.eb)
* prokka/1.14.5-gompi-2021a (prokka-1.14.5-gompi-2021a.eb)
* libgit2/1.1.0-GCCcore-10.3.0 (libgit2-1.1.0-GCCcore-10.3.0.eb)
* GEOS/3.9.1-GCC-10.3.0 (GEOS-3.9.1-GCC-10.3.0.eb)
* PROJ/8.0.1-GCCcore-10.3.0 (PROJ-8.0.1-GCCcore-10.3.0.eb)
* libgeotiff/1.6.0-GCCcore-10.3.0 (libgeotiff-1.6.0-GCCcore-10.3.0.eb)
* libtirpc/1.3.2-GCCcore-10.3.0 (libtirpc-1.3.2-GCCcore-10.3.0.eb)
* HDF/4.2.15-GCCcore-10.3.0 (HDF-4.2.15-GCCcore-10.3.0.eb)
* GDAL/3.3.0-foss-2021a (GDAL-3.3.0-foss-2021a.eb)
* R/4.1.0-foss-2021a (R-4.1.0-foss-2021a.eb)
* vcflib/1.0.3-foss-2021a-R-4.1.0 (vcflib-1.0.3-foss-2021a-R-4.1.0.eb)
* BUSCO/5.4.2-foss-2021a (BUSCO-5.4.2-foss-2021a.eb)
* DAS_Tool/1.1.3-foss-2021a-R-4.1.0 (DAS_Tool-1.1.3-foss-2021a-R-4.1.0.eb)
* freebayes/1.3.6-foss-2021a-R-4.1.0 (freebayes-1.3.6-foss-2021a-R-4.1.0.eb)
* nf-core-mag/20221110-foss-2021a (nf-core-mag-20221110-foss-2021a.eb)

boegel · 2022-11-13T09:52:01Z

R-4.1.0-foss-2021a.eb is a dependency in this stack, and we'll need to pull that easyconfig from a PR to make sure all SHA256 checksums for extensions are correct.
While we're doing that, we might as well enable --parallel-extensions-install, which significantly speeds up the installation of an R easyconfig:

$EB --from-pr 14987 R-4.1.0-foss-2021a.eb -r --parallel-extensions-install --experimental && ...

If not, we'll run into this:

Checksum verification for extension source MASS_7.3-54.tar.gz failed

boegel · 2022-11-13T19:47:43Z

For MetaBAT on aarch64, easybuilders/easybuild-easyconfigs#16633 is also required.

Bowtie2 on aarch64 fails with errors like:

sse_wrap.h:31:10: fatal error: simde/x86/sse2.h: No such file or directory
   31 | #include "simde/x86/sse2.h"
      |          ^~~~~~~~~~~~~~~~~~
compilation terminated.

It seems like SIMDe needs to be added as a dependency (at least on non-x86_64 systems).

…make sure all SHA256 checksums for extensions are correct AND to speed up the installation with --parallel-extensions-install

… instead of True in dependencies

smoretti · 2022-12-22T13:54:32Z

Bowtie2 on aarch64 fails with errors like:
sse_wrap.h:31:10: fatal error: simde/x86/sse2.h: No such file or directory
   31 | #include "simde/x86/sse2.h"
      |          ^~~~~~~~~~~~~~~~~~
compilation terminated.
It seems like SIMDe needs to be added as a dependency (at least on non-x86_64 systems).

Fixed in easybuilders/easybuild-easyconfigs#16946

…9.1.1-foss/2022a {2023.06}[foss/2022a] Pillow V9.1.1

bedroge · 2024-04-16T07:13:15Z

Since this is targeting the old and unsupported pilot repository, I'm going to close this PR. If it's still relevant to add this to EESSI, please open a new PR where this is added to one of the easystack files for the production repository.

Draft of the pipeline installation in EESSI

d31643a

boegel added the BioHack2022 Related to EU BioHackathon 2022 label Nov 10, 2022

ocaisa reviewed Nov 10, 2022

View reviewed changes

smoretti changed the title ~~Draft of the pipeline installation in EESSI~~ Nextflow pipeline https://nf-co.re/mag ported in EESSI Nov 10, 2022

boegel changed the title ~~Nextflow pipeline https://nf-co.re/mag ported in EESSI~~ add software required for 'mag' Nextflow pipeline to EESSI 2021.12 Nov 10, 2022

smoretti added 3 commits November 10, 2022 17:05

Update the Nextflow-pipeline PR

21b6980

Update Nextflow pipeline name

7eeacc9

Add new PRs

e0ec46c

boegel mentioned this pull request Nov 12, 2022

enhance EasyBuild hooks to fix installation of MetaBAT in EESSI #203

Merged

boegel requested changes Nov 12, 2022

View reviewed changes

smoretti added 3 commits November 15, 2022 18:00

Fix the list of PR to only the bundle and the MetaBAT fix

fb934d4

Install R-4.1.0-foss-2021a.eb first for the Nextflow mag pipeline to …

85ea3cd

…make sure all SHA256 checksums for extensions are correct AND to speed up the installation with --parallel-extensions-install

Use the last R-4.1.0-foss-2021a.eb PR to use SYSTEM template constant…

aeab8e0

… instead of True in dependencies

boegel added the pilot-2021.12 label Dec 14, 2022

smoretti added 2 commits December 14, 2022 16:14

Add R 4.1.0 as a separate PR in the EESSI pilot script

6243c5d

Move R 4.1.0 in a separate PR

513c896

boegel changed the base branch from main to 2021.12 June 5, 2023 08:46

TopRichard added a commit to TopRichard/bot-software-layer1 that referenced this pull request Oct 11, 2023

Merge pull request EESSI#202 from TopRichard/nessi.no-2023.06-Pillow/…

58fc0e6

…9.1.1-foss/2022a {2023.06}[foss/2022a] Pillow V9.1.1

bedroge closed this Apr 16, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

add software required for 'mag' Nextflow pipeline to EESSI 2021.12 #202

add software required for 'mag' Nextflow pipeline to EESSI 2021.12 #202

smoretti commented Nov 10, 2022 •

edited

Loading

ocaisa Nov 10, 2022

smoretti Nov 10, 2022

boegel Nov 10, 2022

ocaisa Nov 10, 2022 •

edited

Loading

ocaisa Nov 10, 2022

boegel left a comment

boegel Nov 12, 2022

smoretti Nov 15, 2022

boegel commented Nov 13, 2022

boegel commented Nov 13, 2022

boegel commented Nov 13, 2022

smoretti commented Dec 22, 2022

bedroge commented Apr 16, 2024

add software required for 'mag' Nextflow pipeline to EESSI 2021.12 #202

add software required for 'mag' Nextflow pipeline to EESSI 2021.12 #202

Conversation

smoretti commented Nov 10, 2022 • edited Loading

ocaisa Nov 10, 2022

Choose a reason for hiding this comment

smoretti Nov 10, 2022

Choose a reason for hiding this comment

boegel Nov 10, 2022

Choose a reason for hiding this comment

ocaisa Nov 10, 2022 • edited Loading

Choose a reason for hiding this comment

ocaisa Nov 10, 2022

Choose a reason for hiding this comment

boegel left a comment

Choose a reason for hiding this comment

boegel Nov 12, 2022

Choose a reason for hiding this comment

smoretti Nov 15, 2022

Choose a reason for hiding this comment

boegel commented Nov 13, 2022

boegel commented Nov 13, 2022

boegel commented Nov 13, 2022

smoretti commented Dec 22, 2022

bedroge commented Apr 16, 2024

smoretti commented Nov 10, 2022 •

edited

Loading

ocaisa Nov 10, 2022 •

edited

Loading