From 415f6affd2943e35883479791a97ba1f092fd441 Mon Sep 17 00:00:00 2001
From: <>
Date: Tue, 23 Jan 2024 14:24:24 +0000
Subject: [PATCH] Deployed e0bde22 with MkDocs version: 1.5.3
---
index.html | 22 +++++++++++++++++++++-
search/search_index.json | 2 +-
sitemap.xml.gz | Bin 127 -> 127 bytes
3 files changed, 22 insertions(+), 2 deletions(-)
diff --git a/index.html b/index.html
index f0d3bef..febe3a6 100644
--- a/index.html
+++ b/index.html
@@ -159,6 +159,26 @@ Quick Start
--id_mapping id_mapping_file \
-profile <docker/singularity/podman/shifter/charliecloud/conda/institute>
+Databases and references
+A question that pops up a lot is: Since Euryale requires a lot of reference parameters, where can I find these references?
+Below we provide a short list of places where you can find these databases.
+But, of course, we're not limited to these references: Euryale should be able to process your own databases, should you want to build them yourself.
+Alignment
+For the alignment you can either provide --diamond_db for a pre-built DIAMOND database, or you can provide --reference_fasta.
+For reference fasta, by default Euryale expects something like NCBI-nr, but similarly formatted reference databases should also suffice.
+Taxonomic classification
+At its current version, Euryale doesn't build a reference taxonomic database, but pre-built ones are supported.
+
+- If you're using Kaiju (the default), you can provide a reference database with
--kaiju_db and provide a .tar.gz file like the ones provided in the official Kaiju website.
+ We have extensively tested Euryale with the 2021 version of the nr database and it should work as expected.
+- If you're using Kraken2 (By supplying
--run_kraken2), we expect something like the pre-built .tar.gz databases provided by the Kraken2 developers to be provided to --kraken2_db.
+
+Functional annotation
+We expect an ID mapping reference to be used within annotate. Since we're already expecting by default the NCBI-nr to be used as the alignment reference, the ID mapping data file provided by Uniprot should work well when provided to --id_mapping.
+Host reference
+If you're using metagenomic reads that come from a known host's microbiome, you can also provide the host's genome FASTA to --host_fasta parameter in order to enable our decontamination subworkflow.
+Ensembl provides easy to download genomes that can be used for this purpose.
+Alternatively, you can provide a pre-built BowTie2 database directory to the --bowtie2_db parameter.
Documentation
The dalmolingroup/euryale documentation is split into the following pages:
@@ -246,5 +266,5 @@ Citations
diff --git a/search/search_index.json b/search/search_index.json
index b1c6072..38c0521 100644
--- a/search/search_index.json
+++ b/search/search_index.json
@@ -1 +1 @@
-{"config":{"indexing":"full","lang":["en"],"min_search_length":3,"prebuild_index":false,"separator":"[\\s\\-]+"},"docs":[{"location":"","text":"Introduction dalmolingroup/euryale is a pipeline for taxonomic classification and functional annotation of metagenomic reads. Based on MEDUSA . The pipeline is built using Nextflow , a workflow tool to run tasks across multiple compute infrastructures in a very portable manner. It uses Docker/Singularity containers making installation trivial and results highly reproducible. The Nextflow DSL2 implementation of this pipeline uses one container per process which makes it much easier to maintain and update software dependencies. Where possible, these processes have been submitted to and installed from nf-core/modules in order to make them available to all nf-core pipelines, and to everyone within the Nextflow community! Pipeline summary Pre-processing Read QC ( FastQC ) Read trimming and merging ( fastp ) ( optionally ) Host read removal ( BowTie2 ) Duplicated sequence removal ( fastx collapser ) Present QC and other data ( MultiQC ) Assembly ( optionally ) Read assembly ( MEGAHIT ) Taxonomic classification Sequence classification ( Kaiju ) Sequence classification ( Kraken2 ) Visualization ( Krona ) Functional annotation Sequence alignment ( DIAMOND ) Map alignment matches to functional database ( annotate ) Quick Start Install Nextflow ( >=22.10.1 ) Install any of Docker , Singularity (you can follow this tutorial ), Podman , Shifter or Charliecloud for full pipeline reproducibility (you can use Conda both to install Nextflow itself and also to manage software within pipelines. Please only use it within pipelines as a last resort; see docs ) . Download the pipeline and test it on a minimal dataset with a single command: nextflow run dalmolingroup/euryale -profile test,YOURPROFILE --outdir Note that some form of configuration will be needed so that Nextflow knows how to fetch the required software. This is usually done in the form of a config profile ( YOURPROFILE in the example command above). You can chain multiple config profiles in a comma-separated string. The pipeline comes with config profiles called docker , singularity , podman , shifter , charliecloud and conda which instruct the pipeline to use the named tool for software management. For example, -profile test,docker . Please check nf-core/configs to see if a custom config file to run nf-core pipelines already exists for your Institute. If so, you can simply use -profile in your command. This will enable either docker or singularity and set the appropriate execution settings for your local compute environment. If you are using singularity , please use the nf-core download command to download images first, before running the pipeline. Setting the NXF_SINGULARITY_CACHEDIR or singularity.cacheDir Nextflow options enables you to store and re-use the images from a central location for future pipeline runs. If you are using conda , it is highly recommended to use the NXF_CONDA_CACHEDIR or conda.cacheDir settings to store the environments in a central location for future pipeline runs. Start running your own analysis! nextflow run dalmolingroup/euryale \\ --input samplesheet.csv \\ --outdir \\ --kaiju_db kaiju_reference \\ --diamond_db diamond_db \\ --reference_fasta diamond_fasta \\ --host_fasta host_reference_fasta \\ --id_mapping id_mapping_file \\ -profile Documentation The dalmolingroup/euryale documentation is split into the following pages: Usage An overview of how the pipeline works, how to run it and a description of all of the different command-line flags. Output An overview of the different results produced by the pipeline and how to interpret them. Credits dalmolingroup/euryale was originally written by Jo\u00e3o Cavalcante. We thank the following people for their extensive assistance in the development of this pipeline: Diego Morais (for developing the original MEDUSA pipeline) Citations Morais DAA, Cavalcante JVF, Monteiro SS, Pasquali MAB and Dalmolin RJS (2022) MEDUSA: A Pipeline for Sensitive Taxonomic Classification and Flexible Functional Annotation of Metagenomic Shotgun Sequences. Front. Genet. 13:814437. doi: 10.3389/fgene.2022.814437 This pipeline uses code and infrastructure developed and maintained by the nf-core community, reused here under the MIT license . The nf-core framework for community-curated bioinformatics pipelines. Philip Ewels, Alexander Peltzer, Sven Fillinger, Harshil Patel, Johannes Alneberg, Andreas Wilm, Maxime Ulysse Garcia, Paolo Di Tommaso & Sven Nahnsen. Nat Biotechnol. 2020 Feb 13. doi: 10.1038/s41587-020-0439-x .","title":"Home"},{"location":"#introduction","text":"dalmolingroup/euryale is a pipeline for taxonomic classification and functional annotation of metagenomic reads. Based on MEDUSA . The pipeline is built using Nextflow , a workflow tool to run tasks across multiple compute infrastructures in a very portable manner. It uses Docker/Singularity containers making installation trivial and results highly reproducible. The Nextflow DSL2 implementation of this pipeline uses one container per process which makes it much easier to maintain and update software dependencies. Where possible, these processes have been submitted to and installed from nf-core/modules in order to make them available to all nf-core pipelines, and to everyone within the Nextflow community!","title":"Introduction"},{"location":"#pipeline-summary","text":"","title":"Pipeline summary"},{"location":"#pre-processing","text":"Read QC ( FastQC ) Read trimming and merging ( fastp ) ( optionally ) Host read removal ( BowTie2 ) Duplicated sequence removal ( fastx collapser ) Present QC and other data ( MultiQC )","title":"Pre-processing"},{"location":"#assembly","text":"( optionally ) Read assembly ( MEGAHIT )","title":"Assembly"},{"location":"#taxonomic-classification","text":"Sequence classification ( Kaiju ) Sequence classification ( Kraken2 ) Visualization ( Krona )","title":"Taxonomic classification"},{"location":"#functional-annotation","text":"Sequence alignment ( DIAMOND ) Map alignment matches to functional database ( annotate )","title":"Functional annotation"},{"location":"#quick-start","text":"Install Nextflow ( >=22.10.1 ) Install any of Docker , Singularity (you can follow this tutorial ), Podman , Shifter or Charliecloud for full pipeline reproducibility (you can use Conda both to install Nextflow itself and also to manage software within pipelines. Please only use it within pipelines as a last resort; see docs ) . Download the pipeline and test it on a minimal dataset with a single command: nextflow run dalmolingroup/euryale -profile test,YOURPROFILE --outdir Note that some form of configuration will be needed so that Nextflow knows how to fetch the required software. This is usually done in the form of a config profile ( YOURPROFILE in the example command above). You can chain multiple config profiles in a comma-separated string. The pipeline comes with config profiles called docker , singularity , podman , shifter , charliecloud and conda which instruct the pipeline to use the named tool for software management. For example, -profile test,docker . Please check nf-core/configs to see if a custom config file to run nf-core pipelines already exists for your Institute. If so, you can simply use -profile in your command. This will enable either docker or singularity and set the appropriate execution settings for your local compute environment. If you are using singularity , please use the nf-core download command to download images first, before running the pipeline. Setting the NXF_SINGULARITY_CACHEDIR or singularity.cacheDir Nextflow options enables you to store and re-use the images from a central location for future pipeline runs. If you are using conda , it is highly recommended to use the NXF_CONDA_CACHEDIR or conda.cacheDir settings to store the environments in a central location for future pipeline runs. Start running your own analysis! nextflow run dalmolingroup/euryale \\ --input samplesheet.csv \\ --outdir \\ --kaiju_db kaiju_reference \\ --diamond_db diamond_db \\ --reference_fasta diamond_fasta \\ --host_fasta host_reference_fasta \\ --id_mapping id_mapping_file \\ -profile ","title":"Quick Start"},{"location":"#documentation","text":"The dalmolingroup/euryale documentation is split into the following pages: Usage An overview of how the pipeline works, how to run it and a description of all of the different command-line flags. Output An overview of the different results produced by the pipeline and how to interpret them.","title":"Documentation"},{"location":"#credits","text":"dalmolingroup/euryale was originally written by Jo\u00e3o Cavalcante. We thank the following people for their extensive assistance in the development of this pipeline: Diego Morais (for developing the original MEDUSA pipeline)","title":"Credits"},{"location":"#citations","text":"Morais DAA, Cavalcante JVF, Monteiro SS, Pasquali MAB and Dalmolin RJS (2022) MEDUSA: A Pipeline for Sensitive Taxonomic Classification and Flexible Functional Annotation of Metagenomic Shotgun Sequences. Front. Genet. 13:814437. doi: 10.3389/fgene.2022.814437 This pipeline uses code and infrastructure developed and maintained by the nf-core community, reused here under the MIT license . The nf-core framework for community-curated bioinformatics pipelines. Philip Ewels, Alexander Peltzer, Sven Fillinger, Harshil Patel, Johannes Alneberg, Andreas Wilm, Maxime Ulysse Garcia, Paolo Di Tommaso & Sven Nahnsen. Nat Biotechnol. 2020 Feb 13. doi: 10.1038/s41587-020-0439-x .","title":"Citations"},{"location":"CITATIONS/","text":"dalmolingroup/euryale: Citations nf-core Ewels PA, Peltzer A, Fillinger S, Patel H, Alneberg J, Wilm A, Garcia MU, Di Tommaso P, Nahnsen S. The nf-core framework for community-curated bioinformatics pipelines. Nat Biotechnol. 2020 Mar;38(3):276-278. doi: 10.1038/s41587-020-0439-x. PubMed PMID: 32055031. Nextflow Di Tommaso P, Chatzou M, Floden EW, Barja PP, Palumbo E, Notredame C. Nextflow enables reproducible computational workflows. Nat Biotechnol. 2017 Apr 11;35(4):316-319. doi: 10.1038/nbt.3820. PubMed PMID: 28398311. Pipeline tools FastQC MultiQC Ewels P, Magnusson M, Lundin S, K\u00e4ller M. MultiQC: summarize analysis results for multiple tools and samples in a single report. Bioinformatics. 2016 Oct 1;32(19):3047-8. doi: 10.1093/bioinformatics/btw354. Epub 2016 Jun 16. PubMed PMID: 27312411; PubMed Central PMCID: PMC5039924. Software packaging/containerisation tools Anaconda Anaconda Software Distribution. Computer software. Vers. 2-2.4.0. Anaconda, Nov. 2016. Web. Bioconda Gr\u00fcning B, Dale R, Sj\u00f6din A, Chapman BA, Rowe J, Tomkins-Tinch CH, Valieris R, K\u00f6ster J; Bioconda Team. Bioconda: sustainable and comprehensive software distribution for the life sciences. Nat Methods. 2018 Jul;15(7):475-476. doi: 10.1038/s41592-018-0046-7. PubMed PMID: 29967506. BioContainers da Veiga Leprevost F, Gr\u00fcning B, Aflitos SA, R\u00f6st HL, Uszkoreit J, Barsnes H, Vaudel M, Moreno P, Gatto L, Weber J, Bai M, Jimenez RC, Sachsenberg T, Pfeuffer J, Alvarez RV, Griss J, Nesvizhskii AI, Perez-Riverol Y. BioContainers: an open-source and community-driven framework for software standardization. Bioinformatics. 2017 Aug 15;33(16):2580-2582. doi: 10.1093/bioinformatics/btx192. PubMed PMID: 28379341; PubMed Central PMCID: PMC5870671. Docker Singularity Kurtzer GM, Sochat V, Bauer MW. Singularity: Scientific containers for mobility of compute. PLoS One. 2017 May 11;12(5):e0177459. doi: 10.1371/journal.pone.0177459. eCollection 2017. PubMed PMID: 28494014; PubMed Central PMCID: PMC5426675.","title":"Citations"},{"location":"CITATIONS/#dalmolingroupeuryale-citations","text":"","title":"dalmolingroup/euryale: Citations"},{"location":"CITATIONS/#nf-core","text":"Ewels PA, Peltzer A, Fillinger S, Patel H, Alneberg J, Wilm A, Garcia MU, Di Tommaso P, Nahnsen S. The nf-core framework for community-curated bioinformatics pipelines. Nat Biotechnol. 2020 Mar;38(3):276-278. doi: 10.1038/s41587-020-0439-x. PubMed PMID: 32055031.","title":"nf-core"},{"location":"CITATIONS/#nextflow","text":"Di Tommaso P, Chatzou M, Floden EW, Barja PP, Palumbo E, Notredame C. Nextflow enables reproducible computational workflows. Nat Biotechnol. 2017 Apr 11;35(4):316-319. doi: 10.1038/nbt.3820. PubMed PMID: 28398311.","title":"Nextflow"},{"location":"CITATIONS/#pipeline-tools","text":"FastQC MultiQC Ewels P, Magnusson M, Lundin S, K\u00e4ller M. MultiQC: summarize analysis results for multiple tools and samples in a single report. Bioinformatics. 2016 Oct 1;32(19):3047-8. doi: 10.1093/bioinformatics/btw354. Epub 2016 Jun 16. PubMed PMID: 27312411; PubMed Central PMCID: PMC5039924.","title":"Pipeline tools"},{"location":"CITATIONS/#software-packagingcontainerisation-tools","text":"Anaconda Anaconda Software Distribution. Computer software. Vers. 2-2.4.0. Anaconda, Nov. 2016. Web. Bioconda Gr\u00fcning B, Dale R, Sj\u00f6din A, Chapman BA, Rowe J, Tomkins-Tinch CH, Valieris R, K\u00f6ster J; Bioconda Team. Bioconda: sustainable and comprehensive software distribution for the life sciences. Nat Methods. 2018 Jul;15(7):475-476. doi: 10.1038/s41592-018-0046-7. PubMed PMID: 29967506. BioContainers da Veiga Leprevost F, Gr\u00fcning B, Aflitos SA, R\u00f6st HL, Uszkoreit J, Barsnes H, Vaudel M, Moreno P, Gatto L, Weber J, Bai M, Jimenez RC, Sachsenberg T, Pfeuffer J, Alvarez RV, Griss J, Nesvizhskii AI, Perez-Riverol Y. BioContainers: an open-source and community-driven framework for software standardization. Bioinformatics. 2017 Aug 15;33(16):2580-2582. doi: 10.1093/bioinformatics/btx192. PubMed PMID: 28379341; PubMed Central PMCID: PMC5870671. Docker Singularity Kurtzer GM, Sochat V, Bauer MW. Singularity: Scientific containers for mobility of compute. PLoS One. 2017 May 11;12(5):e0177459. doi: 10.1371/journal.pone.0177459. eCollection 2017. PubMed PMID: 28494014; PubMed Central PMCID: PMC5426675.","title":"Software packaging/containerisation tools"},{"location":"output/","text":"dalmolingroup/euryale: Output Introduction This document describes the output produced by the pipeline. Most of the plots are taken from the MultiQC report, which summarises results at the end of the pipeline. The directories listed below will be created in the results directory after the pipeline has finished. All paths are relative to the top-level results directory. Pipeline overview The pipeline is built using Nextflow and processes data using the following steps (steps in italics don't run by default): Kaiju - Taxonomically classify reads or contigs Krona - Visualize the taxonomic classification for each sample. Diamond - Alignment reads and contigs against a reference database (such as NCBI-nr). Annotate - Functional annotation of alignment matches. MEGAHIT - Assembled contigs. MultiQC - Aggregate report describing results and QC from the whole pipeline Pipeline information - Report metrics generated during the workflow execution Kaiju Output files - `taxonomy/` - `${sample}.tsv`: Kaiju classification output. - `${sample}.txt`: Kaiju2Table TSV output. Kaiju is a software to perform fast taxonomic classification of metagenomic sequencing reads using a protein reference database. Krona Output files - `taxonomy/` - `${sample}.html`: Krona visualization for the sample. Krona is a tool to interactively explore metagenomes and more from a web browser. Diamond Output files - `alignment/${sample}/` - `${sample}.txt`: Alignment matches in blast tabular output format. - `${sample}.log`: DIAMOND execution log. DIAMOND is an accelerated BLAST compatible local sequence aligner. Annotate Output files - `functional/${sample}/` - `${sample}_annotated.txt`: Alignment matches annotated to chosen functional database (e.g. GO). Annotate is a tool to annotate each query using the best alignment for which a mapping is known. MEGAHIT Output files - `assembly/${sample}/` - `${sample}.contigs.fa.gz`: Contigs assembled for the sample. MEGAHIT is an ultra-fast and memory-efficient (meta-)genome assembler DIAMOND database Output files - `diamond_db/` - `${database_name}.dmnd`: DIAMOND database for the reference fasta file. This output is present if you add the --save_db parameter. DIAMOND is an accelerated BLAST compatible local sequence aligner. MultiQC Output files - `multiqc/` - `multiqc_report.html`: a standalone HTML file that can be viewed in your web browser. - `multiqc_data/`: directory containing parsed statistics from the different tools used in the pipeline. - `multiqc_plots/`: directory containing static images from the report in various formats. MultiQC is a visualization tool that generates a single HTML report summarising all samples in your project. Most of the pipeline QC results are visualised in the report and further statistics are available in the report data directory. Results generated by MultiQC collate pipeline QC from supported tools e.g. FastQC. The pipeline has special steps which also allow the software versions to be reported in the MultiQC output for future traceability. For more information about how to use MultiQC reports, see http://multiqc.info . Pipeline information Output files - `pipeline_info/` - Reports generated by Nextflow: `execution_report.html`, `execution_timeline.html`, `execution_trace.txt` and `pipeline_dag.dot`/`pipeline_dag.svg`. - Reports generated by the pipeline: `pipeline_report.html`, `pipeline_report.txt` and `software_versions.yml`. The `pipeline_report*` files will only be present if the `--email` / `--email_on_fail` parameter's are used when running the pipeline. - Reformatted samplesheet files used as input to the pipeline: `samplesheet.valid.csv`. Nextflow provides excellent functionality for generating various reports relevant to the running and execution of the pipeline. This will allow you to troubleshoot errors with the running of the pipeline, and also provide you with other information such as launch commands, run times and resource usage.","title":"Output"},{"location":"output/#dalmolingroupeuryale-output","text":"","title":"dalmolingroup/euryale: Output"},{"location":"output/#introduction","text":"This document describes the output produced by the pipeline. Most of the plots are taken from the MultiQC report, which summarises results at the end of the pipeline. The directories listed below will be created in the results directory after the pipeline has finished. All paths are relative to the top-level results directory.","title":"Introduction"},{"location":"output/#pipeline-overview","text":"The pipeline is built using Nextflow and processes data using the following steps (steps in italics don't run by default): Kaiju - Taxonomically classify reads or contigs Krona - Visualize the taxonomic classification for each sample. Diamond - Alignment reads and contigs against a reference database (such as NCBI-nr). Annotate - Functional annotation of alignment matches. MEGAHIT - Assembled contigs. MultiQC - Aggregate report describing results and QC from the whole pipeline Pipeline information - Report metrics generated during the workflow execution","title":"Pipeline overview"},{"location":"output/#kaiju","text":"Output files - `taxonomy/` - `${sample}.tsv`: Kaiju classification output. - `${sample}.txt`: Kaiju2Table TSV output. Kaiju is a software to perform fast taxonomic classification of metagenomic sequencing reads using a protein reference database.","title":"Kaiju"},{"location":"output/#krona","text":"Output files - `taxonomy/` - `${sample}.html`: Krona visualization for the sample. Krona is a tool to interactively explore metagenomes and more from a web browser.","title":"Krona"},{"location":"output/#diamond","text":"Output files - `alignment/${sample}/` - `${sample}.txt`: Alignment matches in blast tabular output format. - `${sample}.log`: DIAMOND execution log. DIAMOND is an accelerated BLAST compatible local sequence aligner.","title":"Diamond"},{"location":"output/#annotate","text":"Output files - `functional/${sample}/` - `${sample}_annotated.txt`: Alignment matches annotated to chosen functional database (e.g. GO). Annotate is a tool to annotate each query using the best alignment for which a mapping is known.","title":"Annotate"},{"location":"output/#megahit","text":"Output files - `assembly/${sample}/` - `${sample}.contigs.fa.gz`: Contigs assembled for the sample. MEGAHIT is an ultra-fast and memory-efficient (meta-)genome assembler","title":"MEGAHIT"},{"location":"output/#diamond-database","text":"Output files - `diamond_db/` - `${database_name}.dmnd`: DIAMOND database for the reference fasta file. This output is present if you add the --save_db parameter. DIAMOND is an accelerated BLAST compatible local sequence aligner.","title":"DIAMOND database"},{"location":"output/#multiqc","text":"Output files - `multiqc/` - `multiqc_report.html`: a standalone HTML file that can be viewed in your web browser. - `multiqc_data/`: directory containing parsed statistics from the different tools used in the pipeline. - `multiqc_plots/`: directory containing static images from the report in various formats. MultiQC is a visualization tool that generates a single HTML report summarising all samples in your project. Most of the pipeline QC results are visualised in the report and further statistics are available in the report data directory. Results generated by MultiQC collate pipeline QC from supported tools e.g. FastQC. The pipeline has special steps which also allow the software versions to be reported in the MultiQC output for future traceability. For more information about how to use MultiQC reports, see http://multiqc.info .","title":"MultiQC"},{"location":"output/#pipeline-information","text":"Output files - `pipeline_info/` - Reports generated by Nextflow: `execution_report.html`, `execution_timeline.html`, `execution_trace.txt` and `pipeline_dag.dot`/`pipeline_dag.svg`. - Reports generated by the pipeline: `pipeline_report.html`, `pipeline_report.txt` and `software_versions.yml`. The `pipeline_report*` files will only be present if the `--email` / `--email_on_fail` parameter's are used when running the pipeline. - Reformatted samplesheet files used as input to the pipeline: `samplesheet.valid.csv`. Nextflow provides excellent functionality for generating various reports relevant to the running and execution of the pipeline. This will allow you to troubleshoot errors with the running of the pipeline, and also provide you with other information such as launch commands, run times and resource usage.","title":"Pipeline information"},{"location":"params/","text":"dalmolingroup/euryale pipeline parameters A pipeline for metagenomic taxonomic classification and functional annotation. Based on MEDUSA. Input/output options Define where the pipeline should find input data and save output data. Parameter Description Type Default Required Hidden input Path to comma-separated file containing information about the samples in the experiment. Help You will need to create a design file with information about the samples in your experiment before running the pipeline. Use this parameter to specify its location. It has to be a comma-separated file with 3 columns, and a header row. string True outdir The output directory where the results will be saved. You have to use absolute paths to storage on Cloud infrastructure. string True email Email address for completion summary. Help Set this parameter to your e-mail address to get a summary e-mail with details of the run sent to you when the workflow exits. If set in your user config file ( ~/.nextflow/config ) then you don't need to specify this on the command line for every run. string multiqc_title MultiQC report title. Printed as page header, used for filename if not otherwise specified. string save_dbs Save DIAMOND db to results directory after construction boolean Skip Steps Choose to skip pipeline steps Parameter Description Type Default Required Hidden skip_classification Skip taxonomic classification boolean skip_alignment Skip alignment boolean skip_functional Skip functional annotation boolean Decontamination Parameter Description Type Default Required Hidden host_fasta Host FASTA to use for decontamination string bowtie2_db Pre-built bowtie2 index. Directory where index is located. string Alignment Parameter Description Type Default Required Hidden reference_fasta Path to FASTA genome file. string diamond_db Path to pre-built DIAMOND db. string Taxonomy Parameter Description Type Default Required Hidden kaiju_db Kaiju database string True kraken2_db Kraken2 database string run_kaiju Run Kaiju classifier boolean True run_kraken2 Run Kraken2 classifier boolean Functional Parameter Description Type Default Required Hidden id_mapping Path to ID mapping file to be used for the Functional annotation string True minimum_bitscore Minimum bitscore of a match to be used for annotation integer 50 minimum_pident Minimum identity of a match to be used for annotation integer 80 minimum_alen Minimum alignment length of a match to be used for annotation integer 50 maximum_evalue Maximum evalue of a match to be used for annotation number 0.0001 Assembly Parameter Description Type Default Required Hidden assembly_based boolean Reference genome options Reference genome related files and options required for the workflow. Parameter Description Type Default Required Hidden genome Name of iGenomes reference. Help If using a reference genome configured in the pipeline using iGenomes, use this parameter to give the ID for the reference. This is then used to build the full paths for all required reference genome files e.g. --genome GRCh38 . See the nf-core website docs for more details. string igenomes_base Directory / URL base for iGenomes references. string s3://ngi-igenomes/igenomes True igenomes_ignore Do not load the iGenomes reference config. Help Do not load igenomes.config when running the pipeline. You may choose this option if you observe clashes between custom parameters and those supplied in igenomes.config . boolean True fasta string Max job request options Set the top limit for requested resources for any single job. Parameter Description Type Default Required Hidden max_cpus Maximum number of CPUs that can be requested for any single job. Help Use to set an upper-limit for the CPU requirement for each process. Should be an integer e.g. --max_cpus 1 integer 16 True max_memory Maximum amount of memory that can be requested for any single job. Help Use to set an upper-limit for the memory requirement for each process. Should be a string in the format integer-unit e.g. --max_memory '8.GB' string 128.GB True max_time Maximum amount of time that can be requested for any single job. Help Use to set an upper-limit for the time requirement for each process. Should be a string in the format integer-unit e.g. --max_time '2.h' string 240.h True Generic options Less common options for the pipeline, typically set in a config file. Parameter Description Type Default Required Hidden help Display help text. boolean True version Display version and exit. boolean True publish_dir_mode Method used to save pipeline results to output directory. Help The Nextflow publishDir option specifies which intermediate files should be saved to the output directory. This option tells the pipeline what method should be used to move these files. See Nextflow docs for details. string copy True email_on_fail Email address for completion summary, only when pipeline fails. Help An email address to send a summary email to when the pipeline is completed - ONLY sent if the pipeline does not exit successfully. string True plaintext_email Send plain-text email instead of HTML. boolean True max_multiqc_email_size File size limit when attaching MultiQC reports to summary emails. string 25.MB True monochrome_logs Do not use coloured log outputs. boolean True hook_url Incoming hook URL for messaging service Help Incoming hook URL for messaging service. Currently, MS Teams and Slack are supported. string True multiqc_config Custom config file to supply to MultiQC. string True multiqc_logo Custom logo file to supply to MultiQC. File name must also be set in the MultiQC config file string True multiqc_methods_description Custom MultiQC yaml file containing HTML including a methods description. string tracedir Directory to keep pipeline Nextflow logs and reports. string ${params.outdir}/pipeline_info True validate_params Boolean whether to validate parameters against the schema at runtime boolean True True show_hidden_params Show all params when using --help Help By default, parameters set as hidden in the schema are not shown on the command line when a user runs with --help . Specifying this option will tell the pipeline to show all parameters. boolean True schema_ignore_params string genomes True","title":"Parameters"},{"location":"params/#dalmolingroupeuryale-pipeline-parameters","text":"A pipeline for metagenomic taxonomic classification and functional annotation. Based on MEDUSA.","title":"dalmolingroup/euryale pipeline parameters"},{"location":"params/#inputoutput-options","text":"Define where the pipeline should find input data and save output data. Parameter Description Type Default Required Hidden input Path to comma-separated file containing information about the samples in the experiment. Help You will need to create a design file with information about the samples in your experiment before running the pipeline. Use this parameter to specify its location. It has to be a comma-separated file with 3 columns, and a header row. string True outdir The output directory where the results will be saved. You have to use absolute paths to storage on Cloud infrastructure. string True email Email address for completion summary. Help Set this parameter to your e-mail address to get a summary e-mail with details of the run sent to you when the workflow exits. If set in your user config file ( ~/.nextflow/config ) then you don't need to specify this on the command line for every run. string multiqc_title MultiQC report title. Printed as page header, used for filename if not otherwise specified. string save_dbs Save DIAMOND db to results directory after construction boolean","title":"Input/output options"},{"location":"params/#skip-steps","text":"Choose to skip pipeline steps Parameter Description Type Default Required Hidden skip_classification Skip taxonomic classification boolean skip_alignment Skip alignment boolean skip_functional Skip functional annotation boolean","title":"Skip Steps"},{"location":"params/#decontamination","text":"Parameter Description Type Default Required Hidden host_fasta Host FASTA to use for decontamination string bowtie2_db Pre-built bowtie2 index. Directory where index is located. string","title":"Decontamination"},{"location":"params/#alignment","text":"Parameter Description Type Default Required Hidden reference_fasta Path to FASTA genome file. string diamond_db Path to pre-built DIAMOND db. string","title":"Alignment"},{"location":"params/#taxonomy","text":"Parameter Description Type Default Required Hidden kaiju_db Kaiju database string True kraken2_db Kraken2 database string run_kaiju Run Kaiju classifier boolean True run_kraken2 Run Kraken2 classifier boolean","title":"Taxonomy"},{"location":"params/#functional","text":"Parameter Description Type Default Required Hidden id_mapping Path to ID mapping file to be used for the Functional annotation string True minimum_bitscore Minimum bitscore of a match to be used for annotation integer 50 minimum_pident Minimum identity of a match to be used for annotation integer 80 minimum_alen Minimum alignment length of a match to be used for annotation integer 50 maximum_evalue Maximum evalue of a match to be used for annotation number 0.0001","title":"Functional"},{"location":"params/#assembly","text":"Parameter Description Type Default Required Hidden assembly_based boolean","title":"Assembly"},{"location":"params/#reference-genome-options","text":"Reference genome related files and options required for the workflow. Parameter Description Type Default Required Hidden genome Name of iGenomes reference. Help If using a reference genome configured in the pipeline using iGenomes, use this parameter to give the ID for the reference. This is then used to build the full paths for all required reference genome files e.g. --genome GRCh38 . See the nf-core website docs for more details. string igenomes_base Directory / URL base for iGenomes references. string s3://ngi-igenomes/igenomes True igenomes_ignore Do not load the iGenomes reference config. Help Do not load igenomes.config when running the pipeline. You may choose this option if you observe clashes between custom parameters and those supplied in igenomes.config . boolean True fasta string","title":"Reference genome options"},{"location":"params/#max-job-request-options","text":"Set the top limit for requested resources for any single job. Parameter Description Type Default Required Hidden max_cpus Maximum number of CPUs that can be requested for any single job. Help Use to set an upper-limit for the CPU requirement for each process. Should be an integer e.g. --max_cpus 1 integer 16 True max_memory Maximum amount of memory that can be requested for any single job. Help Use to set an upper-limit for the memory requirement for each process. Should be a string in the format integer-unit e.g. --max_memory '8.GB' string 128.GB True max_time Maximum amount of time that can be requested for any single job. Help Use to set an upper-limit for the time requirement for each process. Should be a string in the format integer-unit e.g. --max_time '2.h' string 240.h True","title":"Max job request options"},{"location":"params/#generic-options","text":"Less common options for the pipeline, typically set in a config file. Parameter Description Type Default Required Hidden help Display help text. boolean True version Display version and exit. boolean True publish_dir_mode Method used to save pipeline results to output directory. Help The Nextflow publishDir option specifies which intermediate files should be saved to the output directory. This option tells the pipeline what method should be used to move these files. See Nextflow docs for details. string copy True email_on_fail Email address for completion summary, only when pipeline fails. Help An email address to send a summary email to when the pipeline is completed - ONLY sent if the pipeline does not exit successfully. string True plaintext_email Send plain-text email instead of HTML. boolean True max_multiqc_email_size File size limit when attaching MultiQC reports to summary emails. string 25.MB True monochrome_logs Do not use coloured log outputs. boolean True hook_url Incoming hook URL for messaging service Help Incoming hook URL for messaging service. Currently, MS Teams and Slack are supported. string True multiqc_config Custom config file to supply to MultiQC. string True multiqc_logo Custom logo file to supply to MultiQC. File name must also be set in the MultiQC config file string True multiqc_methods_description Custom MultiQC yaml file containing HTML including a methods description. string tracedir Directory to keep pipeline Nextflow logs and reports. string ${params.outdir}/pipeline_info True validate_params Boolean whether to validate parameters against the schema at runtime boolean True True show_hidden_params Show all params when using --help Help By default, parameters set as hidden in the schema are not shown on the command line when a user runs with --help . Specifying this option will tell the pipeline to show all parameters. boolean True schema_ignore_params string genomes True","title":"Generic options"},{"location":"usage/","text":"dalmolingroup/euryale: Usage Documentation of pipeline parameters is generated automatically from the pipeline schema and can no longer be found in markdown files. Introduction Samplesheet input You will need to create a samplesheet with information about the samples you would like to analyse before running the pipeline. Use this parameter to specify its location. It has to be a comma-separated file with 3 columns, and a header row as shown in the examples below. --input '[path to samplesheet file]' Multiple runs of the same sample The sample identifiers have to be the same when you have re-sequenced the same sample more than once e.g. to increase sequencing depth. The pipeline will concatenate the raw reads before performing any downstream analysis. Below is an example for the same sample sequenced across 3 lanes: sample,fastq_1,fastq_2 CONTROL_REP1,AEG588A1_S1_L002_R1_001.fastq.gz,AEG588A1_S1_L002_R2_001.fastq.gz CONTROL_REP1,AEG588A1_S1_L003_R1_001.fastq.gz,AEG588A1_S1_L003_R2_001.fastq.gz CONTROL_REP1,AEG588A1_S1_L004_R1_001.fastq.gz,AEG588A1_S1_L004_R2_001.fastq.gz Full samplesheet The pipeline will auto-detect whether a sample is single- or paired-end using the information provided in the samplesheet. The samplesheet can have as many columns as you desire, however, there is a strict requirement for the first 3 columns to match those defined in the table below. A final samplesheet file consisting of both single- and paired-end data may look something like the one below. This is for 6 samples, where TREATMENT_REP3 has been sequenced twice. sample,fastq_1,fastq_2 CONTROL_REP1,AEG588A1_S1_L002_R1_001.fastq.gz,AEG588A1_S1_L002_R2_001.fastq.gz CONTROL_REP2,AEG588A2_S2_L002_R1_001.fastq.gz,AEG588A2_S2_L002_R2_001.fastq.gz CONTROL_REP3,AEG588A3_S3_L002_R1_001.fastq.gz,AEG588A3_S3_L002_R2_001.fastq.gz TREATMENT_REP1,AEG588A4_S4_L003_R1_001.fastq.gz, TREATMENT_REP2,AEG588A5_S5_L003_R1_001.fastq.gz, TREATMENT_REP3,AEG588A6_S6_L003_R1_001.fastq.gz, TREATMENT_REP3,AEG588A6_S6_L004_R1_001.fastq.gz, Column Description sample Custom sample name. This entry will be identical for multiple sequencing libraries/runs from the same sample. Spaces in sample names are automatically converted to underscores ( _ ). fastq_1 Full path to FastQ file for Illumina short reads 1. File has to be gzipped and have the extension \".fastq.gz\" or \".fq.gz\". fastq_2 Full path to FastQ file for Illumina short reads 2. File has to be gzipped and have the extension \".fastq.gz\" or \".fq.gz\". An example samplesheet has been provided with the pipeline. Running the pipeline The typical command for running the pipeline is as follows: nextflow run dalmolingroup/euryale --input samplesheet.csv --outdir --genome GRCh37 -profile docker This will launch the pipeline with the docker configuration profile. See below for more information about profiles. Note that the pipeline will create the following files in your working directory: work # Directory containing the nextflow working files # Finished results in specified location (defined with --outdir) .nextflow_log # Log file from Nextflow # Other nextflow hidden files, eg. history of pipeline runs and old logs. Updating the pipeline When you run the above command, Nextflow automatically pulls the pipeline code from GitHub and stores it as a cached version. When running the pipeline after this, it will always use the cached version if available - even if the pipeline has been updated since. To make sure that you're running the latest version of the pipeline, make sure that you regularly update the cached version of the pipeline: nextflow pull dalmolingroup/euryale Reproducibility It is a good idea to specify a pipeline version when running the pipeline on your data. This ensures that a specific version of the pipeline code and software are used when you run your pipeline. If you keep using the same tag, you'll be running the same version of the pipeline, even if there have been changes to the code since. First, go to the dalmolingroup/euryale releases page and find the latest pipeline version - numeric only (eg. 1.3.1 ). Then specify this when running the pipeline with -r (one hyphen) - eg. -r 1.3.1 . Of course, you can switch to another version by changing the number after the -r flag. This version number will be logged in reports when you run the pipeline, so that you'll know what you used when you look back in the future. For example, at the bottom of the MultiQC reports. Core Nextflow arguments NB: These options are part of Nextflow and use a single hyphen (pipeline parameters use a double-hyphen). -profile Use this parameter to choose a configuration profile. Profiles can give configuration presets for different compute environments. Several generic profiles are bundled with the pipeline which instruct the pipeline to use software packaged using different methods (Docker, Singularity, Podman, Shifter, Charliecloud, Conda) - see below. We highly recommend the use of Docker or Singularity containers for full pipeline reproducibility, however when this is not possible, Conda is also supported. Note that multiple profiles can be loaded, for example: -profile test,docker - the order of arguments is important! They are loaded in sequence, so later profiles can overwrite earlier profiles. If -profile is not specified, the pipeline will run locally and expect all software to be installed and available on the PATH . This is not recommended, since it can lead to different results on different machines dependent on the computer enviroment. test A profile with a complete configuration for automated testing Includes links to test data so needs no other parameters docker A generic configuration profile to be used with Docker singularity A generic configuration profile to be used with Singularity podman A generic configuration profile to be used with Podman shifter A generic configuration profile to be used with Shifter charliecloud A generic configuration profile to be used with Charliecloud conda A generic configuration profile to be used with Conda . Please only use Conda as a last resort i.e. when it's not possible to run the pipeline with Docker, Singularity, Podman, Shifter or Charliecloud. -resume Specify this when restarting a pipeline. Nextflow will use cached results from any pipeline steps where the inputs are the same, continuing from where it got to previously. For input to be considered the same, not only the names must be identical but the files' contents as well. For more info about this parameter, see this blog post . You can also supply a run name to resume a specific run: -resume [run-name] . Use the nextflow log command to show previous run names. -c Specify the path to a specific config file (this is a core Nextflow command). See the nf-core website documentation for more information. Custom configuration Resource requests Whilst the default requirements set within the pipeline will hopefully work for most people and with most input data, you may find that you want to customise the compute resources that the pipeline requests. Each step in the pipeline has a default set of requirements for number of CPUs, memory and time. For most of the steps in the pipeline, if the job exits with any of the error codes specified here it will automatically be resubmitted with higher requests (2 x original, then 3 x original). If it still fails after the third attempt then the pipeline execution is stopped. For example, if the nf-core/rnaseq pipeline is failing after multiple re-submissions of the STAR_ALIGN process due to an exit code of 137 this would indicate that there is an out of memory issue: [62/149eb0] NOTE: Process `NFCORE_RNASEQ:RNASEQ:ALIGN_STAR:STAR_ALIGN (WT_REP1)` terminated with an error exit status (137) -- Execution is retried (1) Error executing process > 'NFCORE_RNASEQ:RNASEQ:ALIGN_STAR:STAR_ALIGN (WT_REP1)' Caused by: Process `NFCORE_RNASEQ:RNASEQ:ALIGN_STAR:STAR_ALIGN (WT_REP1)` terminated with an error exit status (137) Command executed: STAR \\ --genomeDir star \\ --readFilesIn WT_REP1_trimmed.fq.gz \\ --runThreadN 2 \\ --outFileNamePrefix WT_REP1. \\ Command exit status: 137 Command output: (empty) Command error: .command.sh: line 9: 30 Killed STAR --genomeDir star --readFilesIn WT_REP1_trimmed.fq.gz --runThreadN 2 --outFileNamePrefix WT_REP1. Work dir: /home/pipelinetest/work/9d/172ca5881234073e8d76f2a19c88fb Tip: you can replicate the issue by changing to the process work dir and entering the command `bash .command.run` For beginners A first step to bypass this error, you could try to increase the amount of CPUs, memory, and time for the whole pipeline. Therefor you can try to increase the resource for the parameters --max_cpus , --max_memory , and --max_time . Based on the error above, you have to increase the amount of memory. Therefore you can go to the parameter documentation of rnaseq and scroll down to the show hidden parameter button to get the default value for --max_memory . In this case 128GB, you than can try to run your pipeline again with --max_memory 200GB -resume to skip all process, that were already calculated. If you can not increase the resource of the complete pipeline, you can try to adapt the resource for a single process as mentioned below. Advanced option on process level To bypass this error you would need to find exactly which resources are set by the STAR_ALIGN process. The quickest way is to search for process STAR_ALIGN in the nf-core/rnaseq Github repo . We have standardised the structure of Nextflow DSL2 pipelines such that all module files will be present in the modules/ directory and so, based on the search results, the file we want is modules/nf-core/star/align/main.nf . If you click on the link to that file you will notice that there is a label directive at the top of the module that is set to label process_high . The Nextflow label directive allows us to organise workflow processes in separate groups which can be referenced in a configuration file to select and configure subset of processes having similar computing requirements. The default values for the process_high label are set in the pipeline's base.config which in this case is defined as 72GB. Providing you haven't set any other standard nf-core parameters to cap the maximum resources used by the pipeline then we can try and bypass the STAR_ALIGN process failure by creating a custom config file that sets at least 72GB of memory, in this case increased to 100GB. The custom config below can then be provided to the pipeline via the -c parameter as highlighted in previous sections. process { withName: 'NFCORE_RNASEQ:RNASEQ:ALIGN_STAR:STAR_ALIGN' { memory = 100.GB } } NB: We specify the full process name i.e. NFCORE_RNASEQ:RNASEQ:ALIGN_STAR:STAR_ALIGN in the config file because this takes priority over the short name ( STAR_ALIGN ) and allows existing configuration using the full process name to be correctly overridden. If you get a warning suggesting that the process selector isn't recognised check that the process name has been specified correctly. Updating containers (advanced users) The Nextflow DSL2 implementation of this pipeline uses one container per process which makes it much easier to maintain and update software dependencies. If for some reason you need to use a different version of a particular tool with the pipeline then you just need to identify the process name and override the Nextflow container definition for that process using the withName declaration. For example, in the nf-core/viralrecon pipeline a tool called Pangolin has been used during the COVID-19 pandemic to assign lineages to SARS-CoV-2 genome sequenced samples. Given that the lineage assignments change quite frequently it doesn't make sense to re-release the nf-core/viralrecon everytime a new version of Pangolin has been released. However, you can override the default container used by the pipeline by creating a custom config file and passing it as a command-line argument via -c custom.config . Check the default version used by the pipeline in the module file for Pangolin Find the latest version of the Biocontainer available on Quay.io Create the custom config accordingly: For Docker: nextflow process { withName: PANGOLIN { container = 'quay.io/biocontainers/pangolin:3.0.5--pyhdfd78af_0' } } For Singularity: nextflow process { withName: PANGOLIN { container = 'https://depot.galaxyproject.org/singularity/pangolin:3.0.5--pyhdfd78af_0' } } For Conda: nextflow process { withName: PANGOLIN { conda = 'bioconda::pangolin=3.0.5' } } NB: If you wish to periodically update individual tool-specific results (e.g. Pangolin) generated by the pipeline then you must ensure to keep the work/ directory otherwise the -resume ability of the pipeline will be compromised and it will restart from scratch. Running in the background Nextflow handles job submissions and supervises the running jobs. The Nextflow process must run until the pipeline is finished. The Nextflow -bg flag launches Nextflow in the background, detached from your terminal so that the workflow does not stop if you log out of your session. The logs are saved to a file. Alternatively, you can use screen / tmux or similar tool to create a detached session which you can log back into at a later time. Some HPC setups also allow you to run nextflow within a cluster job submitted your job scheduler (from where it submits more jobs). Nextflow memory requirements In some cases, the Nextflow Java virtual machines can start to request a large amount of memory. We recommend adding the following line to your environment to limit this (typically in ~/.bashrc or ~./bash_profile ): NXF_OPTS='-Xms1g -Xmx4g'","title":"Usage"},{"location":"usage/#dalmolingroupeuryale-usage","text":"Documentation of pipeline parameters is generated automatically from the pipeline schema and can no longer be found in markdown files.","title":"dalmolingroup/euryale: Usage"},{"location":"usage/#introduction","text":"","title":"Introduction"},{"location":"usage/#samplesheet-input","text":"You will need to create a samplesheet with information about the samples you would like to analyse before running the pipeline. Use this parameter to specify its location. It has to be a comma-separated file with 3 columns, and a header row as shown in the examples below. --input '[path to samplesheet file]'","title":"Samplesheet input"},{"location":"usage/#multiple-runs-of-the-same-sample","text":"The sample identifiers have to be the same when you have re-sequenced the same sample more than once e.g. to increase sequencing depth. The pipeline will concatenate the raw reads before performing any downstream analysis. Below is an example for the same sample sequenced across 3 lanes: sample,fastq_1,fastq_2 CONTROL_REP1,AEG588A1_S1_L002_R1_001.fastq.gz,AEG588A1_S1_L002_R2_001.fastq.gz CONTROL_REP1,AEG588A1_S1_L003_R1_001.fastq.gz,AEG588A1_S1_L003_R2_001.fastq.gz CONTROL_REP1,AEG588A1_S1_L004_R1_001.fastq.gz,AEG588A1_S1_L004_R2_001.fastq.gz","title":"Multiple runs of the same sample"},{"location":"usage/#full-samplesheet","text":"The pipeline will auto-detect whether a sample is single- or paired-end using the information provided in the samplesheet. The samplesheet can have as many columns as you desire, however, there is a strict requirement for the first 3 columns to match those defined in the table below. A final samplesheet file consisting of both single- and paired-end data may look something like the one below. This is for 6 samples, where TREATMENT_REP3 has been sequenced twice. sample,fastq_1,fastq_2 CONTROL_REP1,AEG588A1_S1_L002_R1_001.fastq.gz,AEG588A1_S1_L002_R2_001.fastq.gz CONTROL_REP2,AEG588A2_S2_L002_R1_001.fastq.gz,AEG588A2_S2_L002_R2_001.fastq.gz CONTROL_REP3,AEG588A3_S3_L002_R1_001.fastq.gz,AEG588A3_S3_L002_R2_001.fastq.gz TREATMENT_REP1,AEG588A4_S4_L003_R1_001.fastq.gz, TREATMENT_REP2,AEG588A5_S5_L003_R1_001.fastq.gz, TREATMENT_REP3,AEG588A6_S6_L003_R1_001.fastq.gz, TREATMENT_REP3,AEG588A6_S6_L004_R1_001.fastq.gz, Column Description sample Custom sample name. This entry will be identical for multiple sequencing libraries/runs from the same sample. Spaces in sample names are automatically converted to underscores ( _ ). fastq_1 Full path to FastQ file for Illumina short reads 1. File has to be gzipped and have the extension \".fastq.gz\" or \".fq.gz\". fastq_2 Full path to FastQ file for Illumina short reads 2. File has to be gzipped and have the extension \".fastq.gz\" or \".fq.gz\". An example samplesheet has been provided with the pipeline.","title":"Full samplesheet"},{"location":"usage/#running-the-pipeline","text":"The typical command for running the pipeline is as follows: nextflow run dalmolingroup/euryale --input samplesheet.csv --outdir --genome GRCh37 -profile docker This will launch the pipeline with the docker configuration profile. See below for more information about profiles. Note that the pipeline will create the following files in your working directory: work # Directory containing the nextflow working files # Finished results in specified location (defined with --outdir) .nextflow_log # Log file from Nextflow # Other nextflow hidden files, eg. history of pipeline runs and old logs.","title":"Running the pipeline"},{"location":"usage/#updating-the-pipeline","text":"When you run the above command, Nextflow automatically pulls the pipeline code from GitHub and stores it as a cached version. When running the pipeline after this, it will always use the cached version if available - even if the pipeline has been updated since. To make sure that you're running the latest version of the pipeline, make sure that you regularly update the cached version of the pipeline: nextflow pull dalmolingroup/euryale","title":"Updating the pipeline"},{"location":"usage/#reproducibility","text":"It is a good idea to specify a pipeline version when running the pipeline on your data. This ensures that a specific version of the pipeline code and software are used when you run your pipeline. If you keep using the same tag, you'll be running the same version of the pipeline, even if there have been changes to the code since. First, go to the dalmolingroup/euryale releases page and find the latest pipeline version - numeric only (eg. 1.3.1 ). Then specify this when running the pipeline with -r (one hyphen) - eg. -r 1.3.1 . Of course, you can switch to another version by changing the number after the -r flag. This version number will be logged in reports when you run the pipeline, so that you'll know what you used when you look back in the future. For example, at the bottom of the MultiQC reports.","title":"Reproducibility"},{"location":"usage/#core-nextflow-arguments","text":"NB: These options are part of Nextflow and use a single hyphen (pipeline parameters use a double-hyphen).","title":"Core Nextflow arguments"},{"location":"usage/#-profile","text":"Use this parameter to choose a configuration profile. Profiles can give configuration presets for different compute environments. Several generic profiles are bundled with the pipeline which instruct the pipeline to use software packaged using different methods (Docker, Singularity, Podman, Shifter, Charliecloud, Conda) - see below. We highly recommend the use of Docker or Singularity containers for full pipeline reproducibility, however when this is not possible, Conda is also supported. Note that multiple profiles can be loaded, for example: -profile test,docker - the order of arguments is important! They are loaded in sequence, so later profiles can overwrite earlier profiles. If -profile is not specified, the pipeline will run locally and expect all software to be installed and available on the PATH . This is not recommended, since it can lead to different results on different machines dependent on the computer enviroment. test A profile with a complete configuration for automated testing Includes links to test data so needs no other parameters docker A generic configuration profile to be used with Docker singularity A generic configuration profile to be used with Singularity podman A generic configuration profile to be used with Podman shifter A generic configuration profile to be used with Shifter charliecloud A generic configuration profile to be used with Charliecloud conda A generic configuration profile to be used with Conda . Please only use Conda as a last resort i.e. when it's not possible to run the pipeline with Docker, Singularity, Podman, Shifter or Charliecloud.","title":"-profile"},{"location":"usage/#-resume","text":"Specify this when restarting a pipeline. Nextflow will use cached results from any pipeline steps where the inputs are the same, continuing from where it got to previously. For input to be considered the same, not only the names must be identical but the files' contents as well. For more info about this parameter, see this blog post . You can also supply a run name to resume a specific run: -resume [run-name] . Use the nextflow log command to show previous run names.","title":"-resume"},{"location":"usage/#-c","text":"Specify the path to a specific config file (this is a core Nextflow command). See the nf-core website documentation for more information.","title":"-c"},{"location":"usage/#custom-configuration","text":"","title":"Custom configuration"},{"location":"usage/#resource-requests","text":"Whilst the default requirements set within the pipeline will hopefully work for most people and with most input data, you may find that you want to customise the compute resources that the pipeline requests. Each step in the pipeline has a default set of requirements for number of CPUs, memory and time. For most of the steps in the pipeline, if the job exits with any of the error codes specified here it will automatically be resubmitted with higher requests (2 x original, then 3 x original). If it still fails after the third attempt then the pipeline execution is stopped. For example, if the nf-core/rnaseq pipeline is failing after multiple re-submissions of the STAR_ALIGN process due to an exit code of 137 this would indicate that there is an out of memory issue: [62/149eb0] NOTE: Process `NFCORE_RNASEQ:RNASEQ:ALIGN_STAR:STAR_ALIGN (WT_REP1)` terminated with an error exit status (137) -- Execution is retried (1) Error executing process > 'NFCORE_RNASEQ:RNASEQ:ALIGN_STAR:STAR_ALIGN (WT_REP1)' Caused by: Process `NFCORE_RNASEQ:RNASEQ:ALIGN_STAR:STAR_ALIGN (WT_REP1)` terminated with an error exit status (137) Command executed: STAR \\ --genomeDir star \\ --readFilesIn WT_REP1_trimmed.fq.gz \\ --runThreadN 2 \\ --outFileNamePrefix WT_REP1. \\ Command exit status: 137 Command output: (empty) Command error: .command.sh: line 9: 30 Killed STAR --genomeDir star --readFilesIn WT_REP1_trimmed.fq.gz --runThreadN 2 --outFileNamePrefix WT_REP1. Work dir: /home/pipelinetest/work/9d/172ca5881234073e8d76f2a19c88fb Tip: you can replicate the issue by changing to the process work dir and entering the command `bash .command.run`","title":"Resource requests"},{"location":"usage/#for-beginners","text":"A first step to bypass this error, you could try to increase the amount of CPUs, memory, and time for the whole pipeline. Therefor you can try to increase the resource for the parameters --max_cpus , --max_memory , and --max_time . Based on the error above, you have to increase the amount of memory. Therefore you can go to the parameter documentation of rnaseq and scroll down to the show hidden parameter button to get the default value for --max_memory . In this case 128GB, you than can try to run your pipeline again with --max_memory 200GB -resume to skip all process, that were already calculated. If you can not increase the resource of the complete pipeline, you can try to adapt the resource for a single process as mentioned below.","title":"For beginners"},{"location":"usage/#advanced-option-on-process-level","text":"To bypass this error you would need to find exactly which resources are set by the STAR_ALIGN process. The quickest way is to search for process STAR_ALIGN in the nf-core/rnaseq Github repo . We have standardised the structure of Nextflow DSL2 pipelines such that all module files will be present in the modules/ directory and so, based on the search results, the file we want is modules/nf-core/star/align/main.nf . If you click on the link to that file you will notice that there is a label directive at the top of the module that is set to label process_high . The Nextflow label directive allows us to organise workflow processes in separate groups which can be referenced in a configuration file to select and configure subset of processes having similar computing requirements. The default values for the process_high label are set in the pipeline's base.config which in this case is defined as 72GB. Providing you haven't set any other standard nf-core parameters to cap the maximum resources used by the pipeline then we can try and bypass the STAR_ALIGN process failure by creating a custom config file that sets at least 72GB of memory, in this case increased to 100GB. The custom config below can then be provided to the pipeline via the -c parameter as highlighted in previous sections. process { withName: 'NFCORE_RNASEQ:RNASEQ:ALIGN_STAR:STAR_ALIGN' { memory = 100.GB } } NB: We specify the full process name i.e. NFCORE_RNASEQ:RNASEQ:ALIGN_STAR:STAR_ALIGN in the config file because this takes priority over the short name ( STAR_ALIGN ) and allows existing configuration using the full process name to be correctly overridden. If you get a warning suggesting that the process selector isn't recognised check that the process name has been specified correctly.","title":"Advanced option on process level"},{"location":"usage/#updating-containers-advanced-users","text":"The Nextflow DSL2 implementation of this pipeline uses one container per process which makes it much easier to maintain and update software dependencies. If for some reason you need to use a different version of a particular tool with the pipeline then you just need to identify the process name and override the Nextflow container definition for that process using the withName declaration. For example, in the nf-core/viralrecon pipeline a tool called Pangolin has been used during the COVID-19 pandemic to assign lineages to SARS-CoV-2 genome sequenced samples. Given that the lineage assignments change quite frequently it doesn't make sense to re-release the nf-core/viralrecon everytime a new version of Pangolin has been released. However, you can override the default container used by the pipeline by creating a custom config file and passing it as a command-line argument via -c custom.config . Check the default version used by the pipeline in the module file for Pangolin Find the latest version of the Biocontainer available on Quay.io Create the custom config accordingly: For Docker: nextflow process { withName: PANGOLIN { container = 'quay.io/biocontainers/pangolin:3.0.5--pyhdfd78af_0' } } For Singularity: nextflow process { withName: PANGOLIN { container = 'https://depot.galaxyproject.org/singularity/pangolin:3.0.5--pyhdfd78af_0' } } For Conda: nextflow process { withName: PANGOLIN { conda = 'bioconda::pangolin=3.0.5' } } NB: If you wish to periodically update individual tool-specific results (e.g. Pangolin) generated by the pipeline then you must ensure to keep the work/ directory otherwise the -resume ability of the pipeline will be compromised and it will restart from scratch.","title":"Updating containers (advanced users)"},{"location":"usage/#running-in-the-background","text":"Nextflow handles job submissions and supervises the running jobs. The Nextflow process must run until the pipeline is finished. The Nextflow -bg flag launches Nextflow in the background, detached from your terminal so that the workflow does not stop if you log out of your session. The logs are saved to a file. Alternatively, you can use screen / tmux or similar tool to create a detached session which you can log back into at a later time. Some HPC setups also allow you to run nextflow within a cluster job submitted your job scheduler (from where it submits more jobs).","title":"Running in the background"},{"location":"usage/#nextflow-memory-requirements","text":"In some cases, the Nextflow Java virtual machines can start to request a large amount of memory. We recommend adding the following line to your environment to limit this (typically in ~/.bashrc or ~./bash_profile ): NXF_OPTS='-Xms1g -Xmx4g'","title":"Nextflow memory requirements"}]}
\ No newline at end of file
+{"config":{"indexing":"full","lang":["en"],"min_search_length":3,"prebuild_index":false,"separator":"[\\s\\-]+"},"docs":[{"location":"","text":"Introduction dalmolingroup/euryale is a pipeline for taxonomic classification and functional annotation of metagenomic reads. Based on MEDUSA . The pipeline is built using Nextflow , a workflow tool to run tasks across multiple compute infrastructures in a very portable manner. It uses Docker/Singularity containers making installation trivial and results highly reproducible. The Nextflow DSL2 implementation of this pipeline uses one container per process which makes it much easier to maintain and update software dependencies. Where possible, these processes have been submitted to and installed from nf-core/modules in order to make them available to all nf-core pipelines, and to everyone within the Nextflow community! Pipeline summary Pre-processing Read QC ( FastQC ) Read trimming and merging ( fastp ) ( optionally ) Host read removal ( BowTie2 ) Duplicated sequence removal ( fastx collapser ) Present QC and other data ( MultiQC ) Assembly ( optionally ) Read assembly ( MEGAHIT ) Taxonomic classification Sequence classification ( Kaiju ) Sequence classification ( Kraken2 ) Visualization ( Krona ) Functional annotation Sequence alignment ( DIAMOND ) Map alignment matches to functional database ( annotate ) Quick Start Install Nextflow ( >=22.10.1 ) Install any of Docker , Singularity (you can follow this tutorial ), Podman , Shifter or Charliecloud for full pipeline reproducibility (you can use Conda both to install Nextflow itself and also to manage software within pipelines. Please only use it within pipelines as a last resort; see docs ) . Download the pipeline and test it on a minimal dataset with a single command: nextflow run dalmolingroup/euryale -profile test,YOURPROFILE --outdir Note that some form of configuration will be needed so that Nextflow knows how to fetch the required software. This is usually done in the form of a config profile ( YOURPROFILE in the example command above). You can chain multiple config profiles in a comma-separated string. The pipeline comes with config profiles called docker , singularity , podman , shifter , charliecloud and conda which instruct the pipeline to use the named tool for software management. For example, -profile test,docker . Please check nf-core/configs to see if a custom config file to run nf-core pipelines already exists for your Institute. If so, you can simply use -profile in your command. This will enable either docker or singularity and set the appropriate execution settings for your local compute environment. If you are using singularity , please use the nf-core download command to download images first, before running the pipeline. Setting the NXF_SINGULARITY_CACHEDIR or singularity.cacheDir Nextflow options enables you to store and re-use the images from a central location for future pipeline runs. If you are using conda , it is highly recommended to use the NXF_CONDA_CACHEDIR or conda.cacheDir settings to store the environments in a central location for future pipeline runs. Start running your own analysis! nextflow run dalmolingroup/euryale \\ --input samplesheet.csv \\ --outdir \\ --kaiju_db kaiju_reference \\ --diamond_db diamond_db \\ --reference_fasta diamond_fasta \\ --host_fasta host_reference_fasta \\ --id_mapping id_mapping_file \\ -profile Databases and references A question that pops up a lot is: Since Euryale requires a lot of reference parameters, where can I find these references? Below we provide a short list of places where you can find these databases. But, of course, we're not limited to these references: Euryale should be able to process your own databases, should you want to build them yourself. Alignment For the alignment you can either provide --diamond_db for a pre-built DIAMOND database, or you can provide --reference_fasta . For reference fasta, by default Euryale expects something like NCBI-nr , but similarly formatted reference databases should also suffice. Taxonomic classification At its current version, Euryale doesn't build a reference taxonomic database, but pre-built ones are supported. If you're using Kaiju (the default), you can provide a reference database with --kaiju_db and provide a .tar.gz file like the ones provided in the official Kaiju website . We have extensively tested Euryale with the 2021 version of the nr database and it should work as expected. If you're using Kraken2 (By supplying --run_kraken2 ), we expect something like the pre-built .tar.gz databases provided by the Kraken2 developers to be provided to --kraken2_db . Functional annotation We expect an ID mapping reference to be used within annotate. Since we're already expecting by default the NCBI-nr to be used as the alignment reference, the ID mapping data file provided by Uniprot should work well when provided to --id_mapping . Host reference If you're using metagenomic reads that come from a known host's microbiome, you can also provide the host's genome FASTA to --host_fasta parameter in order to enable our decontamination subworkflow. Ensembl provides easy to download genomes that can be used for this purpose. Alternatively, you can provide a pre-built BowTie2 database directory to the --bowtie2_db parameter. Documentation The dalmolingroup/euryale documentation is split into the following pages: Usage An overview of how the pipeline works, how to run it and a description of all of the different command-line flags. Output An overview of the different results produced by the pipeline and how to interpret them. Credits dalmolingroup/euryale was originally written by Jo\u00e3o Cavalcante. We thank the following people for their extensive assistance in the development of this pipeline: Diego Morais (for developing the original MEDUSA pipeline) Citations Morais DAA, Cavalcante JVF, Monteiro SS, Pasquali MAB and Dalmolin RJS (2022) MEDUSA: A Pipeline for Sensitive Taxonomic Classification and Flexible Functional Annotation of Metagenomic Shotgun Sequences. Front. Genet. 13:814437. doi: 10.3389/fgene.2022.814437 This pipeline uses code and infrastructure developed and maintained by the nf-core community, reused here under the MIT license . The nf-core framework for community-curated bioinformatics pipelines. Philip Ewels, Alexander Peltzer, Sven Fillinger, Harshil Patel, Johannes Alneberg, Andreas Wilm, Maxime Ulysse Garcia, Paolo Di Tommaso & Sven Nahnsen. Nat Biotechnol. 2020 Feb 13. doi: 10.1038/s41587-020-0439-x .","title":"Home"},{"location":"#introduction","text":"dalmolingroup/euryale is a pipeline for taxonomic classification and functional annotation of metagenomic reads. Based on MEDUSA . The pipeline is built using Nextflow , a workflow tool to run tasks across multiple compute infrastructures in a very portable manner. It uses Docker/Singularity containers making installation trivial and results highly reproducible. The Nextflow DSL2 implementation of this pipeline uses one container per process which makes it much easier to maintain and update software dependencies. Where possible, these processes have been submitted to and installed from nf-core/modules in order to make them available to all nf-core pipelines, and to everyone within the Nextflow community!","title":"Introduction"},{"location":"#pipeline-summary","text":"","title":"Pipeline summary"},{"location":"#pre-processing","text":"Read QC ( FastQC ) Read trimming and merging ( fastp ) ( optionally ) Host read removal ( BowTie2 ) Duplicated sequence removal ( fastx collapser ) Present QC and other data ( MultiQC )","title":"Pre-processing"},{"location":"#assembly","text":"( optionally ) Read assembly ( MEGAHIT )","title":"Assembly"},{"location":"#taxonomic-classification","text":"Sequence classification ( Kaiju ) Sequence classification ( Kraken2 ) Visualization ( Krona )","title":"Taxonomic classification"},{"location":"#functional-annotation","text":"Sequence alignment ( DIAMOND ) Map alignment matches to functional database ( annotate )","title":"Functional annotation"},{"location":"#quick-start","text":"Install Nextflow ( >=22.10.1 ) Install any of Docker , Singularity (you can follow this tutorial ), Podman , Shifter or Charliecloud for full pipeline reproducibility (you can use Conda both to install Nextflow itself and also to manage software within pipelines. Please only use it within pipelines as a last resort; see docs ) . Download the pipeline and test it on a minimal dataset with a single command: nextflow run dalmolingroup/euryale -profile test,YOURPROFILE --outdir Note that some form of configuration will be needed so that Nextflow knows how to fetch the required software. This is usually done in the form of a config profile ( YOURPROFILE in the example command above). You can chain multiple config profiles in a comma-separated string. The pipeline comes with config profiles called docker , singularity , podman , shifter , charliecloud and conda which instruct the pipeline to use the named tool for software management. For example, -profile test,docker . Please check nf-core/configs to see if a custom config file to run nf-core pipelines already exists for your Institute. If so, you can simply use -profile in your command. This will enable either docker or singularity and set the appropriate execution settings for your local compute environment. If you are using singularity , please use the nf-core download command to download images first, before running the pipeline. Setting the NXF_SINGULARITY_CACHEDIR or singularity.cacheDir Nextflow options enables you to store and re-use the images from a central location for future pipeline runs. If you are using conda , it is highly recommended to use the NXF_CONDA_CACHEDIR or conda.cacheDir settings to store the environments in a central location for future pipeline runs. Start running your own analysis! nextflow run dalmolingroup/euryale \\ --input samplesheet.csv \\ --outdir \\ --kaiju_db kaiju_reference \\ --diamond_db diamond_db \\ --reference_fasta diamond_fasta \\ --host_fasta host_reference_fasta \\ --id_mapping id_mapping_file \\ -profile ","title":"Quick Start"},{"location":"#databases-and-references","text":"A question that pops up a lot is: Since Euryale requires a lot of reference parameters, where can I find these references? Below we provide a short list of places where you can find these databases. But, of course, we're not limited to these references: Euryale should be able to process your own databases, should you want to build them yourself.","title":"Databases and references"},{"location":"#alignment","text":"For the alignment you can either provide --diamond_db for a pre-built DIAMOND database, or you can provide --reference_fasta . For reference fasta, by default Euryale expects something like NCBI-nr , but similarly formatted reference databases should also suffice.","title":"Alignment"},{"location":"#taxonomic-classification_1","text":"At its current version, Euryale doesn't build a reference taxonomic database, but pre-built ones are supported. If you're using Kaiju (the default), you can provide a reference database with --kaiju_db and provide a .tar.gz file like the ones provided in the official Kaiju website . We have extensively tested Euryale with the 2021 version of the nr database and it should work as expected. If you're using Kraken2 (By supplying --run_kraken2 ), we expect something like the pre-built .tar.gz databases provided by the Kraken2 developers to be provided to --kraken2_db .","title":"Taxonomic classification"},{"location":"#functional-annotation_1","text":"We expect an ID mapping reference to be used within annotate. Since we're already expecting by default the NCBI-nr to be used as the alignment reference, the ID mapping data file provided by Uniprot should work well when provided to --id_mapping .","title":"Functional annotation"},{"location":"#host-reference","text":"If you're using metagenomic reads that come from a known host's microbiome, you can also provide the host's genome FASTA to --host_fasta parameter in order to enable our decontamination subworkflow. Ensembl provides easy to download genomes that can be used for this purpose. Alternatively, you can provide a pre-built BowTie2 database directory to the --bowtie2_db parameter.","title":"Host reference"},{"location":"#documentation","text":"The dalmolingroup/euryale documentation is split into the following pages: Usage An overview of how the pipeline works, how to run it and a description of all of the different command-line flags. Output An overview of the different results produced by the pipeline and how to interpret them.","title":"Documentation"},{"location":"#credits","text":"dalmolingroup/euryale was originally written by Jo\u00e3o Cavalcante. We thank the following people for their extensive assistance in the development of this pipeline: Diego Morais (for developing the original MEDUSA pipeline)","title":"Credits"},{"location":"#citations","text":"Morais DAA, Cavalcante JVF, Monteiro SS, Pasquali MAB and Dalmolin RJS (2022) MEDUSA: A Pipeline for Sensitive Taxonomic Classification and Flexible Functional Annotation of Metagenomic Shotgun Sequences. Front. Genet. 13:814437. doi: 10.3389/fgene.2022.814437 This pipeline uses code and infrastructure developed and maintained by the nf-core community, reused here under the MIT license . The nf-core framework for community-curated bioinformatics pipelines. Philip Ewels, Alexander Peltzer, Sven Fillinger, Harshil Patel, Johannes Alneberg, Andreas Wilm, Maxime Ulysse Garcia, Paolo Di Tommaso & Sven Nahnsen. Nat Biotechnol. 2020 Feb 13. doi: 10.1038/s41587-020-0439-x .","title":"Citations"},{"location":"CITATIONS/","text":"dalmolingroup/euryale: Citations nf-core Ewels PA, Peltzer A, Fillinger S, Patel H, Alneberg J, Wilm A, Garcia MU, Di Tommaso P, Nahnsen S. The nf-core framework for community-curated bioinformatics pipelines. Nat Biotechnol. 2020 Mar;38(3):276-278. doi: 10.1038/s41587-020-0439-x. PubMed PMID: 32055031. Nextflow Di Tommaso P, Chatzou M, Floden EW, Barja PP, Palumbo E, Notredame C. Nextflow enables reproducible computational workflows. Nat Biotechnol. 2017 Apr 11;35(4):316-319. doi: 10.1038/nbt.3820. PubMed PMID: 28398311. Pipeline tools FastQC MultiQC Ewels P, Magnusson M, Lundin S, K\u00e4ller M. MultiQC: summarize analysis results for multiple tools and samples in a single report. Bioinformatics. 2016 Oct 1;32(19):3047-8. doi: 10.1093/bioinformatics/btw354. Epub 2016 Jun 16. PubMed PMID: 27312411; PubMed Central PMCID: PMC5039924. Software packaging/containerisation tools Anaconda Anaconda Software Distribution. Computer software. Vers. 2-2.4.0. Anaconda, Nov. 2016. Web. Bioconda Gr\u00fcning B, Dale R, Sj\u00f6din A, Chapman BA, Rowe J, Tomkins-Tinch CH, Valieris R, K\u00f6ster J; Bioconda Team. Bioconda: sustainable and comprehensive software distribution for the life sciences. Nat Methods. 2018 Jul;15(7):475-476. doi: 10.1038/s41592-018-0046-7. PubMed PMID: 29967506. BioContainers da Veiga Leprevost F, Gr\u00fcning B, Aflitos SA, R\u00f6st HL, Uszkoreit J, Barsnes H, Vaudel M, Moreno P, Gatto L, Weber J, Bai M, Jimenez RC, Sachsenberg T, Pfeuffer J, Alvarez RV, Griss J, Nesvizhskii AI, Perez-Riverol Y. BioContainers: an open-source and community-driven framework for software standardization. Bioinformatics. 2017 Aug 15;33(16):2580-2582. doi: 10.1093/bioinformatics/btx192. PubMed PMID: 28379341; PubMed Central PMCID: PMC5870671. Docker Singularity Kurtzer GM, Sochat V, Bauer MW. Singularity: Scientific containers for mobility of compute. PLoS One. 2017 May 11;12(5):e0177459. doi: 10.1371/journal.pone.0177459. eCollection 2017. PubMed PMID: 28494014; PubMed Central PMCID: PMC5426675.","title":"Citations"},{"location":"CITATIONS/#dalmolingroupeuryale-citations","text":"","title":"dalmolingroup/euryale: Citations"},{"location":"CITATIONS/#nf-core","text":"Ewels PA, Peltzer A, Fillinger S, Patel H, Alneberg J, Wilm A, Garcia MU, Di Tommaso P, Nahnsen S. The nf-core framework for community-curated bioinformatics pipelines. Nat Biotechnol. 2020 Mar;38(3):276-278. doi: 10.1038/s41587-020-0439-x. PubMed PMID: 32055031.","title":"nf-core"},{"location":"CITATIONS/#nextflow","text":"Di Tommaso P, Chatzou M, Floden EW, Barja PP, Palumbo E, Notredame C. Nextflow enables reproducible computational workflows. Nat Biotechnol. 2017 Apr 11;35(4):316-319. doi: 10.1038/nbt.3820. PubMed PMID: 28398311.","title":"Nextflow"},{"location":"CITATIONS/#pipeline-tools","text":"FastQC MultiQC Ewels P, Magnusson M, Lundin S, K\u00e4ller M. MultiQC: summarize analysis results for multiple tools and samples in a single report. Bioinformatics. 2016 Oct 1;32(19):3047-8. doi: 10.1093/bioinformatics/btw354. Epub 2016 Jun 16. PubMed PMID: 27312411; PubMed Central PMCID: PMC5039924.","title":"Pipeline tools"},{"location":"CITATIONS/#software-packagingcontainerisation-tools","text":"Anaconda Anaconda Software Distribution. Computer software. Vers. 2-2.4.0. Anaconda, Nov. 2016. Web. Bioconda Gr\u00fcning B, Dale R, Sj\u00f6din A, Chapman BA, Rowe J, Tomkins-Tinch CH, Valieris R, K\u00f6ster J; Bioconda Team. Bioconda: sustainable and comprehensive software distribution for the life sciences. Nat Methods. 2018 Jul;15(7):475-476. doi: 10.1038/s41592-018-0046-7. PubMed PMID: 29967506. BioContainers da Veiga Leprevost F, Gr\u00fcning B, Aflitos SA, R\u00f6st HL, Uszkoreit J, Barsnes H, Vaudel M, Moreno P, Gatto L, Weber J, Bai M, Jimenez RC, Sachsenberg T, Pfeuffer J, Alvarez RV, Griss J, Nesvizhskii AI, Perez-Riverol Y. BioContainers: an open-source and community-driven framework for software standardization. Bioinformatics. 2017 Aug 15;33(16):2580-2582. doi: 10.1093/bioinformatics/btx192. PubMed PMID: 28379341; PubMed Central PMCID: PMC5870671. Docker Singularity Kurtzer GM, Sochat V, Bauer MW. Singularity: Scientific containers for mobility of compute. PLoS One. 2017 May 11;12(5):e0177459. doi: 10.1371/journal.pone.0177459. eCollection 2017. PubMed PMID: 28494014; PubMed Central PMCID: PMC5426675.","title":"Software packaging/containerisation tools"},{"location":"output/","text":"dalmolingroup/euryale: Output Introduction This document describes the output produced by the pipeline. Most of the plots are taken from the MultiQC report, which summarises results at the end of the pipeline. The directories listed below will be created in the results directory after the pipeline has finished. All paths are relative to the top-level results directory. Pipeline overview The pipeline is built using Nextflow and processes data using the following steps (steps in italics don't run by default): Kaiju - Taxonomically classify reads or contigs Krona - Visualize the taxonomic classification for each sample. Diamond - Alignment reads and contigs against a reference database (such as NCBI-nr). Annotate - Functional annotation of alignment matches. MEGAHIT - Assembled contigs. MultiQC - Aggregate report describing results and QC from the whole pipeline Pipeline information - Report metrics generated during the workflow execution Kaiju Output files - `taxonomy/` - `${sample}.tsv`: Kaiju classification output. - `${sample}.txt`: Kaiju2Table TSV output. Kaiju is a software to perform fast taxonomic classification of metagenomic sequencing reads using a protein reference database. Krona Output files - `taxonomy/` - `${sample}.html`: Krona visualization for the sample. Krona is a tool to interactively explore metagenomes and more from a web browser. Diamond Output files - `alignment/${sample}/` - `${sample}.txt`: Alignment matches in blast tabular output format. - `${sample}.log`: DIAMOND execution log. DIAMOND is an accelerated BLAST compatible local sequence aligner. Annotate Output files - `functional/${sample}/` - `${sample}_annotated.txt`: Alignment matches annotated to chosen functional database (e.g. GO). Annotate is a tool to annotate each query using the best alignment for which a mapping is known. MEGAHIT Output files - `assembly/${sample}/` - `${sample}.contigs.fa.gz`: Contigs assembled for the sample. MEGAHIT is an ultra-fast and memory-efficient (meta-)genome assembler DIAMOND database Output files - `diamond_db/` - `${database_name}.dmnd`: DIAMOND database for the reference fasta file. This output is present if you add the --save_db parameter. DIAMOND is an accelerated BLAST compatible local sequence aligner. MultiQC Output files - `multiqc/` - `multiqc_report.html`: a standalone HTML file that can be viewed in your web browser. - `multiqc_data/`: directory containing parsed statistics from the different tools used in the pipeline. - `multiqc_plots/`: directory containing static images from the report in various formats. MultiQC is a visualization tool that generates a single HTML report summarising all samples in your project. Most of the pipeline QC results are visualised in the report and further statistics are available in the report data directory. Results generated by MultiQC collate pipeline QC from supported tools e.g. FastQC. The pipeline has special steps which also allow the software versions to be reported in the MultiQC output for future traceability. For more information about how to use MultiQC reports, see http://multiqc.info . Pipeline information Output files - `pipeline_info/` - Reports generated by Nextflow: `execution_report.html`, `execution_timeline.html`, `execution_trace.txt` and `pipeline_dag.dot`/`pipeline_dag.svg`. - Reports generated by the pipeline: `pipeline_report.html`, `pipeline_report.txt` and `software_versions.yml`. The `pipeline_report*` files will only be present if the `--email` / `--email_on_fail` parameter's are used when running the pipeline. - Reformatted samplesheet files used as input to the pipeline: `samplesheet.valid.csv`. Nextflow provides excellent functionality for generating various reports relevant to the running and execution of the pipeline. This will allow you to troubleshoot errors with the running of the pipeline, and also provide you with other information such as launch commands, run times and resource usage.","title":"Output"},{"location":"output/#dalmolingroupeuryale-output","text":"","title":"dalmolingroup/euryale: Output"},{"location":"output/#introduction","text":"This document describes the output produced by the pipeline. Most of the plots are taken from the MultiQC report, which summarises results at the end of the pipeline. The directories listed below will be created in the results directory after the pipeline has finished. All paths are relative to the top-level results directory.","title":"Introduction"},{"location":"output/#pipeline-overview","text":"The pipeline is built using Nextflow and processes data using the following steps (steps in italics don't run by default): Kaiju - Taxonomically classify reads or contigs Krona - Visualize the taxonomic classification for each sample. Diamond - Alignment reads and contigs against a reference database (such as NCBI-nr). Annotate - Functional annotation of alignment matches. MEGAHIT - Assembled contigs. MultiQC - Aggregate report describing results and QC from the whole pipeline Pipeline information - Report metrics generated during the workflow execution","title":"Pipeline overview"},{"location":"output/#kaiju","text":"Output files - `taxonomy/` - `${sample}.tsv`: Kaiju classification output. - `${sample}.txt`: Kaiju2Table TSV output. Kaiju is a software to perform fast taxonomic classification of metagenomic sequencing reads using a protein reference database.","title":"Kaiju"},{"location":"output/#krona","text":"Output files - `taxonomy/` - `${sample}.html`: Krona visualization for the sample. Krona is a tool to interactively explore metagenomes and more from a web browser.","title":"Krona"},{"location":"output/#diamond","text":"Output files - `alignment/${sample}/` - `${sample}.txt`: Alignment matches in blast tabular output format. - `${sample}.log`: DIAMOND execution log. DIAMOND is an accelerated BLAST compatible local sequence aligner.","title":"Diamond"},{"location":"output/#annotate","text":"Output files - `functional/${sample}/` - `${sample}_annotated.txt`: Alignment matches annotated to chosen functional database (e.g. GO). Annotate is a tool to annotate each query using the best alignment for which a mapping is known.","title":"Annotate"},{"location":"output/#megahit","text":"Output files - `assembly/${sample}/` - `${sample}.contigs.fa.gz`: Contigs assembled for the sample. MEGAHIT is an ultra-fast and memory-efficient (meta-)genome assembler","title":"MEGAHIT"},{"location":"output/#diamond-database","text":"Output files - `diamond_db/` - `${database_name}.dmnd`: DIAMOND database for the reference fasta file. This output is present if you add the --save_db parameter. DIAMOND is an accelerated BLAST compatible local sequence aligner.","title":"DIAMOND database"},{"location":"output/#multiqc","text":"Output files - `multiqc/` - `multiqc_report.html`: a standalone HTML file that can be viewed in your web browser. - `multiqc_data/`: directory containing parsed statistics from the different tools used in the pipeline. - `multiqc_plots/`: directory containing static images from the report in various formats. MultiQC is a visualization tool that generates a single HTML report summarising all samples in your project. Most of the pipeline QC results are visualised in the report and further statistics are available in the report data directory. Results generated by MultiQC collate pipeline QC from supported tools e.g. FastQC. The pipeline has special steps which also allow the software versions to be reported in the MultiQC output for future traceability. For more information about how to use MultiQC reports, see http://multiqc.info .","title":"MultiQC"},{"location":"output/#pipeline-information","text":"Output files - `pipeline_info/` - Reports generated by Nextflow: `execution_report.html`, `execution_timeline.html`, `execution_trace.txt` and `pipeline_dag.dot`/`pipeline_dag.svg`. - Reports generated by the pipeline: `pipeline_report.html`, `pipeline_report.txt` and `software_versions.yml`. The `pipeline_report*` files will only be present if the `--email` / `--email_on_fail` parameter's are used when running the pipeline. - Reformatted samplesheet files used as input to the pipeline: `samplesheet.valid.csv`. Nextflow provides excellent functionality for generating various reports relevant to the running and execution of the pipeline. This will allow you to troubleshoot errors with the running of the pipeline, and also provide you with other information such as launch commands, run times and resource usage.","title":"Pipeline information"},{"location":"params/","text":"dalmolingroup/euryale pipeline parameters A pipeline for metagenomic taxonomic classification and functional annotation. Based on MEDUSA. Input/output options Define where the pipeline should find input data and save output data. Parameter Description Type Default Required Hidden input Path to comma-separated file containing information about the samples in the experiment. Help You will need to create a design file with information about the samples in your experiment before running the pipeline. Use this parameter to specify its location. It has to be a comma-separated file with 3 columns, and a header row. string True outdir The output directory where the results will be saved. You have to use absolute paths to storage on Cloud infrastructure. string True email Email address for completion summary. Help Set this parameter to your e-mail address to get a summary e-mail with details of the run sent to you when the workflow exits. If set in your user config file ( ~/.nextflow/config ) then you don't need to specify this on the command line for every run. string multiqc_title MultiQC report title. Printed as page header, used for filename if not otherwise specified. string save_dbs Save DIAMOND db to results directory after construction boolean Skip Steps Choose to skip pipeline steps Parameter Description Type Default Required Hidden skip_classification Skip taxonomic classification boolean skip_alignment Skip alignment boolean skip_functional Skip functional annotation boolean Decontamination Parameter Description Type Default Required Hidden host_fasta Host FASTA to use for decontamination string bowtie2_db Pre-built bowtie2 index. Directory where index is located. string Alignment Parameter Description Type Default Required Hidden reference_fasta Path to FASTA genome file. string diamond_db Path to pre-built DIAMOND db. string Taxonomy Parameter Description Type Default Required Hidden kaiju_db Kaiju database string True kraken2_db Kraken2 database string run_kaiju Run Kaiju classifier boolean True run_kraken2 Run Kraken2 classifier boolean Functional Parameter Description Type Default Required Hidden id_mapping Path to ID mapping file to be used for the Functional annotation string True minimum_bitscore Minimum bitscore of a match to be used for annotation integer 50 minimum_pident Minimum identity of a match to be used for annotation integer 80 minimum_alen Minimum alignment length of a match to be used for annotation integer 50 maximum_evalue Maximum evalue of a match to be used for annotation number 0.0001 Assembly Parameter Description Type Default Required Hidden assembly_based boolean Reference genome options Reference genome related files and options required for the workflow. Parameter Description Type Default Required Hidden genome Name of iGenomes reference. Help If using a reference genome configured in the pipeline using iGenomes, use this parameter to give the ID for the reference. This is then used to build the full paths for all required reference genome files e.g. --genome GRCh38 . See the nf-core website docs for more details. string igenomes_base Directory / URL base for iGenomes references. string s3://ngi-igenomes/igenomes True igenomes_ignore Do not load the iGenomes reference config. Help Do not load igenomes.config when running the pipeline. You may choose this option if you observe clashes between custom parameters and those supplied in igenomes.config . boolean True fasta string Max job request options Set the top limit for requested resources for any single job. Parameter Description Type Default Required Hidden max_cpus Maximum number of CPUs that can be requested for any single job. Help Use to set an upper-limit for the CPU requirement for each process. Should be an integer e.g. --max_cpus 1 integer 16 True max_memory Maximum amount of memory that can be requested for any single job. Help Use to set an upper-limit for the memory requirement for each process. Should be a string in the format integer-unit e.g. --max_memory '8.GB' string 128.GB True max_time Maximum amount of time that can be requested for any single job. Help Use to set an upper-limit for the time requirement for each process. Should be a string in the format integer-unit e.g. --max_time '2.h' string 240.h True Generic options Less common options for the pipeline, typically set in a config file. Parameter Description Type Default Required Hidden help Display help text. boolean True version Display version and exit. boolean True publish_dir_mode Method used to save pipeline results to output directory. Help The Nextflow publishDir option specifies which intermediate files should be saved to the output directory. This option tells the pipeline what method should be used to move these files. See Nextflow docs for details. string copy True email_on_fail Email address for completion summary, only when pipeline fails. Help An email address to send a summary email to when the pipeline is completed - ONLY sent if the pipeline does not exit successfully. string True plaintext_email Send plain-text email instead of HTML. boolean True max_multiqc_email_size File size limit when attaching MultiQC reports to summary emails. string 25.MB True monochrome_logs Do not use coloured log outputs. boolean True hook_url Incoming hook URL for messaging service Help Incoming hook URL for messaging service. Currently, MS Teams and Slack are supported. string True multiqc_config Custom config file to supply to MultiQC. string True multiqc_logo Custom logo file to supply to MultiQC. File name must also be set in the MultiQC config file string True multiqc_methods_description Custom MultiQC yaml file containing HTML including a methods description. string tracedir Directory to keep pipeline Nextflow logs and reports. string ${params.outdir}/pipeline_info True validate_params Boolean whether to validate parameters against the schema at runtime boolean True True show_hidden_params Show all params when using --help Help By default, parameters set as hidden in the schema are not shown on the command line when a user runs with --help . Specifying this option will tell the pipeline to show all parameters. boolean True schema_ignore_params string genomes True","title":"Parameters"},{"location":"params/#dalmolingroupeuryale-pipeline-parameters","text":"A pipeline for metagenomic taxonomic classification and functional annotation. Based on MEDUSA.","title":"dalmolingroup/euryale pipeline parameters"},{"location":"params/#inputoutput-options","text":"Define where the pipeline should find input data and save output data. Parameter Description Type Default Required Hidden input Path to comma-separated file containing information about the samples in the experiment. Help You will need to create a design file with information about the samples in your experiment before running the pipeline. Use this parameter to specify its location. It has to be a comma-separated file with 3 columns, and a header row. string True outdir The output directory where the results will be saved. You have to use absolute paths to storage on Cloud infrastructure. string True email Email address for completion summary. Help Set this parameter to your e-mail address to get a summary e-mail with details of the run sent to you when the workflow exits. If set in your user config file ( ~/.nextflow/config ) then you don't need to specify this on the command line for every run. string multiqc_title MultiQC report title. Printed as page header, used for filename if not otherwise specified. string save_dbs Save DIAMOND db to results directory after construction boolean","title":"Input/output options"},{"location":"params/#skip-steps","text":"Choose to skip pipeline steps Parameter Description Type Default Required Hidden skip_classification Skip taxonomic classification boolean skip_alignment Skip alignment boolean skip_functional Skip functional annotation boolean","title":"Skip Steps"},{"location":"params/#decontamination","text":"Parameter Description Type Default Required Hidden host_fasta Host FASTA to use for decontamination string bowtie2_db Pre-built bowtie2 index. Directory where index is located. string","title":"Decontamination"},{"location":"params/#alignment","text":"Parameter Description Type Default Required Hidden reference_fasta Path to FASTA genome file. string diamond_db Path to pre-built DIAMOND db. string","title":"Alignment"},{"location":"params/#taxonomy","text":"Parameter Description Type Default Required Hidden kaiju_db Kaiju database string True kraken2_db Kraken2 database string run_kaiju Run Kaiju classifier boolean True run_kraken2 Run Kraken2 classifier boolean","title":"Taxonomy"},{"location":"params/#functional","text":"Parameter Description Type Default Required Hidden id_mapping Path to ID mapping file to be used for the Functional annotation string True minimum_bitscore Minimum bitscore of a match to be used for annotation integer 50 minimum_pident Minimum identity of a match to be used for annotation integer 80 minimum_alen Minimum alignment length of a match to be used for annotation integer 50 maximum_evalue Maximum evalue of a match to be used for annotation number 0.0001","title":"Functional"},{"location":"params/#assembly","text":"Parameter Description Type Default Required Hidden assembly_based boolean","title":"Assembly"},{"location":"params/#reference-genome-options","text":"Reference genome related files and options required for the workflow. Parameter Description Type Default Required Hidden genome Name of iGenomes reference. Help If using a reference genome configured in the pipeline using iGenomes, use this parameter to give the ID for the reference. This is then used to build the full paths for all required reference genome files e.g. --genome GRCh38 . See the nf-core website docs for more details. string igenomes_base Directory / URL base for iGenomes references. string s3://ngi-igenomes/igenomes True igenomes_ignore Do not load the iGenomes reference config. Help Do not load igenomes.config when running the pipeline. You may choose this option if you observe clashes between custom parameters and those supplied in igenomes.config . boolean True fasta string","title":"Reference genome options"},{"location":"params/#max-job-request-options","text":"Set the top limit for requested resources for any single job. Parameter Description Type Default Required Hidden max_cpus Maximum number of CPUs that can be requested for any single job. Help Use to set an upper-limit for the CPU requirement for each process. Should be an integer e.g. --max_cpus 1 integer 16 True max_memory Maximum amount of memory that can be requested for any single job. Help Use to set an upper-limit for the memory requirement for each process. Should be a string in the format integer-unit e.g. --max_memory '8.GB' string 128.GB True max_time Maximum amount of time that can be requested for any single job. Help Use to set an upper-limit for the time requirement for each process. Should be a string in the format integer-unit e.g. --max_time '2.h' string 240.h True","title":"Max job request options"},{"location":"params/#generic-options","text":"Less common options for the pipeline, typically set in a config file. Parameter Description Type Default Required Hidden help Display help text. boolean True version Display version and exit. boolean True publish_dir_mode Method used to save pipeline results to output directory. Help The Nextflow publishDir option specifies which intermediate files should be saved to the output directory. This option tells the pipeline what method should be used to move these files. See Nextflow docs for details. string copy True email_on_fail Email address for completion summary, only when pipeline fails. Help An email address to send a summary email to when the pipeline is completed - ONLY sent if the pipeline does not exit successfully. string True plaintext_email Send plain-text email instead of HTML. boolean True max_multiqc_email_size File size limit when attaching MultiQC reports to summary emails. string 25.MB True monochrome_logs Do not use coloured log outputs. boolean True hook_url Incoming hook URL for messaging service Help Incoming hook URL for messaging service. Currently, MS Teams and Slack are supported. string True multiqc_config Custom config file to supply to MultiQC. string True multiqc_logo Custom logo file to supply to MultiQC. File name must also be set in the MultiQC config file string True multiqc_methods_description Custom MultiQC yaml file containing HTML including a methods description. string tracedir Directory to keep pipeline Nextflow logs and reports. string ${params.outdir}/pipeline_info True validate_params Boolean whether to validate parameters against the schema at runtime boolean True True show_hidden_params Show all params when using --help Help By default, parameters set as hidden in the schema are not shown on the command line when a user runs with --help . Specifying this option will tell the pipeline to show all parameters. boolean True schema_ignore_params string genomes True","title":"Generic options"},{"location":"usage/","text":"dalmolingroup/euryale: Usage Documentation of pipeline parameters is generated automatically from the pipeline schema and can no longer be found in markdown files. Introduction Samplesheet input You will need to create a samplesheet with information about the samples you would like to analyse before running the pipeline. Use this parameter to specify its location. It has to be a comma-separated file with 3 columns, and a header row as shown in the examples below. --input '[path to samplesheet file]' Multiple runs of the same sample The sample identifiers have to be the same when you have re-sequenced the same sample more than once e.g. to increase sequencing depth. The pipeline will concatenate the raw reads before performing any downstream analysis. Below is an example for the same sample sequenced across 3 lanes: sample,fastq_1,fastq_2 CONTROL_REP1,AEG588A1_S1_L002_R1_001.fastq.gz,AEG588A1_S1_L002_R2_001.fastq.gz CONTROL_REP1,AEG588A1_S1_L003_R1_001.fastq.gz,AEG588A1_S1_L003_R2_001.fastq.gz CONTROL_REP1,AEG588A1_S1_L004_R1_001.fastq.gz,AEG588A1_S1_L004_R2_001.fastq.gz Full samplesheet The pipeline will auto-detect whether a sample is single- or paired-end using the information provided in the samplesheet. The samplesheet can have as many columns as you desire, however, there is a strict requirement for the first 3 columns to match those defined in the table below. A final samplesheet file consisting of both single- and paired-end data may look something like the one below. This is for 6 samples, where TREATMENT_REP3 has been sequenced twice. sample,fastq_1,fastq_2 CONTROL_REP1,AEG588A1_S1_L002_R1_001.fastq.gz,AEG588A1_S1_L002_R2_001.fastq.gz CONTROL_REP2,AEG588A2_S2_L002_R1_001.fastq.gz,AEG588A2_S2_L002_R2_001.fastq.gz CONTROL_REP3,AEG588A3_S3_L002_R1_001.fastq.gz,AEG588A3_S3_L002_R2_001.fastq.gz TREATMENT_REP1,AEG588A4_S4_L003_R1_001.fastq.gz, TREATMENT_REP2,AEG588A5_S5_L003_R1_001.fastq.gz, TREATMENT_REP3,AEG588A6_S6_L003_R1_001.fastq.gz, TREATMENT_REP3,AEG588A6_S6_L004_R1_001.fastq.gz, Column Description sample Custom sample name. This entry will be identical for multiple sequencing libraries/runs from the same sample. Spaces in sample names are automatically converted to underscores ( _ ). fastq_1 Full path to FastQ file for Illumina short reads 1. File has to be gzipped and have the extension \".fastq.gz\" or \".fq.gz\". fastq_2 Full path to FastQ file for Illumina short reads 2. File has to be gzipped and have the extension \".fastq.gz\" or \".fq.gz\". An example samplesheet has been provided with the pipeline. Running the pipeline The typical command for running the pipeline is as follows: nextflow run dalmolingroup/euryale --input samplesheet.csv --outdir --genome GRCh37 -profile docker This will launch the pipeline with the docker configuration profile. See below for more information about profiles. Note that the pipeline will create the following files in your working directory: work # Directory containing the nextflow working files # Finished results in specified location (defined with --outdir) .nextflow_log # Log file from Nextflow # Other nextflow hidden files, eg. history of pipeline runs and old logs. Updating the pipeline When you run the above command, Nextflow automatically pulls the pipeline code from GitHub and stores it as a cached version. When running the pipeline after this, it will always use the cached version if available - even if the pipeline has been updated since. To make sure that you're running the latest version of the pipeline, make sure that you regularly update the cached version of the pipeline: nextflow pull dalmolingroup/euryale Reproducibility It is a good idea to specify a pipeline version when running the pipeline on your data. This ensures that a specific version of the pipeline code and software are used when you run your pipeline. If you keep using the same tag, you'll be running the same version of the pipeline, even if there have been changes to the code since. First, go to the dalmolingroup/euryale releases page and find the latest pipeline version - numeric only (eg. 1.3.1 ). Then specify this when running the pipeline with -r (one hyphen) - eg. -r 1.3.1 . Of course, you can switch to another version by changing the number after the -r flag. This version number will be logged in reports when you run the pipeline, so that you'll know what you used when you look back in the future. For example, at the bottom of the MultiQC reports. Core Nextflow arguments NB: These options are part of Nextflow and use a single hyphen (pipeline parameters use a double-hyphen). -profile Use this parameter to choose a configuration profile. Profiles can give configuration presets for different compute environments. Several generic profiles are bundled with the pipeline which instruct the pipeline to use software packaged using different methods (Docker, Singularity, Podman, Shifter, Charliecloud, Conda) - see below. We highly recommend the use of Docker or Singularity containers for full pipeline reproducibility, however when this is not possible, Conda is also supported. Note that multiple profiles can be loaded, for example: -profile test,docker - the order of arguments is important! They are loaded in sequence, so later profiles can overwrite earlier profiles. If -profile is not specified, the pipeline will run locally and expect all software to be installed and available on the PATH . This is not recommended, since it can lead to different results on different machines dependent on the computer enviroment. test A profile with a complete configuration for automated testing Includes links to test data so needs no other parameters docker A generic configuration profile to be used with Docker singularity A generic configuration profile to be used with Singularity podman A generic configuration profile to be used with Podman shifter A generic configuration profile to be used with Shifter charliecloud A generic configuration profile to be used with Charliecloud conda A generic configuration profile to be used with Conda . Please only use Conda as a last resort i.e. when it's not possible to run the pipeline with Docker, Singularity, Podman, Shifter or Charliecloud. -resume Specify this when restarting a pipeline. Nextflow will use cached results from any pipeline steps where the inputs are the same, continuing from where it got to previously. For input to be considered the same, not only the names must be identical but the files' contents as well. For more info about this parameter, see this blog post . You can also supply a run name to resume a specific run: -resume [run-name] . Use the nextflow log command to show previous run names. -c Specify the path to a specific config file (this is a core Nextflow command). See the nf-core website documentation for more information. Custom configuration Resource requests Whilst the default requirements set within the pipeline will hopefully work for most people and with most input data, you may find that you want to customise the compute resources that the pipeline requests. Each step in the pipeline has a default set of requirements for number of CPUs, memory and time. For most of the steps in the pipeline, if the job exits with any of the error codes specified here it will automatically be resubmitted with higher requests (2 x original, then 3 x original). If it still fails after the third attempt then the pipeline execution is stopped. For example, if the nf-core/rnaseq pipeline is failing after multiple re-submissions of the STAR_ALIGN process due to an exit code of 137 this would indicate that there is an out of memory issue: [62/149eb0] NOTE: Process `NFCORE_RNASEQ:RNASEQ:ALIGN_STAR:STAR_ALIGN (WT_REP1)` terminated with an error exit status (137) -- Execution is retried (1) Error executing process > 'NFCORE_RNASEQ:RNASEQ:ALIGN_STAR:STAR_ALIGN (WT_REP1)' Caused by: Process `NFCORE_RNASEQ:RNASEQ:ALIGN_STAR:STAR_ALIGN (WT_REP1)` terminated with an error exit status (137) Command executed: STAR \\ --genomeDir star \\ --readFilesIn WT_REP1_trimmed.fq.gz \\ --runThreadN 2 \\ --outFileNamePrefix WT_REP1. \\ Command exit status: 137 Command output: (empty) Command error: .command.sh: line 9: 30 Killed STAR --genomeDir star --readFilesIn WT_REP1_trimmed.fq.gz --runThreadN 2 --outFileNamePrefix WT_REP1. Work dir: /home/pipelinetest/work/9d/172ca5881234073e8d76f2a19c88fb Tip: you can replicate the issue by changing to the process work dir and entering the command `bash .command.run` For beginners A first step to bypass this error, you could try to increase the amount of CPUs, memory, and time for the whole pipeline. Therefor you can try to increase the resource for the parameters --max_cpus , --max_memory , and --max_time . Based on the error above, you have to increase the amount of memory. Therefore you can go to the parameter documentation of rnaseq and scroll down to the show hidden parameter button to get the default value for --max_memory . In this case 128GB, you than can try to run your pipeline again with --max_memory 200GB -resume to skip all process, that were already calculated. If you can not increase the resource of the complete pipeline, you can try to adapt the resource for a single process as mentioned below. Advanced option on process level To bypass this error you would need to find exactly which resources are set by the STAR_ALIGN process. The quickest way is to search for process STAR_ALIGN in the nf-core/rnaseq Github repo . We have standardised the structure of Nextflow DSL2 pipelines such that all module files will be present in the modules/ directory and so, based on the search results, the file we want is modules/nf-core/star/align/main.nf . If you click on the link to that file you will notice that there is a label directive at the top of the module that is set to label process_high . The Nextflow label directive allows us to organise workflow processes in separate groups which can be referenced in a configuration file to select and configure subset of processes having similar computing requirements. The default values for the process_high label are set in the pipeline's base.config which in this case is defined as 72GB. Providing you haven't set any other standard nf-core parameters to cap the maximum resources used by the pipeline then we can try and bypass the STAR_ALIGN process failure by creating a custom config file that sets at least 72GB of memory, in this case increased to 100GB. The custom config below can then be provided to the pipeline via the -c parameter as highlighted in previous sections. process { withName: 'NFCORE_RNASEQ:RNASEQ:ALIGN_STAR:STAR_ALIGN' { memory = 100.GB } } NB: We specify the full process name i.e. NFCORE_RNASEQ:RNASEQ:ALIGN_STAR:STAR_ALIGN in the config file because this takes priority over the short name ( STAR_ALIGN ) and allows existing configuration using the full process name to be correctly overridden. If you get a warning suggesting that the process selector isn't recognised check that the process name has been specified correctly. Updating containers (advanced users) The Nextflow DSL2 implementation of this pipeline uses one container per process which makes it much easier to maintain and update software dependencies. If for some reason you need to use a different version of a particular tool with the pipeline then you just need to identify the process name and override the Nextflow container definition for that process using the withName declaration. For example, in the nf-core/viralrecon pipeline a tool called Pangolin has been used during the COVID-19 pandemic to assign lineages to SARS-CoV-2 genome sequenced samples. Given that the lineage assignments change quite frequently it doesn't make sense to re-release the nf-core/viralrecon everytime a new version of Pangolin has been released. However, you can override the default container used by the pipeline by creating a custom config file and passing it as a command-line argument via -c custom.config . Check the default version used by the pipeline in the module file for Pangolin Find the latest version of the Biocontainer available on Quay.io Create the custom config accordingly: For Docker: nextflow process { withName: PANGOLIN { container = 'quay.io/biocontainers/pangolin:3.0.5--pyhdfd78af_0' } } For Singularity: nextflow process { withName: PANGOLIN { container = 'https://depot.galaxyproject.org/singularity/pangolin:3.0.5--pyhdfd78af_0' } } For Conda: nextflow process { withName: PANGOLIN { conda = 'bioconda::pangolin=3.0.5' } } NB: If you wish to periodically update individual tool-specific results (e.g. Pangolin) generated by the pipeline then you must ensure to keep the work/ directory otherwise the -resume ability of the pipeline will be compromised and it will restart from scratch. Running in the background Nextflow handles job submissions and supervises the running jobs. The Nextflow process must run until the pipeline is finished. The Nextflow -bg flag launches Nextflow in the background, detached from your terminal so that the workflow does not stop if you log out of your session. The logs are saved to a file. Alternatively, you can use screen / tmux or similar tool to create a detached session which you can log back into at a later time. Some HPC setups also allow you to run nextflow within a cluster job submitted your job scheduler (from where it submits more jobs). Nextflow memory requirements In some cases, the Nextflow Java virtual machines can start to request a large amount of memory. We recommend adding the following line to your environment to limit this (typically in ~/.bashrc or ~./bash_profile ): NXF_OPTS='-Xms1g -Xmx4g'","title":"Usage"},{"location":"usage/#dalmolingroupeuryale-usage","text":"Documentation of pipeline parameters is generated automatically from the pipeline schema and can no longer be found in markdown files.","title":"dalmolingroup/euryale: Usage"},{"location":"usage/#introduction","text":"","title":"Introduction"},{"location":"usage/#samplesheet-input","text":"You will need to create a samplesheet with information about the samples you would like to analyse before running the pipeline. Use this parameter to specify its location. It has to be a comma-separated file with 3 columns, and a header row as shown in the examples below. --input '[path to samplesheet file]'","title":"Samplesheet input"},{"location":"usage/#multiple-runs-of-the-same-sample","text":"The sample identifiers have to be the same when you have re-sequenced the same sample more than once e.g. to increase sequencing depth. The pipeline will concatenate the raw reads before performing any downstream analysis. Below is an example for the same sample sequenced across 3 lanes: sample,fastq_1,fastq_2 CONTROL_REP1,AEG588A1_S1_L002_R1_001.fastq.gz,AEG588A1_S1_L002_R2_001.fastq.gz CONTROL_REP1,AEG588A1_S1_L003_R1_001.fastq.gz,AEG588A1_S1_L003_R2_001.fastq.gz CONTROL_REP1,AEG588A1_S1_L004_R1_001.fastq.gz,AEG588A1_S1_L004_R2_001.fastq.gz","title":"Multiple runs of the same sample"},{"location":"usage/#full-samplesheet","text":"The pipeline will auto-detect whether a sample is single- or paired-end using the information provided in the samplesheet. The samplesheet can have as many columns as you desire, however, there is a strict requirement for the first 3 columns to match those defined in the table below. A final samplesheet file consisting of both single- and paired-end data may look something like the one below. This is for 6 samples, where TREATMENT_REP3 has been sequenced twice. sample,fastq_1,fastq_2 CONTROL_REP1,AEG588A1_S1_L002_R1_001.fastq.gz,AEG588A1_S1_L002_R2_001.fastq.gz CONTROL_REP2,AEG588A2_S2_L002_R1_001.fastq.gz,AEG588A2_S2_L002_R2_001.fastq.gz CONTROL_REP3,AEG588A3_S3_L002_R1_001.fastq.gz,AEG588A3_S3_L002_R2_001.fastq.gz TREATMENT_REP1,AEG588A4_S4_L003_R1_001.fastq.gz, TREATMENT_REP2,AEG588A5_S5_L003_R1_001.fastq.gz, TREATMENT_REP3,AEG588A6_S6_L003_R1_001.fastq.gz, TREATMENT_REP3,AEG588A6_S6_L004_R1_001.fastq.gz, Column Description sample Custom sample name. This entry will be identical for multiple sequencing libraries/runs from the same sample. Spaces in sample names are automatically converted to underscores ( _ ). fastq_1 Full path to FastQ file for Illumina short reads 1. File has to be gzipped and have the extension \".fastq.gz\" or \".fq.gz\". fastq_2 Full path to FastQ file for Illumina short reads 2. File has to be gzipped and have the extension \".fastq.gz\" or \".fq.gz\". An example samplesheet has been provided with the pipeline.","title":"Full samplesheet"},{"location":"usage/#running-the-pipeline","text":"The typical command for running the pipeline is as follows: nextflow run dalmolingroup/euryale --input samplesheet.csv --outdir --genome GRCh37 -profile docker This will launch the pipeline with the docker configuration profile. See below for more information about profiles. Note that the pipeline will create the following files in your working directory: work # Directory containing the nextflow working files # Finished results in specified location (defined with --outdir) .nextflow_log # Log file from Nextflow # Other nextflow hidden files, eg. history of pipeline runs and old logs.","title":"Running the pipeline"},{"location":"usage/#updating-the-pipeline","text":"When you run the above command, Nextflow automatically pulls the pipeline code from GitHub and stores it as a cached version. When running the pipeline after this, it will always use the cached version if available - even if the pipeline has been updated since. To make sure that you're running the latest version of the pipeline, make sure that you regularly update the cached version of the pipeline: nextflow pull dalmolingroup/euryale","title":"Updating the pipeline"},{"location":"usage/#reproducibility","text":"It is a good idea to specify a pipeline version when running the pipeline on your data. This ensures that a specific version of the pipeline code and software are used when you run your pipeline. If you keep using the same tag, you'll be running the same version of the pipeline, even if there have been changes to the code since. First, go to the dalmolingroup/euryale releases page and find the latest pipeline version - numeric only (eg. 1.3.1 ). Then specify this when running the pipeline with -r (one hyphen) - eg. -r 1.3.1 . Of course, you can switch to another version by changing the number after the -r flag. This version number will be logged in reports when you run the pipeline, so that you'll know what you used when you look back in the future. For example, at the bottom of the MultiQC reports.","title":"Reproducibility"},{"location":"usage/#core-nextflow-arguments","text":"NB: These options are part of Nextflow and use a single hyphen (pipeline parameters use a double-hyphen).","title":"Core Nextflow arguments"},{"location":"usage/#-profile","text":"Use this parameter to choose a configuration profile. Profiles can give configuration presets for different compute environments. Several generic profiles are bundled with the pipeline which instruct the pipeline to use software packaged using different methods (Docker, Singularity, Podman, Shifter, Charliecloud, Conda) - see below. We highly recommend the use of Docker or Singularity containers for full pipeline reproducibility, however when this is not possible, Conda is also supported. Note that multiple profiles can be loaded, for example: -profile test,docker - the order of arguments is important! They are loaded in sequence, so later profiles can overwrite earlier profiles. If -profile is not specified, the pipeline will run locally and expect all software to be installed and available on the PATH . This is not recommended, since it can lead to different results on different machines dependent on the computer enviroment. test A profile with a complete configuration for automated testing Includes links to test data so needs no other parameters docker A generic configuration profile to be used with Docker singularity A generic configuration profile to be used with Singularity podman A generic configuration profile to be used with Podman shifter A generic configuration profile to be used with Shifter charliecloud A generic configuration profile to be used with Charliecloud conda A generic configuration profile to be used with Conda . Please only use Conda as a last resort i.e. when it's not possible to run the pipeline with Docker, Singularity, Podman, Shifter or Charliecloud.","title":"-profile"},{"location":"usage/#-resume","text":"Specify this when restarting a pipeline. Nextflow will use cached results from any pipeline steps where the inputs are the same, continuing from where it got to previously. For input to be considered the same, not only the names must be identical but the files' contents as well. For more info about this parameter, see this blog post . You can also supply a run name to resume a specific run: -resume [run-name] . Use the nextflow log command to show previous run names.","title":"-resume"},{"location":"usage/#-c","text":"Specify the path to a specific config file (this is a core Nextflow command). See the nf-core website documentation for more information.","title":"-c"},{"location":"usage/#custom-configuration","text":"","title":"Custom configuration"},{"location":"usage/#resource-requests","text":"Whilst the default requirements set within the pipeline will hopefully work for most people and with most input data, you may find that you want to customise the compute resources that the pipeline requests. Each step in the pipeline has a default set of requirements for number of CPUs, memory and time. For most of the steps in the pipeline, if the job exits with any of the error codes specified here it will automatically be resubmitted with higher requests (2 x original, then 3 x original). If it still fails after the third attempt then the pipeline execution is stopped. For example, if the nf-core/rnaseq pipeline is failing after multiple re-submissions of the STAR_ALIGN process due to an exit code of 137 this would indicate that there is an out of memory issue: [62/149eb0] NOTE: Process `NFCORE_RNASEQ:RNASEQ:ALIGN_STAR:STAR_ALIGN (WT_REP1)` terminated with an error exit status (137) -- Execution is retried (1) Error executing process > 'NFCORE_RNASEQ:RNASEQ:ALIGN_STAR:STAR_ALIGN (WT_REP1)' Caused by: Process `NFCORE_RNASEQ:RNASEQ:ALIGN_STAR:STAR_ALIGN (WT_REP1)` terminated with an error exit status (137) Command executed: STAR \\ --genomeDir star \\ --readFilesIn WT_REP1_trimmed.fq.gz \\ --runThreadN 2 \\ --outFileNamePrefix WT_REP1. \\ Command exit status: 137 Command output: (empty) Command error: .command.sh: line 9: 30 Killed STAR --genomeDir star --readFilesIn WT_REP1_trimmed.fq.gz --runThreadN 2 --outFileNamePrefix WT_REP1. Work dir: /home/pipelinetest/work/9d/172ca5881234073e8d76f2a19c88fb Tip: you can replicate the issue by changing to the process work dir and entering the command `bash .command.run`","title":"Resource requests"},{"location":"usage/#for-beginners","text":"A first step to bypass this error, you could try to increase the amount of CPUs, memory, and time for the whole pipeline. Therefor you can try to increase the resource for the parameters --max_cpus , --max_memory , and --max_time . Based on the error above, you have to increase the amount of memory. Therefore you can go to the parameter documentation of rnaseq and scroll down to the show hidden parameter button to get the default value for --max_memory . In this case 128GB, you than can try to run your pipeline again with --max_memory 200GB -resume to skip all process, that were already calculated. If you can not increase the resource of the complete pipeline, you can try to adapt the resource for a single process as mentioned below.","title":"For beginners"},{"location":"usage/#advanced-option-on-process-level","text":"To bypass this error you would need to find exactly which resources are set by the STAR_ALIGN process. The quickest way is to search for process STAR_ALIGN in the nf-core/rnaseq Github repo . We have standardised the structure of Nextflow DSL2 pipelines such that all module files will be present in the modules/ directory and so, based on the search results, the file we want is modules/nf-core/star/align/main.nf . If you click on the link to that file you will notice that there is a label directive at the top of the module that is set to label process_high . The Nextflow label directive allows us to organise workflow processes in separate groups which can be referenced in a configuration file to select and configure subset of processes having similar computing requirements. The default values for the process_high label are set in the pipeline's base.config which in this case is defined as 72GB. Providing you haven't set any other standard nf-core parameters to cap the maximum resources used by the pipeline then we can try and bypass the STAR_ALIGN process failure by creating a custom config file that sets at least 72GB of memory, in this case increased to 100GB. The custom config below can then be provided to the pipeline via the -c parameter as highlighted in previous sections. process { withName: 'NFCORE_RNASEQ:RNASEQ:ALIGN_STAR:STAR_ALIGN' { memory = 100.GB } } NB: We specify the full process name i.e. NFCORE_RNASEQ:RNASEQ:ALIGN_STAR:STAR_ALIGN in the config file because this takes priority over the short name ( STAR_ALIGN ) and allows existing configuration using the full process name to be correctly overridden. If you get a warning suggesting that the process selector isn't recognised check that the process name has been specified correctly.","title":"Advanced option on process level"},{"location":"usage/#updating-containers-advanced-users","text":"The Nextflow DSL2 implementation of this pipeline uses one container per process which makes it much easier to maintain and update software dependencies. If for some reason you need to use a different version of a particular tool with the pipeline then you just need to identify the process name and override the Nextflow container definition for that process using the withName declaration. For example, in the nf-core/viralrecon pipeline a tool called Pangolin has been used during the COVID-19 pandemic to assign lineages to SARS-CoV-2 genome sequenced samples. Given that the lineage assignments change quite frequently it doesn't make sense to re-release the nf-core/viralrecon everytime a new version of Pangolin has been released. However, you can override the default container used by the pipeline by creating a custom config file and passing it as a command-line argument via -c custom.config . Check the default version used by the pipeline in the module file for Pangolin Find the latest version of the Biocontainer available on Quay.io Create the custom config accordingly: For Docker: nextflow process { withName: PANGOLIN { container = 'quay.io/biocontainers/pangolin:3.0.5--pyhdfd78af_0' } } For Singularity: nextflow process { withName: PANGOLIN { container = 'https://depot.galaxyproject.org/singularity/pangolin:3.0.5--pyhdfd78af_0' } } For Conda: nextflow process { withName: PANGOLIN { conda = 'bioconda::pangolin=3.0.5' } } NB: If you wish to periodically update individual tool-specific results (e.g. Pangolin) generated by the pipeline then you must ensure to keep the work/ directory otherwise the -resume ability of the pipeline will be compromised and it will restart from scratch.","title":"Updating containers (advanced users)"},{"location":"usage/#running-in-the-background","text":"Nextflow handles job submissions and supervises the running jobs. The Nextflow process must run until the pipeline is finished. The Nextflow -bg flag launches Nextflow in the background, detached from your terminal so that the workflow does not stop if you log out of your session. The logs are saved to a file. Alternatively, you can use screen / tmux or similar tool to create a detached session which you can log back into at a later time. Some HPC setups also allow you to run nextflow within a cluster job submitted your job scheduler (from where it submits more jobs).","title":"Running in the background"},{"location":"usage/#nextflow-memory-requirements","text":"In some cases, the Nextflow Java virtual machines can start to request a large amount of memory. We recommend adding the following line to your environment to limit this (typically in ~/.bashrc or ~./bash_profile ): NXF_OPTS='-Xms1g -Xmx4g'","title":"Nextflow memory requirements"}]}
\ No newline at end of file
diff --git a/sitemap.xml.gz b/sitemap.xml.gz
index 95e73b6b9af1663a454d8a2d4c423d03b82a8a18..1d1d721fd233019a4ea06a233fa43d8ed5b48ef6 100644
GIT binary patch
delta 13
Ucmb=gXP58h;E*`8ej