Merge pull request #17 from nf-core/dev

v1.0.0

.github/CONTRIBUTING.md

@@ -1,12 +1,12 @@
-# nf-core/mag Contributing Guidelines
+# nf-core/mag: Contributing Guidelines
 
 Hi there! Many thanks for taking an interest in improving nf-core/mag.
 
-We try to manage the required tasks for nf-core/mag using GitHub issues, you probably came to this page when creating one. Please use the prefilled template to save time.
+We try to manage the required tasks for nf-core/mag using GitHub issues, you probably came to this page when creating one. Please use the pre-filled template to save time.
 
 However, don't be put off by this template - other more general issues and suggestions are welcome! Contributions to the code are even more welcome ;)
 
-> If you need help using nf-core/mag then the best place to go is the Gitter chatroom where you can ask us questions directly: https://gitter.im/nf-core/Lobby
+> If you need help using or modifying nf-core/mag then the best place to ask is on the pipeline channel on [Slack](https://nf-co.re/join/slack).
 
 ## Contribution workflow
 If you'd like to write some code for nf-core/mag, the standard workflow
@@ -15,11 +15,30 @@ is as follows:
 1. Check that there isn't already an issue about your idea in the
    [nf-core/mag issues](https://github.com/nf-core/mag/issues) to avoid
    duplicating work.
-    * Feel free to add a new issue here for the same reason.
+    * If there isn't one already, please create one so that others know you're working on this
 2. Fork the [nf-core/mag repository](https://github.com/nf-core/mag) to your GitHub account
 3. Make the necessary changes / additions within your forked repository
-4. Submit a Pull Request against the master branch and wait for the code to be reviewed and merged.
+4. Submit a Pull Request against the `dev` branch and wait for the code to be reviewed and merged.
 
 If you're not used to this workflow with git, you can start with some [basic docs from GitHub](https://help.github.com/articles/fork-a-repo/) or even their [excellent interactive tutorial](https://try.github.io/).
 
-For further information/help, please consult the [nf-core/mag documentation](https://github.com/nf-core/mag#documentation) and don't hesitate to get in touch on [Gitter](https://gitter.im/nf-core/Lobby)
+## Tests
+When you create a pull request with changes, [Travis CI](https://travis-ci.org/) will run automatic tests.
+Typically, pull-requests are only fully reviewed when these tests are passing, though of course we can help out before then.
+
+There are typically two types of tests that run:
+
+### Lint Tests
+The nf-core has a [set of guidelines](http://nf-co.re/guidelines) which all pipelines must adhere to.
+To enforce these and ensure that all pipelines stay in sync, we have developed a helper tool which runs checks on the pipeline code. This is in the [nf-core/tools repository](https://github.com/nf-core/tools) and once installed can be run locally with the `nf-core lint <pipeline-directory>` command.
+
+If any failures or warnings are encountered, please follow the listed URL for more documentation.
+
+### Pipeline Tests
+Each nf-core pipeline should be set up with a minimal set of test-data.
+Travis CI then runs the pipeline on this data to ensure that it exists successfully.
+If there are any failures then the automated tests fail.
+These tests are run both with the latest available version of Nextflow and also the minimum required version that is stated in the pipeline code.
+
+## Getting help
+For further information/help, please consult the [nf-core/mag documentation](https://github.com/nf-core/mag#documentation) and don't hesitate to get in touch on the pipeline channel on [Slack](https://nfcore.slack.com/channels/mag).

.github/ISSUE_TEMPLATE/bug_report.md

@@ -0,0 +1,31 @@
+Hi there!
+
+Thanks for telling us about a problem with the pipeline. Please delete this text and anything that's not relevant from the template below:
+
+#### Describe the bug
+A clear and concise description of what the bug is.
+
+#### Steps to reproduce
+Steps to reproduce the behaviour:
+1. Command line: `nextflow run ...`
+2. See error: _Please provide your error message_
+
+#### Expected behaviour
+A clear and concise description of what you expected to happen.
+
+#### System:
+ - Hardware: [e.g. HPC, Desktop, Cloud...]
+ - Executor: [e.g. slurm, local, awsbatch...]
+ - OS: [e.g. CentOS Linux, macOS, Linux Mint...]
+ - Version [e.g. 7, 10.13.6, 18.3...]
+
+#### Nextflow Installation:
+ - Version: [e.g. 0.31.0]
+
+#### Container engine:
+ - Engine: [e.g. Conda, Docker or Singularity]
+ - version: [e.g. 1.0.0]
+ - Image tag: [e.g. nfcore/mag:1.0.0]
+
+#### Additional context
+Add any other context about the problem here.

.github/ISSUE_TEMPLATE/feature_request.md

@@ -0,0 +1,16 @@
+Hi there!
+
+Thanks for suggesting a new feature for the pipeline! Please delete this text and anything that's not relevant from the template below:
+
+#### Is your feature request related to a problem? Please describe.
+A clear and concise description of what the problem is.
+Ex. I'm always frustrated when [...]
+
+#### Describe the solution you'd like
+A clear and concise description of what you want to happen.
+
+#### Describe alternatives you've considered
+A clear and concise description of any alternative solutions or features you've considered.
+
+#### Additional context
+Add any other context about the feature request here.

.github/pull_request.md → .github/PULL_REQUEST_TEMPLATE.md

@@ -5,7 +5,7 @@ Please fill in the appropriate checklist below (delete whatever is not relevant)
 ## PR checklist
  - [ ] This comment contains a description of changes (with reason)
  - [ ] If you've fixed a bug or added code that should be tested, add tests!
- - [ ] If necessary, also make a PR on the [nf-core/mag branch on the nf-core/test-datasets repo]( https://github.com/nf-core/test-datasets/pull/newnf-core/mag)
+ - [ ] If necessary, also make a PR on the [nf-core/mag branch on the nf-core/test-datasets repo]( https://github.com/nf-core/test-datasets/pull/new/nf-core/mag)
  - [ ] Ensure the test suite passes (`nextflow run . -profile test,docker`).
  - [ ] Make sure your code lints (`nf-core lint .`).
  - [ ] Documentation in `docs` is updated

.github/markdownlint.yml

@@ -0,0 +1,9 @@
+# Markdownlint configuration file
+default: true,
+line-length: false
+no-multiple-blanks: 0
+blanks-around-headers: false
+blanks-around-lists: false
+header-increment: false
+no-duplicate-header:
+    siblings_only: true

.gitignore

@@ -3,5 +3,7 @@ work/
 results/
 .DS_Store
 tests/test_data
+*.pyc
 .vscode
-db/
+db/
+tests/

.travis.yml

@@ -8,30 +8,36 @@ matrix:
   fast_finish: true
 
 before_install:
-  # PRs made to 'master' branch should always orginate from another repo or the 'dev' branch
-  - '[ $TRAVIS_PULL_REQUEST = "false" ] || [ $TRAVIS_BRANCH != "master" ] || ([ $TRAVIS_PULL_REQUEST_SLUG = $TRAVIS_REPO_SLUG ] && [ $TRAVIS_PULL_REQUEST_BRANCH = "dev" ])'
+  # PRs to master are only ok if coming from dev branch
+  - '[ $TRAVIS_PULL_REQUEST = "false" ] || [ $TRAVIS_BRANCH != "master" ] || ([ $TRAVIS_PULL_REQUEST_SLUG = $TRAVIS_REPO_SLUG ] && ([ $TRAVIS_PULL_REQUEST_BRANCH = "dev" ] || [ $TRAVIS_PULL_REQUEST_BRANCH = "patch" ]))'
   # Pull the docker image first so the test doesn't wait for this
   - docker pull nfcore/mag:dev
   # Fake the tag locally so that the pipeline runs properly
+  # Looks weird when this is :dev to :dev, but makes sense when testing code for a release (:dev to :1.0.1)
   - docker tag nfcore/mag:dev nfcore/mag:1.0.0
 
 install:
   # Install Nextflow
-  - mkdir /tmp/nextflow
-  - cd /tmp/nextflow
+  - mkdir /tmp/nextflow && cd /tmp/nextflow
   - wget -qO- get.nextflow.io | bash
   - sudo ln -s /tmp/nextflow/nextflow /usr/local/bin/nextflow
   # Install nf-core/tools
+  - pip install --upgrade pip
   - pip install nf-core
   # Reset
-  - cd ${TRAVIS_BUILD_DIR}
+  - mkdir ${TRAVIS_BUILD_DIR}/tests && cd ${TRAVIS_BUILD_DIR}/tests
+  # Install markdownlint-cli
+  - sudo apt-get install npm && npm install -g markdownlint-cli
 
 env:
-  - NXF_VER='0.32.0' # Specify a minimum NF version that should be tested and work
-  - NXF_VER='' # Plus: get the latest NF version and check, that it works
+  - NXF_VER='19.01.0' # Specify a minimum NF version that should be tested and work
+  - NXF_VER='' # Plus: get the latest NF version and check that it works
 
 script:
   # Lint the pipeline code
   - nf-core lint ${TRAVIS_BUILD_DIR}
-  # Run the pipeline with the test profile
-  - nextflow run main.nf -profile test,docker
+  # Lint the documentation
+  - markdownlint ${TRAVIS_BUILD_DIR} -c ${TRAVIS_BUILD_DIR}/.github/markdownlint.yml
+  # Run the pipeline with the test profile(s)
+  - nextflow run ${TRAVIS_BUILD_DIR} -profile test,docker
+  - nextflow run ${TRAVIS_BUILD_DIR} -profile test_hybrid,docker

CHANGELOG.md

@@ -1,3 +1,14 @@
-## nf-core/mag version 1.0.0 -
+# nf-core/mag: Changelog
+
+## v1.0.0 - 2019/12/20
 
 Initial release of nf-core/mag, created with the [nf-core](http://nf-co.re/) template.
+As this release the pipeline will have the following functionailities:
+
+- short and long reads QC (fastp, porechop, filtlong, fastqc)
+- Lambda and PhiX detection and filtering (bowtie2, nanolyse)
+- Taxonomic classification of reads (centrifuge, kraken2)
+- Short read and hybrid assembly (megahit, metaspades)
+- metagenome binning (metabat2)
+- QC of bins (busco, quast)
+- annotation (cat/bat)

CODE_OF_CONDUCT.md

@@ -34,7 +34,7 @@ This Code of Conduct applies both within project spaces and in public spaces whe
 
 ## Enforcement
 
-Instances of abusive, harassing, or otherwise unacceptable behavior may be reported by contacting the project team on the [Gitter channel](https://gitter.im/nf-core/Lobby). The project team will review and investigate all complaints, and will respond in a way that it deems appropriate to the circumstances. The project team is obligated to maintain confidentiality with regard to the reporter of an incident. Further details of specific enforcement policies may be posted separately.
+Instances of abusive, harassing, or otherwise unacceptable behavior may be reported by contacting the project team on [Slack](nf-co.re/join/slack). The project team will review and investigate all complaints, and will respond in a way that it deems appropriate to the circumstances. The project team is obligated to maintain confidentiality with regard to the reporter of an incident. Further details of specific enforcement policies may be posted separately.
 
 Project maintainers who do not follow or enforce the Code of Conduct in good faith may face temporary or permanent repercussions as determined by other members of the project's leadership.
 

Dockerfile

@@ -1,9 +1,8 @@
-FROM nfcore/base
+FROM nfcore/base:1.7
 
-LABEL maintainer="Hadrien Gourlé <[email protected]>"
-LABEL description="Docker image containing all requirements for nf-core/mag pipeline"
+LABEL authors="Hadrien Gourlé <[email protected]>, Daniel Straub <[email protected]>" \
+    description="Docker image containing all requirements for nf-core/mag pipeline"
 
 COPY environment.yml /
 RUN conda env create -f /environment.yml && conda clean -a
 ENV PATH /opt/conda/envs/nf-core-mag-1.0.0/bin:$PATH
-

LICENSE

@@ -1,6 +1,6 @@
 MIT License
 
-Copyright 2018, Hadrien Gourlé
+Copyright 2018, Hadrien Gourlé, Daniel Straub
 
 Permission is hereby granted, free of charge, to any person obtaining a copy
 of this software and associated documentation files (the "Software"), to deal

README.md

@@ -1,32 +1,43 @@
 # ![mag](https://raw.githubusercontent.com/nf-core/mag/master/docs/images/mag_logo.png)
 
-# nf-core/mag
+**Assembly, binning and annotation of metagenomes**.
 
-**Assembly, binning and annotation of metagenomes**
-
-[![Build Status](https://travis-ci.org/nf-core/mag.svg?branch=master)](https://travis-ci.org/nf-core/mag)
-[![Nextflow](https://img.shields.io/badge/nextflow-%E2%89%A50.32.0-brightgreen.svg)](https://www.nextflow.io/)
+[![Build Status](https://travis-ci.com/nf-core/mag.svg?branch=master)](https://travis-ci.com/nf-core/mag)
+[![Nextflow](https://img.shields.io/badge/nextflow-%E2%89%A519.01.0-brightgreen.svg)](https://www.nextflow.io/)
 
 [![install with bioconda](https://img.shields.io/badge/install%20with-bioconda-brightgreen.svg)](http://bioconda.github.io/)
-[![Docker](https://img.shields.io/docker/automated/hadrieng/mag.svg)](https://hub.docker.com/r/hadrieng/mag)
-![Singularity Container available](https://img.shields.io/badge/singularity-available-7E4C74.svg)
+[![Docker](https://img.shields.io/docker/automated/nfcore/mag.svg)](https://hub.docker.com/r/nfcore/mag)
+
+## Introduction
+
+This pipeline is for assembly, binning and annotation of metagenomes.
+It supports both short and long reads, quality trims the reads and adapters with [https://github.com/OpenGene/fastp](fastp) and [https://github.com/rrwick/Porechop](porechop) and performs basic QC with [https://www.bioinformatics.babraham.ac.uk/projects/fastqc/](fastqc).
 
-### Introduction
+The pipeline then:
 
-The pipeline is built using [Nextflow](https://www.nextflow.io), a workflow tool to run tasks across multiple compute infrastructures in a very portable manner. It comes with docker / singularity containers making installation trivial and results highly reproducible.
+- assigns taxonomy to reads using [https://ccb.jhu.edu/software/centrifuge/](centrifuge) and/or [https://ccb.jhu.edu/software/kraken2/](kraken2)
+- performs assembly using [https://github.com/voutcn/megahit](megahit) and [http://cab.spbu.ru/software/spades/](spades), and checks their quality using [http://quast.sourceforge.net/quast](quast)
+- performs metagenome binning using [https://bitbucket.org/berkeleylab/metabat/src/master/](metabat2), and checks the quality of the genome bins using [https://busco.ezlab.org/](busco)
 
-### Documentation
+Furthermore, the pipeline creates various reports in the results directory specified, including a [https://multiqc.info/](multiqc) report summarizing some of the findings and software versions.
+
+The pipeline is built using [Nextflow](https://www.nextflow.io), a workflow tool to run tasks across multiple compute infrastructures in a very portable manner. It comes with docker containers making installation trivial and results highly reproducible.
+
+## Documentation
 
 The nf-core/mag pipeline comes with documentation about the pipeline, found in the `docs/` directory:
 
-1. [Installation](docs/installation.md)
+1. [Installation](https://nf-co.re/usage/installation)
 2. Pipeline configuration
-   - [Local installation](docs/configuration/local.md)
-   - [Adding your own system](docs/configuration/adding_your_own.md)
+   - [Local installation](https://nf-co.re/usage/local_installation)
+   - [Adding your own system config](https://nf-co.re/usage/adding_own_config)
+   - [Reference genomes](https://nf-co.re/usage/reference_genomes)
 3. [Running the pipeline](docs/usage.md)
 4. [Output and how to interpret the results](docs/output.md)
-5. [Troubleshooting](docs/troubleshooting.md)
+5. [Troubleshooting](https://nf-co.re/usage/troubleshooting)
+
+## Credits
 
-### Credits
+This pipeline was written by [Hadrien Gourlé](https://hadriengourle.com) at [SLU](https://slu.se) and Daniel Straub ([@d4straub](https://github.com/d4straub)).
 
-This pipeline was written by [Hadrien Gourlé](https://hadriengourle.com) at [SLU](https://slu.se).
+Long read processing was inspired by [caspargross/HybridAssembly](https://github.com/caspargross/HybridAssembly) written by Caspar Gross [@caspargross](https://github.com/caspargross)

assets/email_template.txt

@@ -17,23 +17,6 @@ ${errorReport}
 } %>
 
 
-<% if (!success){
-    out << """####################################################
-## nf-core/mag execution completed unsuccessfully! ##
-####################################################
-The exit status of the task that caused the workflow execution to fail was: $exitStatus.
-The full error message was:
-
-${errorReport}
-"""
-} else {
-    out << "## nf-core/mag execution completed successfully! ##"
-}
-%>
-
-
-
-
 The workflow was completed at $dateComplete (duration: $duration)
 
 The command used to launch the workflow was as follows:

conf/multiqc_config.yaml → assets/multiqc_config.yaml

@@ -5,3 +5,5 @@ report_comment: >
 report_section_order:
     nf-core/mag-software-versions:
         order: -1000
+
+export_plots: true