Merge pull request #199 from assemblerflow/dev

Update - version 1.4.1

changelog.md

@@ -2,6 +2,39 @@
 
 ## Changes in upcoming release (`dev` branch) 
 
+
+## 1.4.0
+
+### New features
+
+- Added support for the report system to:
+    - `maxbin2`
+- Added new `manifest.config` with the pipeline metadata
+
+### New components
+
+- `Kraken2`: Taxonomic identification on FastQ files
+
+### Bug fixes
+
+- Fix bug in `momps`component related to added in the introduction of the clear input parameter
+- Fixed bug with the `-ft` parameters not retrieving the dockerhub tags for 
+all the components.
+- Fixed bug in the `megahit` process where the fastg mode would break the process
+- Fix inspect and report mode to fetch the nextflow file independently of its 
+position in the `nextflow run` command inside the .nextflow.log file.
+- Fix parsing of .nextflow.log file when searching for `nextflow run` command.
+- Fixed bug between mash_sketch_fasta and mash_dist.
+
+### Minor/Other changes
+
+- Added option to `dengue_typing` to retrieve closest reference sequence and link it 
+with a secondary channel into `mafft`
+- New version of DEN-IM recipe
+- Now prints an ordered list of components
+- Moved taxonomy results from `results/annotation/` to `results/taxonomy/`
+
+
 ## 1.4.0
 
 ### New features
@@ -41,9 +74,11 @@ resolution
 - Added seed parameter to `downsample_fastq` component.
 - Added bacmet database to `abricate` component.
 - Added default docker option to avoid docker permission errors.
+- Changed the default URL generated by inspect and report commands. 
 - Changed the default URL generated by inspect and report commands.
 - Added directives to `-L` parameter of build module.
 
+
 ### Bug fixes
 
 - Fixed forks with same source process name.

docs/user/available_components.rst

@@ -135,6 +135,9 @@ Taxonomic Profiling
 - :doc:`components/kraken`: Performs taxonomic identification with kraken on FastQ files
   (minikrakenDB2017 as default database)
 
+- :doc:`components/kraken2`: Performs taxonomic identification with kraken2 on FastQ files
+  (minikraken2_v1_8GB as default database)
+
 - :doc:`components/midas_species`: Performs taxonomic identification on FastQ files at the
   species level with midas (requires database)
 

docs/user/components/kraken.rst

@@ -27,7 +27,7 @@ Parameters
 Published results
 -----------------
 
-- ``results/annotation/kraken``: Stores the results of the screening
+- ``results/taxonomy/kraken``: Stores the results of the screening
   for each sample.
 
 Published reports

docs/user/components/kraken2.rst

@@ -0,0 +1,45 @@
+kraken2
+=======
+
+Purpose
+-------
+
+This component performs Kraken2 to assign taxonomic labels to short DNA
+sequences, usually obtained through metagenomic studies.
+
+.. note::
+    Software page: https://ccb.jhu.edu/software/kraken2/
+
+Input/Output type
+------------------
+
+- Input type: ``FastQ``
+- Output type: txt
+
+.. note::
+    The default input parameter for fastq data is ``--fastq``.
+
+Parameters
+----------
+
+- ``kraken2DB``: Specifies kraken2 database. Default: minikraken2_v1_8GB (in path inside the
+default container)
+
+Published results
+-----------------
+
+- ``results/taxonomy/kraken2``: Stores the results of the screening
+  for each sample.
+
+Published reports
+-----------------
+
+None.
+
+Default directives
+------------------
+
+- ``container``: flowcraft/kraken2
+- ``version``: 2.0.7-1
+- ``cpus``: 3
+- ``memory``: 5GB (dynamically increased on retry)

docs/user/components/maxbin2.rst

@@ -45,5 +45,11 @@ Default directives
 
 - ``container``: flowcraft/maxbin2
 - ``version``: 2.2.4-1
-- ``cpus``: 3
-- ``memory``: 5.GB * task.attempt
+- ``cpus``: 4
+- ``memory``: 8.GB (dynamically increased on retry)
+
+
+Template
+^^^^^^^^
+
+:mod:`assemblerflow.templates.maxbin2`

docs/user/components/megahit.rst

@@ -27,6 +27,11 @@ Parameters
   from the maximum read length of each assembly. If 'default', megahit will
   use the default k-mer lengths.
 
+- ``fastg``: When true, it converts megahit intermediate contigs into fastg.
+  Default: False
+
+
+
 Published results
 -----------------
 

docs/user/components/midas_species.rst

@@ -27,7 +27,7 @@ Parameters
 Published results
 -----------------
 
-- ``results/annotation/midas``: Stores the results of the screening
+- ``results/taxonomy/midas``: Stores the results of the screening
   for each sample.
 
 Published reports

docs/user/pipeline_building.rst

@@ -14,7 +14,7 @@ determine the raw input type, and the parameter for providing input data.
 The input type information is provided in the documentation page of each
 component. For instance, if the first component is FastQC, which has an input
 type of ``FastQ``, the parameter for providing the raw input data will be
-``--fasta``. Here are the currently supported input types and their
+``--fastq``. Here are the currently supported input types and their
 respective parameters:
 
 - ``FastQ``: ``--fastq``

docs/user/reports/maxbin2.rst

@@ -0,0 +1,15 @@
+maxbin2
+----
+
+Table data
+^^^^^^^^^^
+
+Metagenomic Binning (sample specific):
+    - **Bin name**: The number of bin.
+    - **Completness**: Estimation of completion of genome in bin (% of Single copy genes present)
+    - **Genome size**: Total size of the bin
+    - **GC content**: Percentage of GC in the bin
+
+.. image:: ../resources/reports/binning.png
+    :scale: 80 %
+    :align: center

flowcraft/__init__.py

@@ -1,6 +1,6 @@
 
-__version__ = "1.4.0"
-__build__ = "18112018"
+__version__ = "1.4.1"
+__build__ = "02032019"
 __author__ = "Diogo N. Silva, Tiago F. Jesus, Ines Mendes, Bruno Ribeiro-Goncalves"
 __copyright__ = "Diogo N. Silva"
 __license__ = "GPL3"

flowcraft/bin/parse_fasta.py

@@ -25,6 +25,9 @@ def getSequence(ref, fasta):
             output_file = open(filename + '.fa', "w")
             output_file.write(">" + fasta_header + "\n" + seq.upper() + "\n")
             output_file.close()
+            header_file = open("header.txt", "w")
+            header_file.write(fasta_header)
+            header_file.close()
 
 def main():
 

flowcraft/flowcraft.py

@@ -387,15 +387,20 @@ def inspect(args):
     try:
         nf_inspect = NextflowInspector(args.trace_file, args.refresh_rate,
                                        args.pretty, args.url)
-    except eh.InspectionError as e:
-        logger.error(colored_print(e.value, "red_bold"))
+        if args.mode == "overview":
+            nf_inspect.display_overview()
+
+        if args.mode == "broadcast":
+            nf_inspect.broadcast_status()
+
+    except eh.InspectionError as ie:
+        logger.error(colored_print(ie.value, "red_bold"))
         sys.exit(1)
 
-    if args.mode == "overview":
-        nf_inspect.display_overview()
+    except eh.LogError as le:
+        logger.error(colored_print(le.value, "red_bold"))
+        sys.exit(1)
 
-    if args.mode == "broadcast":
-        nf_inspect.broadcast_status()
 
 
 def report(args):
@@ -407,11 +412,16 @@ def report(args):
             log_file=args.log_file,
             watch=args.watch,
             ip_addr=args.url)
-    except eh.ReportError as e:
-        logger.error(colored_print(e.value, "red_bold"))
+
+        fc_report.broadcast_report()
+
+    except eh.ReportError as re:
+        logger.error(colored_print(re.value, "red_bold"))
         sys.exit(1)
 
-    fc_report.broadcast_report()
+    except eh.LogError as le:
+        logger.error(colored_print(le.value, "red_bold"))
+        sys.exit(1)
 
 
 def main():

flowcraft/generator/components/alignment.py

@@ -25,6 +25,9 @@ def __init__(self, **kwargs):
         self.params = {
         }
 
+        self.link_end.append({"link": "_ref_seqTyping", "alias": "_ref_seqTyping"})
+
+
         self.directives = {
             "mafft": {
                 "container": "flowcraft/mafft",

flowcraft/generator/components/metagenomics.py

@@ -41,6 +41,43 @@ def __init__(self, **kwargs):
             "kraken"
         ]
 
+class Kraken2(Process):
+    """kraken2 process template interface
+
+            This process is set with:
+
+                - ``input_type``: fastq
+                - ``output_type``: txt
+                - ``ptype``: taxonomic classification
+    """
+    def __init__(self, **kwargs):
+
+        super().__init__(**kwargs)
+
+        self.input_type = "fastq"
+        self.output_type = None
+
+        self.params = {
+            "kraken2DB": {
+                "default": "'minikraken2_v1_8GB'",
+                "description": "Specifies kraken2 database. Requires full path if database not on "
+                               "KRAKEN2_DB_PATH."
+            }
+        }
+
+        self.directives = {
+            "kraken2": {
+                "container": "flowcraft/kraken2",
+                "version": "2.0.7-1",
+                "memory": "{8.Gb*task.attempt}",
+                "cpus": 4
+            }
+        }
+
+        self.status_channels = [
+            "kraken2"
+        ]
+
 
 class Maxbin2(Process):
     """MaxBin2, a metagenomics binning software
@@ -102,7 +139,8 @@ def __init__(self, **kwargs):
         }
 
         self.status_channels = [
-            "maxbin2"
+            "maxbin2",
+            "report_maxbin2"
         ]
 
 

flowcraft/generator/components/typing.py

@@ -137,13 +137,13 @@ def __init__(self, **kwargs):
         super().__init__(**kwargs)
 
         self.input_type = "fasta"
-        self.output_type = None
+        self.output_type = "fasta"
 
-        self.link_start = None
+        self.link_start.extend(["_ref_seqTyping"])
 
         self.params = {
             "reference": {
-                "default": "false",
+                "default": "true",
                 "description":
                     "Retrieves the sequence of the closest reference."
             }
@@ -156,4 +156,9 @@ def __init__(self, **kwargs):
             "version": "2.0-1"
         }}
 
+        self.status_channels = [
+            "dengue_typing"
+        ]
+
+