Skip to content

Commit

Permalink
Add fastp component (#3)
Browse files Browse the repository at this point in the history
* initialise fastp component

* add arguments to config

* add test and script

* update help

* update changelog

* fix test data script

* address comments

* Apply suggestions from code review

Co-authored-by: Dries Schaumont <[email protected]>

* extend test

---------

Co-authored-by: Dries Schaumont <[email protected]>
  • Loading branch information
rcannood and DriesSchaumont authored Jan 26, 2024
1 parent d0461db commit b3573a2
Show file tree
Hide file tree
Showing 9 changed files with 870 additions and 0 deletions.
2 changes: 2 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,8 @@

* `arriba`: Detect gene fusions from RNA-seq data (PR #1).

* `fastp`: An ultra-fast all-in-one FASTQ preprocessor (PR #3).

## MAJOR CHANGES

## MINOR CHANGES
Expand Down
574 changes: 574 additions & 0 deletions src/fastp/config.vsh.yaml

Large diffs are not rendered by default.

93 changes: 93 additions & 0 deletions src/fastp/help.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1,93 @@
```bash
fastp --help
```

usage: fastp [options] ...
options:
-i, --in1 read1 input file name (string [=])
-o, --out1 read1 output file name (string [=])
-I, --in2 read2 input file name (string [=])
-O, --out2 read2 output file name (string [=])
--unpaired1 for PE input, if read1 passed QC but read2 not, it will be written to unpaired1. Default is to discard it. (string [=])
--unpaired2 for PE input, if read2 passed QC but read1 not, it will be written to unpaired2. If --unpaired2 is same as --unpaired1 (default mode), both unpaired reads will be written to this same file. (string [=])
--overlapped_out for each read pair, output the overlapped region if it has no any mismatched base. (string [=])
--failed_out specify the file to store reads that cannot pass the filters. (string [=])
-m, --merge for paired-end input, merge each pair of reads into a single read if they are overlapped. The merged reads will be written to the file given by --merged_out, the unmerged reads will be written to the files specified by --out1 and --out2. The merging mode is disabled by default.
--merged_out in the merging mode, specify the file name to store merged output, or specify --stdout to stream the merged output (string [=])
--include_unmerged in the merging mode, write the unmerged or unpaired reads to the file specified by --merge. Disabled by default.
-6, --phred64 indicate the input is using phred64 scoring (it'll be converted to phred33, so the output will still be phred33)
-z, --compression compression level for gzip output (1 ~ 9). 1 is fastest, 9 is smallest, default is 4. (int [=4])
--stdin input from STDIN. If the STDIN is interleaved paired-end FASTQ, please also add --interleaved_in.
--stdout stream passing-filters reads to STDOUT. This option will result in interleaved FASTQ output for paired-end output. Disabled by default.
--interleaved_in indicate that <in1> is an interleaved FASTQ which contains both read1 and read2. Disabled by default.
--reads_to_process specify how many reads/pairs to be processed. Default 0 means process all reads. (int [=0])
--dont_overwrite don't overwrite existing files. Overwritting is allowed by default.
--fix_mgi_id the MGI FASTQ ID format is not compatible with many BAM operation tools, enable this option to fix it.
-V, --verbose output verbose log information (i.e. when every 1M reads are processed).
-A, --disable_adapter_trimming adapter trimming is enabled by default. If this option is specified, adapter trimming is disabled
-a, --adapter_sequence the adapter for read1. For SE data, if not specified, the adapter will be auto-detected. For PE data, this is used if R1/R2 are found not overlapped. (string [=auto])
--adapter_sequence_r2 the adapter for read2 (PE data only). This is used if R1/R2 are found not overlapped. If not specified, it will be the same as <adapter_sequence> (string [=auto])
--adapter_fasta specify a FASTA file to trim both read1 and read2 (if PE) by all the sequences in this FASTA file (string [=])
--detect_adapter_for_pe by default, the auto-detection for adapter is for SE data input only, turn on this option to enable it for PE data.
-f, --trim_front1 trimming how many bases in front for read1, default is 0 (int [=0])
-t, --trim_tail1 trimming how many bases in tail for read1, default is 0 (int [=0])
-b, --max_len1 if read1 is longer than max_len1, then trim read1 at its tail to make it as long as max_len1. Default 0 means no limitation (int [=0])
-F, --trim_front2 trimming how many bases in front for read2. If it's not specified, it will follow read1's settings (int [=0])
-T, --trim_tail2 trimming how many bases in tail for read2. If it's not specified, it will follow read1's settings (int [=0])
-B, --max_len2 if read2 is longer than max_len2, then trim read2 at its tail to make it as long as max_len2. Default 0 means no limitation. If it's not specified, it will follow read1's settings (int [=0])
-D, --dedup enable deduplication to drop the duplicated reads/pairs
--dup_calc_accuracy accuracy level to calculate duplication (1~6), higher level uses more memory (1G, 2G, 4G, 8G, 16G, 24G). Default 1 for no-dedup mode, and 3 for dedup mode. (int [=0])
--dont_eval_duplication don't evaluate duplication rate to save time and use less memory.
-g, --trim_poly_g force polyG tail trimming, by default trimming is automatically enabled for Illumina NextSeq/NovaSeq data
--poly_g_min_len the minimum length to detect polyG in the read tail. 10 by default. (int [=10])
-G, --disable_trim_poly_g disable polyG tail trimming, by default trimming is automatically enabled for Illumina NextSeq/NovaSeq data
-x, --trim_poly_x enable polyX trimming in 3' ends.
--poly_x_min_len the minimum length to detect polyX in the read tail. 10 by default. (int [=10])
-5, --cut_front move a sliding window from front (5') to tail, drop the bases in the window if its mean quality < threshold, stop otherwise.
-3, --cut_tail move a sliding window from tail (3') to front, drop the bases in the window if its mean quality < threshold, stop otherwise.
-r, --cut_right move a sliding window from front to tail, if meet one window with mean quality < threshold, drop the bases in the window and the right part, and then stop.
-W, --cut_window_size the window size option shared by cut_front, cut_tail or cut_sliding. Range: 1~1000, default: 4 (int [=4])
-M, --cut_mean_quality the mean quality requirement option shared by cut_front, cut_tail or cut_sliding. Range: 1~36 default: 20 (Q20) (int [=20])
--cut_front_window_size the window size option of cut_front, default to cut_window_size if not specified (int [=4])
--cut_front_mean_quality the mean quality requirement option for cut_front, default to cut_mean_quality if not specified (int [=20])
--cut_tail_window_size the window size option of cut_tail, default to cut_window_size if not specified (int [=4])
--cut_tail_mean_quality the mean quality requirement option for cut_tail, default to cut_mean_quality if not specified (int [=20])
--cut_right_window_size the window size option of cut_right, default to cut_window_size if not specified (int [=4])
--cut_right_mean_quality the mean quality requirement option for cut_right, default to cut_mean_quality if not specified (int [=20])
-Q, --disable_quality_filtering quality filtering is enabled by default. If this option is specified, quality filtering is disabled
-q, --qualified_quality_phred the quality value that a base is qualified. Default 15 means phred quality >=Q15 is qualified. (int [=15])
-u, --unqualified_percent_limit how many percents of bases are allowed to be unqualified (0~100). Default 40 means 40% (int [=40])
-n, --n_base_limit if one read's number of N base is >n_base_limit, then this read/pair is discarded. Default is 5 (int [=5])
-e, --average_qual if one read's average quality score <avg_qual, then this read/pair is discarded. Default 0 means no requirement (int [=0])
-L, --disable_length_filtering length filtering is enabled by default. If this option is specified, length filtering is disabled
-l, --length_required reads shorter than length_required will be discarded, default is 15. (int [=15])
--length_limit reads longer than length_limit will be discarded, default 0 means no limitation. (int [=0])
-y, --low_complexity_filter enable low complexity filter. The complexity is defined as the percentage of base that is different from its next base (base[i] != base[i+1]).
-Y, --complexity_threshold the threshold for low complexity filter (0~100). Default is 30, which means 30% complexity is required. (int [=30])
--filter_by_index1 specify a file contains a list of barcodes of index1 to be filtered out, one barcode per line (string [=])
--filter_by_index2 specify a file contains a list of barcodes of index2 to be filtered out, one barcode per line (string [=])
--filter_by_index_threshold the allowed difference of index barcode for index filtering, default 0 means completely identical. (int [=0])
-c, --correction enable base correction in overlapped regions (only for PE data), default is disabled
--overlap_len_require the minimum length to detect overlapped region of PE reads. This will affect overlap analysis based PE merge, adapter trimming and correction. 30 by default. (int [=30])
--overlap_diff_limit the maximum number of mismatched bases to detect overlapped region of PE reads. This will affect overlap analysis based PE merge, adapter trimming and correction. 5 by default. (int [=5])
--overlap_diff_percent_limit the maximum percentage of mismatched bases to detect overlapped region of PE reads. This will affect overlap analysis based PE merge, adapter trimming and correction. Default 20 means 20%. (int [=20])
-U, --umi enable unique molecular identifier (UMI) preprocessing
--umi_loc specify the location of UMI, can be (index1/index2/read1/read2/per_index/per_read, default is none (string [=])
--umi_len if the UMI is in read1/read2, its length should be provided (int [=0])
--umi_prefix if specified, an underline will be used to connect prefix and UMI (i.e. prefix=UMI, UMI=AATTCG, final=UMI_AATTCG). No prefix by default (string [=])
--umi_skip if the UMI is in read1/read2, fastp can skip several bases following UMI, default is 0 (int [=0])
--umi_delim delimiter to use between the read name and the UMI, default is : (string [=:])
-p, --overrepresentation_analysis enable overrepresented sequence analysis.
-P, --overrepresentation_sampling one in (--overrepresentation_sampling) reads will be computed for overrepresentation analysis (1~10000), smaller is slower, default is 20. (int [=20])
-j, --json the json format report file name (string [=fastp.json])
-h, --html the html format report file name (string [=fastp.html])
-R, --report_title should be quoted with ' or ", default is "fastp report" (string [=fastp report])
-w, --thread worker thread number, default is 3 (int [=3])
-s, --split split output by limiting total split file number with this option (2~999), a sequential number prefix will be added to output name ( 0001.out.fq, 0002.out.fq...), disabled by default (int [=0])
-S, --split_by_lines split output by limiting lines of each file with this option(>=1000), a sequential number prefix will be added to output name ( 0001.out.fq, 0002.out.fq...), disabled by default (long [=0])
-d, --split_prefix_digits the digits for the sequential number padding (1~10), default is 4, so the filename will be padded as 0001.xxx, 0 to disable padding (int [=4])
--cut_by_quality5 DEPRECATED, use --cut_front instead.
--cut_by_quality3 DEPRECATED, use --cut_tail instead.
--cut_by_quality_aggressive DEPRECATED, use --cut_right instead.
--discard_unmerged DEPRECATED, no effect now, see the introduction for merging.
-?, --help print this message
105 changes: 105 additions & 0 deletions src/fastp/script.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,105 @@
#!/bin/bash

## VIASH START
## VIASH END

# disable flags
[[ "$par_disable_adapter_trimming" == "false" ]] && unset par_disable_adapter_trimming
[[ "$par_detect_adapter_for_pe" == "false" ]] && unset par_detect_adapter_for_pe
[[ "$par_merge" == "false" ]] && unset par_merge
[[ "$par_include_unmerged" == "false" ]] && unset par_include_unmerged
[[ "$par_interleaved_in" == "false" ]] && unset par_interleaved_in
[[ "$par_fix_mgi_id" == "false" ]] && unset par_fix_mgi_id
[[ "$par_phred64" == "false" ]] && unset par_phred64
[[ "$par_dont_overwrite" == "false" ]] && unset par_dont_overwrite
[[ "$par_verbose" == "false" ]] && unset par_verbose
[[ "$par_dedup" == "false" ]] && unset par_dedup
[[ "$par_dont_eval_duplication" == "false" ]] && unset par_dont_eval_duplication
[[ "$par_trim_poly_g" == "false" ]] && unset par_trim_poly_g
[[ "$par_disable_trim_poly_g" == "false" ]] && unset par_disable_trim_poly_g
[[ "$par_trim_poly_x" == "false" ]] && unset par_trim_poly_x
[[ "$par_disable_quality_filtering" == "false" ]] && unset par_disable_quality_filtering
[[ "$par_disable_length_filtering" == "false" ]] && unset par_disable_length_filtering
[[ "$par_low_complexity_filter" == "false" ]] && unset par_low_complexity_filter
[[ "$par_umi" == "false" ]] && unset par_umi
[[ "$par_overrepresentation_analysis" == "false" ]] && unset par_overrepresentation_analysis

# run command
fastp \
-i "$par_in1" \
-o "$par_out1" \
${par_in2:+--in2 "${par_in2}"} \
${par_out2:+--out2 "${par_out2}"} \
${par_unpaired1:+--unpaired1 "${par_unpaired1}"} \
${par_unpaired2:+--unpaired2 "${par_unpaired2}"} \
${par_failed_out:+--failed_out "${par_failed_out}"} \
${par_overlapped_out:+--overlapped_out "${par_overlapped_out}"} \
${par_json:+--json "${par_json}"} \
${par_html:+--html "${par_html}"} \
${par_report_title:+--report_title "${par_report_title}"} \
${par_disable_adapter_trimming:+--disable_adapter_trimming} \
${par_detect_adapter_for_pe:+--detect_adapter_for_pe} \
${par_adapter_sequence:+--adapter_sequence "${par_adapter_sequence}"} \
${par_adapter_sequence_r2:+--adapter_sequence_r2 "${par_adapter_sequence_r2}"} \
${par_adapter_fasta:+--adapter_fasta "${par_adapter_fasta}"} \
${par_trim_front1:+--trim_front1 "${par_trim_front1}"} \
${par_trim_tail1:+--trim_tail1 "${par_trim_tail1}"} \
${par_max_len1:+--max_len1 "${par_max_len1}"} \
${par_trim_front2:+--trim_front2 "${par_trim_front2}"} \
${par_trim_tail2:+--trim_tail2 "${par_trim_tail2}"} \
${par_max_len2:+--max_len2 "${par_max_len2}"} \
${par_merge:+--merge} \
${par_merged_out:+--merged_out "${par_merged_out}"} \
${par_include_unmerged:+--include_unmerged} \
${par_interleaved_in:+--interleaved_in} \
${par_fix_mgi_id:+--fix_mgi_id} \
${par_phred64:+--phred64} \
${par_compression:+--compression "${par_compression}"} \
${par_dont_overwrite:+--dont_overwrite} \
${par_verbose:+--verbose} \
${par_reads_to_process:+--reads_to_process "${par_reads_to_process}"} \
${par_dedup:+--dedup} \
${par_dup_calc_accuracy:+--dup_calc_accuracy "${par_dup_calc_accuracy}"} \
${par_dont_eval_duplication:+--dont_eval_duplication} \
${par_trim_poly_g:+--trim_poly_g} \
${par_poly_g_min_len:+--poly_g_min_len "${par_poly_g_min_len}"} \
${par_disable_trim_poly_g:+--disable_trim_poly_g} \
${par_trim_poly_x:+--trim_poly_x} \
${par_poly_x_min_len:+--poly_x_min_len "${par_poly_x_min_len}"} \
${par_cut_front:+--cut_front "${par_cut_front}"} \
${par_cut_tail:+--cut_tail "${par_cut_tail}"} \
${par_cut_right:+--cut_right "${par_cut_right}"} \
${par_cut_window_size:+--cut_window_size "${par_cut_window_size}"} \
${par_cut_mean_quality:+--cut_mean_quality "${par_cut_mean_quality}"} \
${par_cut_front_window_size:+--cut_front_window_size "${par_cut_front_window_size}"} \
${par_cut_front_mean_quality:+--cut_front_mean_quality "${par_cut_front_mean_quality}"} \
${par_cut_tail_window_size:+--cut_tail_window_size "${par_cut_tail_window_size}"} \
${par_cut_tail_mean_quality:+--cut_tail_mean_quality "${par_cut_tail_mean_quality}"} \
${par_cut_right_window_size:+--cut_right_window_size "${par_cut_right_window_size}"} \
${par_cut_right_mean_quality:+--cut_right_mean_quality "${par_cut_right_mean_quality}"} \
${par_disable_quality_filtering:+--disable_quality_filtering} \
${par_qualified_quality_phred:+--qualified_quality_phred "${par_qualified_quality_phred}"} \
${par_unqualified_percent_limit:+--unqualified_percent_limit "${par_unqualified_percent_limit}"} \
${par_n_base_limit:+--n_base_limit "${par_n_base_limit}"} \
${par_average_qual:+--average_qual "${par_average_qual}"} \
${par_disable_length_filtering:+--disable_length_filtering} \
${par_length_required:+--length_required "${par_length_required}"} \
${par_length_limit:+--length_limit "${par_length_limit}"} \
${par_low_complexity_filter:+--low_complexity_filter} \
${par_complexity_threshold:+--complexity_threshold "${par_complexity_threshold}"} \
${par_filter_by_index1:+--filter_by_index1 "${par_filter_by_index1}"} \
${par_filter_by_index2:+--filter_by_index2 "${par_filter_by_index2}"} \
${par_filter_by_index_threshold:+--filter_by_index_threshold "${par_filter_by_index_threshold}"} \
${par_correction:+--correction} \
${par_overlap_len_require:+--overlap_len_require "${par_overlap_len_require}"} \
${par_overlap_diff_limit:+--overlap_diff_limit "${par_overlap_diff_limit}"} \
${par_overlap_diff_percent_limit:+--overlap_diff_percent_limit "${par_overlap_diff_percent_limit}"} \
${par_umi:+--umi} \
${par_umi_loc:+--umi_loc "${par_umi_loc}"} \
${par_umi_len:+--umi_len "${par_umi_len}"} \
${par_umi_prefix:+--umi_prefix "${par_umi_prefix}"} \
${par_umi_skip:+--umi_skip "${par_umi_skip}"} \
${par_umi_delim:+--umi_delim "${par_umi_delim}"} \
${par_overrepresentation_analysis:+--overrepresentation_analysis} \
${par_overrepresentation_sampling:+--overrepresentation_sampling "${par_overrepresentation_sampling}"} \
${meta_cpus:+--thread "${meta_cpus}"}
Loading

0 comments on commit b3573a2

Please sign in to comment.