Main dev branch #2

yuxuanlan · 2023-04-26T15:39:40Z

Update nextflow pipeline from DSL1 to DSL2.
Update README for current version.
Other changes:

For building the gene level expression matrix, the pipeline now loads a prebuilt transcript to geneID lookup table. A biomart DB is no longer needed.
scqc_from_matrix step used to generate a QC_meanexp_vs_freq.pdf file that is often challenging for computer rendering. The current pipeline converts the PDF to PNG to overcome this issue.

gemygk

@yuxuanlan
Thanks for the PR 👍
I had a quick look and have added my comments to the files I have reviewed. I will check again once you resolve these.

gemygk · 2023-04-26T17:06:05Z

README.md

-* singularity 2.4.2
-
-
+* kallisto 0.44.0


we are going back to an earlier version here for Kallisto (0.45.1 to 0.44.0). Are we sure this is the case? This should be 0.45.1 or above.

Now updated to kallisto 0.48.0

gemygk · 2023-04-26T17:35:35Z

README.md

+        done
+    fi
+done
+```


The bash lines can be converted into a script (python or bash) by taking a sample sheet as input and then moving to the /Scripts, folder for e.g, /Scripts/format_reads.py. The script can then be mentioned here in the README.md

This feature will be included in a later version.

gemygk · 2023-04-26T17:44:12Z

PIP-3144.config

+executor.pollInterval='5 min'
+executor.queueStatInterval='8 min'
+executor.exitReadTimeout='5 min'
+executor.dumpInterval='10 min'


Is this config required as part of the installation? If yes, you can move it to a new sample_data/ folder

gemygk · 2023-04-26T17:51:17Z

README.md

+    --mail-type=ALL [email protected] \
+    --wrap "source /ei/cb/common/Scripts/singlecellQC/v1.0/scqc_reqs-1.0 && \
+    nextflow run scqc_nf.sh -c config_file -with-report -resume"
+```


The software should be under /ei/software/cb location, i.e, /ei/cb/common/Scripts/singlecellQC/v1.0/scqc_reqs-1.0 should be here /ei/software/cb/singlecellQC/1.0/x86_64/bin

gemygk · 2023-04-26T17:53:23Z

README.md

-* mtnamefile - In case of non-human species, .rds file for mitochondrial gene. Leave empty ('') if human.
-    This is a vector in R, containing the list of Ensembl transcript IDs, saved as .rds (using saveRDS())
+    
+* species - Kept for legacy code, this parameter is only used if is 'Hsapiens' or 'Bos_taurus'.


Change: 'if is' > 'if it is'
Also, what about support for 'Mmusculus'?

gemygk · 2023-04-26T17:58:57Z

Scripts/QCreport.Rmd

- * nextflow 19.04.1
- * singularity 2.4.2
+
+  -   kallisto 0.44.0


same as earlier, the Kallisto version requires an update.

gemygk · 2023-04-26T18:10:29Z

scqc_nf.sh

+k_scrape_script='/ei/cb/development/lany/git/singlecellQC/main_dev_branch/Scripts/kallisto_mapping_scrape.R'
+doc_script='/ei/cb/development/lany/git/singlecellQC/main_dev_branch/Scripts/QCreport.Rmd'
+plate_matrix_merge_script='/ei/cb/development/lany/git/singlecellQC/main_dev_branch/Scripts/plate_merge.R'
+tx2g_script='/ei/cb/development/lany/git/singlecellQC/main_dev_branch/Scripts/est_counts_tx2gene.R'


lines 12-17, the path should be changed from
/ei/cb/development/lany/git/singlecellQC/main_dev_branch
to
/ei/cb/common/Scripts/scqc
OR
/ei/software/cb/singlecellQC/1.0/x86_64/bin.

we should not use /ei/cb/development for hosting production scripts as this location will be cleaned up and files will get removed.

scripts are now under /ei/software/cb/singlecellQC/1.0/x86_64/bin/Scripts

gemygk · 2023-04-26T18:15:24Z

scqc_nf.sh

-	singularity exec ${image_scater} Rscript ${plate_matrix_merge_script} ${params.quantificationsoutdir} \'${id_list}\';
-	echo \$(ls -d -1 ${params.quantificationsoutdir}*.* | grep all) > matrix_location.txt
+	singularity exec ${image_doc} Rscript ${plate_matrix_merge_script} ${params.quantificationsoutdir} ${q_merge_col} \'${doc_complete_check}\';
+	ln -s -v -f ${params.quantificationsoutdir}all_plates.tsv all_plates.tsv


I have not run the pipeline myself, but here you are using force (-f, with ln -s -v ) to create a symlink. It will overwrite an existing file. Is that safe in this instance?

Here it overwrites symbolic link file in the nextflow temporary run directory.

gemygk · 2023-04-26T18:43:35Z

scqc_reqs-1.0

+#!/bin/bash
+
+PATH=/ei/software/cb/eipap/2.14.2/x86_64/bin:/ei/software/cb/eimap/2.1.4/x86_64/bin:/ei/software/cb/python_miniconda/4.9.2_py3.8_pap/x86_64/bin:/ei/software/cb/git/2.26.2_CBG/x86_64/bin:/ei/cb/common/Scripts:/tgac/software/production/bin/core/../..//hpccore/5/x86_64/bin:/usr/users/ga002/tgacpap/bin:/ei/software/cb/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/tgac/software/production/bin:/tgac/software/production/libraries/bin \
+source nextflow-22.04.0_CBG


We do not need this file here. The PATH's are from PAP node, with eipap and eimap.

This should be part of the wrapper and sit under /ei/software/cb location.

cd /ei/software/cb/bin cat > singlecellQC-1.0 #!/bin/bash tool="singlecellQC/1.0" location="/ei/software/cb" echo "${tool} is sourced from ${location} location" source nextflow-22.04.0_CBG echo "Usage:" echo " nextflow run scqc_nf.sh -c config_file" export PATH=/ei/software/cb/singlecellQC/1.0/x86_64/bin:$PATH

And scqc_nf.sh should be located under /ei/software/cb/singlecellQC/1.0/x86_64/bin

…atics/singlecellQC into main_dev_branch Conflicts: README.md

and p_kal, to fix issue #3.

… Now fastq name is represented as sampleID everywhere in the pipeline output, including kallisto_quant dir and tsv files.

yuxuanlan added 3 commits April 12, 2023 11:35

Major update: nextflow DSL1 - >DSL2

c279d86

docs: update README for current version

6457f7d

docs: update README, add preprocessing section

27a7464

yuxuanlan requested a review from gemygk April 26, 2023 15:39

gemygk requested changes Apr 26, 2023

View reviewed changes

yuxuanlan added 9 commits December 14, 2023 12:21

Update to scqc-1.1

7c8966a

Merge branch 'main_dev_branch' of https://github.com/EI-CoreBioinform…

adaf98d

…atics/singlecellQC into main_dev_branch Conflicts: README.md

Update with Gemy's comments.

4032f73

minor change to README.md and remove old files.

f79e0e6

style: delete comments and codes no longer needed.

9724564

doc: add hpc sbatch example.

c04a230

fix: move all process outputs to qc_dir/$plate_id, expect quantification

833a952

and p_kal, to fix issue #3.

feat: load sample.csv for input fastq reads (format: sampleId,R1,R2).…

d73c758

… Now fastq name is represented as sampleID everywhere in the pipeline output, including kallisto_quant dir and tsv files.

doc: add sample.csv instruction.

91a2e55

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Main dev branch #2

Main dev branch #2

yuxuanlan commented Apr 26, 2023

gemygk left a comment

gemygk Apr 26, 2023

yuxuanlan Dec 14, 2023

gemygk Apr 26, 2023

yuxuanlan Dec 15, 2023

gemygk Apr 26, 2023

yuxuanlan Dec 14, 2023

gemygk Apr 26, 2023

yuxuanlan Dec 15, 2023

gemygk Apr 26, 2023

yuxuanlan Dec 14, 2023

gemygk Apr 26, 2023

yuxuanlan Dec 14, 2023

gemygk Apr 26, 2023

yuxuanlan Dec 14, 2023

gemygk Apr 26, 2023

yuxuanlan Dec 14, 2023

gemygk Apr 26, 2023

yuxuanlan Dec 14, 2023

Main dev branch #2

Are you sure you want to change the base?

Main dev branch #2

Conversation

yuxuanlan commented Apr 26, 2023

gemygk left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment