Skip to content

Latest commit

 

History

History
165 lines (123 loc) · 8.75 KB

README.md

File metadata and controls

165 lines (123 loc) · 8.75 KB
Raw

📚 Snakemake pipeline: genome-reference-builder

This README provides detailed instructions for using the Genome Reference Builder, a Snakemake pipeline that automates the process of genome indexing for organisms such as human and mouse. It integrates tools like STAR, RSEM, and Cell Ranger, streamlining the genome preparation process for downstream analysis.

🌟 Introduction

The Genome Reference Builder pipeline simplifies and automates the creation of genome indices, which are essential for various genomic analyses. This pipeline integrates multiple bioinformatics tools to ensure a smooth and efficient genome indexing process. The pipeline is built around GENCODE annotation data and is currently set to use October 2024 versions:

  • Human: GENCODE Release 47 (GRCh38.p14)
  • Mouse: GENCODE Release M36 (GRCm39)

⚠️ Note: These versions will be used for the foreseeable future in all my personal analyses.

🔄 Pipeline overview

Genome Reference Builder Pipeline

The following flowchart shows the main steps of the Genome Reference Builder pipeline and their interrelationships:

  1. Download reference data: Fetch the latest GENCODE annotation and genome sequence files.
  2. Filter GTF files: Apply filtering criteria to GTF files to retain only relevant gene and transcript types.
  3. Generate genome indices: Create genome indices using various bioinformatics tools such as STAR, HISAT2, Kallisto, Salmon, RSEM, and Cell Ranger.
  4. Validate outputs: Ensure the generated indices are correct and consistent with the input data.

Each step is interconnected, ensuring a streamlined and automated process for preparing genome references.

📋 Prerequisites

Before running the pipeline, ensure that you have installed the following prerequisites:

  • Mamba (Mamba or Micromamba distribution)
  • Snakemake (version 8.24.0 or higher)

Additionally, the pipeline relies on specific versions of bioinformatics software:

Software Version Reference (PMID)
HISAT2 2.2.1 31375807
STAR 2.7.11b 23104886
Kallisto 0.51.1 27043002
Salmon 1.10.3 28263959
RSEM 1.3.3 21816040
Cell Ranger 8.0.1 28091601

⚠️ Note: To ensure consistency between Bulk RNA-seq and Single-cell RNA-seq data analyses, the genome indexes constructed by this pipeline are based on filtered GTF files following the 10x Genomics filtering scheme. When building the reference genome, the following software utilize the filtered GTF files as input: RSEM, STAR, and Cell Ranger.

The Genome Reference Builder pipeline filters GTF files based on specific gene types and transcript tags to ensure high-quality annotations. Below are the detailed criteria used during the filtering process:

Category Criteria Description
Gene Types protein_coding, protein_coding_LoF, lncRNA, IG_C_gene, IG_D_gene, IG_J_gene, IG_LV_gene, IG_V_gene, IG_V_pseudogene, IG_J_pseudogene, IG_C_pseudogene, TR_C_gene, TR_D_gene, TR_J_gene, TR_V_gene, TR_V_pseudogene, TR_J_pseudogene Only genes belonging to these types are retained.
Transcript Types protein_coding, protein_coding_LoF, lncRNA, IG_C_gene, IG_D_gene, IG_J_gene, IG_LV_gene, IG_V_gene, IG_V_pseudogene, IG_J_pseudogene, IG_C_pseudogene, TR_C_gene, TR_D_gene, TR_J_gene, TR_V_gene, TR_V_pseudogene, TR_J_pseudogene Only transcripts of these types are retained.
Excluded Tags readthrough_transcript Transcripts tagged as readthrough are excluded.
Version Suffix Removed from gene_id, transcript_id, and exon_id Ensures consistency by removing version numbers from IDs.

To evaluate the effectiveness of the filtering process, we present a summary of gene counts before and after filtering for both human and mouse GTF files (October 2024 version).

  • Human

    GTF Total Protein-coding lncRNA
    Unfiltered 77,307 20,089 35,045
    Filtered 54,907 19,446 34,818

  • Mouse

    GTF Total Protein-coding lncRNA
    Unfiltered 77,981 21,729 32,947
    Filtered 55,095 21,530 32,873

🛠️ Installation

  1. Install Snakemake using Mamba 🐍: 
    mamba install -c bioconda snakemake
  2. Clone the pipeline repository from GitHub 🐙: 
    git clone [email protected]:catplot/genome-reference-builder.git
  3. Navigate to the pipeline directory 📂: 
    cd genome-reference-builder

⚙️ Configuration

The pipeline uses a configuration file (config.yaml) to specify the reference genomes and indexing tools. This file is included in the repository and is already set up for the October 2024 GENCODE versions of human and mouse genomes. No further modifications to the configuration are required unless you want to adjust specific paths or parameters.

🚀 Usage

There are two ways to run the Genome Reference Builder pipeline: directly or via a SLURM job scheduler.

Option 1: Direct execution

You can directly execute the pipeline using the following command:

snakemake \
    --configfile config/config.yaml \
    --cores 16 \
    --use-conda \
    --rerun-incomplete \
    --keep-going

Option 2: Using SLURM Job Scheduler

To run the pipeline on a SLURM cluster, use the following command to submit jobs to SLURM:

snakemake \
    --configfile config/config.yaml \
    --cores 16 \
    --use-conda \
    --rerun-incomplete \
    --keep-going \
    --cluster "sbatch --job-name=genome-indexing --mem={resources.mem_mb} --cpus-per-task={threads} --time={resources.runtime} --output=logs/snakemake_%j.out --error=logs/snakemake_%j.err" \
    --jobs 10

📂 Output

The pipeline generates indexed genome files essential for downstream genomic analysis. The output files are organized in the directory specified in the configuration file, following this structure:

/datapool/reference_genomes/
└── gencode/
    ├── human/
    │   └── 47/
    │       ├── GRCh38.primary_assembly.genome.fa
    │       ├── GRCh38.primary_assembly.genome.fa.fai
    │       ├── GRCh38.v47.transcripts.fa
    │       ├── GRCh38.v47.primary_assembly.annotation.gtf
    │       ├── index/
    │       │   ├── star/
    │       │   ├── hisat2/
    │       │   ├── kallisto/
    │       │   ├── salmon/
    │       │   ├── rsem/
    │       │   └── cellranger/
    └── mouse/
        └── M36/
            ├── GRCm39.primary_assembly.genome.fa
            ├── GRCm39.primary_assembly.genome.fa.fai
            ├── GRCm39.vM36.transcripts.fa
            ├── GRCm39.vM36.primary_assembly.annotation.gtf
            ├── index/
            │   ├── star/
            │   ├── hisat2/
            │   ├── kallisto/
            │   ├── salmon/
            │   ├── rsem/
            │   └── cellranger/

🐛 Troubleshooting

  • Ensure all dependencies are installed correctly and the paths in your config file are properly set.
  • For issues related to Snakemake execution, refer to the Snakemake documentation 📖.

📞 Further help

For more details or help with specific tools, consult the documentation for each bioinformatics tool used in this pipeline. You can also open an issue on the repository's GitHub page for additional assistance.