+
+
+
+We've all been here +
+ +A pipeline or process may seem easy or fast when you have 1-3 samples but totally daunting when you have 50. When scaling up you need to consider file overwriting, computational resources, and time. + +One easy way to scale up is to use the array feature in slurm. + +## What is a job array? + +Atlassian says this about job arrays on O2: "Job arrays can be leveraged to quickly submit a number of similar jobs. For example, you can use job arrays to start multiple instances of the same program on different input files, or with different input parameters. A job array is technically one job, but with multiple tasks." [link](https://harvardmed.atlassian.net/wiki/spaces/O2/pages/1586793632/Using+Slurm+Basic#Job-Arrays). + +Array jobs run simultaneously rather than one at a time which means they are very fast! Additionally, running a job array is very simple! + +```bash +sbatch --array=1-10 my_script.sh +``` + +This will run my_script.sh 10 times with the job IDs 1,2,3,4,5,6,7,8,9,10 + +We can also put this directly into the bash script itself (although we will continue with the command line version here). +```bash +$SBATCH --array=1-10 +``` + +We can specify any job IDs we want. + +```bash +sbatch --array=1,7,12 my_script.sh +``` +This will run my_script.sh 3 times with the job IDs 1,7,12 + +Of course we don't want to run the same job on the same input files over and over, that would be pointless. We can use the job IDs within our script to specify different input or output files. In bash the job id is given a special variable `${SLURM_ARRAY_TASK_ID}` + + +## How can I use ${SLURM_ARRAY_TASK_ID}? + +The value of `${SLURM_ARRAY_TASK_ID}` is simply job ID. If I run + +```bash +sbatch --array=1,7 my_script.sh +``` +This will start two jobs, one where `${SLURM_ARRAY_TASK_ID}` is 1 and one where it is 7 + +There are several ways we can use this. If we plan ahead and name our files with these numbers (e.g., sample_1.fastq, sample_2.fastq) we can directly refer to these files in our script: `sample_${SLURM_ARRAY_TASK_ID}.fastq` However, using the ID for input files is often not a great idea as it means you need to strip away most of the information that you might put in these names. + +Instead we can keep our sample names in a separate file and use [awk](awk.md) to pull the file names. + +here is our complete list of long sample names which is found in our file `samples.txt`: + +``` +DMSO_control_day1_rep1 +DMSO_control_day1_rep2 +DMSO_control_day2_rep1 +DMSO_control_day2_rep2 +DMSO_KO_day1_rep1 +DMSO_KO_day1_rep2 +DMSO_KO_day2_rep1 +DMSO_KO_day2_rep2 +Drug_control_day1_rep1 +Drug_control_day1_rep2 +Drug_control_day2_rep1 +Drug_control_day2_rep2 +Drug_KO_day1_rep1 +Drug_KO_day1_rep2 +Drug_KO_day2_rep1 +Drug_KO_day2_rep2 +``` + +If we renamed all of these to 1-16 we would lose a lot of information that may be helpful to have on hand. If these are all sam files and we want to convert them to bam files our script could look like this + +```bash + +file=$(awk -v awkvar="${SLURM_ARRAY_TASK_ID}" 'NR==awkvar' samples.txt) + +samtools view -S -b ${file}.sam > ${file}.bam + +``` + +Since we have sixteen samples we would run this as + +```bash +sbatch --array=1-16 my_script.sh +``` + +So what is this script doing? `file=$(awk -v awkvar="${SLURM_ARRAY_TASK_ID}" 'NR==awkvar' samples.txt)` pulls the line of `samples.txt` that matched the job ID. Then we assign that to a variable called `${file}` and use that to run our command. + +Job IDs can also be helpful for output files or folders. We saw above how we used the job ID to help name our output bam file. But creating and naming folders is helpful in some instances as well. + +```bash + +file=$(awk -v awkvar="${SLURM_ARRAY_TASK_ID}" 'NR==awkvar' samples.txt) + +PREFIX="Folder_${SLURM_ARRAY_TASK_ID}" + mkdir $PREFIX + cd $PREFIX + +samtools view -S -b ../${file}.sam > ${file}.bam + +``` + +This script differs from our previous one in that it makes a folder with the job ID (Folder_1 for job ID 1) then moves inside of it to execute the command. Instead of getting all 16 of our bam files output in a single folder each of them will be in its own folder labled Folder_1 to Folder_16. + +**NOTE** That we define `${file}` BEFORE we move into our new folder as samples.txt is only present in the main directory. + + + + + + diff --git a/Accelerate_with_automation/lessons/loops_and_scripts.md b/Accelerate_with_automation/lessons/loops_and_scripts.md new file mode 100644 index 00000000..cc478953 --- /dev/null +++ b/Accelerate_with_automation/lessons/loops_and_scripts.md @@ -0,0 +1,357 @@ +--- +title: "The Shell: Loops & Scripts" +author: "Bob Freeman, Mary Piper, Radhika Khetani" +--- + +Approximate time: 60 minutes + +## Learning Objectives + +* Capture multiple commands into a script to re-run as one single command +* Understanding variables and storing information in variables +* Learn how to use variables to operate on multiple files + +## Shell scripts + +Within the command-line interface we have at our fingertips, access to various commands which allow you to interrogate your data (i.e `cat`, `less`, `wc`). + +> **NOTE:** If you are unsure about any of these commands and what they do, you may want to review the [Exploring Basics lesson](https://hbctraining.github.io/Training-modules/Intermediate_shell/lessons/exploring_basics.html). + +When you are working with data, it can often be useful to run a set of commands one after another. Further, you may want to re-run this set of commands on every single set of data that you have. Wouldn't it be great if you could do all of this by simply typing out one single command? + +Welcome to the beauty and purpose of shell scripts. + +Shell scripts are **text files that contain commands we want to run**. As with any file, you can give a shell script any name and usually have the extension `.sh`. For historical reasons, a bunch of commands saved in a file is usually called a shell script, but make no mistake, this is actually a small program. + +Since we now know how to create text files in the command-line interface, we are going to use that knowledge to create a shell script and see what makes the shell such a powerful programming environment. We will use commands that you should be familiar with and save them into a file so that we can **re-run all those operations** again later by typing **one single command**. Let's write a shell script that will do two things: + +1. Tell us our current working directory +2. List the contents of the directory + +First open a new file using `vim`: + +```bash +$ vim listing.sh +``` + +Then type in the following lines in the `listing.sh` file: + +``` +echo "Your current working directory is:" +pwd +echo "These are the contents of this directory:" +ls -l +``` + +Exit `vim` and save the file. Now let's run the new script we have created. To run a shell script you usually use the `bash` or `sh` command. + +```bash +$ sh listing.sh +``` + +Now, let's run this script when we are in a different folder. + +```bash +$ cd ../raw_fastq/ + +$ sh ../other/listing.sh +``` + +> Did it work like you expected? +> +> Were the `echo` commands helpful in letting you know what came next? + +This is a very simple shell script. In this lesson, we will be learning how to write more complex ones and show you how to use the power of scripts to make our lives much easier. + +## Bash variables + +A *variable* is a common concept shared by many programming languages. Variables are essentially a symbolic/temporary name for, or a reference to, some information. Variables are analogous to "buckets", where information can be stored, maintained and modified without too much hassle. + +Extending the bucket analogy: the bucket has a name associated with it, i.e. the name of the variable, and when referring to the information in the bucket, we use the name of the bucket, and do not directly refer to the actual data stored in it. + +Let's start with a simple variable that has a single number stored in it: + +```bash +$ num=25 +``` + +*How do we know that we actually created the bash variable?* We can use the `echo` command to print to terminal: + +```bash +$ echo num +``` + +What do you see in the terminal? The `echo` utility takes what arguments you provide and prints to terminal. In this case it interpreted `num` as a a character string and simply printed it back to us. This is because **when trying to retrieve the value stored in the variable, we explicitly use a `$` in front of it**: + +```bash +$ echo $num +``` + +Now you should see the number 25 returned to you. Did you notice that when we created the variable we just typed in the variable name? This is standard shell notation (syntax) for defining and using variables. When defining the variable (i.e. setting the value) you can just type it as is, but when **retrieving the value of a variable don't forget the `$`!** + +Variables can also store a string of character values. In the example below, we define a variable or a 'bucket' called `file`. We will put a filename `Mov10_oe_1.subset.fq` as the value inside the bucket. + +```bash +$ file=Mov10_oe_1.subset.fq +``` + +Once you press return, you should be back at the command prompt. Let's check what's stored inside `file`, but first move into the `raw_fastq` directory:: + +```bash +$ cd ~/unix_lesson/raw_fastq +$ echo $file +``` + +Let's try another command using the variable that we have created. We can also count the number of lines in `Mov10_oe_1.subset.fq` by referencing the `file` variable: + +```bash +$ wc -l $file +``` + +> *NOTE:* The variables we create in a session are system-wide, and independent of where you are in the filesystem. This is why we can reference it from any directory. However, it is only available for your current session. If you exit the close your Terminal and come back again at a later time, the variables you have created will no longer exist. + +*** + +**Exercise** + +* Reuse the `$file` variable to store a different file name, and rerun the commands we ran above (`wc -l`, `echo`) + +*** + +Ok, so we know variables are like buckets, and so far we have seen that bucket filled with a single value. **Variables can store more than just a single value.** They can store multiple values and in this way can be useful to carry out many things at once. Let's create a new variable called `filenames` and this time we will store *all of the filenames* in the `raw_fastq` directory as values. + +To list all the filenames in the directory that have a `.fq` extension, we know the command is: + +```bash +$ ls *.fq +``` + +Now we want to *assign* the output of `ls` to the variable: + +```bash +$ filenames=`ls *.fq` +``` + +> Note the syntax for assigning output of commands to variables, i.e. the backticks around the `ls` command. + +Check and see what's stored inside our newly created variable using `echo`: + +```bash +$ echo $filenames +``` + +Let's try the `wc -l` command again, but this time using our new variable `filenames` as the argument: + +```bash +$ wc -l $filenames +``` + +What just happened? Because our variable contains multiple values, the shell runs the command on each value stored in `filenames` and prints the results to screen. + +*** + +**Exercise** + +* Use some of the other commands you are familiar with (i.e. `head`, `tail`) on the `filenames` variable. + +*** + +## Loops + +Another powerful concept in the Unix shell and useful when writing scripts is the concept of "Loops". We have just shown you that you can run a single command on multiple files by creating a variable whose values are the filenames that you wish to work on. But what if you want to **run a sequence of multiple commands, on multiple files**? This is where loops come in handy! + +Looping is a concept shared by several programming languages, and its implementation in *bash* is very similar to other languages. + +The structure or the syntax of (*for*) loops in bash is as follows: + +```bash +for (variable_name) in (list) +do +(command1 $variable_name) +. +. +done +``` + +where the ***variable_name*** defines (or initializes) a variable that takes the value of every member of the specified ***list*** one at a time. At each iteration, the loop retrieves the value stored in the variable (which is a member of the input list) and runs through the commands indicated between the `do` and `done` one at a time. *This syntax/structure is virtually set in stone.* + + +#### What does this loop do? + +```bash +for x in *.fq + do + echo $x + wc -l $x + done +``` + +Most simply, it writes to the terminal (`echo`) the name of the file and the number of lines (`wc -l`) for each files that end in `.fq` in the current directory. The output is almost identical to what we had before. + +In this case the list of files is specified using the asterisk wildcard: `*.fq`, i.e. all files that end in `.fq`. + +Then, we execute 2 commands between the `do` and `done`. With a loop, we execute these commands for each file at a time. Once the commands are executed for one file, the loop then executes the same commands on the next file in the list. + +Essentially, **the number of items in the list (variable name) == number of times the code will loop through**. + +In our case that is 6 times since we have 6 files in `~/unix_lesson/raw_fastq` that end in `.fq`, and these filenames are stored in the `filename` variable. + +It doesn't matter what variable name we use in a loop structure, but it is advisable to make it something intuitive. In the long run, it's best to use a name that will help point out a variable's functionality, so your future self will understand what you are thinking now. + +### The `basename` command + +Before we get started on creating more complex scripts, we want to introduce you to a command that will be useful for future shell scripting. The `basename` command is used for extracting the base name of a file, which is accomplished using **string splitting to strip the directory and any suffix from filenames**. Let's try an example, by first moving back to your home directory: + +```bash +$ cd +``` + +Then we will run the `basename` command on one of the FASTQ files. Be sure to specify the path to the file: + +```bash +$ basename ~/unix_lesson/raw_fastq/Mov10_oe_1.subset.fq +``` + +What is returned to you? The filename was split into the path `unix_lesson/raw_fastq/` and the filename `Mov10_oe_1.subset.fq`. The command returns only the filename. Now, suppose we wanted to also trim off the file extension (i.e. remove `.fq` leaving only the file *base name*). We can do this by adding a parameter to the command to specify what string of characters we want trimmed. + +```bash +$ basename ~/unix_lesson/raw_fastq/Mov10_oe_1.subset.fq .fq +``` + +You should now see that only `Mov10_oe_1.subset` is returned. + +*** + +**Exercise** + +* How would you modify the above `basename` command to only return `Mov10_oe_1`? + +*** + +## Automating with Scripts + +Now that you've learned how to use loops and variables, let's put this processing power to work. Imagine, if you will, a script that will run a series of commands that would do the following for us each time we get a new data set: + +- Use the `for` loop to iterate over each FASTQ file +- Generate a prefix to use for naming our output files +- Dump out bad reads into a new file +- Get the count of the number of bad reads and generate a summary for each file + +You might not realize it, but this is something that you now know how to do. Let's get started... + +Rather than doing all of this in the terminal we are going to create a script file with all relevant commands. Move back into `unix_lesson` and use `vim` to create our new script file: + +```bash +$ cd ~/unix_lesson + +$ vim generate_bad_reads_summary.sh +``` + +We always want to start our scripts with a shebang line: + +```bash +#!/bin/bash +``` + +This line is the absolute path to the Bash interpreter on almost all computers. The shebang line ensures that the bash shell interprets the script even if it is executed using a different shell. + +> Your script will still work without the shebang line if you run it with the `sh` or `bash` commands, but it is best practice to have it in your shell script. + +After the shebang line, we enter the commands we want to execute. First, we want to move into our `raw_fastq` directory: + +```bash +# enter directory with raw FASTQs +cd ~/unix_lesson/raw_fastq +``` + +And now we loop over all the FASTQs: + +```bash +# count bad reads for each FASTQ file in our directory +for filename in *.fq +``` + +For each file that we process we can use `basename` to create a variable that will uniquely identify our output file based on where it originated from: + +```bash +do + # create a prefix for all output files + base=`basename $filename .subset.fq` +``` + +and then we execute the following commands for each file: + +```bash + # tell us what file we're working on + echo $filename + + # grab all the bad read records into new file + grep -B1 -A2 NNNNNNNNNN $filename > $base-badreads.fastq +``` + +We'll also count the number of these reads and put that in a new file, using the count flag of `grep`: + +```bash + # grab the number of bad reads and write it to a summary file + grep -cH NNNNNNNNNN $filename > $base-badreads.count.summary +done +``` + +> **NOTE:** If you've noticed, we used a new `grep` flag `-H` above; this flag will report the filename the search was performed on along with the matching string. + +Save and exit `vim`, and voila! You now have a script you can use to assess the quality of all your new datasets. Your finished script, complete with comments, should look like the following: + +```bash +#!/bin/bash + +# enter directory with raw FASTQs +cd ~/unix_lesson/raw_fastq + +# count bad reads for each FASTQ file in our directory +for filename in *.fq +do + + # create a prefix for all output files + base=`basename $filename .subset.fq` + + # tell us what file we're working on + echo $filename + + # grab all the bad read records + grep -B1 -A2 NNNNNNNNNN $filename > $base-badreads.fastq + + # grab the number of bad reads and write it to a summary file + grep -cH NNNNNNNNNN $filename > $base-badreads.count.summary +done + +``` + +To run this script, we simply enter the following command: + +```bash +$ sh generate_bad_reads_summary.sh +``` + +How do we know if the script worked? Take a look inside the `raw_fastq` directory, we should see that for every one of the original FASTQ files we have two associated bad read files. + +```bash +$ ls -l ~/unix_lesson/raw_fastq +``` + +To keep our data organized, let's move all of the bad read files out of the `raw_fastq` directory into a new directory called `other`, and the script to a new directory called `scripts`. + +```bash +$ mv raw_fastq/*bad* other/ + +$ mkdir scripts +$ mv *.sh scripts/ +``` + +--- +*This lesson has been developed by members of the teaching team at the [Harvard Chan Bioinformatics Core (HBC)](http://bioinformatics.sph.harvard.edu/). These are open access materials distributed under the terms of the [Creative Commons Attribution license](https://creativecommons.org/licenses/by/4.0/) (CC BY 4.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.* + +* *The materials used in this lesson were derived from work that is Copyright © Data Carpentry (http://datacarpentry.org/). +All Data Carpentry instructional material is made available under the [Creative Commons Attribution license](https://creativecommons.org/licenses/by/4.0/) (CC BY 4.0).* +* *Adapted from the lesson by Tracy Teal. Original contributors: Paul Wilson, Milad Fatenejad, Sasha Wood and Radhika Khetani for Software Carpentry (http://software-carpentry.org/)* + + diff --git a/Accelerate_with_automation/lessons/positional_params.md b/Accelerate_with_automation/lessons/positional_params.md new file mode 100644 index 00000000..9fbb7c33 --- /dev/null +++ b/Accelerate_with_automation/lessons/positional_params.md @@ -0,0 +1,319 @@ +--- +title: "Introduction to Positional Parameters and Variables" +author: "Emma Berdan" +--- + + +## Learning Objectives: + +* Distinguish between variables and positional parameters +* Recognize variables and positional parameters in code written by someone else +* Implement positional parameters and variables in a bash script +* Integrate for loops and variables + +## What is a variable? + +"A **variable** is a character string to which we assign a value. The value assigned could be a number, text, filename, device, or any other type of data. +A variable is nothing more than a pointer to the actual data. The shell enables you to create, assign, and delete variables.” ([Source](https://www.tutorialspoint.com/unix/unix-using-variables.htm)) + +It is easy to identify a variable in any bash script as they will always have the $ in front of them. Here is my very cleverly named variable: `$Variable` + +## Positional parameters are a special kind of variable + +“A **positional parameter** is an argument specified on the command line, used to launch the current process in a shell. Positional parameter values are stored in a special set of variables maintained by the shell.” ([Source](https://www.computerhope.com/jargon/p/positional-parameter.htm)) + +So rather than a variable that is identified inside the bash script, a positional parameter is given when you run your script. This makes it more flexible as it can be changed without modifying the script itself. + +
+
+