diff --git a/.gitignore b/.gitignore index a60901c..2a3fd83 100644 --- a/.gitignore +++ b/.gitignore @@ -1,3 +1,4 @@ +.DS_Store .token /.quarto/ config.toml diff --git a/_quarto.yml b/_quarto.yml index 15ec833..f7ad0d5 100644 --- a/_quarto.yml +++ b/_quarto.yml @@ -38,7 +38,14 @@ website: - labguide/environment/working_remotely.md - labguide/environment/survey.md - labguide/general_lab_policies.md - - labguide/computing/computing.md + - section: "Computing" + contents: + - labguide/computing/etiquette.md + - section: "Sherlock" + contents: + - labguide/computing/sherlock/access-and-resources.md + - labguide/computing/sherlock/job-submission.md + - labguide/computing/sherlock/data-management.md - section: "Research practices" contents: - labguide/research/human_subjects.md diff --git a/labguide/computing/computing.md b/labguide/computing/etiquette.md similarity index 94% rename from labguide/computing/computing.md rename to labguide/computing/etiquette.md index 931ab73..c5fc1f0 100644 --- a/labguide/computing/computing.md +++ b/labguide/computing/etiquette.md @@ -3,6 +3,7 @@ title: "Computing" format: html: page-layout: full +number-sections: true --- ## Computer security @@ -50,7 +51,7 @@ environment properly: - If you encounter an error in the documentation, then flag it and/or fix it - You can refer to [this website](https://www.sherlock.stanford.edu/docs/overview/introduction/) for more information on Sherlock, and if you have any further - questions you can contact the Sherlock admis via email/Slack or meet + questions you can contact the Sherlock admins via email/Slack or meet during specific Office Hours as specified[here](https://www.sherlock.stanford.edu/docs/overview/introduction/#office-hours). - Each TACC system has its own dedicated user guide that provides substantial information about how @@ -58,4 +59,4 @@ environment properly: - Users of shared systems should clean up directories as projects are completed, removing intermediate files that are not necessary to keep. The storage of redundant data -should be avoided, except where required for backups. \ No newline at end of file +should be avoided, except where required for backups. diff --git a/labguide/computing/sherlock/access-and-resources.md b/labguide/computing/sherlock/access-and-resources.md new file mode 100644 index 0000000..f0f36c1 --- /dev/null +++ b/labguide/computing/sherlock/access-and-resources.md @@ -0,0 +1,90 @@ +## Access and resources + +This section describes getting initial access to Sherlock and monitoring available resources. + +### Compute basics + +#### Acquiring an account and logging in + +If you are a new member of a lab at stanford, you will need to have your PI email Sherlock's support to get your SUNet account configured for use with computing resources. +See the [Sherlock getting started guide](https://www.sherlock.stanford.edu/docs/getting-started/#prerequisites) for details. + +Once you have an account set up with your SUNet ID ``, you can access Sherlock via any SSH client client. +If you are using a UNIX-like system (e.g., MacOS) and you are using terminal to connect to sherlock, a useful resource is to set up an ssh config file. +You can do this by editing or creating the file `~/.ssh/config`, and adding the following lines: + +```{.default filename="~/.ssh/config"} +Host sherlock + HostName login.sherlock.stanford.edu + User + KexAlgorithms +diffie-hellman-group-exchange-sha1 +``` + +Navigating to terminal, you can login to Sherlock using: + +```bash +$ ssh sherlock +``` + +and then follow the remainder of the instructions [Sherlock connection guide](https://www.sherlock.stanford.edu/docs/getting-started/connecting/#credentials). + +#### Storage Monitoring + +The Stanford filesystems have fixed allocations for individuals and groups. +As such, it will be useful for you to be able to determine how much space you/the group have, so that you can optimally manage your resources. +For extended details on storage with Sherlock, check out [Sherlock storage guide](https://www.sherlock.stanford.edu/docs/storage/#quotas-and-limits). + +There are several commands that we find extremely useful for working on Sherlock. +We will go over several of them. + +Sherlock has fixed allocations for the storage of individuals and groups. +As such, you will be required to properly manage your storage allocations, re-allocating data to group-level directories as necessary. + +To check your quotas for your group ``, you can use the `sh_quota` command: + +```default +$ sh_quota ++---------------------------------------------------------------------------+ +| Disk usage for user (group: ) | ++---------------------------------------------------------------------------+ +| Filesystem | volume / limit | inodes / limit | ++---------------------------------------------------------------------------+ + HOME | 9.4GB / 15.0GB [|||||| 62%] | - / - ( -%) + GROUP_HOME | 562.6GB / 1.0TB [||||| 56%] | - / - ( -%) + SCRATCH | 65.0GB / 100.0TB [ 0%] | 143.8K / 20.0M ( 0%) + GROUP_SCRATCH | 172.2GB / 100.0TB [ 0%] | 53.4K / 20.0M ( 0%) + OAK | 30.8TB / 240.0TB [| 12%] | 6.6M / 36.0M ( 18%) ++---------------------------------------------------------------------------+ +``` + +`sh_quota` is a Sherlock-specific command that provides a general overview for all partitions a user has access to. +[Documentation is provided on their wiki](https://www.sherlock.stanford.edu/docs/storage/?h=sh_quota#checking-quotas). +When your home directory begins to get filled, it may be valuable to consider moving files to scratch directories, or group directories. +`HOME`, `GROUP_HOME`, and `OAK` are persistent storage; `*SCRATCH` directories are subject to purging. + +Another useful tool is the disk usage command `du`. +A useful and more interacrtive version of this command is `ncdu`. +To use `ncdu`, add the following line to the bottom of your `~/.bash_profile`, which will load the `ncdu` module each time you log in to Sherlock: + +```bash +ml system ncdu +``` + +and then update your `~/.bash_profile` for the current session with: + +```bash +$ source ~/.bash_profile +``` + +which will allow the current session to load the new module updated in your bash profile for `ncdu`. +For future login sessions, re-sourcing your `~/.bash_profile` is unnecessary. + +In the present and future login sessions, you can access the `ncdu` command via: + +```bash +$ ncdu +``` + +which will launch an interactive window for monitoring directory sizes from the folder specified by ``. +Sherlock [recommends running it in an interactive job](https://www.sherlock.stanford.edu/docs/storage/?h=ncdu#locating-large-directories). +This can be useful when identifying where quota usage is being allocated. diff --git a/labguide/computing/sherlock/data-management.md b/labguide/computing/sherlock/data-management.md new file mode 100644 index 0000000..c6d8320 --- /dev/null +++ b/labguide/computing/sherlock/data-management.md @@ -0,0 +1,49 @@ +## Data management on Sherlock + +This section describes how the lab manages datasets on Sherlock, including setting permissions (i.e., who else in the lab can access the dataset). + +Datasets that are considered to be common lab assets (which includes any new studies within the lab and any openly shared datasets) should be placed into the primary data directory on the relevant filesystem. +Datasets that are in process of acquisition should go into the “inprocess” directory. +Once the dataset is finalized, it should be moved into the “data” directory. + +Once a dataset has been installed in the data directory, it should be changed to be read-only for owner and group, using the following commands: + +```bash +find -type d -exec chmod 550 +find -type f -exec chmod 440 +``` + +Datasets that are temporary, or files generated for analyses that are not intended to be reused or shared, should be placed within the user directory. + +#### Restricting access + +Some data resources cannot be shared across the lab and instead need to be restricted to lab members with Data Usage Agreement (DUA) access. +This can be done via access control lists (ACLs), which allow a resource to be owned by a particular owner/group, but applies an additional more specific set of permissions. +The following can be adapted to restrict ACLs to an appropriate subset of lab members: + +```{.bash filename="protect_access.sh"} +#!/bin/bash + +echo "Using ACLs to restrict folder access on oak for russpold folders" +echo -e "\t https://www.sherlock.stanford.edu/docs/storage/data-sharing/#posix-acls " +sleep 1 +echo +# get user input for directory + user +read -p "Enter the folder path: " dir_name +if [ ! -d "$dir_name" ]; then + echo "Error: ${dir_name} doesn't exist" + exit 1 +fi + +read -p "Enter the username: " user_name + +# set restrictions, repeating once for each desired user +echo -e "Setting restrictions for ${user_name} as rxw for folder: /n ${dir_name}" +setfacl -R -m u:$user_name:rwx $dir_name +setfacl -R -d -m u:$user_name:rwx $dir_name +# repeat the above commands, replacing $user_name with the usernames +# for additional members + +# rm default permissions for the group -- oak_russpold +setfacl -m d::group:oak_russpold:--- $dir_name +``` diff --git a/labguide/computing/sherlock/job-submission.md b/labguide/computing/sherlock/job-submission.md new file mode 100644 index 0000000..7f318c7 --- /dev/null +++ b/labguide/computing/sherlock/job-submission.md @@ -0,0 +1,166 @@ +## Job submission on Sherlock + +::: {.callout-important} +Login nodes are shared among many users and therefore must not be used to run computationally intensive tasks. +Those should be submitted to the scheduler which will dispatch them on compute nodes. +::: + +### Determining where to submit your jobs + +You can check available resources with the `sh_part` command: + +```default +$ sh_part + QUEUE STA FREE TOTAL FREE TOTAL RESORC OTHER MAXJOBTIME CORES NODE GRES + PARTITION TUS CORES CORES NODES NODES PENDNG PENDNG DAY-HR:MN /NODE MEM-GB (COUNT) + normal * 153 1792 0 84 23k 127 7-00:00 20-24 128-191 - + bigmem 29 88 0 2 0 8 1-00:00 32-56 512-3072 - + dev 31 40 0 2 0 0 0-02:00 20 128 - + gpu 47 172 0 8 116 1 7-00:00 20-24 191-256 gpu:4(S:0-1)(2),gpu:4(S:0)(6) +``` +This will list out the compute resources available to you, so that you can determine the optimal resource to use. + + +### Running interactive jobs with `sh_dev` + +`sh_dev` sessions run on dedicated compute nodes, ensuring minimal wait times when you need to access a node for testing script, debug code or any kind of interactive work. +`sh_dev` also provides X11 forwarding via the submission host (typically the login node you're connected to) and can thus be used to run GUI applications. + +Users can [specify `sh_dev` calls](https://www.sherlock.stanford.edu/docs/user-guide/running-jobs/#compute-nodes) with specific memory requests. + +#### Interactive jobs from existing scripts: `salloc` + +If you prefer to submit an existing job script or other executable as an interactive job, you can use the `salloc` command: + +```bash +salloc script.sh +``` + +`salloc` will start a Slurm job and allocate resources for it, but it will not automatically connect you to the allocated node(s). +It will only start a new shell on the same node you launched salloc from, and set up the appropriate `$SLURM_*` environment variables. + +### Submitting to the scheduler + +Most large, long-running jobs on Sherlock should be submitted via the job scheduler. + +Most jobs can be submitted the the scheduler using the `sbatch` command. +Sherlock provides [documentation](https://www.sherlock.stanford.edu/docs/getting-started/submitting/#batch-scripts) for how to generate `sbatch` submission scripts. +There is also an [experimental slurm-o-matic](http://slurm-o-matic.stanford.edu/) tool to help in generating these scripts interactively. + +#### Best practice in submitting jobs + +Best practice before launching large, long-running jobs on Sherlock is to run a short test job to evaluate the time memory requirements. +The basic idea is to run a small test job with minimal resource requirements---so the job will run quickly---then re-queue the job with optimized resource requests. + +Jobs can be evaluated along three dimensions: memory, parallelization (i.e., number of nodes and CPUs), and run time. +We briefly highlight why each axis is important below, as well as how to evaluate its requirements. + +- **Memory:** Evaluating memory requirements of completed jobs is straightforward tools such as `sacct` (see @sec-acct). + Requesting excessive memory and not using it will count against your [FairShare score](https://www.sherlock.stanford.edu/docs/glossary/#fairshare). + +- **Parallelization:** Evaluating how well the job performance scales with added CPUs can be done using `seff` (see @sec-seff). + Requesting CPUs then not using them will still count against your [FairShare score](https://www.sherlock.stanford.edu/docs/glossary/#fairshare). + +- **Run Time:** Requesting excess time for your jobs will _not_ count against your [FairShare score](https://www.sherlock.stanford.edu/docs/glossary/#fairshare), but it will affect how quickly the scheduler allocates resources to your job. + +Below, we provide more detailed resources (and example commands) for monitoring jobs. + +#### Monitoring running jobs + +##### Using `squeue` + +Users can monitor the status of their currently running jobs on Sherlock with `squeue`: + +```bash +squeue -u $USER +``` + +This command will monitor how long a job has been sitting on queue or actively running, depending on status. + +##### Using `clush` + +More detailed information on running jobs can be found using [`clush`](https://www.sherlock.stanford.edu/docs/software/using/clustershell/), which can be loaded via the module system: + +```bash +ml system py-clustershell +clush -N -w @job: top -b -n 1 +``` + +On Sherlock, it provides an easy way to run commands on nodes your jobs are running on, and collect back information. +In the above example, we use `clush` to execute the `top` command, which provides real-time monitoring of job resource usage. +Many other uses are possible; for example, + +```bash +clush -w @j: ps -u$USER -o%cpu,rss,cmd +``` + +Will return the CPU and memory usage of all processes for a given job ID. + +#### Example of fMRIPrep job submissions + +One application commonly run within Poldracklab is [`fMRIPrep`](https://fmriprep.org/en/stable/). +Since lab members will often submit many fMRIPrep jobs at the same time, it is best practice to submit these as an [array job](https://rcpedia.stanford.edu/training/10_job_array.html). + +We provide an example fMRIPrep array job script below for running 5 individual subjects. + +```{.bash filename="submit_fmriprep_array.sh"} +#!/bin/bash +# +#SBATCH --job-name=fMRIPrep +#SBATCH --output=fmriprep.%j.out +#SBATCH --time=1-00:00 +#SBATCH --cpus-per-task=16 +#SBATCH --mem-per-cpu=8GB +#SBATCH --mail-user=@stanford.edu +#SBATCH --mail-type=FAIL +#SBATCH --array=1-5 +#SBATCH -p russpold,owners,normal + + +# Define directories + +DATADIR= +SCRATCH= +SIMGDIR= + +# Begin work section +subj_list=(`find $DATADIR -maxdepth 1 -type d -name 'sub-*' -printf '%f\n' | sort -n -ts -k2.1`) +sub="${subj_list[$SLURM_ARRAY_TASK_ID]}" +echo "SUBJECT_ID: " $sub + +singularity run --cleanenv -B ${DATADIR}:/data:ro \ + -B ${SCRATCH}:/out \ + -B ${DATADIR}/license.txt:/license/license.txt:ro \ + ${SIMGDIR}/fmriprep-23-2-0.simg \ + /data /out participant \ + --participant-label ${sub} \ + --output-space MNI152NLin2009cAsym:res-2 \ + -w /out/workdir \ + --notrack \ + --fs-license-file /license/license.txt +``` + +### Evaluating completed job resources + +Regardless of whether your job was run in an interactive (i.e., using `sh_dev`) or non-interactive (i.e., using `sbatch` session), it is useful to evaluate how many resources they consumed after running: + +#### Using `seff` {#sec-seff} + +Nominally, the fastest and easiest way to get a summary report, for a given job, is the “SLURM efficiency” tool, `seff`. +This tool returns a simple, human-readable format report that includes both allocated as well as actually used resources (nodes, CPUs, memory, wall time). + +```bash +seff +``` + +Generally speaking, `seff` reports can be used to determine how well (if at all) a job parallelizes, how much memory to request for future implementations of the job, and how much time to request. +More granular reporting, however, is possible using the `sacct` command. + +#### Using `sacct` {#sec-sacct} + +More rigorous resource analysis can be performed after a job has completed by using SLURM accounting, or `sacct`. +Again, SLURM provides a rigorous documentation, including using `--format=` to define which columns to output and the various options that can constrain a query. + +```bash +sacct --user=$USER --start=2023-09-01 --end=2023-09-03 --format=jobid,jobname,partition,account,nnodes,ncpus,reqmem,maxrss,elapsed,totalcpu,state,reason +```