diff --git a/config/templates/hpc.template b/config/templates/hpc.template index 30a3ba10ed8..9bb1657b1eb 100644 --- a/config/templates/hpc.template +++ b/config/templates/hpc.template @@ -98,6 +98,7 @@ markdown_extensions: - def_list - footnotes - abbr + - pymdownx.details - pymdownx.snippets: auto_append: - ./includes/abbreviations.md diff --git a/mkdocs/docs/HPC/linux-tutorial/beyond_the_basics.md b/mkdocs/docs/HPC/linux-tutorial/beyond_the_basics.md index 2810bab0fe5..c48f0822d5a 100644 --- a/mkdocs/docs/HPC/linux-tutorial/beyond_the_basics.md +++ b/mkdocs/docs/HPC/linux-tutorial/beyond_the_basics.md @@ -89,7 +89,7 @@ For example, to see the number of files in a directory, we can pipe the count the number of lines with the `-l` flag). ``` $ ls | wc -l - 42 +42 ``` A common pattern is to pipe the output of a command to `less` so you @@ -561,25 +561,45 @@ overwrite any pragmas present in the script. ## Exercises -1. Create a file that contains this message: "Hello, I am <user>", - where `` is replaced by your username. Don't cheat by using an - editor, use a command to create the file. -2. Use another command to add this line to the same file: "I am on - system <hostname> in directory <current directory>". Words - between `<>` should be replaced with their value (hint: use - environment variables). +??? abstract "Create a file `hello.txt` that contains this message: 'Hello, I am [user]'" + ```bash + echo "Hello, I am $USER" > hello.txt + ``` -3. How many files and directories are in `/tmp`? +??? abstract "Use another command to add this line to the same file: 'I am on system [hostname] in directory [current directory]'" + ```bash + echo "I am on system $(hostname) in directory $(pwd)" >> hello.txt + ``` -4. What's the name of the 5th file/directory in alphabetical order in - `/tmp`? +??? abstract "How many files and directories are in /tmp?" + ```bash + ls /tmp | wc -l + ``` -5. List all files that start with `t` in `/tmp`. +??? abstract "What's the name of the 5th file/directory in alphabetical order in /tmp?" + ```bash + ls /tmp | sort | head -5 | tail -1 + ``` -6. Create a file containing "My home directory <home> is available - using $HOME". `` should be replaced with your home directory, - but `$HOME` should remain as-is. +??? abstract "List all files that start with 't' in /tmp." + ```bash + ls /tmp/t* + ``` -7. How many processes are you currently running? How many are you - allowed to run? Where are they coming from? +??? abstract "Create a file containing 'My home directory [home] is available using $HOME'." + ```bash + echo "My home directory $HOME is available using \$HOME" > home_info.txt + ``` + +??? abstract "How many processes are you currently running? How many are you allowed to run? Where are they coming from?" + ```bash + echo "Number of processes currently running:" + ps -u $USER | wc -l + + echo "Maximum number of processes allowed:" + ulimit -u + + echo "Processes currently running:" + ps -u $USER + ``` \ No newline at end of file diff --git a/mkdocs/docs/HPC/linux-tutorial/common_pitfalls.md b/mkdocs/docs/HPC/linux-tutorial/common_pitfalls.md index fa2cc852955..4c2d58a29cd 100644 --- a/mkdocs/docs/HPC/linux-tutorial/common_pitfalls.md +++ b/mkdocs/docs/HPC/linux-tutorial/common_pitfalls.md @@ -53,6 +53,21 @@ But, this is tedious, and you can prevent errors by simply colouring within the lines and not using spaces in filenames. ### Missing/mistyped environment variables + +A common pitfall is the (accidental) use of non-defined variables. +Contrary to what you may expect, this does *not* result in error +messages, but the variable is considered to be *empty* instead. + +!!! tip + For job scripts, use `set -e -u` to avoid using empty variables accidentally. + + The `-e` option will result in the script getting stopped if any command + fails. + + The `-u` option will result in the script getting stopped if empty + variables are used. (see for a more + detailed explanation and more options) + If you use a command like `rm -r` with environment variables you need to be careful to make sure that the environment variable exists. If you mistype an environment variable then it will resolve into a blank string. diff --git a/mkdocs/docs/HPC/linux-tutorial/getting_started.md b/mkdocs/docs/HPC/linux-tutorial/getting_started.md index 5335b374b28..3b68b06a3b3 100644 --- a/mkdocs/docs/HPC/linux-tutorial/getting_started.md +++ b/mkdocs/docs/HPC/linux-tutorial/getting_started.md @@ -1,18 +1,9 @@ # Getting Started -## Logging in - To get started with the HPC-UGent infrastructure, you need to obtain a -VSC account, see [HPC manual](../account.md). **Keep in mind that you -must keep your private key to yourself!** - -You can look at your public/private key pair as a lock and a key: you -give us the lock (your public key), we put it on the door, and then you -can use your key to open the door and get access to the HPC -infrastructure. **Anyone who has your key can use your VSC account!** +VSC account, see [HPC manual](../account.md). +Details on connecting to the HPC infrastructure are available in -Details on connecting to the HPC infrastructure are available in [HPC -manual connecting section](../connecting.md). ## Getting help @@ -79,9 +70,7 @@ you want to rerun it with different arguments. ## Variables -[//]: # (sec:environment-variables()) - -At the prompt we also have access to shell variables, which have both a +At the prompt we also have access to *shell variables*, which have both a *name* and a *value*. They can be thought of as placeholders for things we need to remember. @@ -96,8 +85,6 @@ $ echo $HOME This prints the value of this variable. -### Defining variables - There are several variables already defined for you when you start your session, such as `$HOME` which contains the path to your home directory. @@ -113,6 +100,12 @@ HOME=/user/home/gent/vsc400/vsc40000 ... ``` +!!! info + In Linux, the pipe operator (`|`) is used to pass output from one command as input to another. + This is known as a pipeline. Here, `env | sort` will take the output of `env` and use it as the input for `sort`. + This can be extremely useful for chaining together commands and processing data. + + You can also use the `grep` command to search for a piece of text. The following command will output all VSC-specific variable names and their values: @@ -121,17 +114,20 @@ and their values: $ env | sort | grep VSC ``` -But we can also define our own. this is done with the -`export` command (note: variables are always all-caps as a -convention): +### Defining variables + + +It is also possible to define your own variables. This is done with the `export` command: ``` $ export MYVARIABLE="value" ``` -It is important you don't include spaces around the `=` -sign. Also note the lack of `$` sign in front of the -variable name. + +!!! note + Notice the lack of the `$` sign and spaces around the `=` sign. + By convention, variables should be all-caps. + If we then do ``` @@ -142,65 +138,12 @@ this will output `value`. Note that the quotes are not included, they were only used when defining the variable to escape potential spaces in the value. -#### Changing your prompt using `$PS1` - -You can change what your prompt looks like by redefining the -special-purpose variable `$PS1`. - -For example: to include the current location in your prompt: -``` -$ export PS1='\w $' -~ $ cd test -~/test $ -``` - -Note that `~` is short representation of your home -directory. - -To make this persistent across session, you can define this custom value -for `$PS1` in your `.profile` startup script: -``` -$ echo 'export PS1="\w $ " ' >> ~/.profile -``` - -### Using non-defined variables - -One common pitfall is the (accidental) use of non-defined variables. -Contrary to what you may expect, this does *not* result in error -messages, but the variable is considered to be *empty* instead. - -This may lead to surprising results, for example: -``` -$ export WORKDIR=/tmp/test -$ pwd -/user/home/gent/vsc400/vsc40000 -$ echo $HOME -/user/home/gent/vsc400/vsc40000 -``` - -To understand what's going on here, see the section on `cd` below. - -The moral here is: **be very careful to not use empty variables -unintentionally**. - -**Tip for job scripts: use `set -e -u` to avoid using empty variables -accidentally.** - -The `-e` option will result in the script getting stopped if -any command fails. - -The `-u` option will result in the script getting stopped if -empty variables are used. (see for -a more detailed explanation and more options) - -More information can be found at -. - ### Restoring your default environment -If you've made a mess of your environment, you shouldn't waste too -much time trying to fix it. Just log out and log in again and you will -be given a pristine environment. +If you've made a mess of your environment, you shouldn't waste too much +time trying to fix it. Just log out and log in again, and you will be +given a pristine environment. + ## Basic system information @@ -225,10 +168,47 @@ Linux gligar01.gligar.os 2.6.32-573.8.1.el6.ug.x86_64 #1 SMP Mon Nov 16 15:12:09 ## Exercises -- Print the full path to your home directory -- Determine the name of the environment variable to your personal scratch directory -- What's the name of the system you\'re logged into? Is it the same for everyone? -- Figure out how to print the value of a variable without including a newline -- How do you get help on using the `man` command? +??? abstract "Print the full path to your home directory" + ```bash + $ echo $HOME + ``` + +??? abstract "Determine the name of the environment variable to your personal scratch directory" + ```bash + $ env | grep SCRATCH + ``` + +??? abstract "What's the name of the system you're logged into? Is it the same for everyone?" + ```bash + $ hostname + ``` + + Not everyone will be logged in to the same node, so the output will differ. + +??? abstract "Figure out how to print the value of a variable without including a newline" + + We can use the `man` command to find relevant information on the echo command with: + + ```bash + $ man echo + ``` + + We find the following line in the manual: + + ``` + -n do not output the trailing newline + ``` + + So we can use the `-n` flag to suppress the newline: + + ```bash + $ echo -n $HOME + ``` + +??? abstract "How do you get help on using the `man` command?" + + ```bash + $ man man + ``` -Next [chapter](navigating.md) teaches you on how to navigate. +Next [chapter](navigating.md) teaches you how to navigate the filesystem. diff --git a/mkdocs/docs/HPC/linux-tutorial/hpc_infrastructure.md b/mkdocs/docs/HPC/linux-tutorial/hpc_infrastructure.md index 2de27c6f5db..d777d4391ef 100644 --- a/mkdocs/docs/HPC/linux-tutorial/hpc_infrastructure.md +++ b/mkdocs/docs/HPC/linux-tutorial/hpc_infrastructure.md @@ -18,7 +18,7 @@ scratch filesystems, which you can share with other members in the VO. Space is limited on the cluster's storage. To check your quota, see section [Pre-defined quota](../running_jobs_with_input_output_data.md#pre-defined-quotas). -To figure out where your quota is being spent, the `du` (isk sage) +To figure out where your quota is being spent, the `du` (**d**isk **u**sage) command can come in useful: ``` $ du -sh test @@ -30,7 +30,7 @@ data are stored, since that will: 1. take a long time -2. result in increased load on the shared storage since (the metadata +2. result in an increased load on the shared storage since (the metadata of) every file in those directories will have to be inspected. ## Modules @@ -60,16 +60,69 @@ Detailed information is available in section ## Exercises -Create and submit a job script that computes the sum of 1-100 using -Python, and prints the numbers to a *unique* output file in -`$VSC_SCRATCH`. - -Hint: `python -c "print(sum(range(1, 101)))"` - -- How many modules are available for Python version 3.6.4? -- How many modules get loaded when you load the `Python/3.6.4-intel-2018a` module? -- Which `cluster` modules are available? - -- What's the full path to your personal home/data/scratch directories? -- Determine how large your personal directories are. -- What's the difference between the size reported by `du -sh $HOME` and by `ls -ld $HOME`? +??? abstract "Create and submit a job script that computes the sum of 1-100 using Python, and prints the result to the standard output." + + We make the following job script: + + ```bash title="jobscript.pbs" + #!/bin/bash + + # Basic parameters + #PBS -N sum_1_to_100 ## Job name + #PBS -l nodes=1:ppn=1 ## 1 node, 1 processor per node + #PBS -l walltime=00:00:15 ## Max time your job will run (no more than 72:00:00) + + module load Python/3.10.4-GCCcore-11.3.0 + + cd $PBS_O_WORKDIR # Change working directory to the location where the job was submmitted + + python -c "print(sum(range(1, 101)))" + ``` + + We optionally switch to a cluster of choice, for example `skitty`: + + ```bashand prints the numbers to a *unique* output file in `$VSC_SCRATCH`. + $ module swap cluster/skitty + ``` + + We submit the job script: + + ```bash + $ qsub jobscript.pbs + ``` + + after some time, two files (`sum_1_to_100.e[JOBID]` and `sum_1_to_100.o[JOBID]`) should appear. + The first one contains the error output, and is empty in this case. + The second one contains the output of the Python command. + + +??? abstract "How many modules are available for Python version 3.10?" + + We can use the `module avail` command to list all available modules. + To filter the list for Python 3.10, we can use `module avail Python/3.10`. + + ```bash + $ module avail Python/3.10 + ``` + +??? abstract "How many modules get loaded when you load the `Python/3.10.4-GCCcore-11.3.0` module?" + + We can use the `module load` command to load the Python module. + After loading the module, we can use the `module list` command to list all loaded modules. + + ```bash + $ module load Python/3.10.4-GCCcore-11.3.0 + $ module list + ``` + + These are the modules the python module depends on. + + +??? abstract "Which `cluster` modules are available?" + + We can use the `module avail` command to list all available modules. + To filter the list for modules with the name `cluster`, we can use `module avail cluster`. + + ```bash + $ module avail cluster + ``` diff --git a/mkdocs/docs/HPC/linux-tutorial/manipulating_files_and_directories.md b/mkdocs/docs/HPC/linux-tutorial/manipulating_files_and_directories.md index 32ed9395d67..d2c3c73bb39 100644 --- a/mkdocs/docs/HPC/linux-tutorial/manipulating_files_and_directories.md +++ b/mkdocs/docs/HPC/linux-tutorial/manipulating_files_and_directories.md @@ -72,56 +72,42 @@ directories as files. ## Removing files: "rm" -Note: there are NO backups, there is no 'trash bin'. If you -remove files/directories, they are gone. ``` -$ rm filename +rm filename ``` -`rm` will remove a file or directory. (`rm -rf directory` will remove -every file inside a given directory). WARNING: files removed will be -lost forever, there are no backups, so beware when using this command! +`rm` will remove a file or directory. (`rm -rf ` will remove every file inside a given directory). -#### Removing a directory: "rmdir" +!!! danger + There are NO backups, there is no 'trash bin'. If you remove files/directories, they are gone. -You can remove directories using `rm -r directory`, however, this is -error-prone and can ruin your day if you make a mistake in typing. To -prevent this type of error, you can remove the contents of a directory -using `rm` and then finally removing the directory with: -``` -$ rmdir directory -``` -## Changing permissions: "chmod" -[//]: # (#sec:chmod) +## Permissions -Every file, directory, and link has a set of permissions. These -permissions consist of permission groups and permission types. The -permission groups are: +Each file and directory has particular *permissions* set on it, which +can be queried using `ls -l`. -1. User - a particular user (account) +For example: -2. Group - a particular group of users (may be user-specific group with - only one member) +``` +$ ls -l afile.txt +-rwxrw-r-- 1 vsc40000 agroup 2929176 Apr 12 13:29 afile.sh +``` -3. Other - other users in the system +Here, the output `-rwxrw-r--` indicates the permissions of the file. It can be broken down into 4 parts: -The permission types are: +| type | permissions user | permissions group | permissions others | +|------------------------------------|-------------------------------|-----------------------|--------------------| +| `-`: is a file (`d` for directory) | `rwx`: can read/write/execute | `rw-`: can read/write | `r--`: can read | -1. Read - For files, this gives permission to read the contents of a - file +In this example, the file `afile.sh` is a regular file, and the owner `vsc40000` has read/write/execute permissions, +users in the group `agroup` have read/write permissions, +and all others only have read permissions. -2. Write - For files, this gives permission to write data to the file. - For directories, it allows users to add or remove files to a - directory. +The default permission settings for new files/directories are determined +by the so-called *umask* setting, and are by default `rw-rw-r--` for files and `rwxrwxr-x` for directories. -3. Execute - For files this gives permission to execute a file as - through it were a script. For directories, it allows users to open - the directory and look at the contents. +## Changing permissions: "chmod" -Any time you run `ls -l` you'll see a familiar line of `-rwx------` or -similar combination of the letters `r`, `w`, `x` and `-` (dashes). These -are the permissions for the file or directory. (See also the -[previous section on permissions](navigating.md#permissions)) ``` $ ls -l total 1 @@ -129,21 +115,7 @@ total 1 drwxr-x---. 2 vsc40000 mygroup 40 Apr 12 15:00 Project_GoldenDragon ``` -Here, we see that `articleTable.csv` is a file (beginning the line with -`-`) has read and write permission for the user `vsc40000` (`rw-`), and read -permission for the group `mygroup` as well as all other users (`r--` and -`r--`). - -The next entry is `Project_GoldenDragon`. We see it is a directory -because the line begins with a `d`. It also has read, write, and execute -permission for the `vsc40000` user (`rwx`). So that user can look into the -directory and add or remove files. Users in the `mygroup` can also look -into the directory and read the files. But they can't add or remove -files (`r-x`). Finally, other users can read files in the directory, but -other users have no permissions to look in the directory at all (`---`). - -Maybe we have a colleague who wants to be able to add files to the -directory. We use `chmod` to change the modifiers to the directory to +We use `chmod` to change the modifiers to the directory to let people in the group write to the directory: ``` $ chmod g+w Project_GoldenDragon @@ -205,13 +177,14 @@ $ ls -lh myfile.gz -rw-r--r--. 1 vsc40000 vsc40000 1.1M Dec 2 11:14 myfile.gz ``` -Note: if you zip a file, the original file will be removed. If you unzip -a file, the compressed file will be removed. To keep both, we send the -data to `stdout` and redirect it to the target file: -``` -$ gzip -c myfile > myfile.gz -$ gunzip -c myfile.gz > myfile -``` +!!! note + If you zip a file, the original file will be removed. If you unzip + a file, the compressed file will be removed. To keep both, we send the + data to `stdout` and redirect it to the target file: + ``` + $ gzip -c myfile > myfile.gz + $ gunzip -c myfile.gz > myfile + ``` ### "zip" and "unzip" @@ -272,26 +245,54 @@ $ tar -c source1 source2 source3 -f tarfile.tar ## Exercises -1. Create a subdirectory in your home directory named `test` containing - a single, empty file named `one.txt`. - -2. Copy `/etc/hostname` into the `test` directory and then check what's - in it. Rename the file to `hostname.txt`. - -3. Make a new directory named `another` and copy the entire `test` - directory to it. `another/test/one.txt` should then be an empty - file. - -4. Remove the `another/test` directory with a single command. - -5. Rename `test` to `test2`. Move `test2/hostname.txt` to your home - directory. - -6. Change the permission of `test2` so only you can access it. - -7. Create an empty job script named `job.sh`, and make it executable. -8. gzip `hostname.txt`, see how much smaller it becomes, then unzip it - again. +??? abstract "Create a subdirectory in your home directory named `test` containing a single, empty file named `one.txt`." + ```bash + mkdir ~/test + touch ~/test/one.txt + ``` + +??? abstract "Copy `/etc/hostname` into the `test` directory and then check what's in it. Rename the file to `hostname.txt`." + ```bash + cp /etc/hostname ~/test/ + cat ~/test/hostname + mv ~/test/hostname ~/test/hostname.txt + ``` + +??? abstract "Make a new directory named `another` and copy the entire `test` directory to it. `another/test/one.txt` should then be an empty file." + ```bash + mkdir ~/another + cp -r ~/test ~/another/ + ``` + +??? abstract "Remove the `another/test` directory with a single command." + ```bash + rm -r ~/another/test + ``` + +??? abstract "Rename `test` to `test2`. Move `test2/hostname.txt` to your home directory." + ```bash + mv ~/test ~/test2 + mv ~/test2/hostname.txt ~/ + ``` + +??? abstract "Change the permission of `test2` so only you can access it." + ```bash + chmod u+rwx ~/test2 # Add read, write, and execute permissions for the user (owner) + chmod go-rwx ~/test2 # Remove read, write, and execute permissions for the group and others + ``` + +??? abstract "Create an empty job script named `job.sh`, and make it executable." + ```bash + touch ~/job.sh + chmod +x ~/job.sh + ``` + +??? abstract "gzip `hostname.txt`, see how much smaller it becomes, then unzip it again." + ```bash + gzip ~/hostname.txt + ls -lh ~/hostname.txt.gz + gunzip ~/hostname.txt.gz + ``` The next [chapter](uploading_files.md) is on uploading files, especially important when using HPC-infrastructure. diff --git a/mkdocs/docs/HPC/linux-tutorial/navigating.md b/mkdocs/docs/HPC/linux-tutorial/navigating.md index 5bbfb7ba326..51442f8ebdf 100644 --- a/mkdocs/docs/HPC/linux-tutorial/navigating.md +++ b/mkdocs/docs/HPC/linux-tutorial/navigating.md @@ -1,111 +1,136 @@ # Navigating -This chapter serves as a guide to navigating within a Linux shell, -giving users essential techniques to traverse directories. A very -important skill. +In its most basic form, the linux file system consists of directories and files. +A directory can contain multiple files and subdirectories. +In Linux, the current and parent directory are respectively denoted by `.` and `..`. -## Current directory: "pwd" and "$PWD" +This chapter serves as a guide to navigating within a Linux shell. -To print the current directory, use `pwd` or `\$PWD`: -``` -$ cd $HOME +## The current directory + +To print the current directory, use the `pwd` (**p**rint **w**orking **d**irectory) command: + +```bash $ pwd -/user/home/gent/vsc400/vsc40000 -$ echo "The current directory is: $PWD" -The current directory is: /user/home/gent/vsc400/vsc40000 +/user/home/gent/vsc400/{{ userid }} ``` +`pwd` prints the working directory starting from the root directory (denoted by `/`). +The root directory is the top-level directory in the filesystem hierarchy. +After that, each directory is separated by a `/`. + +## Absolute vs relative file paths + +An *absolute* filepath starts with `/`, the *root* of the filesystem. + +!!! example + absolute path to your home directory: `/user/home/gent/vsc400/vsc40000`. + +A *relative* path starts from the current directory, and points to +another location up or down the filesystem hierarchy. + +!!! example + `some_directory/one.txt` points to the file `one.txt` that is + located in the subdirectory named `some_directory` of the current + directory. + + `../file.txt` points to the file `file.txt` that is located in the + parent directory of the current directory. + + ## Listing files and directories: "ls" A very basic and commonly used command is `ls`, which can be used to list files and directories. -In its basic usage, it just prints the names of files and directories in -the current directory. For example: -``` +In its basic usage, it just prints the names of files and directories +in the current directory. For example: +```bash $ ls -afile.txt some_directory +afile.txt some_directory ``` -When provided an argument, it can be used to list the contents of a -directory: -``` -$ ls some_directory +When provided an argument, it can be used to list the contents of a directory: +```bash +$ ls some_directory one.txt two.txt ``` -A couple of commonly used options include: +In Linux, the semantics of commands can often be customized by adding *options* (or *flags*). +Options are usually preceded by a `-` character. -- detailed listing using `ls -l`: +Some common options for `ls` are: +- detailed listing using `ls -l`: ``` $ ls -l - total 4224 - -rw-rw-r-- 1 vsc40000 vsc40000 2157404 Apr 12 13:17 afile.txt - drwxrwxr-x 2 vsc40000 vsc40000 512 Apr 12 12:51 some_directory + total 4224 + -rw-rw-r-- 1 vsc40000 vsc40000 2157404 Apr 12 13:17 afile.txt + drwxrwxr-x 2 vsc40000 vsc40000 512 Apr 12 12:51 some_directory ``` -- To print the size information in human-readable form, use the `-h` flag: - +- printing the size information in human-readable form, using the `-h` flag: ``` $ ls -lh - total 4.1M - -rw-rw-r-- 1 vsc40000 vsc40000 2.1M Apr 12 13:16 afile.txt - drwxrwxr-x 2 vsc40000 vsc40000 512 Apr 12 12:51 some_directory - ``` + total 4.1M + -rw-rw-r-- 1 vsc40000 vsc40000 2.1M Apr 12 13:16 afile.txt + drwxrwxr-x 2 vsc40000 vsc40000 512 Apr 12 12:51 some_directory + ``` -- also listing hidden files using the `-a` flag: +- listing hidden files using the `-a` flag: ``` $ ls -lah - total 3.9M - drwxrwxr-x 3 vsc40000 vsc40000 512 Apr 12 13:11 . - drwx------ 188 vsc40000 vsc40000 128K Apr 12 12:41 .. - -rw-rw-r-- 1 vsc40000 vsc40000 1.8M Apr 12 13:12 afile.txt - -rw-rw-r-- 1 vsc40000 vsc40000 0 Apr 12 13:11 .hidden_file.txt - drwxrwxr-x 2 vsc40000 vsc40000 512 Apr 12 12:51 some_directory + total 3.9M + drwxrwxr-x 3 vsc40000 vsc40000 512 Apr 12 13:11 . + drwx------ 188 vsc40000 vsc40000 128K Apr 12 12:41 .. + -rw-rw-r-- 1 vsc40000 vsc40000 1.8M Apr 12 13:12 afile.txt + -rw-rw-r-- 1 vsc40000 vsc40000 0 Apr 12 13:11 .hidden_file.txt + drwxrwxr-x 2 vsc40000 vsc40000 512 Apr 12 12:51 some_directory ``` + + !!! info + In Linux, files and directories starting with a `.` are considered hidden files. -- ordering files by the most recent change using `-rt`: +- ordering files by the most recent change using `-t`: ``` - $ ls -lrth - total 4.0M - drwxrwxr-x 2 vsc40000 vsc40000 512 Apr 12 12:51 some_directory - -rw-rw-r-- 1 vsc40000 vsc40000 2.0M Apr 12 13:15 afile.txt + $ ls -lth + total 4.0M + -rw-rw-r-- 1 vsc40000 vsc40000 2.0M Apr 12 13:15 afile.txt + drwxrwxr-x 2 vsc40000 vsc40000 512 Apr 12 12:51 some_directory ``` -If you try to use `ls` on a file that doesn't exist, you -will get a clear error message: +If you try to use `ls` on a file that doesn't exist, you will get a clear error message: -``` -$ ls nosuchfile +```bash +$ ls nosuchfile ls: cannot access nosuchfile: No such file or directory ``` ## Changing directory: "cd" -To change to a different directory, you can use the `cd` -command: -``` +To change to a different directory, you can use the `cd` command: + +```bash $ cd some_directory ``` -To change back to the previous directory you were in, there's a -shortcut: `cd -` +To change back to the previous directory you were in, use the shortcut: `cd -` -Using `cd` without an argument results in returning back to -your home directory: -``` -$ cd +Using `cd` without an argument results in returning back to your home +directory: + +```bash +$ cd $ pwd -/user/home/gent/vsc400/vsc40000 +/user/home/gent/vsc400/vsc40000 ``` -## Inspecting file type: "file" +## Inspecting files + -The `file` command can be used to inspect what type of file -you're dealing with: +The `file` command can be used to inspect what type of file you're dealing with: ``` $ file afile.txt @@ -115,111 +140,82 @@ $ file some_directory some_directory: directory ``` -## Absolute vs relative file paths - -An *absolute* filepath starts with `/` (or a variable which -value starts with `/`), which is also called the *root* of -the filesystem. - -Example: absolute path to your home directory: -`/user/home/gent/vsc400/vsc40000`. - -A *relative* path starts from the current directory, and points to -another location up or down the filesystem hierarchy. - -Example: `some_directory/one.txt` points to the file -`one.txt` that is located in the subdirectory named -`some_directory` of the current directory. - -There are two special relative paths worth mentioning: - -- `.` is a shorthand for the current directory -- `..` is a shorthand for the parent of the current - directory - -You can also use `..` when constructing relative paths, for -example: - -``` -$ cd $HOME/some_directory -$ ls ../afile.txt -../afile.txt -``` - -## Permissions - -[//]: # (sec:permissions) - -Each file and directory has particular *permissions* set on it, which -can be queried using `ls -l`. - -For example: -``` -$ ls -l afile.txt --rw-rw-r-- 1 vsc40000 agroup 2929176 Apr 12 13:29 afile.txt -``` - -The `-rwxrw-r--` specifies both the type of file -(`-` for files, `d` for directories (see first -character)), and the permissions for user/group/others: - -1. each triple of characters indicates whether the read - (`r`), write (`w`), execute - (`x`) permission bits are set or not -2. the 1st part `rwx` indicates that the *owner* - "vsc40000" of the file has all the rights -3. the 2nd part `rw-` indicates the members of the *group* - "agroup" only have read/write permissions (not execute) -4. the 3rd part `r--` indicates that *other* users only - have read permissions - -The default permission settings for new files/directories are determined -by the so-called *umask* setting, and are by default: - -1. read-write permission on files for user/group (no execute), - read-only for others (no write/execute) -2. read-write-execute permission for directories on user/group, - read/execute-only for others (no write) - -See also [the chmod command](manipulating_files_and_directories.md#changing-permissions---chmod--sec--chmod) -later in this manual. - ## Finding files/directories: "find" `find` will crawl a series of directories and lists files matching given criteria. -For example, to look for the file named `one.txt`: +For example, to look for the file named `one.txt` in the current directory and its subdirectories: + ``` -$ cd $HOME -$ find . -name one.txt -./some_directory/one.txt +$ cd $HOME +find . -name one.txt +./some_directory/one.txt ``` -To look for files using incomplete names, you can use a wildcard -`*`; note that you need to escape the `*` to -avoid that Bash *expands* it into `afile.txt` by adding -double quotes: +To look for files using incomplete names, you can use a wildcard `*`; +note that you need to escape the `*` to avoid that Bash *expands* it +into `afile.txt` by adding double quotes: + ``` $ find . -name "*.txt" -./.hidden_file.txt -./afile.txt +./.hidden_file.txt +./afile.txt ./some_directory/one.txt -./some_directory/two.txt +./some_directory/two.txt ``` -A more advanced use of the `find` command is to use the -`-exec` flag to perform actions on the found file(s), rather -than just printing their paths (see `man find`). +For more advanced uses of the `find` command, use `man find` to display its **man**ual. ## Exercises -- Go to `/tmp`, then back to your home directory. How many - different ways to do this can you come up with? -- When was your home directory created or last changed? -- Determine the name of the last changed file in `/tmp`. -- See how home directories are organised. Can you access the home - directory of other users? +??? abstract "Go to `/tmp`, then back to your home directory. How many different ways to do this can you come up with?" + + Some ways to go from the home directory to `/tmp` and back are: + + 1. Using the `cd -` shortcut: + ```bash + $ cd /tmp + $ cd - + ``` + + 2. Using absolute paths: + ```bash + $ cd /tmp + $ cd /user/home/gent/vsc400/vsc40000 + ``` + + 3. Using the `~` shortcut: + ```bash + $ cd /tmp + $ cd ~ + ``` + + 4. Using variable expansion: + ```bash + $ cd /tmp + $ cd $HOME + ``` + +??? abstract "When was your home directory last changed?" + + To find out when your home directory was last changed, you can use the `ls` command, + along with the `-l` flag to get a detailed listing and the `-a` flag to list hidden files: + ``` + $ ls -al ~ + total 3.9M + drwxrwxr-x 3 vsc40000 vsc40000 512 Apr 12 13:11 . + ... + ``` + + Here, `.` denotes the home directory. The directory was last changed on April 12th at 13:11. + + +??? abstract "What is the name of the last changed file in `/tmp`?" + + The output of the `ls -lt /tmp` command will sort the files in `/tmp` by the time of last change. + Check the first letter of the line to determine if it is a file (`-`) or a directory (`d`). + The first line with `-` is the last changed file in `/tmp`. The [next](manipulating_files_and_directories.md) chapter will teach you how to interact with files and directories. diff --git a/mkdocs/docs/HPC/linux-tutorial/uploading_files.md b/mkdocs/docs/HPC/linux-tutorial/uploading_files.md index 59948c9b063..b5fced76644 100644 --- a/mkdocs/docs/HPC/linux-tutorial/uploading_files.md +++ b/mkdocs/docs/HPC/linux-tutorial/uploading_files.md @@ -89,8 +89,8 @@ For example, to copy a folder with lots of CSV files: $ rsync -rzv testfolder vsc40000@login.hpc.ugent.be:data/ ``` -will copy the folder `testfolder` and its contents to `$VSC_DATA` on the -, assuming the `data` symlink is present in your home directory, see +will copy the folder `testfolder` and its contents to `$VSC_DATA`, +assuming the `data` symlink is present in your home directory, see [symlinks section](uploading_files.md#symlinks-for-datascratch). The `-r` flag means "recursively", the `-z` flag means that compression @@ -112,10 +112,26 @@ See `man rsync` or for more information about rsync. ## Exercises -1. Download the file `/etc/hostname` to your local computer. -2. Upload a file to a subdirectory of your personal `$VSC_DATA` space. +??? abstract "Download the file `/etc/hostname` to your local computer." + ```bash + rsync -rzv vsc40000@login.hpc.ugent.be:/etc/hostname local_folder + ``` -3. Create a file named `hello.txt` and edit it using `nano`. + Don't forget to change `vsc40000` to your own username. + +??? abstract "Upload a file to a subdirectory of your personal `$VSC_DATA` space." + ```bash + rsync -rzv testfolder vsc40000@login.hpc.ugent.be:$VSC_DATA/subdirectory/ + ``` + + Don't forget to change `vsc40000` to your own username. + +??? abstract "Create a file named `hello.txt` and edit it using `nano`." + ```bash + touch hello.txt + nano hello.txt + ``` + The `touch` command creates a new file named `hello.txt`, and the `nano` command opens this file in the nano editor for editing. Now you have a basic understanding, see next [chapter](beyond_the_basics.md) for some more in depth concepts.