Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Refine linux tutorial #647

Open
wants to merge 16 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from 10 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions config/templates/hpc.template
Original file line number Diff line number Diff line change
Expand Up @@ -95,6 +95,7 @@ markdown_extensions:
- def_list
- footnotes
- abbr
- pymdownx.details
- pymdownx.snippets:
auto_append:
- ./includes/abbreviations.md
Expand Down
54 changes: 37 additions & 17 deletions mkdocs/docs/HPC/linux-tutorial/beyond_the_basics.md
Original file line number Diff line number Diff line change
Expand Up @@ -83,7 +83,7 @@ For example, to see the number of files in a directory, we can pipe the
(`stdout`) output of `ls` to `wc` (**w**ord **c**ount, but can also be used to
count the number of lines with the `-l` flag).
<pre><code>$ <b>ls | wc -l</b>
42
42
</code></pre>

A common pattern is to pipe the output of a command to `less` so you
Expand Down Expand Up @@ -521,25 +521,45 @@ overwrite any pragmas present in the script.

## Exercises

1. Create a file that contains this message: "Hello, I am &lt;user&gt;",
where `<user>` is replaced by your username. Don't cheat by using an
editor, use a command to create the file.

2. Use another command to add this line to the same file: "I am on
system &lt;hostname&gt; in directory &lt;current&nbsp;directory&gt;". Words
between `<>` should be replaced with their value (hint: use
environment variables).
??? abstract "Create a file `hello.txt` that contains this message: 'Hello, I am [user]'"
```bash
echo "Hello, I am $USER" > hello.txt
```

3. How many files and directories are in `/tmp`?
??? abstract "Use another command to add this line to the same file: 'I am on system [hostname] in directory [current directory]'"
```bash
echo "I am on system $(hostname) in directory $(pwd)" >> hello.txt
```

4. What's the name of the 5th file/directory in alphabetical order in
`/tmp`?
??? abstract "How many files and directories are in /tmp?"
```bash
ls /tmp | wc -l
```

5. List all files that start with `t` in `/tmp`.
??? abstract "What's the name of the 5th file/directory in alphabetical order in /tmp?"
```bash
ls /tmp | sort | head -5 | tail -1
```

6. Create a file containing "My home directory &lt;home&gt; is available
using $HOME". `<home>` should be replaced with your home directory,
but `$HOME` should remain as-is.
??? abstract "List all files that start with 't' in /tmp."
```bash
ls /tmp/t*
```

7. How many processes are you currently running? How many are you
allowed to run? Where are they coming from?
??? abstract "Create a file containing 'My home directory [home] is available using $HOME'."
```bash
echo "My home directory $HOME is available using \$HOME" > home_info.txt
```

??? abstract "How many processes are you currently running? How many are you allowed to run? Where are they coming from?"
```bash
echo "Number of processes currently running:"
ps -u $USER | wc -l

echo "Maximum number of processes allowed:"
ulimit -u

echo "Processes currently running:"
ps -u $USER
```
23 changes: 23 additions & 0 deletions mkdocs/docs/HPC/linux-tutorial/common_pitfalls.md
Original file line number Diff line number Diff line change
Expand Up @@ -49,6 +49,29 @@ But, this is tedious, and you can prevent errors by simply colouring
within the lines and not using spaces in filenames.

### Missing/mistyped environment variables

A common pitfall is the (accidental) use of non-defined variables.
Contrary to what you may expect, this does *not* result in error
messages, but the variable is considered to be *empty* instead.

This may lead to surprising results, for example:
<pre><code>$ <b>export WORKDIR=/tmp/test</b>
lbarraga marked this conversation as resolved.
Show resolved Hide resolved
$ <b>pwd</b>
/user/home/gent/vsc400/vsc40000
$ <b>echo $HOME</b>
/user/home/gent/vsc400/vsc40000
</code></pre>

!!! tip
For job scripts, use `set -e -u` to avoid using empty variables accidentally.

The `-e` option will result in the script getting stopped if any command
fails.

The `-u` option will result in the script getting stopped if empty
variables are used. (see <https://ss64.com/bash/set.html> for a more
detailed explanation and more options)

If you use a command like `rm -r` with environment variables you need to
be careful to make sure that the environment variable exists. If you
mistype an environment variable then it will resolve into a blank string.
Expand Down
143 changes: 60 additions & 83 deletions mkdocs/docs/HPC/linux-tutorial/getting_started.md
Original file line number Diff line number Diff line change
@@ -1,23 +1,11 @@
Getting Started
===============

Logging in
----------
# Getting Started

To get started with the HPC-UGent infrastructure, you need to obtain a
VSC account, see [HPC manual](../account.md).
**Keep in mind that you must keep your private key to yourself!**

You can look at your public/private key pair as a lock and a key: you
give us the lock (your public key), we put it on the door, and then you
can use your key to open the door and get access to the HPC
infrastructure. **Anyone who has your key can use your VSC account!**

VSC account, see [HPC manual](../account.md).
Details on connecting to the HPC infrastructure are available in
[HPC manual connecting section](../connecting.md).

Getting help
------------
## Getting help

To get help:

Expand All @@ -42,8 +30,7 @@ carefully and try to act on it. Try googling the error first to find any
possible solution, but if you can't come up with something in 15
minutes, don't hesitate to mail <a href="mailto:{{hpcinfo}}">{{hpcinfo}}</a>.

Basic terminal usage
--------------------
## Basic terminal usage

The basic interface is the so-called shell prompt, typically ending with
`$` (for `bash` shells).
Expand Down Expand Up @@ -77,11 +64,11 @@ If for any reason you want to stop a command from executing, press
`Ctrl-C`. For example, if a command is taking too long, or you want to
rerun it with different arguments.

Variables
---------
## Variables

[//]: # (sec:environment-variables())

At the prompt we also have access to shell variables, which have both a
At the prompt we also have access to *shell variables*, which have both a
*name* and a *value*.

They can be thought of as placeholders for things we need to remember.
Expand All @@ -95,8 +82,6 @@ shell variable named `HOME`:

This prints the value of this variable.

### Defining variables

There are several variables already defined for you when you start your
session, such as `$HOME` which contains the path to your home directory.

Expand All @@ -110,19 +95,27 @@ HOME=/user/home/gent/vsc400/vsc40000
...
</code></pre>

!!! info
In Linux, the pipe operator (`|`) is used to pass output from one command as input to another.
This is known as a pipeline. Here, `env | sort` will take the output of `env` and use it as the input for `sort`.
This can be extremely useful for chaining together commands and processing data.

You can also use the `grep` command to search for a piece of text. The
following command will output all VSC-specific variable names and their
values:

<pre><code>$ <b>env | sort | grep VSC</b></code></pre>

But we can also define our own. this is done with the `export` command
(note: variables are always all-caps as a convention):
### Defining variables

It is also possible to define your own variables. this is done with the `export` command:

<pre><code>$ <b>export MYVARIABLE="value"</b></code></pre>

It is important you don't include spaces around the `=` sign. Also note
the lack of `$` sign in front of the variable name.

!!! note
Notice the lack of the `$` sign and spaces around the `=` sign.
By convention, variables should be all-caps.

If we then do
<pre><code>$ <b>echo $MYVARIABLE</b></code></pre>
Expand All @@ -131,56 +124,11 @@ this will output `value`. Note that the quotes are not included, they
were only used when defining the variable to escape potential spaces in
the value.

#### Changing your prompt using `$PS1`

You can change what your prompt looks like by redefining the
special-purpose variable `$PS1`.

For example: to include the current location in your prompt:
<pre><code>$ <b>export PS1='\w $'</b>
~ $ cd test
~/test $
</code></pre>

Note that `~` is short representation of your home directory.

To make this persistent across session, you can define this custom value
for `$PS1` in your `.profile` startup script:
<pre><code>$ <b>echo 'export PS1="\w $ " ' >> ~/.profile</b></code></pre>

### Using non-defined variables

One common pitfall is the (accidental) use of non-defined variables.
Contrary to what you may expect, this does *not* result in error
messages, but the variable is considered to be *empty* instead.

This may lead to surprising results, for example:
<pre><code>$ <b>export WORKDIR=/tmp/test</b>
$ <b>pwd</b>
/user/home/gent/vsc400/vsc40000
$ <b>echo $HOME</b>
/user/home/gent/vsc400/vsc40000
</code></pre>

To understand what's going on here, see the section on `cd` below.

The moral here is: **be very careful to not use empty variables unintentionally**.

**Tip for job scripts: use `set -e -u` to avoid using empty variables accidentally.**

The `-e` option will result in the script getting stopped if any command
fails.

The `-u` option will result in the script getting stopped if empty
variables are used. (see <https://ss64.com/bash/set.html> for a more
detailed explanation and more options)

More information can be found at <http://www.tldp.org/LDP/abs/html/variables.html>.

### Restoring your default environment

If you've made a mess of your environment, you shouldn't waste too much
time trying to fix it. Just log out and log in again and you will be
time trying to fix it. Just log out and log in again, and you will be
given a pristine environment.

Basic system information
Expand All @@ -205,20 +153,49 @@ Linux gligar01.gligar.os 2.6.32-573.8.1.el6.ug.x86_64 #1 SMP Mon Nov 16 15:12:09



Exercises
---------
## Exercises

- Print the full path to your home directory
??? abstract "Print the full path to your home directory"
```bash
$ echo $HOME
```

- Determine the name of the environment variable to your personal
scratch directory
??? abstract "Determine the name of the environment variable to your personal scratch directory"
```bash
$ env | grep SCRATCH
```

- What's the name of the system you're logged into? Is it the same for
everyone?
??? abstract "What's the name of the system you're logged into? Is it the same for everyone?"
```bash
$ hostname
```

Not everyone will be logged in to the same node, so the output will differ.

??? abstract "Figure out how to print the value of a variable without including a newline"

We can use the `man` command to find relevant information on the echo command with:

- Figure out how to print the value of a variable without including a
newline
```bash
$ man echo
```

- How do you get help on using the `man` command?
We find the following line in the manual:

```
-n do not output the trailing newline
```

So we can use the `-n` flag to suppress the newline:

```bash
$ echo -n $HOME
```

??? abstract "How do you get help on using the `man` command?"

```bash
$ man man
```

Next [chapter](navigating.md) teaches you on how to navigate.
Next [chapter](navigating.md) teaches you how to navigate the filesystem.
Loading
Loading