From 300e8c2028cc955047dd34cb4852326377bd74fb Mon Sep 17 00:00:00 2001 From: Lukas Barragan Torres Date: Wed, 3 Jul 2024 12:07:49 +0200 Subject: [PATCH 01/74] added missing link to join VO --- mkdocs/docs/HPC/running_jobs_with_input_output_data.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/mkdocs/docs/HPC/running_jobs_with_input_output_data.md b/mkdocs/docs/HPC/running_jobs_with_input_output_data.md index f95ee6e9b6e..ca0d5666b7c 100644 --- a/mkdocs/docs/HPC/running_jobs_with_input_output_data.md +++ b/mkdocs/docs/HPC/running_jobs_with_input_output_data.md @@ -299,7 +299,7 @@ scratch space instead. {% if site == gent %} If you are running out of quota on your -_$VSC_DATA filesystem you can request a VO. See on how to do this. +_$VSC_DATA filesystem you can request to join a VO. See [this section](./#joining-an-existing-vo) on how to do this. {% endif %} ### Your scratch space ($VSC_SCRATCH) From 6ae320f3e728899ebe63bc6bfda8a351c2efff4d Mon Sep 17 00:00:00 2001 From: Lukas Barragan Torres Date: Wed, 3 Jul 2024 13:39:42 +0200 Subject: [PATCH 02/74] remove double sentence --- mkdocs/docs/HPC/running_jobs_with_input_output_data.md | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/mkdocs/docs/HPC/running_jobs_with_input_output_data.md b/mkdocs/docs/HPC/running_jobs_with_input_output_data.md index fc2911996a1..1690d991f63 100644 --- a/mkdocs/docs/HPC/running_jobs_with_input_output_data.md +++ b/mkdocs/docs/HPC/running_jobs_with_input_output_data.md @@ -324,9 +324,7 @@ Each type of scratch has its own use: **Node scratch ($VSC_SCRATCH_NODE).** Every node has its own scratch space, which is completely separated from the other nodes. On some clusters, it will be on a local disk in the node, while on other clusters it will be emulated through - another file server. In many cases, it will be significantly slower - than the cluster scratch as it typically consists of just a single - disk. Some **drawbacks** are that the storage can only be accessed on that + another file server. Some **drawbacks** are that the storage can only be accessed on that particular node and that the capacity is often very limited (e.g., 100 GB). The performance will depend a lot on the particular implementation in the cluster. In many cases, it will be From 83c35b435da026170296628c6f654f5c2921f38b Mon Sep 17 00:00:00 2001 From: Lukas Barragan Torres Date: Wed, 3 Jul 2024 13:45:14 +0200 Subject: [PATCH 03/74] added word 'conda' --- mkdocs/docs/HPC/running_jobs_with_input_output_data.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/mkdocs/docs/HPC/running_jobs_with_input_output_data.md b/mkdocs/docs/HPC/running_jobs_with_input_output_data.md index fc2911996a1..0707c43e68c 100644 --- a/mkdocs/docs/HPC/running_jobs_with_input_output_data.md +++ b/mkdocs/docs/HPC/running_jobs_with_input_output_data.md @@ -429,7 +429,7 @@ needed: If you get an error "*Unknown credential cache type while getting -default ccache*" (or similar) and you use , then please deactivate conda +default ccache*" (or similar) and you use conda, then please deactivate conda before you use the commands in this chapter.
$ conda deactivate

From a0c1fa59ef0cbac61777c01b6cf68843dc965f6c Mon Sep 17 00:00:00 2001
From: Lukas Barragan Torres 
Date: Wed, 3 Jul 2024 14:09:56 +0200
Subject: [PATCH 04/74] --max-depth 1 instead of -s

---
 .../HPC/running_jobs_with_input_output_data.md | 18 ++++++++++--------
 1 file changed, 10 insertions(+), 8 deletions(-)

diff --git a/mkdocs/docs/HPC/running_jobs_with_input_output_data.md b/mkdocs/docs/HPC/running_jobs_with_input_output_data.md
index fc2911996a1..f697f84eab1 100644
--- a/mkdocs/docs/HPC/running_jobs_with_input_output_data.md
+++ b/mkdocs/docs/HPC/running_jobs_with_input_output_data.md
@@ -758,13 +758,14 @@ for a summary of the current directory:
 If you want to see the size of any file or top-level subdirectory in the
 current directory, you could use the following command:
 
-
$ du -s -h *
-1.5M ex01-matlab
-512K ex02-python
-768K ex03-python
-768K ex04-python
-256K example.sh
-1.5M intro-HPC.pdf
+
$ du -h --max-depth 1
+1.5M ./ex01-matlab
+512K ./ex02-python
+768K ./ex03-python
+768K ./ex04-python
+256K ./example.sh
+1.5M ./intro-HPC.pdf
+700M ./.cache
 
Finally, if you don't want to know the size of the data in your current @@ -773,11 +774,12 @@ just pass this directory as a parameter. The command below will show the disk use in your home directory, even if you are currently in a different directory: -
$ du -h $VSC_HOME/*
+
$ du -h --max-depth 1 $VSC_HOME/*
 22M {{ homedir }}/dataset01
 36M {{ homedir }}/dataset02
 22M {{ homedir }}/dataset03
 3.5M {{ homedir }}/primes.txt
+24M {{ homedir }}/.cache
 
{% if site == gent %} From fbf70e9049e88e9b9dcf294c3ca186536b1b2acd Mon Sep 17 00:00:00 2001 From: Lukas Barragan Torres Date: Wed, 3 Jul 2024 14:23:37 +0200 Subject: [PATCH 05/74] remove * from VSC_HOME --- mkdocs/docs/HPC/running_jobs_with_input_output_data.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/mkdocs/docs/HPC/running_jobs_with_input_output_data.md b/mkdocs/docs/HPC/running_jobs_with_input_output_data.md index f697f84eab1..4d7750f77c5 100644 --- a/mkdocs/docs/HPC/running_jobs_with_input_output_data.md +++ b/mkdocs/docs/HPC/running_jobs_with_input_output_data.md @@ -774,7 +774,7 @@ just pass this directory as a parameter. The command below will show the disk use in your home directory, even if you are currently in a different directory: -
$ du -h --max-depth 1 $VSC_HOME/*
+
$ du -h --max-depth 1 $VSC_HOME
 22M {{ homedir }}/dataset01
 36M {{ homedir }}/dataset02
 22M {{ homedir }}/dataset03

From e06c1c9148626ce4f32416015181493cbd634be8 Mon Sep 17 00:00:00 2001
From: Lukas Barragan Torres 
Date: Wed, 3 Jul 2024 14:45:59 +0200
Subject: [PATCH 06/74] add note to copy the example to the home directory

---
 mkdocs/docs/HPC/running_jobs_with_input_output_data.md | 6 ++++++
 1 file changed, 6 insertions(+)

diff --git a/mkdocs/docs/HPC/running_jobs_with_input_output_data.md b/mkdocs/docs/HPC/running_jobs_with_input_output_data.md
index fc2911996a1..24f7cfb7ba0 100644
--- a/mkdocs/docs/HPC/running_jobs_with_input_output_data.md
+++ b/mkdocs/docs/HPC/running_jobs_with_input_output_data.md
@@ -16,6 +16,12 @@ First go to the directory:
 
$ cd ~/{{ exampledir }}
 
+!!! note + If the example directory is not yet present, copy it to your home directory: + +
$ cp -r {{ examplesdir }} ~/
+ + List and check the contents with:
ls -l

From 3507e5c7b7ef4409629f8c57d8ad9adbfe8093ad Mon Sep 17 00:00:00 2001
From: Lukas Barragan Torres 
Date: Wed, 3 Jul 2024 16:39:42 +0200
Subject: [PATCH 07/74] added 'not'

---
 mkdocs/docs/HPC/multi_core_jobs.md | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/mkdocs/docs/HPC/multi_core_jobs.md b/mkdocs/docs/HPC/multi_core_jobs.md
index f5754f69d73..67325745743 100644
--- a/mkdocs/docs/HPC/multi_core_jobs.md
+++ b/mkdocs/docs/HPC/multi_core_jobs.md
@@ -110,7 +110,7 @@ MPI.
     This queues a job that claims 2 nodes and 10 cores.
 
 !!! warning
-    Just requesting more nodes and/or cores does mean that your job will automatically run faster.
+    Just requesting more nodes and/or cores does not mean that your job will automatically run faster.
     You can find more about this [here](troubleshooting.md#job_does_not_run_faster).
 
 ## Parallel Computing with threads

From c15604bc8c1f544338e5e9b13c684e449ab854db Mon Sep 17 00:00:00 2001
From: Lukas Barragan Torres 
Date: Wed, 3 Jul 2024 16:51:14 +0200
Subject: [PATCH 08/74] added note: if example directory is not present, copy
 with this command: ...

---
 mkdocs/docs/HPC/multi_core_jobs.md | 5 +++++
 1 file changed, 5 insertions(+)

diff --git a/mkdocs/docs/HPC/multi_core_jobs.md b/mkdocs/docs/HPC/multi_core_jobs.md
index f5754f69d73..b497fd60a07 100644
--- a/mkdocs/docs/HPC/multi_core_jobs.md
+++ b/mkdocs/docs/HPC/multi_core_jobs.md
@@ -153,6 +153,11 @@ Go to the example directory:
 
$ cd ~/{{ exampledir }}
 
+!!! note + If the example directory is not yet present, copy it to your home directory: + +
$ cp -r {{ examplesdir }} ~/
+ Study the example first:
-- T_hello.c --
From 388a878e38cf968fd41abf3b657afa5f8e29c410 Mon Sep 17 00:00:00 2001 From: Lukas Barragan Torres Date: Wed, 3 Jul 2024 16:59:33 +0200 Subject: [PATCH 09/74] changed the name of the example folder for multi core jobs tutorial --- mkdocs/docs/HPC/multi_core_jobs.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/mkdocs/docs/HPC/multi_core_jobs.md b/mkdocs/docs/HPC/multi_core_jobs.md index f5754f69d73..f5d62a751bc 100644 --- a/mkdocs/docs/HPC/multi_core_jobs.md +++ b/mkdocs/docs/HPC/multi_core_jobs.md @@ -1,4 +1,4 @@ -{% set exampledir = 'examples/Multi_core_jobs_Parallel_Computing' %} +{% set exampledir = 'examples/Multi-core-jobs-Parallel-Computing' %} # Multi core jobs/Parallel Computing From 422d2d6a445dc725a10cbfff29d051c47ebcdbb3 Mon Sep 17 00:00:00 2001 From: Lukas Barragan Torres Date: Wed, 3 Jul 2024 17:09:49 +0200 Subject: [PATCH 10/74] add hyperlink to the compilation page --- mkdocs/docs/HPC/multi_core_jobs.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/mkdocs/docs/HPC/multi_core_jobs.md b/mkdocs/docs/HPC/multi_core_jobs.md index f5754f69d73..61a6578be9d 100644 --- a/mkdocs/docs/HPC/multi_core_jobs.md +++ b/mkdocs/docs/HPC/multi_core_jobs.md @@ -453,7 +453,7 @@ and compile it:
mpiicc is a wrapper of the Intel C++ compiler icc to compile MPI -programs (see the chapter on compilation for details). +programs (see [the chapter on compilation](./compiling_your_software.md) for details). Run the parallel program: From f5b63a1538866c74b526f428001f91faf6fb4200 Mon Sep 17 00:00:00 2001 From: Lukas Barragan Torres Date: Wed, 3 Jul 2024 17:17:03 +0200 Subject: [PATCH 11/74] fixed note indentation --- mkdocs/docs/HPC/multi_core_jobs.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/mkdocs/docs/HPC/multi_core_jobs.md b/mkdocs/docs/HPC/multi_core_jobs.md index f5754f69d73..785300af559 100644 --- a/mkdocs/docs/HPC/multi_core_jobs.md +++ b/mkdocs/docs/HPC/multi_core_jobs.md @@ -525,4 +525,4 @@ absolute amount of concurrency available. {% endif %} !!! tip -If you plan engaging in parallel programming using MPI, this book may prove useful: *Parallel Programming with MPI. Peter Pacheo. Morgan Kaufmann. 1996.* + If you plan engaging in parallel programming using MPI, this book may prove useful: *Parallel Programming with MPI. Peter Pacheo. Morgan Kaufmann. 1996.* From 5f22bc913ff7fab1d961e15067291bf1111d6f02 Mon Sep 17 00:00:00 2001 From: Lukas Barragan Torres Date: Thu, 4 Jul 2024 11:58:24 +0200 Subject: [PATCH 12/74] rephrasing + link to virtual organisations --- mkdocs/docs/HPC/running_jobs_with_input_output_data.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/mkdocs/docs/HPC/running_jobs_with_input_output_data.md b/mkdocs/docs/HPC/running_jobs_with_input_output_data.md index ca0d5666b7c..4b94fef894c 100644 --- a/mkdocs/docs/HPC/running_jobs_with_input_output_data.md +++ b/mkdocs/docs/HPC/running_jobs_with_input_output_data.md @@ -299,7 +299,7 @@ scratch space instead. {% if site == gent %} If you are running out of quota on your -_$VSC_DATA filesystem you can request to join a VO. See [this section](./#joining-an-existing-vo) on how to do this. +_$VSC_DATA filesystem you can join an existing VO, or request a new VO. See [the section about virtual organisations](./#virtual-organisations) on how to do this. {% endif %} ### Your scratch space ($VSC_SCRATCH) From ccb61dbec38e33abebcfff3d2184624399b81645 Mon Sep 17 00:00:00 2001 From: Lukas Barragan Torres Date: Thu, 4 Jul 2024 12:41:58 +0200 Subject: [PATCH 13/74] fixing some minor typo's --- mkdocs/docs/HPC/web_portal.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/mkdocs/docs/HPC/web_portal.md b/mkdocs/docs/HPC/web_portal.md index fff8a25b8e7..9c95a20f72d 100644 --- a/mkdocs/docs/HPC/web_portal.md +++ b/mkdocs/docs/HPC/web_portal.md @@ -132,7 +132,7 @@ Here you can: Dir*) in the current directory; - upload files or directories from your local workstation into - your VSC account, in the currect directory (via *Upload*); + your VSC account, in the correct directory (via *Upload*); - show hidden files and directories, of which the name starts with a dot (`.`) (via *Show Dotfiles*); @@ -169,7 +169,7 @@ Here you can: - use the *Delete* button to (**permanently!**) remove the selected files and directories; -For more information, see aslo +For more information, see also . ### Job management @@ -220,7 +220,7 @@ create new jobs: This extensive interface allows you to create jobs from one of the available templates, or by copying an existing job. -You can carefuly prepare your job and the corresponding job script via +You can carefully prepare your job and the corresponding job script via the *Job Options* button and by editing the job script (see lower right). From 1296df87bd95ca8971ed7d8a3a51c19d4461e9b2 Mon Sep 17 00:00:00 2001 From: Lukas Barragan Torres Date: Thu, 4 Jul 2024 13:06:28 +0200 Subject: [PATCH 14/74] immediatelty -> immediately --- mkdocs/docs/HPC/interactive_debug.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/mkdocs/docs/HPC/interactive_debug.md b/mkdocs/docs/HPC/interactive_debug.md index 62e611e7178..11b3a84dfaf 100644 --- a/mkdocs/docs/HPC/interactive_debug.md +++ b/mkdocs/docs/HPC/interactive_debug.md @@ -4,7 +4,7 @@ The purpose of this cluster is to give the user an environment where there should be no waiting in the queue to get access to a limited -number of resources. This environment allows a user to immediatelty +number of resources. This environment allows a user to immediately start working, and is the ideal place for interactive work such as development, debugging and light production workloads (typically sufficient for training and/or courses). From 16c257c4ce403a7d017035f5e2235f6a8f9b2ce3 Mon Sep 17 00:00:00 2001 From: Lukas Barragan Torres Date: Thu, 4 Jul 2024 16:17:54 +0200 Subject: [PATCH 15/74] fixed typos --- mkdocs/docs/HPC/troubleshooting.md | 18 +++++++++--------- 1 file changed, 9 insertions(+), 9 deletions(-) diff --git a/mkdocs/docs/HPC/troubleshooting.md b/mkdocs/docs/HPC/troubleshooting.md index de7e0d170e2..18ca81bdde4 100644 --- a/mkdocs/docs/HPC/troubleshooting.md +++ b/mkdocs/docs/HPC/troubleshooting.md @@ -10,7 +10,7 @@ More information on this in the subsections below. ### Using multiple cores When you want to speed up your jobs by requesting multiple cores, you also need to use software that is actually capable of using them (and use them efficiently, ideally). -Unless particular a parallel programming paradigm like [OpenMP](https://www.openmp.org/about/openmp-faq/#WhatIs) threading +Unless a particular parallel programming paradigm like [OpenMP](https://www.openmp.org/about/openmp-faq/#WhatIs) threading (shared memory) or [MPI](https://en.wikipedia.org/wiki/Message_Passing_Interface) (distributed memory) is used, software will run sequentially (on a single core). @@ -30,7 +30,7 @@ Other reasons why using more cores may not lead to a (significant) speedup inclu you should *not* expect that doubling the amount of cores will result in a 2x speedup. This is due to the fact that time is needed to create, manage and synchronize the threads/processes. When this "bookkeeping" overhead exceeds the time gained by parallelization, you will not observe any speedup (or even see slower runs). -For example, this can happen when you split you program in too many (tiny) tasks to run in parallel - +For example, this can happen when you split your program in too many (tiny) tasks to run in parallel - creating a thread/process for each task may even take longer than actually running the task itself. - **[Amdahl's Law](https://en.wikipedia.org/wiki/Amdahl%27s_law)** is often used in parallel computing to predict the maximum achievable (theoretical) speedup when using multiple cores. @@ -41,7 +41,7 @@ Regardless of how many cores are devoted to a parallelized execution of this pro So when you reach this theoretical limit, using more cores will not help at all to speed up the computational workload. - **Resource contention:** When two or more threads/processes want to access the same resource, they need to wait on each other - this is called resource contention. -As a result, 1 thread/process will need to wait until the other one is is finished using that resource. +As a result, 1 thread/process will need to wait until the other one is finished using that resource. When each thread uses the same resource, it will definitely run slower than if it doesn't need to wait for other threads to finish. - **Software limitations:** It is possible that the software you are using is just not really optimized for parallelization. @@ -71,7 +71,7 @@ This means that just changing `#PBS -l nodes=1:ppn=10` to `#PBS -l nodes=2:ppn=1 Actually using additional nodes is not as straightforward as merely asking for multiple nodes when submitting your job. The resources on these additional nodes often need to discovered, managed, and synchronized. This introduces complexities in distributing work effectively across the nodes. Luckily, there exist some libraries that do this for you. -Using the resources of multiple nodes is often done using an [Message Passing Interface (MPI)](https://en.wikipedia.org/wiki/Message_Passing_Interface) library. +Using the resources of multiple nodes is often done using a [Message Passing Interface (MPI)](https://en.wikipedia.org/wiki/Message_Passing_Interface) library. MPI allows nodes to communicate and coordinate, but it also introduces additional complexity. An example of how you can make beneficial use of multiple nodes can be found [here](multi_core_jobs.md#parallel-computing-with-mpi). @@ -91,7 +91,7 @@ you should check its documentation for instructions on how to run in parallel, or check for options that control how many threads/cores/nodes can be used. If you can not find any information along those lines, the software you are using can probably only use a single core -and thus requesting multiple cores and/or nodes will only result in wasted sources. +and thus requesting multiple cores and/or nodes will only result in wasted resources. ## Walltime issues @@ -136,7 +136,7 @@ If you have errors that look like: or you are experiencing problems with connecting, here is a list of things to do that should help: -1. Keep in mind that it an take up to an hour for your VSC account to +1. Keep in mind that it can take up to an hour for your VSC account to become active after it has been *approved*; until then, logging in to your VSC account will not work. @@ -267,7 +267,7 @@ and include it in the email. Follow the instructions in [Change PuTTY private key for a saved configuration](../troubleshooting/#change-putty-private-key-for-a-saved-configuration) util item 5, then: -1. Single click on the textbox containig the path to your private key, +1. Single click on the textbox containing the path to your private key, then select all text (push ++"Ctrl"++ + ++"a"++ ), then copy the location of the private key (push ++"Ctrl"++ + ++"c"++) @@ -343,7 +343,7 @@ line 21). To do that, open `~/.ssh/known_hosts` in an editor, and remove the line. This results in `ssh` "forgetting" the system you are connecting to. -Alternatively you can use the command that might shown by the warning under +Alternatively you can use the command that might be shown by the warning under `remove with:` and it should be something like this:
ssh-keygen -f "~/.ssh/known_hosts" -R "{{loginnode}}"
@@ -363,7 +363,7 @@ one of the following fingerprints:
 
 **Do not click "Yes" until you verified the fingerprint. Do not press "No" in any case.**
 
-If it the fingerprint matches, click "Yes".
+If the fingerprint matches, click "Yes".
 
 If it doesn't (like in the example) or you are in doubt, take a screenshot, press "Cancel" and contact {{ hpcinfo }}.
 

From 5059adf8e97484da6ea3d16121a96a9321c3f29c Mon Sep 17 00:00:00 2001
From: Lukas Barragan Torres 
Date: Thu, 4 Jul 2024 16:23:08 +0200
Subject: [PATCH 16/74] removed checkpoints sentence

---
 mkdocs/docs/HPC/troubleshooting.md | 4 +---
 1 file changed, 1 insertion(+), 3 deletions(-)

diff --git a/mkdocs/docs/HPC/troubleshooting.md b/mkdocs/docs/HPC/troubleshooting.md
index de7e0d170e2..7698439713a 100644
--- a/mkdocs/docs/HPC/troubleshooting.md
+++ b/mkdocs/docs/HPC/troubleshooting.md
@@ -104,9 +104,7 @@ If you get from your job output an error message similar to this:
 This occurs when your job did not complete within the requested
 walltime. See
 section on [Specifying Walltime](../fine_tuning_job_specifications/#specifying-walltime) for more information
-about how to request the walltime. It is recommended to use
-*checkpointing* if the job requires **72 hours** of walltime or more to be executed.
-
+about how to request the walltime.
 
 ## Out of quota issues
 

From fbdebc0f70dd4a89466e80c225867fc18e50559f Mon Sep 17 00:00:00 2001
From: Lukas Barragan Torres 
Date: Thu, 4 Jul 2024 17:01:32 +0200
Subject: [PATCH 17/74] added note explaining the difference between storage
 and memory

---
 mkdocs/docs/HPC/troubleshooting.md | 6 ++++++
 1 file changed, 6 insertions(+)

diff --git a/mkdocs/docs/HPC/troubleshooting.md b/mkdocs/docs/HPC/troubleshooting.md
index de7e0d170e2..24acf0c2f03 100644
--- a/mkdocs/docs/HPC/troubleshooting.md
+++ b/mkdocs/docs/HPC/troubleshooting.md
@@ -450,6 +450,12 @@ To avoid jobs allocating too much memory, there are memory limits in
 place by default. It is possible to specify higher memory limits if your
 jobs require this.
 
+!!! note
+
+    Memory is not the same as storage. Memory or RAM is used for temporary, 
+    fast access to data when the program is running, while storage is used for long-term data retention.
+    If you are running into problems because you reached your storage quota, see [Out of quota issues](#out-of-quota-issues).
+
 ### How will I know if memory limits are the cause of my problem?
 
 If your program fails with a memory-related issue, there is a good

From 0b04e45ca136ca2c5d390c8668a203b1a9287abd Mon Sep 17 00:00:00 2001
From: Lukas Barragan Torres 
Date: Fri, 5 Jul 2024 13:17:30 +0200
Subject: [PATCH 18/74] fix typos

---
 mkdocs/docs/HPC/fine_tuning_job_specifications.md | 10 +++++-----
 1 file changed, 5 insertions(+), 5 deletions(-)

diff --git a/mkdocs/docs/HPC/fine_tuning_job_specifications.md b/mkdocs/docs/HPC/fine_tuning_job_specifications.md
index e9629915fea..0bec9630c39 100644
--- a/mkdocs/docs/HPC/fine_tuning_job_specifications.md
+++ b/mkdocs/docs/HPC/fine_tuning_job_specifications.md
@@ -14,7 +14,7 @@ can slow down the run time of your application, but also block {{hpc}} resources
 for other users.
 
 Specifying the "optimal" Job Parameters requires some knowledge of your
-application (e.g., how many parallel threads does my application uses,
+application (e.g., how many parallel threads does my application use,
 is there a lot of inter-process communication, how much memory does my
 application need) and also some knowledge about the {{hpc}} infrastructure
 (e.g., what kind of multi-core processors are available, which nodes
@@ -78,7 +78,7 @@ taken to be on the safe side.
 
 It is also wise to check the walltime on different compute nodes or to
 select the "slowest" compute node for your walltime tests. Your estimate
-should appropriate in case your application will run on the "slowest"
+should be appropriate in case your application will run on the "slowest"
 (oldest) compute nodes.
 
 The walltime can be specified in a job scripts as:
@@ -184,7 +184,7 @@ Whereby:
 3.  The third column shows the memory utilisation, expressed in
     percentages of the full available memory. At full memory
     consumption, 19.2% of the memory was being used by our application.
-    With the *"free"* command, we have previously seen that we had a
+    With the `free` command, we have previously seen that we had a
     node of 16 GB in this example. 3 GB is indeed more or less 19.2% of
     the full available memory.
 4.  The fourth column shows you the CPU utilisation, expressed in
@@ -240,7 +240,7 @@ htop
     horizontally to see all processes and their full command lines.
 
 
$ top
-$ htot
+$ htop
### Setting the memory parameter {: #pbs_mem } @@ -295,7 +295,7 @@ are working at full load. The number of core and nodes that a user shall request fully depends on the architecture of the application. Developers design their -applications with a strategy for parallelisation in mind. The +applications with a strategy for parallelization in mind. The application can be designed for a certain fixed number or for a configurable number of nodes and cores. It is wise to target a specific set of compute nodes (e.g., Westmere, Harpertown) for your computing From 54814884716c0facdfe998d7117248d8d1cdd29c Mon Sep 17 00:00:00 2001 From: Lukas Barragan Torres Date: Fri, 5 Jul 2024 13:25:33 +0200 Subject: [PATCH 19/74] note instead of remark --- mkdocs/docs/HPC/fine_tuning_job_specifications.md | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/mkdocs/docs/HPC/fine_tuning_job_specifications.md b/mkdocs/docs/HPC/fine_tuning_job_specifications.md index e9629915fea..f491e861945 100644 --- a/mkdocs/docs/HPC/fine_tuning_job_specifications.md +++ b/mkdocs/docs/HPC/fine_tuning_job_specifications.md @@ -328,9 +328,10 @@ processor : 5 processor : 6 processor : 7
-Remark: Unless you want information of the login nodes, you'll have to issue -these commands on one of the workernodes. This is most easily achieved -in an interactive job, see the chapter on Running interactive jobs. +!!! note + Unless you want information of the login nodes, you'll have to issue + these commands on one of the workernodes. This is most easily achieved + in an interactive job, see [the chapter on Running interactive jobs](./running_interactive_jobs.md). In order to specify the number of nodes and the number of processors per node in your job script, use: From 57f71c000f64582ed1a32484e6da6a2b855e22ac Mon Sep 17 00:00:00 2001 From: Lukas Barragan Torres Date: Fri, 5 Jul 2024 13:45:46 +0200 Subject: [PATCH 20/74] compile eat_cpu before running --- mkdocs/docs/HPC/fine_tuning_job_specifications.md | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/mkdocs/docs/HPC/fine_tuning_job_specifications.md b/mkdocs/docs/HPC/fine_tuning_job_specifications.md index e9629915fea..43aeb9e0b3a 100644 --- a/mkdocs/docs/HPC/fine_tuning_job_specifications.md +++ b/mkdocs/docs/HPC/fine_tuning_job_specifications.md @@ -515,9 +515,10 @@ The **uptime** command will show us the average load 10:14:05 up 86 days, 12:01, 11 users, load average: 0.60, 0.41, 0.41
-Now, start a few instances of the "*eat_cpu*" program in the background, +Now, compile and start a few instances of the "*eat_cpu*" program in the background, and check the effect on the load again: -
$ ./eat_cpu&
+
$ gcc eat_cpu.c -o eat_cpu
+$ ./eat_cpu&
 $ ./eat_cpu&
 $ ./eat_cpu&
 $ uptime

From d7498aafe2cf7d9bd4c3d4496f6d508d737cbf7d Mon Sep 17 00:00:00 2001
From: Lukas Barragan Torres 
Date: Fri, 5 Jul 2024 15:49:33 +0200
Subject: [PATCH 21/74] added compiler iptimizations

---
 mkdocs/docs/HPC/fine_tuning_job_specifications.md | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/mkdocs/docs/HPC/fine_tuning_job_specifications.md b/mkdocs/docs/HPC/fine_tuning_job_specifications.md
index 43aeb9e0b3a..9d0bad53c3d 100644
--- a/mkdocs/docs/HPC/fine_tuning_job_specifications.md
+++ b/mkdocs/docs/HPC/fine_tuning_job_specifications.md
@@ -517,7 +517,7 @@ The **uptime** command will show us the average load
 
 Now, compile and start a few instances of the "*eat_cpu*" program in the background,
 and check the effect on the load again:
-
$ gcc eat_cpu.c -o eat_cpu
+
$ gcc -O2 eat_cpu.c -o eat_cpu
 $ ./eat_cpu&
 $ ./eat_cpu&
 $ ./eat_cpu&

From ded0f450f8fb2c16e4399491d8cdb910f96e1f28 Mon Sep 17 00:00:00 2001
From: Lukas Barragan Torres 
Date: Fri, 5 Jul 2024 16:44:49 +0200
Subject: [PATCH 22/74] fixed typos in multi job submission docs

---
 mkdocs/docs/HPC/multi_job_submission.md | 9 ++++-----
 1 file changed, 4 insertions(+), 5 deletions(-)

diff --git a/mkdocs/docs/HPC/multi_job_submission.md b/mkdocs/docs/HPC/multi_job_submission.md
index 3f8fa2c7f64..e568ecfd2b5 100644
--- a/mkdocs/docs/HPC/multi_job_submission.md
+++ b/mkdocs/docs/HPC/multi_job_submission.md
@@ -22,7 +22,7 @@ huge amounts of small jobs will create a lot of overhead, and can slow
 down the whole cluster. It would be better to bundle those jobs in
 larger sets. In TORQUE, an experimental feature known as "*job arrays*"
 existed to allow the creation of multiple jobs with one *qsub* command,
-but is was not supported by Moab, the current scheduler.
+but is not supported by Moab, the current scheduler.
 
 The "**Worker framework**" has been developed to address this issue.
 
@@ -50,7 +50,7 @@ scenario that can be reduced to a **MapReduce** approach.[^1]
 First go to the right directory:
 
$ cd ~/examples/Multi-job-submission/par_sweep
-Suppose the program the user wishes to run the "*weather*" program, +Suppose the user wishes to run the "*weather*" program, which takes three parameters: a temperature, a pressure and a volume. A typical call of the program looks like:
$ ./weather -t 20 -p 1.05 -v 4.3
@@ -366,7 +366,7 @@ This will summarise the log file every 60 seconds.
 
 ### Time limits for work items
 
-Sometimes, the execution of a work item takes long than expected, or
+Sometimes, the execution of a work item takes longer than expected, or
 worse, some work items get stuck in an infinite loop. This situation is
 unfortunate, since it implies that work items that could successfully
 execute are not even started. Again, the Worker framework offers a
@@ -392,8 +392,7 @@ it can be used outside the Worker framework as well.
 
 ### Resuming a Worker job
 
-Unfortunately, it is not always easy to estimate the walltime for a job,
-and consequently, sometimes the latter is underestimated. When using the
+Unfortunately, walltime is sometimes underestimated. When using the
 Worker framework, this implies that not all work items will have been
 processed. Worker makes it very easy to resume such a job without having
 to figure out which work items did complete successfully, and which

From cfd997d8acbb2cbed1bf43e81a02ef53b08c5ce7 Mon Sep 17 00:00:00 2001
From: Lukas Barragan Torres 
Date: Fri, 5 Jul 2024 17:23:26 +0200
Subject: [PATCH 23/74] fixed some minor inconsistencies

---
 mkdocs/docs/HPC/compiling_your_software.md | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/mkdocs/docs/HPC/compiling_your_software.md b/mkdocs/docs/HPC/compiling_your_software.md
index 018565bb817..26dcbbcb6b6 100644
--- a/mkdocs/docs/HPC/compiling_your_software.md
+++ b/mkdocs/docs/HPC/compiling_your_software.md
@@ -9,7 +9,7 @@ compiled for {{operatingsystem}}. It also means that you first have to install a
 required external software packages on the {{hpc}}.
 
 Most commonly used compilers are already pre-installed on the {{hpc}} and can be
-used straight away. Also many popular external software packages, which
+used straight away. Also, many popular external software packages, which
 are regularly used in the scientific community, are also pre-installed.
 
 ## Check the pre-installed software on the {{hpc}}
@@ -149,7 +149,7 @@ compilation process, even if they seem unimportant so that a code change
 that produces a warning does not go unnoticed.
 
 Let's test this program on the local compute node, which is at your
-disposal after the "qsub --I" command:
+disposal after the `qsub --I` command:
 
$ ./hello
 Hello #0
 Hello #1

From 0f3798c9284368d1f5709ab3508568efcabbd0e6 Mon Sep 17 00:00:00 2001
From: Lukas Barragan Torres 
Date: Fri, 5 Jul 2024 17:37:45 +0200
Subject: [PATCH 24/74] remove -i option sentence

---
 mkdocs/docs/HPC/sites/gent/available-modules.md | 16 ++++++++--------
 1 file changed, 8 insertions(+), 8 deletions(-)

diff --git a/mkdocs/docs/HPC/sites/gent/available-modules.md b/mkdocs/docs/HPC/sites/gent/available-modules.md
index d9c1770ad7f..e59a9f01c13 100644
--- a/mkdocs/docs/HPC/sites/gent/available-modules.md
+++ b/mkdocs/docs/HPC/sites/gent/available-modules.md
@@ -1,4 +1,4 @@
-
$ module av 2>&1 | more
+
$ module av | more
 --- /apps/gent/SL6/sandybridge/modules/all ---
 ABAQUS/6.12.1-linux-x86_64
 AMOS/3.1.0-ictce-4.0.10
@@ -11,11 +11,11 @@ ASE/3.6.0.2515-ictce-5.5.0-Python-2.7.6
 Or when you want to check whether some specific software, some compiler or some
 application (e.g., MATLAB) is installed on the {{hpc}}.
 
-
$ module av 2>&1 | grep -i -e "matlab"
-MATLAB/2010b
-MATLAB/2012b
-MATLAB/2013b
+
$ module av matlab
+--------------------- /apps/gent/RHEL8/zen2-ib/modules/all ---------------------
+   LIBSVM-MATLAB/3.30-GCCcore-11.3.0-MATLAB-2022b-r5
+   MATLAB/2019b
+   MATLAB/2021b
+   MATLAB/2022b-r5                                   (D)
+   SPM/12.5_r7771-MATLAB-2021b
 
- -As you are not aware of the capitals letters in the module name, we looked for -a case-insensitive name with the "-i" option. From e432b718ebbeb8577b7be7d850d3aeca69ede858 Mon Sep 17 00:00:00 2001 From: Lukas Barragan Torres Date: Fri, 5 Jul 2024 17:44:00 +0200 Subject: [PATCH 25/74] update output module av command --- mkdocs/docs/HPC/sites/gent/available-modules.md | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/mkdocs/docs/HPC/sites/gent/available-modules.md b/mkdocs/docs/HPC/sites/gent/available-modules.md index e59a9f01c13..6c38360947e 100644 --- a/mkdocs/docs/HPC/sites/gent/available-modules.md +++ b/mkdocs/docs/HPC/sites/gent/available-modules.md @@ -1,10 +1,10 @@
$ module av | more
---- /apps/gent/SL6/sandybridge/modules/all ---
-ABAQUS/6.12.1-linux-x86_64
-AMOS/3.1.0-ictce-4.0.10
-ant/1.9.0-Java-1.7.0_40
-ASE/3.6.0.2515-ictce-4.1.13-Python-2.7.3
-ASE/3.6.0.2515-ictce-5.5.0-Python-2.7.6
+--- /apps/gent/RHEL8/zen2-ib/modules/all ---
+   ABAQUS/2021-hotfix-2132
+   ABAQUS/2022-hotfix-2214
+   ABAQUS/2022
+   ABAQUS/2023
+   ABAQUS/2024-hotfix-2405                                                (D)
 ...
 
@@ -12,7 +12,7 @@ Or when you want to check whether some specific software, some compiler or some application (e.g., MATLAB) is installed on the {{hpc}}.
$ module av matlab
---------------------- /apps/gent/RHEL8/zen2-ib/modules/all ---------------------
+--- /apps/gent/RHEL8/zen2-ib/modules/all ---
    LIBSVM-MATLAB/3.30-GCCcore-11.3.0-MATLAB-2022b-r5
    MATLAB/2019b
    MATLAB/2021b

From d4fa262b84408c7dad0f4d96be4ff4b374c44f7b Mon Sep 17 00:00:00 2001
From: Lukas Barragan Torres 
Date: Fri, 5 Jul 2024 17:44:56 +0200
Subject: [PATCH 26/74] added tab before ellipsis

---
 mkdocs/docs/HPC/sites/gent/available-modules.md | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/mkdocs/docs/HPC/sites/gent/available-modules.md b/mkdocs/docs/HPC/sites/gent/available-modules.md
index 6c38360947e..13956ab2346 100644
--- a/mkdocs/docs/HPC/sites/gent/available-modules.md
+++ b/mkdocs/docs/HPC/sites/gent/available-modules.md
@@ -5,7 +5,7 @@
    ABAQUS/2022
    ABAQUS/2023
    ABAQUS/2024-hotfix-2405                                                (D)
-...
+   ...
 
Or when you want to check whether some specific software, some compiler or some From 7b8994b3ea75d341807391c8fe720bf20a292754 Mon Sep 17 00:00:00 2001 From: Lukas Barragan Torres Date: Fri, 5 Jul 2024 18:05:38 +0200 Subject: [PATCH 27/74] added 02 compilation flag + explaining --- mkdocs/docs/HPC/compiling_your_software.md | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/mkdocs/docs/HPC/compiling_your_software.md b/mkdocs/docs/HPC/compiling_your_software.md index 018565bb817..f1773d2d3de 100644 --- a/mkdocs/docs/HPC/compiling_your_software.md +++ b/mkdocs/docs/HPC/compiling_your_software.md @@ -127,9 +127,11 @@ We first need to compile this C-file into an executable with the gcc-compiler. First, check the command line options for *"gcc" (GNU C-Compiler)*, then -we compile and list the contents of the directory again: +we compile. the `O2` option enables a moderate level of optimization when compiling the code. +It instructs the compiler to optimize the code for better performance without significantly increasing compilation time. +Finally, list the contents of the directory again:
$ gcc -help
-$ gcc -o hello hello.c
+$ gcc -O2 -o hello hello.c
 $ ls -l
 total 512
 -rwxrwxr-x 1 {{userid}} 7116 Sep 16 11:43 hello*

From 75a6c1aace91a18d8c9841d49e9c9305ec2ac811 Mon Sep 17 00:00:00 2001
From: Lukas Barragan Torres 
Date: Fri, 5 Jul 2024 19:56:32 +0200
Subject: [PATCH 28/74] removed checkpointing

---
 mkdocs/docs/HPC/best_practices.md | 3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)

diff --git a/mkdocs/docs/HPC/best_practices.md b/mkdocs/docs/HPC/best_practices.md
index 8b1aa11a52e..73b1c5e4f64 100644
--- a/mkdocs/docs/HPC/best_practices.md
+++ b/mkdocs/docs/HPC/best_practices.md
@@ -49,8 +49,7 @@
 9.  Submit small jobs by grouping them together. See chapter [Multi-job submission](multi_job_submission.md) for 
     how this is done.
 
-10. The runtime is limited by the maximum walltime of the queues. For
-    longer walltimes, use checkpointing.
+10. The runtime is limited by the maximum walltime of the queues.
 
 11. Requesting many processors could imply long queue times. It's
     advised to only request the resources you'll be able to use.

From 1bf4629a8868b4b16e1eef4d5e8a33d53cb814bd Mon Sep 17 00:00:00 2001
From: Lukas Barragan Torres 
Date: Tue, 9 Jul 2024 11:14:43 +0200
Subject: [PATCH 29/74] fixed some typos

---
 mkdocs/docs/HPC/gpu.md | 12 ++++++------
 1 file changed, 6 insertions(+), 6 deletions(-)

diff --git a/mkdocs/docs/HPC/gpu.md b/mkdocs/docs/HPC/gpu.md
index e70b13ce1b5..53f4ab1ea09 100644
--- a/mkdocs/docs/HPC/gpu.md
+++ b/mkdocs/docs/HPC/gpu.md
@@ -40,14 +40,14 @@ There are 2 main ways to ask for GPUs as part of a job:
     specified via `ppn`) using `-l nodes=X:ppn=Y:gpus=Z` (where the
     `ppn=Y` is optional), or as a separate resource request (similar to
     the amount of memory) via `-l gpus=Z`. Both notations give exactly
-    the same result. The `-l gpus=Z` is convenient is you only need one
+    the same result. The `-l gpus=Z` is convenient if you only need one
     node and you are fine with the default number of cores per GPU. The
     `-l nodes=...:gpus=Z` notation is required if you want to run with
     full control or in multinode cases like MPI jobs. If you do not
     specify the number of GPUs by just using `-l gpus`, you get by
     default 1 GPU.
 
--   As a resource of it's own, via `--gpus X`. In this case however, you
+-   As a resource of its own, via `--gpus X`. In this case however, you
     are *not* guaranteed that the GPUs are on the same node, so your
     script or code must be able to deal with this.
 
@@ -61,7 +61,7 @@ There are 2 main ways to ask for GPUs as part of a job:
 %  need to be processed or increasing the MPI ranks gives a speedup (e.g. when there is a significant portion of CPU work in the code).
 %  Unfortunately, this is not a silver bullet, and might require some experimenting to found out any potential benefits and proper tuning.
 %  TODO: how can a user now that an application is not using the full gpu resources?
-%  TODO this needs testing and there are some constraints (eg one mps job per node and thus one user per node using MPS)
+%  TODO this needs testing and there are some constraints (e.g. one mps job per node and thus one user per node using MPS)
 %  TODO needs proper integration with mypmirun / wurker
 %  TODO add separate section on MPS -->
 
@@ -86,7 +86,7 @@ Some important attention points:
     the MPI tasks, and is different from the usual `mpirun` that is used
     by the `mympirun` wrapper). At some later point, we *might* promote
     the `mypmirun` tool or rename it, to avoid the confusion in the
-    naming).
+    naming.
 
 -   Sharing GPUs requires MPS. The Slurm built-in MPS does not really do
     want you want, so we will provide integration with `mypmirun` and
@@ -95,7 +95,7 @@ Some important attention points:
 -   For parallel work, we are working on a `wurker` wrapper from the
     `vsc-mympirun` module that supports GPU placement and MPS, without
     any limitations wrt the requested resources (i.e. also support the
-    case where GPUs are spread heterogenous over nodes from using the
+    case where GPUs are spread heterogeneous over nodes from using the
     `--gpus Z` option).
 
 -   Both `mypmirun` and `wurker` will try to do the most optimised
@@ -133,7 +133,7 @@ Please consult `module avail Horovod` for a list of installed versions.
 
 Horovod supports TensorFlow, Keras, PyTorch and MxNet (see
 ), but should be run as an MPI
-application with `mypmirun`. (Horovod also provides it's own wrapper
+application with `mypmirun`. (Horovod also provides its own wrapper
 `horovodrun`, not sure if it handles placement and others correctly).
 
 At least for simple TensorFlow benchmarks, it looks like Horovod is a

From 6bc0cd1c79b81a47c0be0465e2aea7bc6fa71cd2 Mon Sep 17 00:00:00 2001
From: Lukas Barragan Torres 
Date: Tue, 9 Jul 2024 11:40:17 +0200
Subject: [PATCH 30/74] fixed some typos in teaching_training.md

---
 mkdocs/docs/HPC/teaching_training.md | 8 ++++----
 1 file changed, 4 insertions(+), 4 deletions(-)

diff --git a/mkdocs/docs/HPC/teaching_training.md b/mkdocs/docs/HPC/teaching_training.md
index 492198ddd37..46062175857 100644
--- a/mkdocs/docs/HPC/teaching_training.md
+++ b/mkdocs/docs/HPC/teaching_training.md
@@ -64,7 +64,7 @@ as described in the [course data policy](./#course-data-policy):
 A _course group_ is created with all students or participants, and the teachers or trainers are
 the group moderators (and also member of this group).
 
-This course group and the moderators group are used to manage the different priviliges:
+This course group and the moderators group are used to manage the different privileges:
 moderators have additional privileges over non-moderator members
 e.g. they have read/write access in specific folders, can manage subgroups, ....
 
@@ -140,7 +140,7 @@ Optionally, we can also create these folders:
     - each of these `group_<01>` folders are owned by a dedicated group
     - teachers are automatically made moderators of these dedicated groups
     - moderators can populate these groups with VSC-ids of group members in the VSC accountpage
-      or ask the students to invite themself via [group edit](https://account.vscentrum.be/django/group/edit).
+      or ask the students to invite themselves via [group edit](https://account.vscentrum.be/django/group/edit).
       When students invite them self, moderators still need to [approve the group invites](https://account.vscentrum.be/django/group/approve).
     - only these VSC-ids will then be able to access a `group_<01>` folder, and will have read/write access.
 
@@ -158,7 +158,7 @@ There are 4 quota settings that you can choose in your _teaching request_ in the
 - overall quota (defaults *10 GB volume* and *20k files*) are for the moderators and can be used for e.g. the `input` folder.
 - member quota (defaults *5 GB volume* and *10k files*) are per student/participant
 
-The course data usage is not accounted for the any other quota (like VO quota). It is solely dependant on these settings.
+The course data usage is not accounted for any other quota (like VO quota). It is solely dependent on these settings.
 
 
 ### Course data policy
@@ -176,7 +176,7 @@ We assume that your course requirements are such that the [interactive cluster](
 If these resources are insufficient, you will need to request and motivate a reservation.
 
 Indicate which cluster you would need and the number of nodes, cores and/or GPUs.
-Also clearly indicate when you would need these resources, i.e. the dates and times of each course session.
+Also, clearly indicate when you would need these resources, i.e. the dates and times of each course session.
 
 Be aware that students will have no access to the reservation outside the course sessions.
 This might be relevant when requesting a custom application.

From 6f6c93f2d2bdeaff8fb49b226f752faa26269420 Mon Sep 17 00:00:00 2001
From: Lukas Barragan Torres 
Date: Tue, 9 Jul 2024 11:50:51 +0200
Subject: [PATCH 31/74] fix: bullet point list was not rendereing correctly

---
 mkdocs/docs/HPC/teaching_training.md | 7 +++++++
 1 file changed, 7 insertions(+)

diff --git a/mkdocs/docs/HPC/teaching_training.md b/mkdocs/docs/HPC/teaching_training.md
index 492198ddd37..2238a57ac49 100644
--- a/mkdocs/docs/HPC/teaching_training.md
+++ b/mkdocs/docs/HPC/teaching_training.md
@@ -213,12 +213,19 @@ It will appear in the `Interactive Apps` menu in the webportal, under the sectio
 After the indicated end date of your course, this application will be removed.
 
 If you would like this for your course, provide more details in your _teaching request_, including:
+
 - what interactive application would you like to get launched (cluster desktop, Jupyter Notebook, ...)
+
 - which cluster you want to use
+
 - how many nodes/cores/GPUs are needed
+
 - which software modules you are loading
+
 - custom code you are launching (e.g. autostart a GUI)
+
 - required environment variables that you are setting
+
 - ...
 
 We will try to make the custom interactive application available before the start of your course/workshop,

From a1cf6d485e8f5ea7478128f2c24b05ed8553abc4 Mon Sep 17 00:00:00 2001
From: Lukas Barragan Torres 
Date: Fri, 12 Jul 2024 17:07:56 +0200
Subject: [PATCH 32/74] added table and example to 'module conflict' section

---
 mkdocs/docs/HPC/troubleshooting.md | 44 ++++++++++++++++++++++++++----
 1 file changed, 38 insertions(+), 6 deletions(-)

diff --git a/mkdocs/docs/HPC/troubleshooting.md b/mkdocs/docs/HPC/troubleshooting.md
index 18c35b728d3..3ae43d5f8fb 100644
--- a/mkdocs/docs/HPC/troubleshooting.md
+++ b/mkdocs/docs/HPC/troubleshooting.md
@@ -479,13 +479,13 @@ memory you request.
 {% if site == gent %}
 ## Module conflicts
 
-Modules that are loaded together must use the same toolchain version: it
-is impossible to load two versions of the same module. In the following
+Modules that are loaded together must use the same toolchain version or common dependencies. In the following
 example, we try to load a module that uses the `intel-2018a` toolchain
 together with one that uses the `intel-2017a` toolchain:
 
-
$ module load Python/2.7.14-intel-2018a
-$ module load  HMMER/3.1b2-intel-2017a
+```bash
+$ module load Python/2.7.14-intel-2018a
+$ module load  HMMER/3.1b2-intel-2017a
 Lmod has detected the following error: A different version of the 'intel' module is already loaded (see output of 'ml'). 
 You should load another 'HMMER' module for that is compatible with the currently loaded version of 'intel'. 
 Use 'ml avail HMMER' to get an overview of the available versions.
@@ -497,14 +497,46 @@ While processing the following module(s):
     ---------------          ---------------
     HMMER/3.1b2-intel-2017a  /apps/gent/CO7/haswell-ib/modules/all/HMMER/3.1b2-intel-2017a.lua
 
+``` -This resulted in an error because we tried to load two different -versions of the `intel` module. +This resulted in an error because we tried to load two modules with different +versions of the `intel` toolchain. To fix this, check if there are other versions of the modules you want to load that have the same version of common dependencies. You can list all versions of a module with `module avail`: for `HMMER`, this command is `module avail HMMER`. +As a rule of thumb, toolchains in the same row are compatible with each other: + +| | | | | +|----------------|--------------------------|-------------------------|-------------| +| GCCcore-13.2.0 | GCC-13.2.0 | gfbf-2023b/gompi-2023b | foss-2023b | +| GCCcore-13.2.0 | intel-compilers-2023.2.1 | iimkl-2023b/iimpi-2023b | intel-2023b | +| GCCcore-12.3.0 | GCC-12.3.0 | gfbf-2023a/gompi-2023a | foss-2023a | +| GCCcore-12.3.0 | intel-compilers-2023.1.0 | iimkl-2023a/iimpi-2023a | intel-2023a | +| GCCcore-12.2.0 | GCC-12.2.0 | gfbf-2022b/gompi-2022b | foss-2022b | +| GCCcore-12.2.0 | intel-compilers-2022.2.1 | iimkl-2022b/iimpi-2022b | intel-2022b | +| GCCcore-11.3.0 | GCC-11.3.0 | gfbf-2022a/gompi-2022a | foss-2022a | +| GCCcore-11.3.0 | intel-compilers-2022.1.0 | iimkl-2022a/iimpi-2022a | intel-2022a | +| GCCcore-11.2.0 | GCC-11.2.0 | gfbf-2021b/gompi-2021b | foss-2021b | +| GCCcore-11.2.0 | intel-compilers-2021.4.0 | iimkl-2021b/iimpi-2021b | intel-2021b | +| GCCcore-10.3.0 | GCC-10.3.0 | gfbf-2021a/gompi-2021a | foss-2021a | +| GCCcore-10.3.0 | intel-compilers-2021.2.0 | iimkl-2021a/iimpi-2021a | intel-2021a | +| GCCcore-10.2.0 | GCC-10.2.0 | gfbf-2020b/gompi-2020b | foss-2020b | +| GCCcore-10.2.0 | iccifort-2020.4.304 | iimkl-2020b/iimpi-2020b | intel-2020b | + +!!! example + we could load the following modules together: + + ```bash + ml XGBoost/1.7.2-foss-2022a + ml scikit-learn/1.1.2-foss-2022a + ml cURL/7.83.0-GCCcore-11.3.0 + ml JupyterNotebook/6.4.0-GCCcore-11.3.0-IPython-8.5.0 + ``` + + + Another common error is:
$ module load cluster/{{othercluster}}

From c73de52f33c2527abdc71f55cc6a7405bf30b25a Mon Sep 17 00:00:00 2001
From: stdweird 
Date: Mon, 15 Jul 2024 14:30:29 +0200
Subject: [PATCH 33/74] shinx in prod

---
 mkdocs/docs/HPC/only/gent/2023/shinx.md | 13 +++++++++----
 1 file changed, 9 insertions(+), 4 deletions(-)

diff --git a/mkdocs/docs/HPC/only/gent/2023/shinx.md b/mkdocs/docs/HPC/only/gent/2023/shinx.md
index 762ee8268af..daa770e8b69 100644
--- a/mkdocs/docs/HPC/only/gent/2023/shinx.md
+++ b/mkdocs/docs/HPC/only/gent/2023/shinx.md
@@ -15,8 +15,8 @@ For software installation requests, please use the [request form](https://www.ug
 
 `shinx` is a new CPU-only cluster.
 
-It replaces `swalot`, which will be retired on **Wednesday 01 November 2023**,
-and `victini`, which will be retired on **Monday 05 February 2024**.
+It replaces `swalot`, which was retired on **Wednesday 01 November 2023**,
+and `victini`, which ws retired on **Monday 05 February 2024**.
 
 It is primarily for regular CPU compute use.
 
@@ -77,7 +77,7 @@ It is not recommended to always set this workaround, only for the specific tools
 
 ---
 
-## Shinx pilot phase (23/10/2023-20/05/2024)
+## Shinx pilot phase (23/10/2023-15/07/2024)
 
 As usual with any pilot phase, you need to be member of the `gpilot` group, and to start using this cluster run:
 
@@ -111,7 +111,7 @@ As such, we will have an extended pilot phase in 3 stages:
 * Racking of last 16 nodes
 * Installation of NDR/NDR-200 infiniband network
 
-### Stage 2 (19/04/2024-20/05/2024)
+### Stage 2 (19/04/2024-15/07/2024)
 
 * Full size cluster
     * 48 nodes (no job size limit)
@@ -123,6 +123,11 @@ As such, we will have an extended pilot phase in 3 stages:
 that period, and the testing of the `EL9` operating system will also take some time.
 
 
+### Stage 3 (15/07/2024 - )
+
+* Cluster in production using EL9 (starting with 9.4). Any user can now submit jobs.
+
+
 ### Using `doduo` software
 
 For benchmarking and/or compatibility testing, you can use try to use `doduo` software stack by adding

From e196e11eeaefc8b4eb7c1f8c4b7aa7ba8c0fceed Mon Sep 17 00:00:00 2001
From: Lukas Barragan Torres 
Date: Thu, 18 Jul 2024 16:28:12 +0200
Subject: [PATCH 34/74] remove checkpointing.md and all references to it

---
 mkdocs/docs/HPC/FAQ.md           |   1 -
 mkdocs/docs/HPC/checkpointing.md | 180 -------------------------------
 2 files changed, 181 deletions(-)
 delete mode 100644 mkdocs/docs/HPC/checkpointing.md

diff --git a/mkdocs/docs/HPC/FAQ.md b/mkdocs/docs/HPC/FAQ.md
index 398bd3e1e67..659f3d6be3f 100644
--- a/mkdocs/docs/HPC/FAQ.md
+++ b/mkdocs/docs/HPC/FAQ.md
@@ -107,7 +107,6 @@ The 72 hour walltime limit will not be extended. However, you can work around th
 * Use a faster [cluster]({{ hpc_infrastructure_url }}).
 * Divide the job into more parallel processes.
 * Divide the job into shorter processes, which you can submit as separate jobs.
-* Use the built-in checkpointing of your software.
 
 ### Job failed: SEGV Segmentation fault
 
diff --git a/mkdocs/docs/HPC/checkpointing.md b/mkdocs/docs/HPC/checkpointing.md
deleted file mode 100644
index e9843f57fdd..00000000000
--- a/mkdocs/docs/HPC/checkpointing.md
+++ /dev/null
@@ -1,180 +0,0 @@
-# Checkpointing { #ch:checkpointing}
-
-## Why checkpointing?
-
-If you want to run jobs that require wall time than the maximum wall
-time per job and/or want to avoid you lose work because of power outages
-or system crashes, you need to resort to checkpointing.
-
-## What is checkpointing?
-
-Checkpointing allows for running jobs that run for weeks or months, by
-splitting the job into smaller parts (called subjobs) which are executed
-consecutively. Each time a subjob is running out of requested wall time,
-a snapshot of the application memory (and much more) is taken and
-stored, after which a subsequent subjob will pick up the checkpoint and
-continue.
-
-## How to use checkpointing?
-
-Using checkpointing is very simple: just use `csub` instead of `qsub` to
-submit a job.
-
-The `csub` command creates a wrapper around your job script, to take
-care of all the checkpointing stuff.
-
-In practice, you (usually) don't need to adjust anything, except for the
-command used to submit your job.
-
-Checkpointing does not require any changes to the application you are
-running, and should support most software.
-
-## Usage and parameters
-
-An overview of the usage and various command line parameters is given
-here.
-
-### Submitting a job
-
-Typically, a job script is submitted with checkpointing support enabled
-by running:
-
-::: prompt
-csub -s job_script.sh
-:::
-
-The `-s` flag specifies the job script to run.
-
-### Caveat: don't create local directories
-
-One important caveat is that the job script (or the applications run in
-the script) should *not* create its own local temporary directories,
-because those will not (always) be restored when the job is restarted
-from checkpoint.
-
-### PBS directives
-
-Most PBS directives (`#PBS …` specified in the job script will be
-ignored. There are a few exceptions however, i.e., `# PBS -N `
-(job name) and all `-l` directives (`# PBS -l`), e.g., `nodes`, `ppn`,
-`vmem` (virtual memory limit), etc. Controlling other job parameters
-(like requested walltime per sub-job) should be specified on the `csub`
-command line.
-
-### Getting help
-
-Help on the various command line parameters supported by `csub` can be
-obtained using `-h` or `--help`.
-
-### Local files (`–pre` / `–post`)
-
-The `--pre` and `--post` parameters control whether local files are
-copied or not. The job submitted using `csub` is (by default) run on the
-local storage provided by a particular workernode. Thus, no changes will
-be made to the files on the shared storage.
-
-If the job script needs (local) access to the files of the directory
-where `csub` is executed, `--pre` flag should be used. This will copy
-all the files in the job script directory to the location where the job
-script will execute.
-
-If the output of the job (`stdout`/`stderr`) that was run, or additional
-output files created by the job in its working directory are required,
-the `--post` flag should be used. This will copy the entire job working
-directory to the location where `csub` was executed, in a directory
-named `result.`. An alternative is to copy the interesting
-files to the shared storage at the end of the job script.
-
-### Running on shared storage (`–shared`)
-
-If the job needs to be run on the shared storage, `--shared` should be
-specified. You should enable this option by default, because it makes
-the execution of the underlying `csub` script more robust: it doesn't
-have to copy from/to the local storage on the workernode. When enabled,
-the job will be run in a subdirectory of `$VSC_SCRATCH/chkpt`. All files
-produced by the job will be in `$VSC_SCRATCH/chkpt//` while the
-job is running.
-
-Note that if files in the directory where the job script is located are
-required to run the job, you should also use `--pre`.
-
-### Job wall time (`–job_time`, `–chkpt_time`)
-
-To specify the requested wall time per subjob, use the `--job-time`
-parameter. The default setting is 10 hours per (sub)job. Lowering this
-will result in more frequent checkpointing, and thus more (sub)jobs.
-
-To specify the time that is reserved for checkpointing the job, use
-`--chkpt_time`. By default, this is set to 15 minutes which should be
-enough for most applications/jobs. *Don't change this unless you really
-need to*.
-
-The total requested wall time per subjob is the sum of both `job_time`
-and `chkpt_time`.
-
-If you would like to time how long the job executes, just prepend the
-main command in your job script with the time command `time`, e.g.:
-
-::: prompt
-time main_command
-:::
-
-The `real` time will not make sense, as it will also include the time
-passed between two checkpointed subjobs. However, the `user` time should
-give a good indication of the actual time it took to run your command,
-even if multiple checkpoints were performed.
-
-### Resuming from last checkpoint (`–resume`)
-
-The `--resume` option allows you to resume a job from the last available
-checkpoint in case something went wrong (e.g., accidentally deleting a
-(sub)job using `qdel`, a power outage or other system failure, ...).
-
-Specify the job name as returned after submission (and as listed in
-`$VSC_SCRATCH/chkpt`). The full job name consists of the specified job
-name (or the script name if no job name was specified), a timestamp and
-two random characters at the end, for example `my_job_name._133755.Hk`
-or `script.sh._133755.Hk`.
-
-Note: When resuming from checkpoint, you can change the wall time
-resources for your job using the `--job_time` and `--chkpt_time`
-options. This should allow you to continue from the last checkpoint in
-case your job crashed due to an excessively long checkpointing time.
-
-In case resuming fails for you, please contact , and include the output
-of the `csub --resume` command in your message.
-
-## Additional options
-
-### Array jobs (`-t`)
-
-`csub` has support for checkpointing array jobs with the `-t `
-flag on the `csub` command line. This behaves the same as `qsub`, see .
-
-### Pro/epilogue mimicking (`–no_mimic_pro_epi`)
-
-The option `--no_mimic_pro_epi` disables the workaround currently
-required to resolve a permissions problem when using actual Torque
-prologue/epilogue scripts. Don't use this option unless you really know
-what you are doing.
-
-### Cleanup checkpoints (`–cleanup_after_restart`)
-
-Specifying this option will make the wrapper script remove the
-checkpoint files after a successful job restart. This may be desirable
-in cause you are short on storage space.
-
-Note that we don't recommend setting this option, because this way you
-won't be able to resume from the last checkpoint when something goes
-wrong. It may also prevent the wrapper script from reattempting to
-resubmit a new job in case an infrequent known failure occurs. So, don't
-set this unless you really need to.
-
-### No cleanup after job completion (`–no_cleanup_chkpt`)
-
-Specifying this option will prevent the wrapper script from cleaning up
-the checkpoints and related information once the job has finished. This
-may be useful for debugging, since this also preserves the
-`stdout`/`stderr` of the wrapper script.
-
-*Don't set this unless you know what you are doing*.

From 2d37b9188d23769ba94b852a75597b1cf7f4c6b3 Mon Sep 17 00:00:00 2001
From: Lukas Barragan Torres 
Date: Thu, 18 Jul 2024 17:59:40 +0200
Subject: [PATCH 35/74] re-add checkpointing in FAQ

---
 mkdocs/docs/HPC/FAQ.md | 1 +
 1 file changed, 1 insertion(+)

diff --git a/mkdocs/docs/HPC/FAQ.md b/mkdocs/docs/HPC/FAQ.md
index 659f3d6be3f..398bd3e1e67 100644
--- a/mkdocs/docs/HPC/FAQ.md
+++ b/mkdocs/docs/HPC/FAQ.md
@@ -107,6 +107,7 @@ The 72 hour walltime limit will not be extended. However, you can work around th
 * Use a faster [cluster]({{ hpc_infrastructure_url }}).
 * Divide the job into more parallel processes.
 * Divide the job into shorter processes, which you can submit as separate jobs.
+* Use the built-in checkpointing of your software.
 
 ### Job failed: SEGV Segmentation fault
 

From 343ae4476c217328cdcf43659bbb66857726fd42 Mon Sep 17 00:00:00 2001
From: Lukas Barragan Torres 
Date: Thu, 18 Jul 2024 18:38:31 +0200
Subject: [PATCH 36/74] add ORTE daemon error in troubleshooting for
 multi_job_submission

---
 mkdocs/docs/HPC/multi_job_submission.md | 14 ++++++++++++++
 1 file changed, 14 insertions(+)

diff --git a/mkdocs/docs/HPC/multi_job_submission.md b/mkdocs/docs/HPC/multi_job_submission.md
index e568ecfd2b5..5177d79fa33 100644
--- a/mkdocs/docs/HPC/multi_job_submission.md
+++ b/mkdocs/docs/HPC/multi_job_submission.md
@@ -455,6 +455,20 @@ specified:
 #                           command
 
+## Troubleshooting + +### Error: An ORTE daemon has unexpectedly failed after launch and before communicating back to mpirun + +When submitting a Worker job, you might encounter the following error: +`An ORTE daemon has unexpectedly failed after launch and before communicating back to mpirun`. +This error can occur when the foss toolchain version of worker is loaded. Instead, try loading an iimpi toolchain version of worker. + +to check for the available versions of worker, use the following command: + +```bash +$ module avail worker +``` + [^1]: MapReduce: 'Map' refers to the map pattern in which every item in a collection is mapped onto a new value by applying a given From 72c68e59713dc35e60a295f7e5776bb4a5ca82c0 Mon Sep 17 00:00:00 2001 From: Lukas Barragan Torres Date: Fri, 19 Jul 2024 18:26:39 +0200 Subject: [PATCH 37/74] typo fixes --- mkdocs/docs/HPC/alphafold.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/mkdocs/docs/HPC/alphafold.md b/mkdocs/docs/HPC/alphafold.md index 253cd895c70..9c502992620 100644 --- a/mkdocs/docs/HPC/alphafold.md +++ b/mkdocs/docs/HPC/alphafold.md @@ -96,9 +96,9 @@ export ALPHAFOLD_DATA_DIR={{directory}}/{{version}} ### Running AlphaFold -AlphaFold provides run script called [run_alphafold.py](https://raw.githubusercontent.com/deepmind/alphafold/main/run_alphafold.py) +AlphaFold provides a script called [run_alphafold.py](https://raw.githubusercontent.com/deepmind/alphafold/main/run_alphafold.py) -A symbolic link named *alphafold* that points to the this script is included, +A symbolic link named *alphafold* that points to this script is included, so you can just use `alphafold` instead of `run_alphafold.py` or `python run_alphafold.py` after loading the AlphaFold module. The `run_alphafold.py` script has also been slightly modified such that defining the `$ALPHAFOLD_DATA_DIR` (see [above](./#setting-up-the-environment)) is sufficient to pick up all the data provided in that location, @@ -158,7 +158,7 @@ This highlights the difference between CPU and GPU performance even more. The following example comes from the official [Examples section]({{readme}}#examples) in the Alphafold [README]({{readme}}). The run command is slightly different (see above: [Running AlphaFold](./running-alphafold)). -Do not forget to setup the environment (see above: [Setting up the environment](./setting-up-the-environment)). +Do not forget to set up the environment (see above: [Setting up the environment](./setting-up-the-environment)). ### Folding a monomer @@ -193,7 +193,7 @@ The main difference between using a GPU or CPU in a job script is what module to For running AlphaFold on GPU, use an AlphaFold module that mentions `CUDA` (or `cuda`), for example `AlphaFold/2.3.1-foss-2022a-CUDA-11.7.0`. -To run the jobs cripts you need to create a file named `T1050.fasta` with the following content: +To run the job scripts you need to create a file named `T1050.fasta` with the following content: ```fasta >T1050 A7LXT1, Bacteroides Ovatus, 779 residues| From 7d59575c4f622207f503320b20fc0ff9dd777da7 Mon Sep 17 00:00:00 2001 From: Lukas Barragan Torres Date: Fri, 19 Jul 2024 18:53:30 +0200 Subject: [PATCH 38/74] typo fixes --- mkdocs/docs/HPC/MATLAB.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/mkdocs/docs/HPC/MATLAB.md b/mkdocs/docs/HPC/MATLAB.md index e518dccf367..84109fe6b24 100644 --- a/mkdocs/docs/HPC/MATLAB.md +++ b/mkdocs/docs/HPC/MATLAB.md @@ -31,7 +31,7 @@ license, licenses would quickly run out. Compiling MATLAB code can only be done from the login nodes, because only login nodes can access the MATLAB license server, workernodes on -clusters can not. +clusters cannot. To access the MATLAB compiler, the `MATLAB` module should be loaded first. Make sure you are using the same `MATLAB` version to compile and @@ -93,7 +93,7 @@ with:
$ export _JAVA_OPTIONS="-Xmx64M"
 
-The MATLAB compiler spawns multiple Java processes, and because of the +The MATLAB compiler spawns multiple Java processes. Because of the default memory limits that are in effect on the login nodes, this might lead to a crash of the compiler if it's trying to create to many Java processes. If we lower the heap size, more Java processes will be able @@ -122,7 +122,7 @@ controlled via the `parpool` function: `parpool(16)` will use 16 workers. It's best to specify the amount of workers, because otherwise you might not harness the full compute power available (if you have too few workers), or you might negatively impact performance (if you have -too much workers). By default, MATLAB uses a fixed number of workers +too many workers). By default, MATLAB uses a fixed number of workers (12). You should use a number of workers that is equal to the number of cores @@ -163,7 +163,7 @@ You should remove the directory at the end of your job script: ## Cache location When running, MATLAB will use a cache for performance reasons. This -location and size of this cache can be changed trough the +location and size of this cache can be changed through the `MCR_CACHE_ROOT` and `MCR_CACHE_SIZE` environment variables. The snippet below would set the maximum cache size to 1024MB and the From 7724f8e3851abfb55c5154c8426f011fe4c260e4 Mon Sep 17 00:00:00 2001 From: Lukas Barragan Torres Date: Fri, 19 Jul 2024 19:02:20 +0200 Subject: [PATCH 39/74] typo fixes --- mkdocs/docs/HPC/openFOAM.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/mkdocs/docs/HPC/openFOAM.md b/mkdocs/docs/HPC/openFOAM.md index 09d04efe709..c03732a9bcc 100644 --- a/mkdocs/docs/HPC/openFOAM.md +++ b/mkdocs/docs/HPC/openFOAM.md @@ -138,7 +138,7 @@ it does not prevent that illegal operations (like a division by zero) are being executed; if `NaN` values appear in your results, floating point errors are occurring. -As such, **you should *not* use this in productions runs. Instead, you should track down the root cause of the floating +As such, **you should *not* use this in production runs. Instead, you should track down the root cause of the floating point errors, and try to prevent them from occurring at all. ## OpenFOAM workflow @@ -285,7 +285,7 @@ specify in `system/controlDict` (see also - instruct OpenFOAM to write out results at a reasonable frequency, **certainly *not*** for every single time step}; you can control this using the `writeControl`, `writeInterval`, etc. keywords; -- consider only retaining results for the last couple of time steps, +- consider only retaining results for the last couple of steps, see the `purgeWrite` keyword; - consider writing results for only part of the domain (e.g., a line @@ -302,7 +302,7 @@ For modest OpenFOAM simulations where a single workernode suffices, consider using the local disk of the workernode as working directory (accessible via `$VSC_SCRATCH_NODE`), rather than the shared `$VSC_SCRATCH` filesystem. **Certainly do not use a subdirectory in `$VSC_HOME` or `$VSC_DATA`, since these shared filesystems are too slow -for these type of workloads. +for these types of workloads. {% if site == gent %} For large parallel OpenFOAM simulations on the {{university}} Tier-2 clusters, consider From 303d067b08667fdd039b30b139b851376f55610f Mon Sep 17 00:00:00 2001 From: Lukas Barragan Torres Date: Fri, 19 Jul 2024 19:06:01 +0200 Subject: [PATCH 40/74] re-add times --- mkdocs/docs/HPC/openFOAM.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/mkdocs/docs/HPC/openFOAM.md b/mkdocs/docs/HPC/openFOAM.md index c03732a9bcc..8f83201d6a4 100644 --- a/mkdocs/docs/HPC/openFOAM.md +++ b/mkdocs/docs/HPC/openFOAM.md @@ -285,7 +285,7 @@ specify in `system/controlDict` (see also - instruct OpenFOAM to write out results at a reasonable frequency, **certainly *not*** for every single time step}; you can control this using the `writeControl`, `writeInterval`, etc. keywords; -- consider only retaining results for the last couple of steps, +- consider only retaining results for the last couple of time steps, see the `purgeWrite` keyword; - consider writing results for only part of the domain (e.g., a line From 74c259b36100e1401f3325007d130685ad23e0ac Mon Sep 17 00:00:00 2001 From: Lara Ramona Peeters <49882639+laraPPr@users.noreply.github.com> Date: Tue, 30 Jul 2024 10:22:26 +0200 Subject: [PATCH 41/74] load foss toolchain instead of intel in mpi examples --- .../Compiling-and-testing-your-software-on-the-HPC/mpihello.pbs | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/intro-HPC/examples/Compiling-and-testing-your-software-on-the-HPC/mpihello.pbs b/intro-HPC/examples/Compiling-and-testing-your-software-on-the-HPC/mpihello.pbs index 9d4783308e9..47aeea0c757 100644 --- a/intro-HPC/examples/Compiling-and-testing-your-software-on-the-HPC/mpihello.pbs +++ b/intro-HPC/examples/Compiling-and-testing-your-software-on-the-HPC/mpihello.pbs @@ -12,6 +12,6 @@ cd $PBS_O_WORKDIR # load the environment module purge -module load intel +module load foss mpirun ./mpihello From 7c89ab1e42aa3e0391abfba2d6e85b4b7a4e9c23 Mon Sep 17 00:00:00 2001 From: Lara Ramona Peeters <49882639+laraPPr@users.noreply.github.com> Date: Tue, 30 Jul 2024 10:27:56 +0200 Subject: [PATCH 42/74] Update TensorFlow_GPU.sh --- intro-HPC/examples/HPC-UGent-GPU-clusters/TensorFlow_GPU.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/intro-HPC/examples/HPC-UGent-GPU-clusters/TensorFlow_GPU.sh b/intro-HPC/examples/HPC-UGent-GPU-clusters/TensorFlow_GPU.sh index ff4e2aa558c..52c8be06ba0 100644 --- a/intro-HPC/examples/HPC-UGent-GPU-clusters/TensorFlow_GPU.sh +++ b/intro-HPC/examples/HPC-UGent-GPU-clusters/TensorFlow_GPU.sh @@ -2,7 +2,7 @@ #PBS -l walltime=5:0:0 #PBS -l nodes=1:ppn=quarter:gpus=1 -module load TensorFlow/2.6.0-foss-2021a-CUDA-11.3.1 +module load TensorFlow/2.11.0-foss-2022a-CUDA-11.7.0 cd $PBS_O_WORKDIR python example.py From b12962f9dd2d89906e25e4c983d68946f9f3a56c Mon Sep 17 00:00:00 2001 From: Lara Ramona Peeters <49882639+laraPPr@users.noreply.github.com> Date: Tue, 30 Jul 2024 10:28:55 +0200 Subject: [PATCH 43/74] Update multi_core.sh --- intro-HPC/examples/Job-script-examples/multi_core.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/intro-HPC/examples/Job-script-examples/multi_core.sh b/intro-HPC/examples/Job-script-examples/multi_core.sh index ce7426bff03..3a38689de0b 100644 --- a/intro-HPC/examples/Job-script-examples/multi_core.sh +++ b/intro-HPC/examples/Job-script-examples/multi_core.sh @@ -2,7 +2,7 @@ #PBS -N mpi_hello ## job name #PBS -l nodes=2:ppn=all ## 2 nodes, all cores per node #PBS -l walltime=2:00:00 ## max. 2h of wall time -module load intel/2017b +module load foss/2023a module load vsc-mympirun ## We don't use a version here, this is on purpose # go to working directory, compile and run MPI hello world cd $PBS_O_WORKDIR From 2ee6a103d7c6288bc632afc41aeb0c42739740f8 Mon Sep 17 00:00:00 2001 From: Lara Ramona Peeters <49882639+laraPPr@users.noreply.github.com> Date: Tue, 30 Jul 2024 10:29:49 +0200 Subject: [PATCH 44/74] Update single_core.sh --- intro-HPC/examples/Job-script-examples/single_core.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/intro-HPC/examples/Job-script-examples/single_core.sh b/intro-HPC/examples/Job-script-examples/single_core.sh index f6ed99259e9..a406b09931d 100644 --- a/intro-HPC/examples/Job-script-examples/single_core.sh +++ b/intro-HPC/examples/Job-script-examples/single_core.sh @@ -2,7 +2,7 @@ #PBS -N count_example ## job name #PBS -l nodes=1:ppn=1 ## single-node job, single core #PBS -l walltime=2:00:00 ## max. 2h of wall time -module load Python/3.6.4-intel-2018a +module load Python/3.11.3-GCCcore-12.3.0 # copy input data from location where job was submitted from cp $PBS_O_WORKDIR/input.txt $TMPDIR # go to temporary working directory (on local disk) & run From 9e3df695617a7051305a26c01a95e6f93b163d59 Mon Sep 17 00:00:00 2001 From: Lara Ramona Peeters <49882639+laraPPr@users.noreply.github.com> Date: Tue, 30 Jul 2024 10:34:39 +0200 Subject: [PATCH 45/74] Update jobscript.sh --- intro-HPC/examples/MATLAB/jobscript.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/intro-HPC/examples/MATLAB/jobscript.sh b/intro-HPC/examples/MATLAB/jobscript.sh index 3a785317b70..849289c329a 100644 --- a/intro-HPC/examples/MATLAB/jobscript.sh +++ b/intro-HPC/examples/MATLAB/jobscript.sh @@ -7,7 +7,7 @@ # # make sure the MATLAB version matches with the one used to compile the MATLAB program! -module load MATLAB/2018a +module load MATLAB/2022b-r5 # use temporary directory (not $HOME) for (mostly useless) MATLAB log files # subdir in $TMPDIR (if defined, or /tmp otherwise) From d3af0c598eb61d05c52b66552557f43d5bc96e4e Mon Sep 17 00:00:00 2001 From: Lara Ramona Peeters <49882639+laraPPr@users.noreply.github.com> Date: Tue, 30 Jul 2024 10:35:14 +0200 Subject: [PATCH 46/74] Update mpi_hello.pbs --- .../examples/Multi-core-jobs-Parallel-Computing/mpi_hello.pbs | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/intro-HPC/examples/Multi-core-jobs-Parallel-Computing/mpi_hello.pbs b/intro-HPC/examples/Multi-core-jobs-Parallel-Computing/mpi_hello.pbs index 7101571d64d..ffb42b14fa7 100644 --- a/intro-HPC/examples/Multi-core-jobs-Parallel-Computing/mpi_hello.pbs +++ b/intro-HPC/examples/Multi-core-jobs-Parallel-Computing/mpi_hello.pbs @@ -11,6 +11,6 @@ cd $PBS_O_WORKDIR # load the environment -module load intel +module load foss mpirun ./mpi_hello From ec5b75a770a577b7bbd0481a097002c8df511a48 Mon Sep 17 00:00:00 2001 From: Lara Ramona Peeters <49882639+laraPPr@users.noreply.github.com> Date: Tue, 30 Jul 2024 10:37:13 +0200 Subject: [PATCH 47/74] Update OpenFOAM_damBreak.sh --- intro-HPC/examples/OpenFOAM/OpenFOAM_damBreak.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/intro-HPC/examples/OpenFOAM/OpenFOAM_damBreak.sh b/intro-HPC/examples/OpenFOAM/OpenFOAM_damBreak.sh index dab57a4cd33..021303e6715 100644 --- a/intro-HPC/examples/OpenFOAM/OpenFOAM_damBreak.sh +++ b/intro-HPC/examples/OpenFOAM/OpenFOAM_damBreak.sh @@ -2,7 +2,7 @@ #PBS -l walltime=1:0:0 #PBS -l nodes=1:ppn=4 # check for more recent OpenFOAM modules with 'module avail OpenFOAM' -module load OpenFOAM/6-intel-2018a +module load OpenFOAM/11-foss-2023a source $FOAM_BASH # purposely not specifying a particular version to use most recent mympirun module load vsc-mympirun From 00e800ba7add40a20432668c8971fba1b7059a33 Mon Sep 17 00:00:00 2001 From: Lara Ramona Peeters <49882639+laraPPr@users.noreply.github.com> Date: Tue, 30 Jul 2024 10:38:04 +0200 Subject: [PATCH 48/74] Update mpihello.pbs --- intro-HPC/examples/Program-examples/04_MPI_C/mpihello.pbs | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/intro-HPC/examples/Program-examples/04_MPI_C/mpihello.pbs b/intro-HPC/examples/Program-examples/04_MPI_C/mpihello.pbs index c31d6002c5b..86e8e49271c 100644 --- a/intro-HPC/examples/Program-examples/04_MPI_C/mpihello.pbs +++ b/intro-HPC/examples/Program-examples/04_MPI_C/mpihello.pbs @@ -13,6 +13,6 @@ cd $PBS_O_WORKDIR # load the environment module purge -module load intel +module load foss mpirun ./mpihello From 68e11c8ee5e34bf307e1357f37693a3deec6a8be Mon Sep 17 00:00:00 2001 From: EwDa291 Date: Fri, 2 Aug 2024 15:25:07 +0200 Subject: [PATCH 49/74] added FAQ about 'No space left on device' error --- mkdocs/docs/HPC/FAQ.md | 18 ++++++++++++++++++ 1 file changed, 18 insertions(+) diff --git a/mkdocs/docs/HPC/FAQ.md b/mkdocs/docs/HPC/FAQ.md index 398bd3e1e67..d62a2f5e503 100644 --- a/mkdocs/docs/HPC/FAQ.md +++ b/mkdocs/docs/HPC/FAQ.md @@ -226,6 +226,24 @@ information, see . {% endif %} +### Why do I get a "No space left on device" error, while there is still available space? + +When trying to create files, errors like this can occur: + +```shell +touch: cannot touch 'filename': No space left on device +``` + +The error "No space left on device" cann mean two different things: +either all allocated memory on the file system in question has been used, +or the inode limit has been reached on that file system. An inode can be seen as a "file slot", +meaning that when the limit is reached, no more files can be created. There is a standard inode limit in place that will be expanded if needed. +The number of inodes used per file system can be checked on . + +Possible solutions to this problem include cleaning up unused files and directories or +[compressing directories with a lot of files into zip- or tar-files](../linux-tutorial/manipulating_files_and_directories/?h=zip#zipping-gzipgunzip-zipunzip). +If the problem persists, feel free to contact support [here](./#i-have-another-questionproblem). + ## Other ### Can I share my account with someone else? From 9154f052ca00734fbbc88e49f2d19a5a05c4c09d Mon Sep 17 00:00:00 2001 From: EwDa291 Date: Fri, 2 Aug 2024 15:32:44 +0200 Subject: [PATCH 50/74] added FAQ about 'No space left on device' error --- mkdocs/docs/HPC/FAQ.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/mkdocs/docs/HPC/FAQ.md b/mkdocs/docs/HPC/FAQ.md index d62a2f5e503..a022a7859e3 100644 --- a/mkdocs/docs/HPC/FAQ.md +++ b/mkdocs/docs/HPC/FAQ.md @@ -234,7 +234,7 @@ When trying to create files, errors like this can occur: touch: cannot touch 'filename': No space left on device ``` -The error "No space left on device" cann mean two different things: +The error "No space left on device" can mean two different things: either all allocated memory on the file system in question has been used, or the inode limit has been reached on that file system. An inode can be seen as a "file slot", meaning that when the limit is reached, no more files can be created. There is a standard inode limit in place that will be expanded if needed. From 4cd0246b6413b8eb79b9f6c34a58d349b68a8051 Mon Sep 17 00:00:00 2001 From: EwDa291 <100782488+EwDa291@users.noreply.github.com> Date: Mon, 5 Aug 2024 10:44:14 +0200 Subject: [PATCH 51/74] Update mkdocs/docs/HPC/FAQ.md Co-authored-by: Kenneth Hoste --- mkdocs/docs/HPC/FAQ.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/mkdocs/docs/HPC/FAQ.md b/mkdocs/docs/HPC/FAQ.md index a022a7859e3..a1123ac5b52 100644 --- a/mkdocs/docs/HPC/FAQ.md +++ b/mkdocs/docs/HPC/FAQ.md @@ -226,7 +226,8 @@ information, see . {% endif %} -### Why do I get a "No space left on device" error, while there is still available space? + +### Why do I get a "No space left on device" error, while I still have storage space left? When trying to create files, errors like this can occur: From 712f7bbee29c944c01aa520e3c3b78021ff0f3fc Mon Sep 17 00:00:00 2001 From: EwDa291 <100782488+EwDa291@users.noreply.github.com> Date: Mon, 5 Aug 2024 10:44:23 +0200 Subject: [PATCH 52/74] Update mkdocs/docs/HPC/FAQ.md Co-authored-by: Kenneth Hoste --- mkdocs/docs/HPC/FAQ.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/mkdocs/docs/HPC/FAQ.md b/mkdocs/docs/HPC/FAQ.md index a1123ac5b52..94680200321 100644 --- a/mkdocs/docs/HPC/FAQ.md +++ b/mkdocs/docs/HPC/FAQ.md @@ -235,7 +235,7 @@ When trying to create files, errors like this can occur: touch: cannot touch 'filename': No space left on device ``` -The error "No space left on device" can mean two different things: +The error "`No space left on device`" can mean two different things: either all allocated memory on the file system in question has been used, or the inode limit has been reached on that file system. An inode can be seen as a "file slot", meaning that when the limit is reached, no more files can be created. There is a standard inode limit in place that will be expanded if needed. From ccb977f81bfaa04d67fd2593cb385755f24b1f0b Mon Sep 17 00:00:00 2001 From: EwDa291 <100782488+EwDa291@users.noreply.github.com> Date: Mon, 5 Aug 2024 10:44:29 +0200 Subject: [PATCH 53/74] Update mkdocs/docs/HPC/FAQ.md Co-authored-by: Kenneth Hoste --- mkdocs/docs/HPC/FAQ.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/mkdocs/docs/HPC/FAQ.md b/mkdocs/docs/HPC/FAQ.md index 94680200321..b2848114c83 100644 --- a/mkdocs/docs/HPC/FAQ.md +++ b/mkdocs/docs/HPC/FAQ.md @@ -236,7 +236,7 @@ touch: cannot touch 'filename': No space left on device ``` The error "`No space left on device`" can mean two different things: -either all allocated memory on the file system in question has been used, +either all available *storage quota* on the file system in question has been used, or the inode limit has been reached on that file system. An inode can be seen as a "file slot", meaning that when the limit is reached, no more files can be created. There is a standard inode limit in place that will be expanded if needed. The number of inodes used per file system can be checked on . From b1b50ccc9bcd9a8e1a0b7cdac687bf00930d6ca0 Mon Sep 17 00:00:00 2001 From: EwDa291 <100782488+EwDa291@users.noreply.github.com> Date: Mon, 5 Aug 2024 10:44:35 +0200 Subject: [PATCH 54/74] Update mkdocs/docs/HPC/FAQ.md Co-authored-by: Kenneth Hoste --- mkdocs/docs/HPC/FAQ.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/mkdocs/docs/HPC/FAQ.md b/mkdocs/docs/HPC/FAQ.md index b2848114c83..dabdf54ae5e 100644 --- a/mkdocs/docs/HPC/FAQ.md +++ b/mkdocs/docs/HPC/FAQ.md @@ -237,7 +237,7 @@ touch: cannot touch 'filename': No space left on device The error "`No space left on device`" can mean two different things: either all available *storage quota* on the file system in question has been used, -or the inode limit has been reached on that file system. An inode can be seen as a "file slot", +or the *inode limit* has been reached on that file system. An inode can be seen as a "file slot", meaning that when the limit is reached, no more files can be created. There is a standard inode limit in place that will be expanded if needed. The number of inodes used per file system can be checked on . From 06560750cd75bebed11208a1d390950c2adc83e5 Mon Sep 17 00:00:00 2001 From: EwDa291 <100782488+EwDa291@users.noreply.github.com> Date: Mon, 5 Aug 2024 10:44:43 +0200 Subject: [PATCH 55/74] Update mkdocs/docs/HPC/FAQ.md Co-authored-by: Kenneth Hoste --- mkdocs/docs/HPC/FAQ.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/mkdocs/docs/HPC/FAQ.md b/mkdocs/docs/HPC/FAQ.md index dabdf54ae5e..338bd7dca89 100644 --- a/mkdocs/docs/HPC/FAQ.md +++ b/mkdocs/docs/HPC/FAQ.md @@ -238,7 +238,8 @@ touch: cannot touch 'filename': No space left on device The error "`No space left on device`" can mean two different things: either all available *storage quota* on the file system in question has been used, or the *inode limit* has been reached on that file system. An inode can be seen as a "file slot", -meaning that when the limit is reached, no more files can be created. There is a standard inode limit in place that will be expanded if needed. +meaning that when the limit is reached, no more additional files can be created. +There is a standard inode limit in place that will be increased if needed. The number of inodes used per file system can be checked on . Possible solutions to this problem include cleaning up unused files and directories or From adb623404028a5f54daa09d0bfc0674d97b6b9eb Mon Sep 17 00:00:00 2001 From: EwDa291 <100782488+EwDa291@users.noreply.github.com> Date: Mon, 5 Aug 2024 10:44:52 +0200 Subject: [PATCH 56/74] Update mkdocs/docs/HPC/FAQ.md Co-authored-by: Kenneth Hoste --- mkdocs/docs/HPC/FAQ.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/mkdocs/docs/HPC/FAQ.md b/mkdocs/docs/HPC/FAQ.md index 338bd7dca89..e5ec0834b33 100644 --- a/mkdocs/docs/HPC/FAQ.md +++ b/mkdocs/docs/HPC/FAQ.md @@ -240,7 +240,7 @@ either all available *storage quota* on the file system in question has been use or the *inode limit* has been reached on that file system. An inode can be seen as a "file slot", meaning that when the limit is reached, no more additional files can be created. There is a standard inode limit in place that will be increased if needed. -The number of inodes used per file system can be checked on . +The number of inodes used per file system can be checked on [the VSC account page](https://account.vscentrum.be). Possible solutions to this problem include cleaning up unused files and directories or [compressing directories with a lot of files into zip- or tar-files](../linux-tutorial/manipulating_files_and_directories/?h=zip#zipping-gzipgunzip-zipunzip). From ed19f7caa439c39d5dff21863c27caf093ced491 Mon Sep 17 00:00:00 2001 From: EwDa291 <100782488+EwDa291@users.noreply.github.com> Date: Mon, 5 Aug 2024 10:44:58 +0200 Subject: [PATCH 57/74] Update mkdocs/docs/HPC/FAQ.md Co-authored-by: Kenneth Hoste --- mkdocs/docs/HPC/FAQ.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/mkdocs/docs/HPC/FAQ.md b/mkdocs/docs/HPC/FAQ.md index e5ec0834b33..6b82ca1ef09 100644 --- a/mkdocs/docs/HPC/FAQ.md +++ b/mkdocs/docs/HPC/FAQ.md @@ -244,7 +244,7 @@ The number of inodes used per file system can be checked on [the VSC account pag Possible solutions to this problem include cleaning up unused files and directories or [compressing directories with a lot of files into zip- or tar-files](../linux-tutorial/manipulating_files_and_directories/?h=zip#zipping-gzipgunzip-zipunzip). -If the problem persists, feel free to contact support [here](./#i-have-another-questionproblem). +If the problem persists, feel free to [contact support](./#i-have-another-questionproblem). ## Other From 35a2163290d1b7eaa05f434f3b7572a1045cc137 Mon Sep 17 00:00:00 2001 From: EwDa291 <100782488+EwDa291@users.noreply.github.com> Date: Mon, 5 Aug 2024 10:51:33 +0200 Subject: [PATCH 58/74] Added list for reasons for error --- mkdocs/docs/HPC/FAQ.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/mkdocs/docs/HPC/FAQ.md b/mkdocs/docs/HPC/FAQ.md index 6b82ca1ef09..6c540dd6f2e 100644 --- a/mkdocs/docs/HPC/FAQ.md +++ b/mkdocs/docs/HPC/FAQ.md @@ -236,9 +236,9 @@ touch: cannot touch 'filename': No space left on device ``` The error "`No space left on device`" can mean two different things: -either all available *storage quota* on the file system in question has been used, -or the *inode limit* has been reached on that file system. An inode can be seen as a "file slot", -meaning that when the limit is reached, no more additional files can be created. +- all available *storage quota* on the file system in question has been used; +- the *inode limit* has been reached on that file system. +An *inode* can be seen as a "file slot", meaning that when the limit is reached, no more additional files can be created. There is a standard inode limit in place that will be increased if needed. The number of inodes used per file system can be checked on [the VSC account page](https://account.vscentrum.be). From e1c0f30de68c76d43e4da0ee8a5d823cff93c0b2 Mon Sep 17 00:00:00 2001 From: lara Date: Mon, 5 Aug 2024 12:50:54 +0200 Subject: [PATCH 59/74] fix link to OpenFOAM example --- intro-HPC/examples/OpenFOAM/OpenFOAM_damBreak.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/intro-HPC/examples/OpenFOAM/OpenFOAM_damBreak.sh b/intro-HPC/examples/OpenFOAM/OpenFOAM_damBreak.sh index 021303e6715..ca1991e1a1e 100644 --- a/intro-HPC/examples/OpenFOAM/OpenFOAM_damBreak.sh +++ b/intro-HPC/examples/OpenFOAM/OpenFOAM_damBreak.sh @@ -15,7 +15,7 @@ export MYMPIRUN_VARIABLESPREFIX=WM_PROJECT,FOAM,MPI export WORKDIR=$VSC_SCRATCH_NODE/$PBS_JOBID # for single-node jobs mkdir -p $WORKDIR # damBreak tutorial, see also https://cfd.direct/openfoam/user-guide/dambreak -cp -r $FOAM_TUTORIALS/multiphase/interFoam/laminar/damBreak/damBreak $WORKDIR +cp -r $FOAM_TUTORIALS/incompressibleVoF/damBreakLaminar/damBreak $WORKDIR cd $WORKDIR/damBreak echo "working directory: $PWD" # pre-processing: generate mesh From 54f871dd6813cba8cba8fb560782fd4e94a39e6c Mon Sep 17 00:00:00 2001 From: EwDa291 <100782488+EwDa291@users.noreply.github.com> Date: Mon, 5 Aug 2024 13:06:57 +0200 Subject: [PATCH 60/74] Update FAQ.md --- mkdocs/docs/HPC/FAQ.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/mkdocs/docs/HPC/FAQ.md b/mkdocs/docs/HPC/FAQ.md index 6c540dd6f2e..11ffd1afe87 100644 --- a/mkdocs/docs/HPC/FAQ.md +++ b/mkdocs/docs/HPC/FAQ.md @@ -232,12 +232,14 @@ information, see . When trying to create files, errors like this can occur: ```shell -touch: cannot touch 'filename': No space left on device +No space left on device ``` The error "`No space left on device`" can mean two different things: + - all available *storage quota* on the file system in question has been used; - the *inode limit* has been reached on that file system. +- An *inode* can be seen as a "file slot", meaning that when the limit is reached, no more additional files can be created. There is a standard inode limit in place that will be increased if needed. The number of inodes used per file system can be checked on [the VSC account page](https://account.vscentrum.be). From c978c3c3cc3a76ca724ca735df7d440e346b7886 Mon Sep 17 00:00:00 2001 From: EwDa291 <100782488+EwDa291@users.noreply.github.com> Date: Mon, 5 Aug 2024 13:07:40 +0200 Subject: [PATCH 61/74] Update FAQ.md: resolving some formatting issues --- mkdocs/docs/HPC/FAQ.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/mkdocs/docs/HPC/FAQ.md b/mkdocs/docs/HPC/FAQ.md index 11ffd1afe87..8dc97c60ac4 100644 --- a/mkdocs/docs/HPC/FAQ.md +++ b/mkdocs/docs/HPC/FAQ.md @@ -245,7 +245,8 @@ There is a standard inode limit in place that will be increased if needed. The number of inodes used per file system can be checked on [the VSC account page](https://account.vscentrum.be). Possible solutions to this problem include cleaning up unused files and directories or -[compressing directories with a lot of files into zip- or tar-files](../linux-tutorial/manipulating_files_and_directories/?h=zip#zipping-gzipgunzip-zipunzip). +[compressing directories with a lot of files into zip- or tar-files](../linux-tutorial/manipulating_files_and_directories/?h=zip#zipping-gzipgunzip-zipunzip). + If the problem persists, feel free to [contact support](./#i-have-another-questionproblem). ## Other From 8e876f563827717e232cad082ec82f69e5c74fe9 Mon Sep 17 00:00:00 2001 From: EwDa291 <100782488+EwDa291@users.noreply.github.com> Date: Mon, 5 Aug 2024 13:38:06 +0200 Subject: [PATCH 62/74] Update FAQ.md: fixing formatting issues --- mkdocs/docs/HPC/FAQ.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/mkdocs/docs/HPC/FAQ.md b/mkdocs/docs/HPC/FAQ.md index 8dc97c60ac4..7e92f45bd93 100644 --- a/mkdocs/docs/HPC/FAQ.md +++ b/mkdocs/docs/HPC/FAQ.md @@ -239,7 +239,7 @@ The error "`No space left on device`" can mean two different things: - all available *storage quota* on the file system in question has been used; - the *inode limit* has been reached on that file system. -- + An *inode* can be seen as a "file slot", meaning that when the limit is reached, no more additional files can be created. There is a standard inode limit in place that will be increased if needed. The number of inodes used per file system can be checked on [the VSC account page](https://account.vscentrum.be). From bb780f668c82784a45d0fc71d343c8188dfb70c7 Mon Sep 17 00:00:00 2001 From: EwDa291 <100782488+EwDa291@users.noreply.github.com> Date: Mon, 5 Aug 2024 15:10:29 +0200 Subject: [PATCH 63/74] Update FAQ.md: changed "Troubleshooting jobs" to "Troubleshooting" --- mkdocs/docs/HPC/FAQ.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/mkdocs/docs/HPC/FAQ.md b/mkdocs/docs/HPC/FAQ.md index 7e92f45bd93..6c855ea06e6 100644 --- a/mkdocs/docs/HPC/FAQ.md +++ b/mkdocs/docs/HPC/FAQ.md @@ -74,7 +74,7 @@ It is possible to use the modules without specifying a version or toolchain. How this will probably cause incompatible modules to be loaded. Don't do it if you use multiple modules. Even if it works now, as more modules get installed on the HPC, your job can suddenly break. -## Troubleshooting jobs +## Troubleshooting ### My modules don't work together From 0d89405f6dab8db11cacc9f8cf5f46628703eb9f Mon Sep 17 00:00:00 2001 From: EwDa291 Date: Mon, 5 Aug 2024 15:43:25 +0200 Subject: [PATCH 64/74] New FAQ for du command --- mkdocs/docs/HPC/FAQ.md | 19 +++++++++++++++++++ 1 file changed, 19 insertions(+) diff --git a/mkdocs/docs/HPC/FAQ.md b/mkdocs/docs/HPC/FAQ.md index 6c855ea06e6..228be00978f 100644 --- a/mkdocs/docs/HPC/FAQ.md +++ b/mkdocs/docs/HPC/FAQ.md @@ -373,6 +373,25 @@ See also: [Your UGent home drive and shares](running_jobs_with_input_output_data {% endif %} +### How do I find the largest files I should remove to make room on my home directory + +To free up space in the home directory, you can make use of the [`du` command](https://docs.hpc.ugent.be/Linux/running_jobs_with_input_output_data/#check-your-quota) to find out what the largest files and subdirectories are: + +```shell +du -h --max-depth 1 $VSC_HOME | egrep '[0-9]{3}M|[0-9]G' +``` + +The `du` command returns the size of every file and subdirectory in the $VSC_HOME directory. +This output is then piped into an [`egrep`](https://docs.hpc.ugent.be/Linux/linux-tutorial/beyond_the_basics/?h=grep#searching-file-contents-grep), +where only entries that are large enough are let through. +The expression that accomplishes this is `[0-9]{3}M|[0-9]G`, which is structured as follows: + +- `[0-9]{3}M` accepts entries with 3 digits followed by an M (So files and subdirectories with a size between 100 and 1023 MB). +- `[0-9]G` accepts entries with a digit followed by a G (So files and subdirectories with a size between 1 and 1023 GB). + +If an entry complies with either of these conditions, it is let through and printed as output. + + ### Why can't I use the `sudo` command? When you attempt to use sudo, you will be prompted for a password. From 459c83fe867ff19f83de88181c439bd327bb491d Mon Sep 17 00:00:00 2001 From: EwDa291 <100782488+EwDa291@users.noreply.github.com> Date: Tue, 6 Aug 2024 11:35:21 +0200 Subject: [PATCH 65/74] Update FAQ.md: changes to the du-command FAQ --- mkdocs/docs/HPC/FAQ.md | 14 ++++---------- 1 file changed, 4 insertions(+), 10 deletions(-) diff --git a/mkdocs/docs/HPC/FAQ.md b/mkdocs/docs/HPC/FAQ.md index 228be00978f..2212aab1006 100644 --- a/mkdocs/docs/HPC/FAQ.md +++ b/mkdocs/docs/HPC/FAQ.md @@ -373,23 +373,17 @@ See also: [Your UGent home drive and shares](running_jobs_with_input_output_data {% endif %} -### How do I find the largest files I should remove to make room on my home directory +### My home directory is (almost) full, and I don't know why -To free up space in the home directory, you can make use of the [`du` command](https://docs.hpc.ugent.be/Linux/running_jobs_with_input_output_data/#check-your-quota) to find out what the largest files and subdirectories are: +Your home directory might be full without looking like it due to hidden files. Hidden files and subdirectories have a name starting with a dot and do not show up when using a normal `ls`. If you want to check where the storage in your home directory is used, you can make use of the [`du` command](https://docs.hpc.ugent.be/Linux/running_jobs_with_input_output_data/#check-your-quota) to find out what the largest files and subdirectories are: ```shell du -h --max-depth 1 $VSC_HOME | egrep '[0-9]{3}M|[0-9]G' ``` -The `du` command returns the size of every file and subdirectory in the $VSC_HOME directory. -This output is then piped into an [`egrep`](https://docs.hpc.ugent.be/Linux/linux-tutorial/beyond_the_basics/?h=grep#searching-file-contents-grep), -where only entries that are large enough are let through. -The expression that accomplishes this is `[0-9]{3}M|[0-9]G`, which is structured as follows: +The `du` command returns the size of every file and subdirectory in the $VSC_HOME directory. This output is then piped into an [`egrep`](https://docs.hpc.ugent.be/Linux/linux-tutorial/beyond_the_basics/?h=grep#searching-file-contents-grep) to filter the lines to the ones that matter the most. -- `[0-9]{3}M` accepts entries with 3 digits followed by an M (So files and subdirectories with a size between 100 and 1023 MB). -- `[0-9]G` accepts entries with a digit followed by a G (So files and subdirectories with a size between 1 and 1023 GB). - -If an entry complies with either of these conditions, it is let through and printed as output. +The `egrep` command will only let entries that match with the specified regular expression `[0-9]{3}M|[0-9]G` through, which corresponds with files that consume between 100 and 1024 MB or more than 1 GB. ### Why can't I use the `sudo` command? From c358fa59fbe1410ee262b4fea6cae83d8e3549c7 Mon Sep 17 00:00:00 2001 From: EwDa291 <100782488+EwDa291@users.noreply.github.com> Date: Tue, 6 Aug 2024 14:01:17 +0200 Subject: [PATCH 66/74] Update mkdocs/docs/HPC/FAQ.md Small tweaks for improved readability Co-authored-by: Kenneth Hoste --- mkdocs/docs/HPC/FAQ.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/mkdocs/docs/HPC/FAQ.md b/mkdocs/docs/HPC/FAQ.md index 2212aab1006..4639f111c8c 100644 --- a/mkdocs/docs/HPC/FAQ.md +++ b/mkdocs/docs/HPC/FAQ.md @@ -375,7 +375,9 @@ See also: [Your UGent home drive and shares](running_jobs_with_input_output_data ### My home directory is (almost) full, and I don't know why -Your home directory might be full without looking like it due to hidden files. Hidden files and subdirectories have a name starting with a dot and do not show up when using a normal `ls`. If you want to check where the storage in your home directory is used, you can make use of the [`du` command](https://docs.hpc.ugent.be/Linux/running_jobs_with_input_output_data/#check-your-quota) to find out what the largest files and subdirectories are: +Your home directory might be full without looking like it due to hidden files. +Hidden files and subdirectories have a name starting with a dot and do not show up when running `ls`. +If you want to check where the storage in your home directory is used, you can make use of the [`du` command](https://docs.hpc.ugent.be/Linux/running_jobs_with_input_output_data/#check-your-quota) to find out what the largest files and subdirectories are: ```shell du -h --max-depth 1 $VSC_HOME | egrep '[0-9]{3}M|[0-9]G' From 94f4c85be7eb54eaaafd8fa88205d10b86d87694 Mon Sep 17 00:00:00 2001 From: EwDa291 <100782488+EwDa291@users.noreply.github.com> Date: Tue, 6 Aug 2024 14:04:26 +0200 Subject: [PATCH 67/74] Update FAQ.md --- mkdocs/docs/HPC/FAQ.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/mkdocs/docs/HPC/FAQ.md b/mkdocs/docs/HPC/FAQ.md index 4639f111c8c..ab9a0cceded 100644 --- a/mkdocs/docs/HPC/FAQ.md +++ b/mkdocs/docs/HPC/FAQ.md @@ -385,7 +385,7 @@ du -h --max-depth 1 $VSC_HOME | egrep '[0-9]{3}M|[0-9]G' The `du` command returns the size of every file and subdirectory in the $VSC_HOME directory. This output is then piped into an [`egrep`](https://docs.hpc.ugent.be/Linux/linux-tutorial/beyond_the_basics/?h=grep#searching-file-contents-grep) to filter the lines to the ones that matter the most. -The `egrep` command will only let entries that match with the specified regular expression `[0-9]{3}M|[0-9]G` through, which corresponds with files that consume between 100 and 1024 MB or more than 1 GB. +The `egrep` command will only let entries that match with the specified regular expression `[0-9]{3}M|[0-9]G` through, which corresponds with files that consume more than 100 MB. ### Why can't I use the `sudo` command? From a7b7a90a5d5807bd6b78390bfb04413256bcb840 Mon Sep 17 00:00:00 2001 From: EwDa291 Date: Tue, 6 Aug 2024 14:32:20 +0200 Subject: [PATCH 68/74] added FAQ about getting more storage space as a VO --- mkdocs/docs/HPC/FAQ.md | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/mkdocs/docs/HPC/FAQ.md b/mkdocs/docs/HPC/FAQ.md index ab9a0cceded..4e5b2496e35 100644 --- a/mkdocs/docs/HPC/FAQ.md +++ b/mkdocs/docs/HPC/FAQ.md @@ -388,6 +388,11 @@ The `du` command returns the size of every file and subdirectory in the $VSC_HOM The `egrep` command will only let entries that match with the specified regular expression `[0-9]{3}M|[0-9]G` through, which corresponds with files that consume more than 100 MB. +### How can I get more storage space + +[By default](https://docs.hpc.ugent.be/running_jobs_with_input_output_data/#quota) you get 3 GB of storage space for your home directory and 25 GB in the data and scratch filesystems. It is not possible to expand these quota. However it is possible to get more storage space through a [Virtual Organisation](https://docs.hpc.ugent.be/running_jobs_with_input_output_data/#virtual-organisations), which will also give you access to the [extra directories](https://docs.hpc.ugent.be/running_jobs_with_input_output_data/#vo-directories) related to that VO. The moderator of a VO can [request more storage](https://docs.hpc.ugent.be/running_jobs_with_input_output_data/#requesting-more-storage-space) for their VO. + + ### Why can't I use the `sudo` command? When you attempt to use sudo, you will be prompted for a password. From 694695a5894622542d8f42bb7bb194404f0a218d Mon Sep 17 00:00:00 2001 From: EwDa291 <100782488+EwDa291@users.noreply.github.com> Date: Tue, 6 Aug 2024 16:38:59 +0200 Subject: [PATCH 69/74] Update mkdocs/docs/HPC/FAQ.md Co-authored-by: Kenneth Hoste --- mkdocs/docs/HPC/FAQ.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/mkdocs/docs/HPC/FAQ.md b/mkdocs/docs/HPC/FAQ.md index 4e5b2496e35..4738fd2122a 100644 --- a/mkdocs/docs/HPC/FAQ.md +++ b/mkdocs/docs/HPC/FAQ.md @@ -388,7 +388,7 @@ The `du` command returns the size of every file and subdirectory in the $VSC_HOM The `egrep` command will only let entries that match with the specified regular expression `[0-9]{3}M|[0-9]G` through, which corresponds with files that consume more than 100 MB. -### How can I get more storage space +### How can I get more storage space? [By default](https://docs.hpc.ugent.be/running_jobs_with_input_output_data/#quota) you get 3 GB of storage space for your home directory and 25 GB in the data and scratch filesystems. It is not possible to expand these quota. However it is possible to get more storage space through a [Virtual Organisation](https://docs.hpc.ugent.be/running_jobs_with_input_output_data/#virtual-organisations), which will also give you access to the [extra directories](https://docs.hpc.ugent.be/running_jobs_with_input_output_data/#vo-directories) related to that VO. The moderator of a VO can [request more storage](https://docs.hpc.ugent.be/running_jobs_with_input_output_data/#requesting-more-storage-space) for their VO. From 8f8b953abb0e79820ebbc44b81cc64b663e09404 Mon Sep 17 00:00:00 2001 From: EwDa291 <100782488+EwDa291@users.noreply.github.com> Date: Tue, 6 Aug 2024 16:39:50 +0200 Subject: [PATCH 70/74] Update mkdocs/docs/HPC/FAQ.md Co-authored-by: Kenneth Hoste --- mkdocs/docs/HPC/FAQ.md | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-) diff --git a/mkdocs/docs/HPC/FAQ.md b/mkdocs/docs/HPC/FAQ.md index 4738fd2122a..0e101e82461 100644 --- a/mkdocs/docs/HPC/FAQ.md +++ b/mkdocs/docs/HPC/FAQ.md @@ -390,7 +390,13 @@ The `egrep` command will only let entries that match with the specified regular ### How can I get more storage space? -[By default](https://docs.hpc.ugent.be/running_jobs_with_input_output_data/#quota) you get 3 GB of storage space for your home directory and 25 GB in the data and scratch filesystems. It is not possible to expand these quota. However it is possible to get more storage space through a [Virtual Organisation](https://docs.hpc.ugent.be/running_jobs_with_input_output_data/#virtual-organisations), which will also give you access to the [extra directories](https://docs.hpc.ugent.be/running_jobs_with_input_output_data/#vo-directories) related to that VO. The moderator of a VO can [request more storage](https://docs.hpc.ugent.be/running_jobs_with_input_output_data/#requesting-more-storage-space) for their VO. + +[By default](https://docs.hpc.ugent.be/running_jobs_with_input_output_data/#quota) you get 3 GB of storage space for your home directory and 25 GB in your personal directories on both the data (`$VSC_DATA`) and scratch (`$VSC_SCRATCH`) filesystems. +It is not possible to expand the storage quota for these personal directories. + +You can get more storage space through a [Virtual Organisation (VO)](https://docs.hpc.ugent.be/running_jobs_with_input_output_data/#virtual-organisations), +which will give you access to the [additional directories](https://docs.hpc.ugent.be/running_jobs_with_input_output_data/#vo-directories) in a subdirectory specific to that VO (`$VSC_DATA_VO` and `$VSC_SCRATCH_VO`). +The moderators of a VO can [request more storage](https://docs.hpc.ugent.be/running_jobs_with_input_output_data/#requesting-more-storage-space) for their VO. ### Why can't I use the `sudo` command? From ce0d2720ea9a17a1e5d2931c32f8c2877b14a2fc Mon Sep 17 00:00:00 2001 From: EwDa291 <100782488+EwDa291@users.noreply.github.com> Date: Wed, 7 Aug 2024 09:17:52 +0200 Subject: [PATCH 71/74] Update mkdocs/docs/HPC/FAQ.md Co-authored-by: Kenneth Hoste --- mkdocs/docs/HPC/FAQ.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/mkdocs/docs/HPC/FAQ.md b/mkdocs/docs/HPC/FAQ.md index 0e101e82461..40701d37084 100644 --- a/mkdocs/docs/HPC/FAQ.md +++ b/mkdocs/docs/HPC/FAQ.md @@ -391,7 +391,7 @@ The `egrep` command will only let entries that match with the specified regular ### How can I get more storage space? -[By default](https://docs.hpc.ugent.be/running_jobs_with_input_output_data/#quota) you get 3 GB of storage space for your home directory and 25 GB in your personal directories on both the data (`$VSC_DATA`) and scratch (`$VSC_SCRATCH`) filesystems. +[By default](running_jobs_with_input_output_data.md#quota) you get 3 GB of storage space for your home directory and 25 GB in your personal directories on both the data (`$VSC_DATA`) and scratch (`$VSC_SCRATCH`) filesystems. It is not possible to expand the storage quota for these personal directories. You can get more storage space through a [Virtual Organisation (VO)](https://docs.hpc.ugent.be/running_jobs_with_input_output_data/#virtual-organisations), From 833ddcc09569f5c6a6e24e873c865c4e5bc7b8be Mon Sep 17 00:00:00 2001 From: EwDa291 <100782488+EwDa291@users.noreply.github.com> Date: Wed, 7 Aug 2024 09:38:18 +0200 Subject: [PATCH 72/74] Update FAQ.md: changed the links from a few FAQ's --- mkdocs/docs/HPC/FAQ.md | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/mkdocs/docs/HPC/FAQ.md b/mkdocs/docs/HPC/FAQ.md index 40701d37084..48e9d705c33 100644 --- a/mkdocs/docs/HPC/FAQ.md +++ b/mkdocs/docs/HPC/FAQ.md @@ -245,9 +245,9 @@ There is a standard inode limit in place that will be increased if needed. The number of inodes used per file system can be checked on [the VSC account page](https://account.vscentrum.be). Possible solutions to this problem include cleaning up unused files and directories or -[compressing directories with a lot of files into zip- or tar-files](../linux-tutorial/manipulating_files_and_directories/?h=zip#zipping-gzipgunzip-zipunzip). +[compressing directories with a lot of files into zip- or tar-files](linux-tutorial/manipulating_files_and_directories.md#zipping-gzipgunzip-zipunzip). -If the problem persists, feel free to [contact support](./#i-have-another-questionproblem). +If the problem persists, feel free to [contact support](FAQ.md#i-have-another-questionproblem). ## Other @@ -377,13 +377,13 @@ See also: [Your UGent home drive and shares](running_jobs_with_input_output_data Your home directory might be full without looking like it due to hidden files. Hidden files and subdirectories have a name starting with a dot and do not show up when running `ls`. -If you want to check where the storage in your home directory is used, you can make use of the [`du` command](https://docs.hpc.ugent.be/Linux/running_jobs_with_input_output_data/#check-your-quota) to find out what the largest files and subdirectories are: +If you want to check where the storage in your home directory is used, you can make use of the [`du` command](running_jobs_with_input_output_data.md#check-your-quota) to find out what the largest files and subdirectories are: ```shell du -h --max-depth 1 $VSC_HOME | egrep '[0-9]{3}M|[0-9]G' ``` -The `du` command returns the size of every file and subdirectory in the $VSC_HOME directory. This output is then piped into an [`egrep`](https://docs.hpc.ugent.be/Linux/linux-tutorial/beyond_the_basics/?h=grep#searching-file-contents-grep) to filter the lines to the ones that matter the most. +The `du` command returns the size of every file and subdirectory in the $VSC_HOME directory. This output is then piped into an [`egrep`](linux-tutorial/beyond_the_basics.md#searching-file-contents-grep) to filter the lines to the ones that matter the most. The `egrep` command will only let entries that match with the specified regular expression `[0-9]{3}M|[0-9]G` through, which corresponds with files that consume more than 100 MB. @@ -394,9 +394,9 @@ The `egrep` command will only let entries that match with the specified regular [By default](running_jobs_with_input_output_data.md#quota) you get 3 GB of storage space for your home directory and 25 GB in your personal directories on both the data (`$VSC_DATA`) and scratch (`$VSC_SCRATCH`) filesystems. It is not possible to expand the storage quota for these personal directories. -You can get more storage space through a [Virtual Organisation (VO)](https://docs.hpc.ugent.be/running_jobs_with_input_output_data/#virtual-organisations), -which will give you access to the [additional directories](https://docs.hpc.ugent.be/running_jobs_with_input_output_data/#vo-directories) in a subdirectory specific to that VO (`$VSC_DATA_VO` and `$VSC_SCRATCH_VO`). -The moderators of a VO can [request more storage](https://docs.hpc.ugent.be/running_jobs_with_input_output_data/#requesting-more-storage-space) for their VO. +You can get more storage space through a [Virtual Organisation (VO)](running_jobs_with_input_output_data.md#virtual-organisations), +which will give you access to the [additional directories](running_jobs_with_input_output_data.md#vo-directories) in a subdirectory specific to that VO (`$VSC_DATA_VO` and `$VSC_SCRATCH_VO`). +The moderators of a VO can [request more storage](running_jobs_with_input_output_data.md#requesting-more-storage-space) for their VO. ### Why can't I use the `sudo` command? From c764da39e9ad807f2a0dd5273f8a8f67b4fff77a Mon Sep 17 00:00:00 2001 From: EwDa291 Date: Wed, 14 Aug 2024 14:34:01 +0200 Subject: [PATCH 73/74] reformatted to use markdown --- .../HPC/linux-tutorial/getting_started.md | 186 ++++++++---------- 1 file changed, 87 insertions(+), 99 deletions(-) diff --git a/mkdocs/docs/HPC/linux-tutorial/getting_started.md b/mkdocs/docs/HPC/linux-tutorial/getting_started.md index 735194999c1..1a9a20aeeac 100644 --- a/mkdocs/docs/HPC/linux-tutorial/getting_started.md +++ b/mkdocs/docs/HPC/linux-tutorial/getting_started.md @@ -1,37 +1,33 @@ -Getting Started -=============== +# Getting Started -Logging in ----------- +## Logging in To get started with the HPC-UGent infrastructure, you need to obtain a -VSC account, see [HPC manual](../account.md). -**Keep in mind that you must keep your private key to yourself!** +VSC account, see [HPC manual](../account.md). **Keep in mind that you +must keep your private key to yourself!** You can look at your public/private key pair as a lock and a key: you give us the lock (your public key), we put it on the door, and then you can use your key to open the door and get access to the HPC infrastructure. **Anyone who has your key can use your VSC account!** -Details on connecting to the HPC infrastructure are available in -[HPC manual connecting section](../connecting.md). +Details on connecting to the HPC infrastructure are available in [HPC +manual connecting section](../connecting.md). -Getting help ------------- +## Getting help To get help: -1. use the documentation available on the system, through the `help`, - `info` and `man` commands (use `q` to exit). +1. use the documentation available on the system, through the + `help`, `info` and `man` commands (use `q` to exit). + ``` + help cd + info ls + man cp ``` - help cd - info ls - man cp - ``` - 2. use Google -3. contact {{hpcinfo}} in case +3. contact {{hpcinfo}} in case of problems or questions (even for basic things!) ### Errors @@ -40,30 +36,30 @@ Sometimes when executing a command, an error occurs. Most likely there will be error output or a message explaining you this. Read this carefully and try to act on it. Try googling the error first to find any possible solution, but if you can't come up with something in 15 -minutes, don't hesitate to mail {{hpcinfo}}. +minutes, don't hesitate to mail +{{hpcinfo}}. -Basic terminal usage --------------------- +## Basic terminal usage The basic interface is the so-called shell prompt, typically ending with `$` (for `bash` shells). -You use the shell by executing commands, and hitting ``. For -example: -
$ echo hello
-hello
-
+You use the shell by executing commands, and hitting +``. For example: +
$ echo hello 
+ hello 
-You can go to the start or end of the command line using `Ctrl-A` or -`Ctrl-E`. +You can go to the start or end of the command line using +`Ctrl-A` or `Ctrl-E`. -To go through previous commands, use `` and ``, rather than -retyping them. +To go through previous commands, use `` and +``, rather than retyping them. ### Command history A powerful feature is that you can "search" through your command -history, either using the `history` command, or using `Ctrl-R`: +history, either using the `history` command, or using +`Ctrl-R`:
$ history
     1 echo hello
 
@@ -74,11 +70,11 @@ history, either using the `history` command, or using `Ctrl-R`:
 ### Stopping commands
 
 If for any reason you want to stop a command from executing, press
-`Ctrl-C`. For example, if a command is taking too long, or you want to
-rerun it with different arguments.
+`Ctrl-C`. For example, if a command is taking too long, or
+you want to rerun it with different arguments.
+
+## Variables
 
-Variables
----------
 [//]: # (sec:environment-variables())
 
 At the prompt we also have access to shell variables, which have both a
@@ -89,7 +85,7 @@ They can be thought of as placeholders for things we need to remember.
 For example, to print the path to your home directory, we can use the
 shell variable named `HOME`:
 
-
$ echo $HOME
+
$ echo $HOME 
 /user/home/gent/vsc400/vsc40000
 
@@ -98,38 +94,40 @@ This prints the value of this variable. ### Defining variables There are several variables already defined for you when you start your -session, such as `$HOME` which contains the path to your home directory. +session, such as `$HOME` which contains the path to your +home directory. For a full overview of defined environment variables in your current -session, you can use the `env` command. You can sort this output with -`sort` to make it easier to search in: +session, you can use the `env` command. You can sort this +output with `sort` to make it easier to search in: -
$ env | sort
-...
-HOME=/user/home/gent/vsc400/vsc40000
+
$ env | sort 
 ...
-
+HOME=/user/home/gent/vsc400/vsc40000 +...
-You can also use the `grep` command to search for a piece of text. The -following command will output all VSC-specific variable names and their -values: +You can also use the `grep` command to search for a piece of +text. The following command will output all VSC-specific variable names +and their values:
$ env | sort | grep VSC
-But we can also define our own. this is done with the `export` command -(note: variables are always all-caps as a convention): +But we can also define our own. this is done with the +`export` command (note: variables are always all-caps as a +convention):
$ export MYVARIABLE="value"
-It is important you don't include spaces around the `=` sign. Also note -the lack of `$` sign in front of the variable name. +It is important you don't include spaces around the `=` +sign. Also note the lack of `$` sign in front of the +variable name. -If we then do +If we then do
$ echo $MYVARIABLE
-this will output `value`. Note that the quotes are not included, they -were only used when defining the variable to escape potential spaces in -the value. +this will output `value`. Note that the quotes are not +included, they were only used when defining the variable to escape +potential spaces in the value. #### Changing your prompt using `$PS1` @@ -138,11 +136,11 @@ special-purpose variable `$PS1`. For example: to include the current location in your prompt:
$ export PS1='\w $'
-~ $ cd test
-~/test $
-
+~ $ cd test +~/test $
-Note that `~` is short representation of your home directory. +Note that `~` is short representation of your home +directory. To make this persistent across session, you can define this custom value for `$PS1` in your `.profile` startup script: @@ -154,71 +152,61 @@ One common pitfall is the (accidental) use of non-defined variables. Contrary to what you may expect, this does *not* result in error messages, but the variable is considered to be *empty* instead. -This may lead to surprising results, for example: -
$ export WORKDIR=/tmp/test
+This may lead to surprising results, for example: 
+
$ export WORKDIR=/tmp/test 
 $ pwd
-/user/home/gent/vsc400/vsc40000
+/user/home/gent/vsc400/vsc40000 
 $ echo $HOME
-/user/home/gent/vsc400/vsc40000
-
+/user/home/gent/vsc400/vsc40000
To understand what's going on here, see the section on `cd` below. -The moral here is: **be very careful to not use empty variables unintentionally**. +The moral here is: **be very careful to not use empty variables +unintentionally**. -**Tip for job scripts: use `set -e -u` to avoid using empty variables accidentally.** +**Tip for job scripts: use `set -e -u` to avoid using empty variables +accidentally.** -The `-e` option will result in the script getting stopped if any command -fails. +The `-e` option will result in the script getting stopped if +any command fails. -The `-u` option will result in the script getting stopped if empty -variables are used. (see for a more -detailed explanation and more options) +The `-u` option will result in the script getting stopped if +empty variables are used. (see for +a more detailed explanation and more options) -More information can be found at . +More information can be found at +. ### Restoring your default environment -If you've made a mess of your environment, you shouldn't waste too much -time trying to fix it. Just log out and log in again and you will be -given a pristine environment. +If you've made a mess of your environment, you shouldn't waste too +much time trying to fix it. Just log out and log in again and you will +be given a pristine environment. -Basic system information ------------------------- +## Basic system information Basic information about the system you are logged into can be obtained in a variety of ways. -We limit ourselves to determining the hostname: -
$ hostname
+We limit ourselves to determining the hostname: 
+
$ hostname 
 gligar01.gligar.os
 
-$ echo $HOSTNAME
-gligar01.gligar.os
+$ echo $HOSTNAME 
+gligar01.gligar.os 
 
And querying some basic information about the Linux kernel: -
$ uname -a
+
$ uname -a 
 Linux gligar01.gligar.os 2.6.32-573.8.1.el6.ug.x86_64 #1 SMP Mon Nov 16 15:12:09
-    CET 2015 x86_64 x86_64 x86_64 GNU/Linux
-
- - - -Exercises ---------- - -- Print the full path to your home directory - -- Determine the name of the environment variable to your personal - scratch directory - -- What's the name of the system you're logged into? Is it the same for - everyone? + CET 2015 x86_64 x86_64 x86_64 GNU/Linux
-- Figure out how to print the value of a variable without including a - newline +## Exercises -- How do you get help on using the `man` command? +- Print the full path to your home directory +- Determine the name of the environment variable to your personal scratch directory +- What's the name of the system you\'re logged into? Is it the same for everyone? +- Figure out how to print the value of a variable without including a newline +- How do you get help on using the `man` command? Next [chapter](navigating.md) teaches you on how to navigate. From d0915be31582efdd8a586f0e0fecff6996983681 Mon Sep 17 00:00:00 2001 From: EwDa291 Date: Wed, 14 Aug 2024 15:16:48 +0200 Subject: [PATCH 74/74] reformatted to use markdown --- mkdocs/docs/HPC/linux-tutorial/navigating.md | 249 +++++++++---------- 1 file changed, 117 insertions(+), 132 deletions(-) diff --git a/mkdocs/docs/HPC/linux-tutorial/navigating.md b/mkdocs/docs/HPC/linux-tutorial/navigating.md index bd882acafbb..030f7b5da54 100644 --- a/mkdocs/docs/HPC/linux-tutorial/navigating.md +++ b/mkdocs/docs/HPC/linux-tutorial/navigating.md @@ -1,210 +1,195 @@ -Navigating -========== +# Navigating -This chapter serves as a guide to navigating within a Linux shell, giving users essential techniques to traverse directories. -A very important skill. +This chapter serves as a guide to navigating within a Linux shell, +giving users essential techniques to traverse directories. A very +important skill. -Current directory: "pwd" and "\$PWD" ------------------------------------- +## Current directory: "pwd" and "$PWD" -To print the current directory, use `pwd` or `$PWD`: -
$ cd $HOME
+To print the current directory, use `pwd` or `\$PWD`:
+
$ cd $HOME 
 $ pwd
-/user/home/gent/vsc400/vsc40000
-$ echo "The current directory is: $PWD"
+/user/home/gent/vsc400/vsc40000 
+$ echo "The current directory is: $PWD" 
 The current directory is: /user/home/gent/vsc400/vsc40000
 
-Listing files and directories: "ls" ------------------------------------ +## Listing files and directories: "ls" -A very basic and commonly used command is `ls`, which can be used to -list files and directories. +A very basic and commonly used command is `ls`, which can be +used to list files and directories. -In its basic usage, it just prints the names of files and directories -in the current directory. For example: -
$ ls
-afile.txt some_directory
-
+In its basic usage, it just prints the names of files and directories in +the current directory. For example:
$ ls
+afile.txt some_directory 
When provided an argument, it can be used to list the contents of a -directory: -
$ ls some_directory
-one.txt two.txt
-
- +directory:
$ ls some_directory 
+one.txt two.txt 
A couple of commonly used options include: -- detailed listing using `ls -l`: -
$ ls -l
-total 4224
--rw-rw-r-- 1 vsc40000 vsc40000 2157404 Apr 12 13:17 afile.txt
-drwxrwxr-x 2 vsc40000 vsc40000 512 Apr 12 12:51 some_directory
- -- To print the size information in human-readable form, use the `-h` - flag: -
$ ls -lh
-total 4.1M
--rw-rw-r-- 1 vsc40000 vsc40000 2.1M Apr 12 13:16 afile.txt
-drwxrwxr-x 2 vsc40000 vsc40000 512 Apr 12 12:51 some_directory
- -- also listing hidden files using the `-a` flag: - -
$ ls -lah
-total 3.9M
-drwxrwxr-x 3 vsc40000 vsc40000 512 Apr 12 13:11 .
-drwx------ 188 vsc40000 vsc40000 128K Apr 12 12:41 ..
--rw-rw-r-- 1 vsc40000 vsc40000 1.8M Apr 12 13:12 afile.txt
--rw-rw-r-- 1 vsc40000 vsc40000 0 Apr 12 13:11 .hidden_file.txt
-drwxrwxr-x 2 vsc40000 vsc40000 512 Apr 12 12:51 some_directory
- -- ordering files by the most recent change using `-rt`: - -
$ ls -lrth
-total 4.0M
-drwxrwxr-x 2 vsc40000 vsc40000 512 Apr 12 12:51 some_directory
--rw-rw-r-- 1 vsc40000 vsc40000 2.0M Apr 12 13:15 afile.txt
- -If you try to use `ls` on a file that doesn't exist, you will get a -clear error message: -
$ ls nosuchfile
+- detailed listing using `ls -l`:
+
+:   
$ ls -l
+	total 4224 
+	-rw-rw-r-- 1 vsc40000 vsc40000 2157404 Apr 12 13:17 afile.txt 
+	drwxrwxr-x 2 vsc40000 vsc40000 512 Apr 12 12:51 some_directory
+ +- To print the size information in human-readable form, use the `-h` flag: + +:
$ ls -lh
+	total 4.1M 
+	-rw-rw-r-- 1 vsc40000 vsc40000 2.1M Apr 12 13:16 afile.txt
+	drwxrwxr-x 2 vsc40000 vsc40000 512 Apr 12 12:51 some_directory
+ +- also listing hidden files using the `-a` flag: + +:
$ ls -lah
+	total 3.9M 
+	drwxrwxr-x 3 vsc40000 vsc40000 512 Apr 12 13:11 .
+	drwx------ 188 vsc40000 vsc40000 128K Apr 12 12:41 .. 
+	-rw-rw-r-- 1 vsc40000 vsc40000 1.8M Apr 12 13:12 afile.txt 
+	-rw-rw-r-- 1 vsc40000 vsc40000 0 Apr 12 13:11 .hidden_file.txt 
+	drwxrwxr-x 2 vsc40000 vsc40000 512 Apr 12 12:51 some_directory
+ +- ordering files by the most recent change using `-rt`: + +:
$ ls -lrth
+	total 4.0M 
+	drwxrwxr-x 2 vsc40000 vsc40000 512 Apr 12 12:51 some_directory 
+	-rw-rw-r-- 1 vsc40000 vsc40000 2.0M Apr 12 13:15 afile.txt
+ +If you try to use `ls` on a file that doesn't exist, you +will get a clear error message: +
$ ls nosuchfile 
 ls: cannot access nosuchfile: No such file or directory
 
-Changing directory: "cd" ------------------------- +## Changing directory: "cd" -To change to a different directory, you can use the `cd` command: +To change to a different directory, you can use the `cd` +command:
$ cd some_directory
+ To change back to the previous directory you were in, there's a shortcut: `cd -` -Using `cd` without an argument results in returning back to your home -directory: -
$ cd
+Using `cd` without an argument results in returning back to
+your home directory: 
+
$ cd 
 $ pwd
-/user/home/gent/vsc400/vsc40000
-
+/user/home/gent/vsc400/vsc40000
+## Inspecting file type: "file" -Inspecting file type: "file" ----------------------------- +The `file` command can be used to inspect what type of file +you're dealing with: -The `file` command can be used to inspect what type of file you're -dealing with:
$ file afile.txt
 afile.txt: ASCII text
 
-$ file some_directory
+$ file some_directory 
 some_directory: directory
 
-Absolute vs relative file paths -------------------------------- +## Absolute vs relative file paths -An *absolute* filepath starts with `/` (or a variable which value starts -with `/`), which is also called the *root* of the filesystem. +An *absolute* filepath starts with `/` (or a variable which +value starts with `/`), which is also called the *root* of +the filesystem. -Example: absolute path to your home directory: `/user/home/gent/vsc400/vsc40000`. +Example: absolute path to your home directory: +`/user/home/gent/vsc400/vsc40000`. A *relative* path starts from the current directory, and points to another location up or down the filesystem hierarchy. -Example: `some_directory/one.txt` points to the file `one.txt` that is -located in the subdirectory named `some_directory` of the current -directory. +Example: `some_directory/one.txt` points to the file +`one.txt` that is located in the subdirectory named +`some_directory` of the current directory. There are two special relative paths worth mentioning: - `.` is a shorthand for the current directory +- `..` is a shorthand for the parent of the current + directory -- `..` is a shorthand for the parent of the current directory +You can also use `..` when constructing relative paths, for +example: -You can also use `..` when constructing relative paths, for example: -
$ cd $HOME/some_directory
-$ ls ../afile.txt
-../afile.txt
-
+
$ cd $HOME/some_directory 
+$ ls ../afile.txt 
+../afile.txt 
-Permissions ------------ +## Permissions [//]: # (sec:permissions) Each file and directory has particular *permissions* set on it, which can be queried using `ls -l`. -For example: -
$ ls -l afile.txt
--rw-rw-r-- 1 vsc40000 agroup 2929176 Apr 12 13:29 afile.txt
-
- -The `-rwxrw-r--` specifies both the type of file (`-` for files, `d` for -directories (see first character)), and the permissions for -user/group/others: +For example: +
$ ls -l afile.txt 
+-rw-rw-r-- 1 vsc40000 agroup 2929176 Apr 12 13:29 afile.txt 
-1. each triple of characters indicates whether the read (`r`), write - (`w`), execute (`x`) permission bits are set or not +The `-rwxrw-r--` specifies both the type of file +(`-` for files, `d` for directories (see first +character)), and the permissions for user/group/others: -2. the 1st part `rwx` indicates that the *owner* "vsc40000" of the file has all - the rights - -3. the 2nd part `rw-` indicates the members of the *group* "agroup" - only have read/write permissions (not execute) - -4. the 3rd part `r--` indicates that *other* users only have read - permissions +1. each triple of characters indicates whether the read + (`r`), write (`w`), execute + (`x`) permission bits are set or not +2. the 1st part `rwx` indicates that the *owner* + "vsc40000" of the file has all the rights +3. the 2nd part `rw-` indicates the members of the *group* + "agroup" only have read/write permissions (not execute) +4. the 3rd part `r--` indicates that *other* users only + have read permissions The default permission settings for new files/directories are determined by the so-called *umask* setting, and are by default: 1. read-write permission on files for user/group (no execute), read-only for others (no write/execute) - 2. read-write-execute permission for directories on user/group, read/execute-only for others (no write) -See also [the chmod command](manipulating_files_and_directories.md#changing-permissions---chmod--sec--chmod) +See also [the chmod command](manipulating_files_and_directories.md#changing-permissions---chmod--sec--chmod) later in this manual. -Finding files/directories: "find" ---------------------------------- +## Finding files/directories: "find" -`find` will crawl a series of directories and lists files matching given -criteria. +`find` will crawl a series of directories and lists files +matching given criteria. -For example, to look for the file named `one.txt`: -
$ cd $HOME
+For example, to look for the file named `one.txt`: 
+
$ cd $HOME 
 $ find . -name one.txt
-./some_directory/one.txt
-
+./some_directory/one.txt
-To look for files using incomplete names, you can use a wildcard `*`; -note that you need to escape the `*` to avoid that Bash *expands* it -into `afile.txt` by adding double quotes: +To look for files using incomplete names, you can use a wildcard +`*`; note that you need to escape the `*` to +avoid that Bash *expands* it into `afile.txt` by adding +double quotes:
$ find . -name "*.txt"
-./.hidden_file.txt
-./afile.txt
+./.hidden_file.txt 
+./afile.txt 
 ./some_directory/one.txt
-./some_directory/two.txt
-
+./some_directory/two.txt
-A more advanced use of the `find` command is to use the `-exec` flag to -perform actions on the found file(s), rather than just printing their -paths (see `man find`). +A more advanced use of the `find` command is to use the +`-exec` flag to perform actions on the found file(s), rather +than just printing their paths (see `man find`). -Exercises ---------- - -- Go to `/tmp`, then back to your home directory. How many different - ways to do this can you come up with? +## Exercises +- Go to `/tmp`, then back to your home directory. How many + different ways to do this can you come up with? - When was your home directory created or last changed? - - Determine the name of the last changed file in `/tmp`. - - See how home directories are organised. Can you access the home directory of other users? -The [next](manipulating_files_and_directories.md) chapter will teach you how to interact with files and directories. +The [next](manipulating_files_and_directories.md) chapter will teach +you how to interact with files and directories.