Skip to content

Commit

Permalink
Merge pull request #669 from lbarraga/html-to-md
Browse files Browse the repository at this point in the history 
HTML to md
  • Loading branch information
@boegel
boegel authored Aug 27, 2024
2 parents bd86bc2 + 24739b2 commit 95741e1
Show file tree
Hide file tree
Showing 43 changed files with 1,721 additions and 2,117 deletions.
7 changes: 4 additions & 3 deletions mkdocs/docs/HPC/FAQ.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -283,10 +283,11 @@ of files so other users can access the data. For example, the following
command will enable a user named "otheruser" to read the file named
`dataset.txt`. See

<pre><code>$ <b>setfacl -m u:otheruser:r dataset.txt</b>
$ <b>ls -l dataset.txt</b>
```
$ setfacl -m u:otheruser:r dataset.txt
$ ls -l dataset.txt
-rwxr-x---+ 2 {{userid}} mygroup 40 Apr 12 15:00 dataset.txt
</code></pre>
```

For more information about `chmod` or `setfacl`, see
[Linux tutorial](linux-tutorial/manipulating_files_and_directories.md#changing-permissions-chmod).
Expand Down
34 changes: 19 additions & 15 deletions mkdocs/docs/HPC/HOD.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -16,8 +16,9 @@ Before using HOD, you first need to load the `hod` module. We don't
specify a version here (this is an exception, for most other modules you
should, see [Using explicit version numbers](../running_batch_jobs/#using-explicit-version-numbers)) because newer versions might include important bug fixes.

<pre><code>$ <b>module load hod</b>
</code></pre>
```
module load hod
```

### Compatibility with login nodes

Expand All @@ -31,16 +32,17 @@ cluster module before loading the `hod` module and subsequently running

For example, this will work as expected:

<pre><code>$ <b>module swap cluster/{{othercluster}}</b>
$ <b>module load hod</b>
$ <b>hod</b>
```
$ module swap cluster/{{othercluster}}
$ module load hod
$ hod
hanythingondemand - Run services within an HPC cluster
usage: hod <subcommand> [subcommand options]
Available subcommands (one of these must be specified!):
batch Submit a job to spawn a cluster on a PBS job controller, run a job script, and tear down the cluster when it's done
clean Remove stale cluster info.
...
</code></pre>
```

Note that also modules named `hanythingondemand/*` are available. These
should however not be used directly, since they may not be compatible
Expand All @@ -52,13 +54,14 @@ for).
The `hod` module will also put a basic configuration in place for HOD,
by defining a couple of `$HOD_*` environment variables:

<pre><code>$ <b>module load hod</b>
$ <b>env | grep HOD | sort</b>
```
$ module load hod
$ env | grep HOD | sort
HOD_BATCH_HOD_MODULE=hanythingondemand/3.2.2-intel-2016b-Python-2.7.12
HOD_BATCH_WORKDIR=$VSC_SCRATCH/hod
HOD_CREATE_HOD_MODULE=hanythingondemand/3.2.2-intel-2016b-Python-2.7.12
HOD_CREATE_WORKDIR=$VSC_SCRATCH/hod
</code></pre>
```

By defining these environment variables, we avoid that you have to
specify `--hod-module` and `--workdir` when using `hod batch` or
Expand All @@ -85,26 +88,27 @@ will be marked as `<job-not-found>`.

You should occasionally clean this up using `hod clean`:

<pre><code>$ <b>module list</b>
```
$ module list
Currently Loaded Modulefiles:
1) cluster/{{defaultcluster}}(default) 2) pbs_python/4.6.0 3) vsc-base/2.4.2 4) hod/3.0.0-cli
$ <b>hod list</b>
$ hod list
Cluster label Job ID State Hosts
example1 {{jobid}} &lt;job-not-found&gt; &lt;none&gt;
$ <b>hod clean</b>
$ hod clean
Removed cluster localworkdir directory /user/scratch/gent/vsc400/vsc40000/hod/hod/{{jobid}} for cluster labeled example1
Removed cluster info directory /user/home/gent/vsc400/vsc40000/.config/hod.d/wordcount for cluster labeled example1
$ <b>module swap cluster/{{othercluster}}</b>
$ module swap cluster/{{othercluster}}
Cluster label Job ID State Hosts
example2 98765.master19.{{othercluster}}.gent.vsc &lt;job-not-found&gt; &lt;none&gt;
$ <b>hod clean</b>
$ hod clean
Removed cluster localworkdir directory /user/scratch/gent/vsc400/vsc40000/hod/hod/98765.master19.{{othercluster}}.gent.vsc for cluster labeled example2
Removed cluster info directory /user/home/gent/vsc400/vsc40000/.config/hod.d/wordcount for cluster labeled example2
</code></pre>
```
Note that **only HOD clusters that were submitted to the currently loaded `cluster` module will be cleaned up**.

## Getting help
Expand Down
62 changes: 35 additions & 27 deletions mkdocs/docs/HPC/MATLAB.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -37,11 +37,12 @@ To access the MATLAB compiler, the `MATLAB` module should be loaded
first. Make sure you are using the same `MATLAB` version to compile and
to run the compiled MATLAB program.

<pre><code>$ <b>module avail MATLAB/</b>
```
$ module avail MATLAB/
----------------------/apps/gent/RHEL8/zen2-ib/modules/all----------------------
MATLAB/2021b MATLAB/2022b-r5 (D)
$ <b>module load MATLAB/2021b</b>
</code></pre>
$ module load MATLAB/2021b
```

After loading the `MATLAB` module, the `mcc` command can be used. To get
help on `mcc`, you can run `mcc -?`.
Expand All @@ -53,12 +54,14 @@ flag means verbose output). To show how `mcc` can be used, we use the
First, we copy the `magicsquare.m` example that comes with MATLAB to
`example.m`:

<pre><code>$ <b>cp $EBROOTMATLAB/extern/examples/compiler/magicsquare.m example.m</b>
</code></pre>
```
cp $EBROOTMATLAB/extern/examples/compiler/magicsquare.m example.m
```

To compile a MATLAB program, use `mcc -mv`:

<pre><code><b>mcc -mv example.m</b>
```
mcc -mv example.m
Opening log file: {{homedir}}/java.log.34090
Compiler version: 8.3 (R2021b)
Dependency analysis by REQUIREMENTS.
Expand All @@ -67,7 +70,7 @@ Parsing file "{{homedir}}/example.m"
Deleting 0 temporary MEX authorization files.
Generating file "{{homedir}}/readme.txt".
Generating file "run\_example.sh".
</code></pre>
```

### Libraries

Expand All @@ -90,8 +93,9 @@ MATLAB program on the login nodes, consider tweaking the default maximum
heap size (128M) of Java using the `_JAVA_OPTIONS` environment variable
with:

<pre><code>$ <b>export _JAVA_OPTIONS="-Xmx64M"</b>
</code></pre>
```
export _JAVA_OPTIONS="-Xmx64M"
```

The MATLAB compiler spawns multiple Java processes. Because of the
default memory limits that are in effect on the login nodes, this might
Expand All @@ -102,14 +106,16 @@ to fit in memory.
Another possible issue is that the heap size is too small. This could
result in errors like:

<pre><code>Error: Out of memory
</code></pre>
```
Error: Out of memory
```

A possible solution to this is by setting the maximum heap size to be
bigger:

<pre><code>$ <b>export _JAVA_OPTIONS="-Xmx512M"</b>
</code></pre>
```
export _JAVA_OPTIONS="-Xmx512M"
```

## Multithreading

Expand All @@ -130,8 +136,7 @@ you requested when submitting your job script (the `ppn` value, see [Generic res
You can determine the right number of workers to use via the following
code snippet in your MATLAB program:

<div style="text-align: center;">-- parpool.m --</div>
```matlab
```matlab title="parpool.m"
{% include "./examples/MATLAB/parpool.m" %}
```

Expand All @@ -143,22 +148,25 @@ documentation](https://nl.mathworks.com/help/distcomp/parpool.html).
Each time MATLAB is executed, it generates a Java log file in the users
home directory. The output log directory can be changed using:

<pre><code>$ <b>MATLAB_LOG_DIR=<i>&lt;OUTPUT_DIR&gt;</i></b>
</code></pre>
```
MATLAB_LOG_DIR=<OUTPUT_DIR>
```

where `<OUTPUT_DIR>` is the name of the desired output directory. To
create and use a temporary directory for these logs:

<pre><code># create unique temporary directory in $TMPDIR (or /tmp/$USER if
```
# create unique temporary directory in $TMPDIR (or /tmp/$USER if
$TMPDIR is not defined)
# instruct MATLAB to use this directory for log files by setting $MATLAB_LOG_DIR
$ <b> export MATLAB_LOG_DIR=$ (mktemp -d -p $TMPDIR:-/tmp/$USER)</b>
</code></pre>
$ export MATLAB_LOG_DIR=$ (mktemp -d -p $TMPDIR:-/tmp/$USER)
```

You should remove the directory at the end of your job script:

<pre><code>$ <b> rm -rf $MATLAB_LOG_DIR</b>
</code></pre>
```
rm -rf $MATLAB_LOG_DIR
```

## Cache location

Expand All @@ -169,9 +177,10 @@ location and size of this cache can be changed through the
The snippet below would set the maximum cache size to 1024MB and the
location to `/tmp/testdirectory`.

<pre><code>$ <b>export MATLAB_CACHE_ROOT=/tmp/testdirectory </b>
$ <b>export MATLAB_CACHE_SIZE=1024M </b>
</code></pre>
```
export MATLAB_CACHE_ROOT=/tmp/testdirectory
export MATLAB_CACHE_SIZE=1024M
```

So when MATLAB is running, it can fill up to 1024MB of cache in
`/tmp/testdirectory`.
Expand All @@ -182,7 +191,6 @@ All of the tweaks needed to get MATLAB working have been implemented in
an example job script. This job script is also available on the HPC.
<!-- %TODO: where? -->

<div style="text-align: center;">-- jobscript.sh --</div>
```bash
```bash title="jobscript.sh"
{% include "./examples/MATLAB/jobscript.sh" %}
```
64 changes: 36 additions & 28 deletions mkdocs/docs/HPC/VNC.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -15,21 +15,22 @@ infrastructure from your own computer.

First login on the login node (see [First time connection to the HPC infrastructure](../connecting/#first-time-connection-to-the-hpc-infrastructure), then start `vncserver` with:

<pre><code>$ <b>vncserver -geometry 1920x1080 -localhost</b>
```
$ vncserver -geometry 1920x1080 -localhost
You will require a password to access your desktops.
Password:<b>&lt;enter a secure password&gt;</b>
Verify:<b>&lt;enter the same password&gt;</b>
Would you like to enter a view-only password (y/n)? <b>n</b>
Password: <enter a secure password>
Verify: <enter the same password>
Would you like to enter a view-only password (y/n)? n
A view-only password is not used
New '<b>{{loginhost}}:6</b> ({{userid}})' desktop is {{loginhost}}:6
New '{{loginhost}}:6 ({{userid}})' desktop is {{loginhost}}:6
Creating default startup script {{homedir}}.vnc/xstartup
Creating default config {{homedir}}.vnc/config
Starting applications specified in {{homedir}}.vnc/xstartup
Log file is {{homedir}}.vnc/{{loginhost}}:6.log
</code></pre>
```

**When prompted for a password, make sure to enter a secure password: if someone can guess your password, they will be able to do anything with your account you can!**

Expand All @@ -46,24 +47,26 @@ each time you want to connect.

You can get a list of running VNC servers on a node with

<pre><code>$ <b>vncserver -list</b>
```
$ vncserver -list
TigerVNC server sessions:
X DISPLAY # PROCESS ID
:6 30713
</code></pre>
```

This only displays the running VNC servers on **the login node you run the command on**.

To see what login nodes you are running a VNC server on, you can run the
`ls .vnc/*.pid` command in your home directory: the files shown have the
hostname of the login node in the filename:

<pre><code>$ <b>cd $HOME</b>
$ <b>ls .vnc/*.pid</b>
```
$ cd $HOME
$ ls .vnc/*.pid
.vnc/{{loginhost}}:6.pid
.vnc/{{altloginhost}}:8.pid
</code></pre>
```

This shows that there is a VNC server running on `{{loginhost}}` on port 5906 and
another one running `{{altloginhost}}` on port 5908 (see also [Determining the source/destination port](./#determining-the-sourcedestination-port)).
Expand Down Expand Up @@ -153,8 +156,9 @@ tunnel, by entering the settings in the and fields in [SSH tunnel](../running_in
{% else %}
Execute the following command to set up the SSH tunnel.

<pre><code>$ <b>ssh -L 5906:localhost:12345 {{userid}}@{{loginnode}}</b>
</code></pre>
```
ssh -L 5906:localhost:12345 {{userid}}@{{loginnode}}
```

**Replace the source port `5906`, destination port `12345` and user ID {{userid}} with your own!**

Expand All @@ -172,21 +176,22 @@ you have picked is actually still available (see [Picking an intermediate port t

You can check using the following command (**do not forget to replace `12345` the value you picked for your intermediate port):

<pre><code>$ <b>netstat -an | grep -i listen | grep tcp | grep 12345</b>
$
</code></pre>
```
netstat -an | grep -i listen | grep tcp | grep 12345
```

If you see no matching lines, then the port you picked is still
available, and you can continue.

If you see one or more matching lines as shown below,
**you must disconnect the first SSH tunnel, pick a different intermediate port, and set up the first SSH tunnel again using the new value**.

<pre><code>$ <b>netstat -an | grep -i listen | grep tcp | grep 12345</b>
```
$ netstat -an | grep -i listen | grep tcp | grep 12345
tcp 0 0 0.0.0.0:12345 0.0.0.0:* LISTEN
tcp6 0 0 :::12345 :::* LISTEN
$
</code></pre>
```

#### Setting up the second SSH tunnel to the correct login node

Expand All @@ -197,10 +202,11 @@ running (`{{loginhost}}` in our running example, see [Starting a VNC server](./#

To do this, run the following command:

<pre><code>$ <b>ssh -L 12345:localhost:5906 {{loginhost}}</b>
$ <b>hostname</b>
```
$ ssh -L 12345:localhost:5906 {{loginhost}}
$ hostname
{{loginhost}}
</code></pre>
```

With this, we are forwarding port `12345` on the login node we are
connected to (which is referred to as `localhost`) through to port
Expand Down Expand Up @@ -251,18 +257,20 @@ When prompted for default or empty panel, choose default.
If you have an empty panel, you can reset your settings with the
following commands:

<pre><code>$ <b>xfce4-panel --quit ; pkill xfconfd</b>
$ <b>mkdir ~/.oldxfcesettings</b>
$ <b>mv ~/.config/xfce4 ~/.oldxfcesettings</b>
$ <b>xfce4-panel</b>
</code></pre>
```
xfce4-panel --quit ; pkill xfconfd
mkdir ~/.oldxfcesettings
mv ~/.config/xfce4 ~/.oldxfcesettings
xfce4-panel
```

## Stopping the VNC server

The VNC server can be killed by running

<pre><code>vncserver -kill :6
</code></pre>
```
vncserver -kill :6
```

where `6` is the port number we noted down earlier. If you forgot, you
can get it with `vncserver -list` (see [List running VNC servers](./#list-running-vnc-servers)).
Expand Down
Loading

0 comments on commit 95741e1

Please sign in to comment.