From dae727bd4d42bae3f40dbdbe7f2922614dafabad Mon Sep 17 00:00:00 2001 From: Lukas Barragan Torres Date: Thu, 29 Aug 2024 09:57:18 +0200 Subject: [PATCH 1/6] Add warning explaining venvs are cluster specific --- mkdocs/docs/HPC/setting_up_python_virtual_environments.md | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/mkdocs/docs/HPC/setting_up_python_virtual_environments.md b/mkdocs/docs/HPC/setting_up_python_virtual_environments.md index e73b59bd0fd..6302a096cc2 100644 --- a/mkdocs/docs/HPC/setting_up_python_virtual_environments.md +++ b/mkdocs/docs/HPC/setting_up_python_virtual_environments.md @@ -25,6 +25,13 @@ python -m venv myenv # Create a new virtual environment named 'myenv' This command creates a new subdirectory named `myenv` in the current working directory. This directory will contain the packages, scripts, and binaries that are needed to manage the virtual environment. +!!! warning + When you create a virtual environment on top of a loaded Python module, + the environment becomes specific to the cluster you're working on. + This is because modules are built and optimized for the operating system and CPUs of the cluster. + This means that you should create a new virtual environment for each cluster you work on. + See [Creating a virtual environment for a specific cluster](#creating-a-virtual-environment-for-a-specific-cluster) for more information. + ### Activating a virtual environment To use the virtual environment, you need to *activate* it. From d05f7382a340e3a66645ac8defdb5bd7739ba4ad Mon Sep 17 00:00:00 2001 From: Lukas Barragan Torres Date: Thu, 29 Aug 2024 10:35:00 +0200 Subject: [PATCH 2/6] convert note about choosing modules to include pure/non-pure explenation --- .../setting_up_python_virtual_environments.md | 16 ++++++++++++---- 1 file changed, 12 insertions(+), 4 deletions(-) diff --git a/mkdocs/docs/HPC/setting_up_python_virtual_environments.md b/mkdocs/docs/HPC/setting_up_python_virtual_environments.md index 6302a096cc2..629b95fb0aa 100644 --- a/mkdocs/docs/HPC/setting_up_python_virtual_environments.md +++ b/mkdocs/docs/HPC/setting_up_python_virtual_environments.md @@ -31,6 +31,7 @@ This directory will contain the packages, scripts, and binaries that are needed This is because modules are built and optimized for the operating system and CPUs of the cluster. This means that you should create a new virtual environment for each cluster you work on. See [Creating a virtual environment for a specific cluster](#creating-a-virtual-environment-for-a-specific-cluster) for more information. + ### Activating a virtual environment @@ -56,10 +57,17 @@ No administrator privileges are required to install packages in a virtual enviro It is now possible to run Python scripts that use the installed packages in the virtual environment. -!!! note - Always prefer to use Python packages that are centrally installed, which are available via the environment modules interface. - Software installations available via the modules interface are compiled and optimized for the HPC cluster, while packages installed with `pip` are not. - To check if a package is available as a module, you can use the following command: +!!! tip + When creating a virtual environment, it's best to install only *pure* Python + packages. Pure Python packages consist solely of Python code and don't require compilation. + The installation method of these packages doesn't impact performance since they're not compiled. + + Compiled libraries with a Python wrapper (*non-pure* Python packages) are better loaded as modules + rather than installed in the virtual environment. + Modules are optimized for the HPC cluster’s specific hardware and operating system. + If a non-pure Python package isn't available as a module, you can submit a software installation request. + + To check if a package is available as a module, use: ```bash module av package_name From 99d73350a11e54ed9dc0025907a87007e02fa161 Mon Sep 17 00:00:00 2001 From: Lukas Barragan Torres Date: Thu, 29 Aug 2024 10:36:15 +0200 Subject: [PATCH 3/6] link to software isntallation request --- mkdocs/docs/HPC/setting_up_python_virtual_environments.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/mkdocs/docs/HPC/setting_up_python_virtual_environments.md b/mkdocs/docs/HPC/setting_up_python_virtual_environments.md index 629b95fb0aa..204a7991ab7 100644 --- a/mkdocs/docs/HPC/setting_up_python_virtual_environments.md +++ b/mkdocs/docs/HPC/setting_up_python_virtual_environments.md @@ -65,7 +65,8 @@ It is now possible to run Python scripts that use the installed packages in the Compiled libraries with a Python wrapper (*non-pure* Python packages) are better loaded as modules rather than installed in the virtual environment. Modules are optimized for the HPC cluster’s specific hardware and operating system. - If a non-pure Python package isn't available as a module, you can submit a software installation request. + If a non-pure Python package isn't available as a module, you can + [submit a software installation request](https://www.ugent.be/hpc/en/support/software-installation-request). To check if a package is available as a module, use: From d4104a32ef966cec198d0092821f1f2fe273ff71 Mon Sep 17 00:00:00 2001 From: Lukas Barragan Torres Date: Thu, 29 Aug 2024 10:44:25 +0200 Subject: [PATCH 4/6] update FAQ to link to venv page --- mkdocs/docs/HPC/FAQ.md | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/mkdocs/docs/HPC/FAQ.md b/mkdocs/docs/HPC/FAQ.md index 398bd3e1e67..1ef9e091f92 100644 --- a/mkdocs/docs/HPC/FAQ.md +++ b/mkdocs/docs/HPC/FAQ.md @@ -292,6 +292,12 @@ Please send an e-mail to {{hpcinfo}} that includes: - The purpose for which you want to install the software +If the software is a Python package, you can manually install it in a virtual environment. +More information can be found [here](./setting_up_python_virtual_environments.md). +Note that it is still preferred to submit a software installation request, +as the software installed by the HPC team will be optimized for the HPC environment. +This can lead to dramatic performance improvements. + {% endif %} ### Is my connection compromised? Remote host identification has changed From 82894dddab9ec18fce498dbe0d09d972a600e7e7 Mon Sep 17 00:00:00 2001 From: Lukas Barragan Torres Date: Thu, 29 Aug 2024 10:45:29 +0200 Subject: [PATCH 5/6] add 'this is because' in explaining sentence --- mkdocs/docs/HPC/setting_up_python_virtual_environments.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/mkdocs/docs/HPC/setting_up_python_virtual_environments.md b/mkdocs/docs/HPC/setting_up_python_virtual_environments.md index 204a7991ab7..1d228c44b05 100644 --- a/mkdocs/docs/HPC/setting_up_python_virtual_environments.md +++ b/mkdocs/docs/HPC/setting_up_python_virtual_environments.md @@ -64,7 +64,7 @@ It is now possible to run Python scripts that use the installed packages in the Compiled libraries with a Python wrapper (*non-pure* Python packages) are better loaded as modules rather than installed in the virtual environment. - Modules are optimized for the HPC cluster’s specific hardware and operating system. + This is because modules are optimized for the HPC cluster’s specific hardware and operating system. If a non-pure Python package isn't available as a module, you can [submit a software installation request](https://www.ugent.be/hpc/en/support/software-installation-request). From c15a0692e0e987fa8353c4f915ff1beff93bd65a Mon Sep 17 00:00:00 2001 From: Lukas Barragan Torres Date: Thu, 29 Aug 2024 11:06:59 +0200 Subject: [PATCH 6/6] correct FAQ venv reference placement --- mkdocs/docs/HPC/FAQ.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/mkdocs/docs/HPC/FAQ.md b/mkdocs/docs/HPC/FAQ.md index 1ef9e091f92..f249887b60b 100644 --- a/mkdocs/docs/HPC/FAQ.md +++ b/mkdocs/docs/HPC/FAQ.md @@ -292,14 +292,14 @@ Please send an e-mail to {{hpcinfo}} that includes: - The purpose for which you want to install the software +{% endif %} + If the software is a Python package, you can manually install it in a virtual environment. More information can be found [here](./setting_up_python_virtual_environments.md). Note that it is still preferred to submit a software installation request, as the software installed by the HPC team will be optimized for the HPC environment. This can lead to dramatic performance improvements. -{% endif %} - ### Is my connection compromised? Remote host identification has changed On Monday 25 April 2022, the login nodes received an update to RHEL8.