From e87b1b7c468f53cdd8f7d1b75612219fc87490ac Mon Sep 17 00:00:00 2001 From: Lukas Barragan Torres Date: Thu, 18 Jul 2024 12:54:05 +0200 Subject: [PATCH 1/9] create conda document --- mkdocs/docs/HPC/conda.md | 36 ++++++++++++++++++++++++++++++++++++ 1 file changed, 36 insertions(+) create mode 100644 mkdocs/docs/HPC/conda.md diff --git a/mkdocs/docs/HPC/conda.md b/mkdocs/docs/HPC/conda.md new file mode 100644 index 00000000000..157761aaf5f --- /dev/null +++ b/mkdocs/docs/HPC/conda.md @@ -0,0 +1,36 @@ +# Conda + +We do not recommend the use of conda environments on the HPC clusters. +This section explains why and highlights some common issues. + +## Performance and Optimization + +Conda packages are pre-compiled binaries that are designed to work on a wide range of systems. +This means they are not optimized for the specific architecture of HPC clusters, +leading to potential performance drawbacks compared to modules compiled specifically for the HPC environment. + +HPC modules on the other hand, are compiled for the specific architecture of the cluster, +and are optimized for performance. + +## Compatibility and Dependency + +Using Conda in conjunction with centrally installed modules can lead to conflicts and unexpected errors, +making it difficult to debug and manage dependencies. + +## Package Availability + +Conda has a smaller repository of available packages compared to PyPI, the repository used by pip. +This can limit the availability of specific tools or libraries needed for certain workflows. + +## Environment and Installation Issues + +### Home Directory Usage + +Conda installs packages and environments in the user's home directory, +which can quickly fill up disk quotas due to the large number of files and directories it creates. +This is particularly problematic in HPC environments where home directory quotas are often limited. + +### Modification of Configuration Files + +Conda modifies the .bashrc file in the user's home directory, +which can lead to conflicts and unintended side effects in the user's environment setup. \ No newline at end of file From c5019aa092e5137d6e1f8cd367a0951327093ff7 Mon Sep 17 00:00:00 2001 From: Lukas Barragan Torres Date: Thu, 18 Jul 2024 12:54:55 +0200 Subject: [PATCH 2/9] added conda document to Advanced Topics --- config/templates/hpc.template | 1 + 1 file changed, 1 insertion(+) diff --git a/config/templates/hpc.template b/config/templates/hpc.template index 5e3a1807199..889514b9c3a 100644 --- a/config/templates/hpc.template +++ b/config/templates/hpc.template @@ -31,6 +31,7 @@ nav: - Best Practices: best_practices.md - Graphical applications with VNC: VNC.md - Graphical applications with X2Go: x2go.md + - Conda: conda.md {%- if site == 'Gent' %} - GPU clusters: gpu.md - Cron scripts: crontab.md From cebe3509baf606f9e2320596fcd1685ea9e7c3aa Mon Sep 17 00:00:00 2001 From: Lukas Barragan Torres Date: Thu, 18 Jul 2024 16:16:03 +0200 Subject: [PATCH 3/9] fix typo --- mkdocs/docs/HPC/conda.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/mkdocs/docs/HPC/conda.md b/mkdocs/docs/HPC/conda.md index 157761aaf5f..7a5f8e06f03 100644 --- a/mkdocs/docs/HPC/conda.md +++ b/mkdocs/docs/HPC/conda.md @@ -5,7 +5,7 @@ This section explains why and highlights some common issues. ## Performance and Optimization -Conda packages are pre-compiled binaries that are designed to work on a wide range of systems. +Conda's packages are pre-compiled binaries that are designed to work on a wide range of systems. This means they are not optimized for the specific architecture of HPC clusters, leading to potential performance drawbacks compared to modules compiled specifically for the HPC environment. From a7aa31a0488ac50a6ef2c0c1905e2a650d6dca74 Mon Sep 17 00:00:00 2001 From: Lukas Barragan Torres Date: Fri, 19 Jul 2024 13:38:45 +0200 Subject: [PATCH 4/9] added conda pkgs export --- mkdocs/docs/HPC/conda.md | 14 ++++++++++++-- 1 file changed, 12 insertions(+), 2 deletions(-) diff --git a/mkdocs/docs/HPC/conda.md b/mkdocs/docs/HPC/conda.md index 7a5f8e06f03..525022286bf 100644 --- a/mkdocs/docs/HPC/conda.md +++ b/mkdocs/docs/HPC/conda.md @@ -26,9 +26,19 @@ This can limit the availability of specific tools or libraries needed for certai ### Home Directory Usage -Conda installs packages and environments in the user's home directory, +by default, conda installs packages in the user's home directory, which can quickly fill up disk quotas due to the large number of files and directories it creates. -This is particularly problematic in HPC environments where home directory quotas are often limited. +This is particularly problematic in the HPC environment where home directory quotas are limited. + +If you do wish to use conda, we recommend setting the conda package directory to a location with more storage +space. This can be done by running: + +```bash +export CONDA_PKGS_DIRS=$VSC_DATA/conda_pkgs +``` + +before installing packages into a conda environment. + ### Modification of Configuration Files From 8a342c76561b2613623ca9c1a8617be590514b34 Mon Sep 17 00:00:00 2001 From: Lukas Barragan Torres Date: Fri, 19 Jul 2024 13:56:13 +0200 Subject: [PATCH 5/9] new intro to better reflect the ideq of condq at HPC --- mkdocs/docs/HPC/conda.md | 10 ++++++++-- 1 file changed, 8 insertions(+), 2 deletions(-) diff --git a/mkdocs/docs/HPC/conda.md b/mkdocs/docs/HPC/conda.md index 525022286bf..b92b4f3ca09 100644 --- a/mkdocs/docs/HPC/conda.md +++ b/mkdocs/docs/HPC/conda.md @@ -1,7 +1,13 @@ # Conda -We do not recommend the use of conda environments on the HPC clusters. -This section explains why and highlights some common issues. +Conda is a software package manager for data science that allows unprivileged (non-administrative) users to search, +fetch, install, upgrade, use, and manage supported open-source software packages and +programming languages/libraries/environments in a directory they have write access to. +Conda allows users to create reproducible scientific software environments + +We do not recommend using conda environments on HPC clusters due to several common issues, +which we will explain in this section. However, we do not forbid the use of conda entirely. In certain situations, +such as testing software, creating new workflows, or teaching a course, it is perfectly acceptable to use it. ## Performance and Optimization From 883f8a256e89e39c5278da88c1a4fd5242e639f1 Mon Sep 17 00:00:00 2001 From: Lukas Barragan Torres Date: Fri, 19 Jul 2024 14:16:51 +0200 Subject: [PATCH 6/9] add use of 'source activate' instead of 'conda activate' --- mkdocs/docs/HPC/conda.md | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/mkdocs/docs/HPC/conda.md b/mkdocs/docs/HPC/conda.md index b92b4f3ca09..2aaede9970b 100644 --- a/mkdocs/docs/HPC/conda.md +++ b/mkdocs/docs/HPC/conda.md @@ -49,4 +49,7 @@ before installing packages into a conda environment. ### Modification of Configuration Files Conda modifies the .bashrc file in the user's home directory, -which can lead to conflicts and unintended side effects in the user's environment setup. \ No newline at end of file +which can lead to conflicts and unintended side effects in the user's environment setup. + +You can avoid the change of the `.bashrc` file by activating the environment with `source activate full/path/to/myenv` +instead of `conda activate myenv`. \ No newline at end of file From 939290602752d9b5333fe1da4b2ed463c789b534 Mon Sep 17 00:00:00 2001 From: Lukas Barragan Torres Date: Fri, 19 Jul 2024 14:24:05 +0200 Subject: [PATCH 7/9] don't use modules when using conda --- mkdocs/docs/HPC/conda.md | 3 +++ 1 file changed, 3 insertions(+) diff --git a/mkdocs/docs/HPC/conda.md b/mkdocs/docs/HPC/conda.md index 2aaede9970b..145e3ca518c 100644 --- a/mkdocs/docs/HPC/conda.md +++ b/mkdocs/docs/HPC/conda.md @@ -23,6 +23,9 @@ and are optimized for performance. Using Conda in conjunction with centrally installed modules can lead to conflicts and unexpected errors, making it difficult to debug and manage dependencies. +If you do wish to use conda, we recommend only using packages from conda itself, even when they are available as modules. +This will degrade performance, but will avoid conflicts between conda and module dependencies. + ## Package Availability Conda has a smaller repository of available packages compared to PyPI, the repository used by pip. From cdd603f1240d04ed1b18b28963c5f4dab49b9bf5 Mon Sep 17 00:00:00 2001 From: Lukas Barragan Torres Date: Fri, 19 Jul 2024 16:51:48 +0200 Subject: [PATCH 8/9] remove word: scientific --- mkdocs/docs/HPC/conda.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/mkdocs/docs/HPC/conda.md b/mkdocs/docs/HPC/conda.md index 145e3ca518c..1d35aee8a7c 100644 --- a/mkdocs/docs/HPC/conda.md +++ b/mkdocs/docs/HPC/conda.md @@ -3,7 +3,7 @@ Conda is a software package manager for data science that allows unprivileged (non-administrative) users to search, fetch, install, upgrade, use, and manage supported open-source software packages and programming languages/libraries/environments in a directory they have write access to. -Conda allows users to create reproducible scientific software environments +Conda allows users to create reproducible software environments We do not recommend using conda environments on HPC clusters due to several common issues, which we will explain in this section. However, we do not forbid the use of conda entirely. In certain situations, From 4f41dbc09f3d2e96e2757768b9a0b7435ee4aee9 Mon Sep 17 00:00:00 2001 From: Lukas Barragan Torres Date: Thu, 29 Aug 2024 12:05:28 +0200 Subject: [PATCH 9/9] add section about Anaconda enforcing TOS --- mkdocs/docs/HPC/conda.md | 13 ++++++++++++- 1 file changed, 12 insertions(+), 1 deletion(-) diff --git a/mkdocs/docs/HPC/conda.md b/mkdocs/docs/HPC/conda.md index 1d35aee8a7c..23fc0e0e2f1 100644 --- a/mkdocs/docs/HPC/conda.md +++ b/mkdocs/docs/HPC/conda.md @@ -55,4 +55,15 @@ Conda modifies the .bashrc file in the user's home directory, which can lead to conflicts and unintended side effects in the user's environment setup. You can avoid the change of the `.bashrc` file by activating the environment with `source activate full/path/to/myenv` -instead of `conda activate myenv`. \ No newline at end of file +instead of `conda activate myenv`. + +## Anaconda Licensing + +If your organization uses Conda packages from Anaconda channels, +be aware that Anaconda may now be actively enforcing its licensing terms. +Academic, non-profit, and research institutions that previously used Anaconda software for free may now be required to +purchase a commercial license if they have over 200 employees or contractors. +Anaconda has sent legal notices to institutions demanding compliance and threatening backdated billing. +This shift follows changes in Anaconda’s terms of service, +updated in March 2024, and may affect your access to Anaconda’s resources. +Ensure your organization reviews its usage and licensing agreements to avoid potential legal issues. \ No newline at end of file