From cd3576fd601c9be364fc430937e1c42ac64fea6a Mon Sep 17 00:00:00 2001 From: Xavier Delaruelle Date: Thu, 2 Nov 2023 08:55:37 +0100 Subject: [PATCH 1/8] Clarify module sort order differences Module version sort issue (e.g., choosing 1.9.0 as a greater version than 1.15.2) affects Environment Modules version <= 3.2.10. Newer Environment Modules versions sort module as expected and as also done by Lmod. Ancient Tcl-only version of Environment Modules also sorts things as expected since version 1.584. --- docs/installation.md | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/docs/installation.md b/docs/installation.md index 246bd2758..9b52b95f0 100644 --- a/docs/installation.md +++ b/docs/installation.md @@ -69,11 +69,11 @@ This is EasyBuild 4.8.0 (framework: 4.8.0, easyblocks: 4.8.0) on host example.lo ``` !!! tip - The Tcl-based or Lmod implementations of environment modules do their default sorting differently. - The former will normally sort in the lexicographic order, while Lmod follows - an approach that is closer to Python's construct `LooseVersion` way of ordering. Such aspects - may make a big difference, if you have installed both versions 1.9.0 and 1.15.2, - with respect to what is the version being loaded by default. + The Tcl/C Environment Modules (version <= 3.2.10) does its default sorting differently than + newer versions of Environment Modules and Lmod. The former will normally sort in the + lexicographic order, while the latters follow an approach that is closer to Python's construct + `LooseVersion` way of ordering. Such aspects may make a big difference, if you have installed + both versions 1.9.0 and 1.15.2, with respect to what is the version being loaded by default. You can also run `eb --show-system-info` to see system information relevant to EasyBuild, or run`eb --show-config` to see the default EasyBuild configuration (see also [Configuring EasyBuild][configuring_easybuild]). From e3b8d379c227b922c8380abffdb08c064c2b49be Mon Sep 17 00:00:00 2001 From: Xavier Delaruelle Date: Thu, 2 Nov 2023 09:06:56 +0100 Subject: [PATCH 2/8] Describe MODULES_CMD env var use to search module tool When using EnvironmentModules as module tool, the modulecmd.tcl binary is now also searched with "$MODULES_CMD" environment variable, like done for Lmod with "$LMOD_CMD". --- docs/configuration.md | 4 +++- docs/installation.md | 6 +++++- 2 files changed, 8 insertions(+), 2 deletions(-) diff --git a/docs/configuration.md b/docs/configuration.md index d8dcc3d41..c93f0062b 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -620,7 +620,9 @@ output of `type -f module` (in a `bash` shell), or `alias module` (in a `tcsh` s The actual module command (i.e., `modulecmd`, `modulecmd.tcl`, `lmod`, ...) must be available via `$PATH` (which is not standard), except when using Lmod -(in that case the `lmod` binary can also be located via `$LMOD_CMD`). +(in that case the `lmod` binary can also be located via `$LMOD_CMD`) or when using +Environment Modules (in that case the `modulecmd.tcl` binary can also be located via +`$MODULES_CMD`). For example, to indicate that EasyBuild should be using `Lmod` as modules tool: diff --git a/docs/installation.md b/docs/installation.md index 246bd2758..34eb313d7 100644 --- a/docs/installation.md +++ b/docs/installation.md @@ -356,12 +356,16 @@ Supported module tools: the `module` function; for example, using `type module` or `type -f module`. !!! note - For Lmod specifically, EasyBuild will try to fall back to finding the `lmod` binary via the `$LMOD_CMD` + For Lmod, EasyBuild will try to fall back to finding the `lmod` binary via the `$LMOD_CMD` environment variable, in case `lmod` is not available in `$PATH`. In EasyBuild versions *prior* to 2.1.1, the path specified by `$LMOD_CMD` was (erroneously) preferred over the (first) `lmod` binary available via `$PATH`. + For modern Tcl-only environment modules (version >= 4.0.0), EasyBuild will try to fall back to finding the + `modulecmd.tcl` binary via the `$MODULES_CMD` environment variable, in case `modulecmd.tcl` is not available + in `$PATH`. + Additional notes: From 38a3f3b573aa6ffe76af232c84250d2c26452f5d Mon Sep 17 00:00:00 2001 From: Xavier Delaruelle Date: Thu, 2 Nov 2023 11:55:47 +0100 Subject: [PATCH 3/8] Adapt unit-tests now Lmod is default module tool The default module tool of Easybuild was changed some years ago and is now Lmod. This commit update the unit-tests documentation to reflect that default module tool change. It is proposed to update the examples to run unit tests against a different module tool, using EnvironmentModules. --- docs/unit-tests.md | 15 +++++++++------ 1 file changed, 9 insertions(+), 6 deletions(-) diff --git a/docs/unit-tests.md b/docs/unit-tests.md index c6d94d192..9effe23f5 100644 --- a/docs/unit-tests.md +++ b/docs/unit-tests.md @@ -126,18 +126,19 @@ instead. Thus, to set a particular configuration option `--foo`, you should defi #### Modules tool to use for running tests {: #unit_tests_modules_tool } One particular configuration option worth mentioning explicitly is the modules tool that is to be used by the EasyBuild -framework, which is by default the traditional Tcl/C environment modules, referred to as `EnvironmentModulesC` in -EasyBuild configuration terms (see `eb --help` and `eb --avail-modules-tools`). +framework, which is by default Lmod, referred to as `Lmod` in EasyBuild configuration terms (see `eb --help` and +`eb --avail-modules-tools`). To run the EasyBuild framework unit tests with a particular modules tool, simply define the `$TEST_EASYBUILD_MODULES_TOOL` environment variable with the corresponding value. For example:: ``` shell -export TEST_EASYBUILD_MODULES_TOOL=Lmod +export TEST_EASYBUILD_MODULES_TOOL=EnvironmentModules ``` Just like for EasyBuild itself, the EasyBuild framework unit test suite expects that the modules tool binary/script -(`modulecmd`, `modulecmd.tcl` or `lmod`) is available via `$PATH`, see [Required modules tool][required_modules_tool]. +(`modulecmd`, `modulecmd.tcl` or `lmod`) is available via `$PATH` or referred in a module tool-specific environment +variable, see [Required modules tool][required_modules_tool]. #### Installing a GitHub token for the unit tests {: #unit_tests_github_token } @@ -173,10 +174,12 @@ other than `suite`, to focus on testing a particular aspect. See for an overview of the available Python modules corresponding to subgroups of tests (note: `__init__.py` and `utilities.py` are *not* such modules). -For example, to run the full EasyBuild framework unit test suite using Lmod as a modules tool:: +For example, to run the full EasyBuild framework unit test suite using Environment Modules as a modules tool and Tcl as +module file syntax:: ``` shell -export TEST_EASYBUILD_MODULES_TOOL=Lmod +export TEST_EASYBUILD_MODULES_TOOL=EnvironmentModules +export TEST_EASYBUILD_MODULE_SYNTAX=Tcl python -m test.framework.suite ``` From f790109fdaa8a56326434c838d09b78bdc5567d3 Mon Sep 17 00:00:00 2001 From: Xavier Delaruelle Date: Thu, 2 Nov 2023 15:51:14 +0100 Subject: [PATCH 4/8] See hidden modules on Environment Modules with --all --all option was introduced on Environment Modules version 4.6.0 to unveil hidden modules in module avail output. --- docs/manipulating-dependencies.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/manipulating-dependencies.md b/docs/manipulating-dependencies.md index 174072713..be64621f6 100644 --- a/docs/manipulating-dependencies.md +++ b/docs/manipulating-dependencies.md @@ -85,8 +85,8 @@ $ eb HDF5-1.8.13-intel-2015a.eb --hide-deps=zlib,Szip -D ``` !!! note - Using Lmod (version >= 5.7.5), hidden modules can be made visible in the output of '`module avail`' using the - `--show-hidden` option. + Using Lmod (version >= 5.7.5) or Environment Modules (version >= 4.6.0), hidden modules can be made + visible in the output of '`module avail`' using respectively the `--show-hidden` or `--all` option. For example: From 1791ff8715a70011cdbd775780aea20b0c2c8db7 Mon Sep 17 00:00:00 2001 From: Simon Branford Date: Fri, 16 Feb 2024 18:30:23 +0000 Subject: [PATCH 5/8] split out default config changes and remove proposed changes --- .../changes-in-default-configuration.md | 52 +++++++++++++++++++ docs/easybuild-v5/enhancements.md | 45 ---------------- docs/easybuild-v5/index.md | 1 + docs/easybuild-v5/overview-of-changes.md | 5 +- docs/easybuild-v5/proposed-changes.md | 9 ---- 5 files changed, 55 insertions(+), 57 deletions(-) create mode 100644 docs/easybuild-v5/changes-in-default-configuration.md delete mode 100644 docs/easybuild-v5/proposed-changes.md diff --git a/docs/easybuild-v5/changes-in-default-configuration.md b/docs/easybuild-v5/changes-in-default-configuration.md new file mode 100644 index 000000000..e71551c20 --- /dev/null +++ b/docs/easybuild-v5/changes-in-default-configuration.md @@ -0,0 +1,52 @@ +# Changes in default configuration in EasyBuild v5.0 + +*(for a full overview of changes in EasyBuild v5.0, see [here](overview-of-changes.md))* + +Various changes in default configuration included in EasyBuild v5.0, including: + +* [Enable RPATH linking by default][rpath] +* [Enable `--trace` by default][trace] + +--- + +## Enable RPATH linking by default {: #rpath } + +[RPATH linking][rpath_support] is enabled by default in EasyBuild v5.0 (see [easybuild-framework PR #4448](https://github.com/easybuilders/easybuild-framework/pull/4448)). + +The benefits for enabling RPATH are explained in [Why RPATH?][rpath_support_why]. + +This enhancement **does not** add any filtering of environment variables. This means `$LD_LIBRARY_PATH` +will continue to be appended by the environment module files EasyBuild generates, +unless it is configured to filter these variables (via `--filter-env-vars`, +see also [Relation to `$LD_LIBRARY_PATH`][rpath_support_LD_LIBRARY_PATH]). + +To disable RPATH linking, either: + +* Use the `--disable-rpath` command line option; +* Set the `$EASYBUILD_DISABLE_RPATH` environment variable; +* Disable RPATH linking in an EasyBuild [configuration file](../configuration.md#configuration_file): + + ``` ini + [override] + rpath=0 + ``` + + +--- + +## Enable `--trace` by default {: #trace } + +The [`--trace` option](../tracing-progress.md) is enabled by default (see [easybuild-framework PR #4250](https://github.com/easybuilders/easybuild-framework/pull/4250)). + +This makes the output produced by the `eb` command more informative, by providing more information about what's going on in the background. + +To disable trace output, either: + +* Use the `--disable-trace` command line option; +* Set the `$EASYBUILD_DISABLE_TRACE` environment variable; +* Disable trace mode in a [configuration file](../configuration.md#configuration_file): + + ``` ini + [override] + trace=0 + ``` diff --git a/docs/easybuild-v5/enhancements.md b/docs/easybuild-v5/enhancements.md index 3d5332cd3..33b3543dd 100644 --- a/docs/easybuild-v5/enhancements.md +++ b/docs/easybuild-v5/enhancements.md @@ -5,8 +5,6 @@ Various significant enhancements are included in EasyBuild v5.0, including: * [`run_shell_cmd` function][run_shell_cmd] -* [Enable RPATH linking by default][rpath] -* [Enable `--trace` by default][trace] --- @@ -14,46 +12,3 @@ Various significant enhancements are included in EasyBuild v5.0, including: See dedicated page on the new [`run_shell_cmd` function](run_shell_cmd.md). ---- - -## Enable RPATH linking by default {: #rpath } - -[RPATH linking][rpath_support] is enabled by default in EasyBuild v5.0 (see [easybuild-framework PR #4448](https://github.com/easybuilders/easybuild-framework/pull/4448)). - -The benefits for enabling RPATH are explained in [Why RPATH?][rpath_support_why]. - -This enhancement **does not** add any filtering of environment variables. This means `$LD_LIBRARY_PATH` -will continue to be appended by the environment module files EasyBuild generates, -unless it is configured to filter these variables (via `--filter-env-vars`, -see also [Relation to `$LD_LIBRARY_PATH`][rpath_support_LD_LIBRARY_PATH]). - -To disable RPATH linking, either: - -* Use the `--disable-rpath` command line option; -* Set the `$EASYBUILD_DISABLE_RPATH` environment variable; -* Disable RPATH linking in an EasyBuild [configuration file](../configuration.md#configuration_file): - - ``` ini - [override] - rpath=0 - ``` - - ---- - -## Enable `--trace` by default {: #trace } - -The [`--trace` option](../tracing-progress.md) is enabled by default (see [easybuild-framework PR #4250](https://github.com/easybuilders/easybuild-framework/pull/4250)). - -This makes the output produced by the `eb` command more informative, by providing more information about what's going on in the background. - -To disable trace output, either: - -* Use the `--disable-trace` command line option; -* Set the `$EASYBUILD_DISABLE_TRACE` environment variable; -* Disable trace mode in a [configuration file](../configuration.md#configuration_file): - - ``` ini - [override] - trace=0 - ``` diff --git a/docs/easybuild-v5/index.md b/docs/easybuild-v5/index.md index f6578511c..b1002acff 100644 --- a/docs/easybuild-v5/index.md +++ b/docs/easybuild-v5/index.md @@ -2,6 +2,7 @@ - [Overview of changes](overview-of-changes.md) - [Backwards-incompatible changes](backwards-incompatible-changes.md) + - [Changes in default configuration](changes-in-default-configuration.md) - [Enhancements](enhancements.md) - [`run_shell_cmd` function](run_shell_cmd.md) - [Deprecated functionality](deprecated-functionality.md) diff --git a/docs/easybuild-v5/overview-of-changes.md b/docs/easybuild-v5/overview-of-changes.md index 3eec0ff9f..9ab66bda3 100644 --- a/docs/easybuild-v5/overview-of-changes.md +++ b/docs/easybuild-v5/overview-of-changes.md @@ -3,16 +3,15 @@ !!! warning EasyBuild 5.0 is currently still under development, via the `5.0.x` branches in the EasyBuild GitHub repositories. - We intend to update this section of the documentation regularly as the planned changes are being implemented, - and when there are [proposed changes](proposed-changes.md) where we are requesting community feedback. + We intend to update this section of the documentation regularly as the planned changes are being implemented. This page provides a concise overview of the most prominent changes in EasyBuild version 5.0, which can be categorized as: * [Enhancements](enhancements.md) +* [Changes in default configuration](changes-in-default-configuration.md) * [Backward-incompatible changes](backwards-incompatible-changes.md) * [Deprecated functionality](deprecated-functionality.md) -* [Proposed changes](proposed-changes.md) * [Policies](policies.md) For in-depth details on a particular change, see the pull requests that are linked diff --git a/docs/easybuild-v5/proposed-changes.md b/docs/easybuild-v5/proposed-changes.md deleted file mode 100644 index d7af4c5df..000000000 --- a/docs/easybuild-v5/proposed-changes.md +++ /dev/null @@ -1,9 +0,0 @@ -# Proposed changes for EasyBuild v5.0 - -*(for a full overview of changes in EasyBuild v5.0, see [here](overview-of-changes.md))* - -There are several proposed changes where the EasyBuild maintainers are seeking community feedback. -If you wish to provide feedback then please comment in the GitHub issue for the proposal. - -* [Minimum supported Lmod version](https://github.com/easybuilders/easybuild/issues/871) -* [Toolchain support policy](https://github.com/easybuilders/easybuild/issues/872) From 7008df91fc8a4123420991ff7ce2065b6e03bec1 Mon Sep 17 00:00:00 2001 From: ocaisa Date: Fri, 23 Feb 2024 20:21:13 +0100 Subject: [PATCH 6/8] Update docs/unit-tests.md --- docs/unit-tests.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/unit-tests.md b/docs/unit-tests.md index 9effe23f5..a9766bce1 100644 --- a/docs/unit-tests.md +++ b/docs/unit-tests.md @@ -137,7 +137,7 @@ export TEST_EASYBUILD_MODULES_TOOL=EnvironmentModules ``` Just like for EasyBuild itself, the EasyBuild framework unit test suite expects that the modules tool binary/script -(`modulecmd`, `modulecmd.tcl` or `lmod`) is available via `$PATH` or referred in a module tool-specific environment +(`modulecmd`, `modulecmd.tcl` or `lmod`) is available via `$PATH` or referenced via a module tool-specific environment variable, see [Required modules tool][required_modules_tool]. From 549561294b9a78097d60ad3fa5642f919bcbd465 Mon Sep 17 00:00:00 2001 From: Simon Branford <4967+branfosj@users.noreply.github.com> Date: Mon, 4 Mar 2024 12:39:39 +0000 Subject: [PATCH 7/8] redirect proposed changes to EB5 index --- mkdocs.yml | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/mkdocs.yml b/mkdocs.yml index 0f79fc30c..1214f11fe 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -196,9 +196,12 @@ plugins: - index.md # necessary for search to work - search - # redirects for original EasyBuild documentation to avoid broken URLs + - redirects: redirect_maps: + # redirect removed pages to sensible alternative + easybuild-v5/proposed-changes/: easybuild-v5/index.md + # redirects for original EasyBuild documentation to avoid broken URLs en/latest/Archived-easyconfigs.html: archived-easyconfigs.md en/latest/Backup_modules.html: backup-modules.md en/latest/Changelog.html: changelog-docs.md From ca2a19483d48d6e7c79e7b945d87e483ac5c4851 Mon Sep 17 00:00:00 2001 From: Simon Branford <4967+branfosj@users.noreply.github.com> Date: Mon, 4 Mar 2024 12:41:01 +0000 Subject: [PATCH 8/8] yaml lint --- mkdocs.yml | 1 - 1 file changed, 1 deletion(-) diff --git a/mkdocs.yml b/mkdocs.yml index 1214f11fe..81db743ee 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -196,7 +196,6 @@ plugins: - index.md # necessary for search to work - search - - redirects: redirect_maps: # redirect removed pages to sensible alternative