From 6d0e3cc94d7a734b67e5b8d76e415d6cdaaf484a Mon Sep 17 00:00:00 2001 From: Kenneth Hoste Date: Mon, 20 Nov 2023 11:29:06 +0100 Subject: [PATCH] restructure documentation on EasyBuild v5.0 --- .../backwards-incompatible-changes.md | 57 ++++++ docs/easybuild-v5/deprecated-functionality.md | 56 ++++++ docs/easybuild-v5/enhancements.md | 33 ++++ docs/easybuild-v5/index.md | 11 +- docs/easybuild-v5/overview-of-changes.md | 168 ++---------------- docs/easybuild-v5/policies.md | 5 + docs/easybuild-v5/proposed-changes.md | 9 + docs/easybuild-v5/run_shell_cmd.md | 27 +++ docs/easybuild-v5/run_shell_cmd.md.bk | 54 ++++++ 9 files changed, 264 insertions(+), 156 deletions(-) create mode 100644 docs/easybuild-v5/backwards-incompatible-changes.md create mode 100644 docs/easybuild-v5/deprecated-functionality.md create mode 100644 docs/easybuild-v5/enhancements.md create mode 100644 docs/easybuild-v5/policies.md create mode 100644 docs/easybuild-v5/proposed-changes.md create mode 100644 docs/easybuild-v5/run_shell_cmd.md create mode 100644 docs/easybuild-v5/run_shell_cmd.md.bk diff --git a/docs/easybuild-v5/backwards-incompatible-changes.md b/docs/easybuild-v5/backwards-incompatible-changes.md new file mode 100644 index 000000000..c39fa5be9 --- /dev/null +++ b/docs/easybuild-v5/backwards-incompatible-changes.md @@ -0,0 +1,57 @@ +# Backwards-incompatible changes in EasyBuild v5.0 + +*(for a full overview of changes in EasyBuild v5.0, see [here](overview-of-changes.md))* + +A number of *backwards-incompatible* changes are being made in EasyBuild v5.0: + +* [Support for Python 2.7 is removed, Python 3.6+ is required][py36] +* [Deprecated EasyBuild bootstrap script is removed][bootstrap_script] +* [Experimental support for the `.yeb` easyconfig format is removed][yeb] + +--- + +## Support for Python 2.7 is removed, Python 3.6+ is required {: #py36 } + +EasyBuild 5.0 requires Python >= 3.6 to run. + +Running EasyBuild with Python 2.7 or a Python 3 version older than Python 3.6 is no longer supported. + +Trying to run EasyBuild with a Python version that is too old will result in an error: + +```log +ERROR: No compatible 'python' command found via $PATH (EasyBuild requires Python 3.6+) +``` + +Python 2.7 has been [end-of-life since 1 Jan 2020](https://www.python.org/doc/sunset-python-2), +and dropping compatibility with Python 2.7 and Python 3.5 enabled some significant code cleanup +(see [easybuild-framework PR #4229](https://github.com/easybuilders/easybuild-framework/pull/4229)). + +The [results of the 6th EasyBuild User Survey (2022)](https://easybuild.io/user_survey) show that the impact of +this breaking change on the EasyBuild community should be very limited, since: + +* Only ~13% of survey participants were still running EasyBuild on top of Python 2.7; +* No survey participants reported using Python 3.5; +* Over 85% of survey participants reported using Python 3.6, or a more recent version of Python 3; +* Only 3 out of 118 survey participants (~2.5%) reported that dropping support for running EasyBuild + on top of Python 2 would be *problematic* for them; + +Along with actively removing code that was only required to retain compatibility with Python 2.7 or 3.5, +the `easybuild.tools.py2vs3` module that was introduced to facilitate supporting both Python 2.7 and Python 3 +has been deprecated (see also [here](deprecated-functionality.md#py2vs3)). + +--- + +## Deprecated EasyBuild bootstrap script is removed {: #bootstrap_script } + +The EasyBuild bootstrap script has been removed (see [easybuild-framework PR #4233](https://github.com/easybuilders/easybuild-framework/pull/4233)). + +Please see the [installation page](../installation.md) for the suggested methods for installing EasyBuild. + +--- + +## Experimental support for the .yeb easyconfig format is removed {: #yeb } + +Support for the experimental `.yeb` easyconfig format has been removed (see [easybuild-framework PR #4237](https://github.com/easybuilders/easybuild-framework/pull/4237)). + +This format allowed easyconfigs to be specified in YAML. However, there has been no recent development of this +format and little suggestion that anyone was using it at all. diff --git a/docs/easybuild-v5/deprecated-functionality.md b/docs/easybuild-v5/deprecated-functionality.md new file mode 100644 index 000000000..ffb3df187 --- /dev/null +++ b/docs/easybuild-v5/deprecated-functionality.md @@ -0,0 +1,56 @@ +# Deprecated functionality in EasyBuild v5.0 + +*(for a full overview of changes in EasyBuild v5.0, see [here](overview-of-changes.md))* + +Some functionality is being deprecated in EasyBuild v5.0, and will no longer be supported in EasyBuild v6.0: + +* [`run_cmd` and `run_cmd_qa` functions][run_cmd] +* [`easybuild.tools.py2vs3` module][py2vs3] + +If you trigger any deprecated functionality when using EasyBuild v5.0, a warning message will be printed. + +--- + +## `run_cmd` and `run_cmd_qa` functions {: #run_cmd } + +The `run_cmd` and `run_cmd_qa` functions will be deprecated. + +You should migrate to the new [`run_shell_cmd`](run_shell_cmd.md) function instead. + + +--- + +## `easybuild.tools.py2vs3` module {: #py2vs3 } + +[easybuild-framework PR #4229](https://github.com/easybuilders/easybuild-framework/pull/4229) + +The following table lists the changes required to replace imports from the the `py2vs3` module. + +| `from easybuild.tools.py2vs3 import ...` | Replacement | +|--|--| +| `ascii_letters` | `from string import ascii_letters` | +| `ascii_lowercase` | `from string import ascii_lowercase` | +| `build_opener` | `from urllib.request import build_opener` | +| `ConfigParser` | `from configparser import ConfigParser` | +| `configparser` | `import configparser` | +| `create_base_metaclass` | `from easybuild.base.wrapper import create_base_metaclass` | +| `extract_method_name` | No import required. Replace `extract_method_name(method)` with `'_'.join(method.__code__.co_names)` | +| `HTMLParser` | `from html.parser import HTMLParser` | +| `HTTPError` | `from urllib.request import HTTPError` | +| `HTTPSHandler` | `from urllib.request import HTTPSHandler` | +| `json_loads` | `from json import loads` and rename `json_loads` to `loads` | +| `Mapping` | `from collections.abc import Mapping` | +| `mk_wrapper_baseclass` | `from easybuild.base.wrapper import mk_wrapper_baseclass` | +| `OrderedDict` | `from collections import OrderedDict` | +| `raise_with_traceback` | No import required. Replace `raise_with_traceback(exception, message, tb)` with `raise exception(message).with_traceback(tb)` | +| `reload` | `from importlib import reload` | +| `Request` | `from urllib.request import Request` | +| `string_type` | No import required. Use `str` directly. | +| `StringIO` | `from io import StringIO` | +| `std_urllib` | `import urllib.request as std_urllib` | +| `subprocess_popen_text` | `from easybuild.tools.run import subprocess_popen_text` | +| `subprocess_terminate` | `from easybuild.tools.run import subprocess_terminate` | +| `urlencode` | `from urllib.parse import urlencode` | +| `URLError` | `from urllib.request import URLError` | +| `urlopen` | `from urllib.request import urlopen` | + diff --git a/docs/easybuild-v5/enhancements.md b/docs/easybuild-v5/enhancements.md new file mode 100644 index 000000000..16660f6c5 --- /dev/null +++ b/docs/easybuild-v5/enhancements.md @@ -0,0 +1,33 @@ +# Enhancements in EasyBuild v5.0 + +*(for a full overview of changes in EasyBuild v5.0, see [here](overview-of-changes.md))* + +Various significant enhancements are included in EasyBuild v5.0, including: + +* [`run_shell_cmd` function][run_shell_cmd] +* [Enable `--trace` by default][trace] + +--- + +## `run_shell_cmd` function { : #run_shell_cmd } + +See dedicated page on the new [`run_shell_cmd` function](run_shell_cmd.md). + +--- + +## Enable `--trace` by default {: #trace } + +The [`--trace` option](../tracing-progress.md) is enabled by default (see [easybuild-framework PR #4250](https://github.com/easybuilders/easybuild-framework/pull/4250)). + +This makes the output produced by the `eb` command more informative, by providing more information about what's going on in the background. + +To disable trace output, either: + +* Use the `--disable-trace` command line option; +* Set the `$EASYBUILD_DISABLE_TRACE` environment variable; +* Disable trace mode in a [configuration file](../configuration.md#configuration_file): + + ``` ini + [override] + trace=0 + ``` diff --git a/docs/easybuild-v5/index.md b/docs/easybuild-v5/index.md index 2146257cf..f6578511c 100644 --- a/docs/easybuild-v5/index.md +++ b/docs/easybuild-v5/index.md @@ -1,5 +1,10 @@ # EasyBuild v5.0 -* [Overview of changes](overview-of-changes.md) -* [GitHub Project board](https://github.com/orgs/easybuilders/projects/18) -* [Talk on EasyBuild 5.0 at EUM'23](https://easybuild.io/eum23/#easybuild5) +- [Overview of changes](overview-of-changes.md) + - [Backwards-incompatible changes](backwards-incompatible-changes.md) + - [Enhancements](enhancements.md) + - [`run_shell_cmd` function](run_shell_cmd.md) + - [Deprecated functionality](deprecated-functionality.md) + - [Policies](policies.md) +- [GitHub Project board](https://github.com/orgs/easybuilders/projects/18) +- [Talk on EasyBuild 5.0 at EasyBuild User Meeting 2023](https://easybuild.io/eum23/#easybuild5) diff --git a/docs/easybuild-v5/overview-of-changes.md b/docs/easybuild-v5/overview-of-changes.md index 24499ff1b..3eec0ff9f 100644 --- a/docs/easybuild-v5/overview-of-changes.md +++ b/docs/easybuild-v5/overview-of-changes.md @@ -1,168 +1,30 @@ -# Overview of changes in EasyBuild version 5.0 {: #overview } +# Overview of changes in EasyBuild version 5.0 !!! warning EasyBuild 5.0 is currently still under development, via the `5.0.x` branches in the EasyBuild GitHub repositories. - We intend to update this page regularly as the [planned changes][eb5_plans] are being implemented and when there - are [proposed changes][eb5_proposals] where we are requesting community feedback. + We intend to update this section of the documentation regularly as the planned changes are being implemented, + and when there are [proposed changes](proposed-changes.md) where we are requesting community feedback. This page provides a concise overview of the most prominent changes in EasyBuild version 5.0, which can be categorized as: -* [Significant enhancements][significant_enhancements] -* [Backward-incompatible changes][backwards_incompatible] -* [Deprecated functionality][deprecated_v5] +* [Enhancements](enhancements.md) +* [Backward-incompatible changes](backwards-incompatible-changes.md) +* [Deprecated functionality](deprecated-functionality.md) +* [Proposed changes](proposed-changes.md) +* [Policies](policies.md) -For in-depth details on a particular change, see the pull requests that are linked from each of the subsections below. +For in-depth details on a particular change, see the pull requests that are linked +from each of the subsections linked above. ---- - -## Planned and proposed changes for EasyBuild v5.0 {: #eb5_changes } - -At the [EasyBuild User Meeting](https://easybuild.io/eum23), Simon Branford set out the +At the [EasyBuild User Meeting 2023](https://easybuild.io/eum23), Simon Branford set out the [roadmap for EasyBuild v5.0](https://easybuild.io/eum23/#easybuild5). -### Proposed changes for EasyBuild v5.0 {: #eb5_proposals } - -There are several proposed changes where the EasyBuild maintainers are seeking community feedback. If you wish to provide -feedback then please comment in the GitHub issue for the proposal. - -* [Minimum supported Lmod Version](https://github.com/easybuilders/easybuild/issues/871) -* [Toolchain Support Policy](https://github.com/easybuilders/easybuild/issues/872) - -### Planned changes for EasyBuild v5.0 {: #eb5_plans } +To track the progress on the development of EasyBuild v5.0, +see the [GitHub Project board for EasyBuild v5.0](https://github.com/orgs/easybuilders/projects/18). !!! note - This list is the major planned changes. It is not intended to be a complete list of all changes that are - planned for EasyBuild v5.0. - -### Tracking development of EasyBuild v5.0 - -* [GitHub Project board for EasyBuild v5.0](https://github.com/orgs/easybuilders/projects/18) - ---- - -## Significant enhancements in EasyBuild v5.0 {: #significant_enhancements } - -Various significant enhancements are included in EasyBuild v5.0, including: - -* [enable `--trace` by default][eb5_trace] - -### `--trace` enabled by default {: #eb5_trace } - -The [`--trace` option][trace] is enabled by default (see [easybuild-framework PR #4250](https://github.com/easybuilders/easybuild-framework/pull/4250)). -This makes the output produced by the `eb` command more informative, by providing more information about what's going on in the background. - -To disable trace output, either: - -* Use the `--disable-trace` command line option; -* Set the `$EASYBUILD_DISABLE_TRACE` environment variable; -* Disable trace mode in a [configuration file][configuration_file]: - - ``` ini - [override] - trace=0 - ``` - ---- - -## Backwards-incompatible changes in EasyBuild v4.0 {: #backwards_incompatible } - -A number of *backwards-incompatible* changes are being made in EasyBuild v5.0: - -* [Support for Python 2.7 is removed -- Python 3.6+ is required][py36] -* [Deprecated EasyBuild bootstrap script is removed][bootstrap_script] -* [Experimental support for the .yeb easyconfig format is removed][yeb] - ---- - -### Support for Python 2.7 is removed -- Python 3.6+ is required {: #py36 } - -EasyBuild 5.0 requires Python >= 3.6 to run. - -Running EasyBuild with Python 2.7 or a Python 3 version older than Python 3.6 is no longer supported. - -Trying to run EasyBuild with a Python version that is too old will result in an error: - -```log -ERROR: No compatible 'python' command found via $PATH (EasyBuild requires Python 3.6+) -``` - -Python 2.7 has been [end-of-life since 1 Jan 2020](https://www.python.org/doc/sunset-python-2), -and dropping compatibility with Python 2.7 and Python 3.5 enabled some significant code cleanup -(see [easybuild-framework PR #4229](https://github.com/easybuilders/easybuild-framework/pull/4229)). - -The [results of the 6th EasyBuild User Survey (2022)](https://easybuild.io/user_survey) show that the impact of -this breaking change on the EasyBuild community should be very limited, since: - -* Only ~13% of survey participants were still running EasyBuild on top of Python 2.7; -* No survey participants reported using Python 3.5; -* Over 85% of survey participants reported using Python 3.6, or a more recent version of Python 3; -* Only 3 out of 118 survey participants (~2.5%) reported that dropping support for running EasyBuild - on top of Python 2 would be *problematic* for them; - -Along with actively removing code that was only required to retain compatibility with Python 2.7 or 3.5, -the `easybuild.tools.py2vs3` module that was introduced to facilitate supporting both Python 2.7 and Python 3 -has been deprecated ([see also below][py2vs3]). - ---- - -### Deprecated EasyBuild bootstrap script is removed {: #bootstrap_script } - -The EasyBuild bootstrap script has been removed (see [easybuild-framework PR #4233](https://github.com/easybuilders/easybuild-framework/pull/4233)). -Please see the [installation page][installation] for the suggested methods for installing EasyBuild. - ---- - -### Experimental support for the .yeb easyconfig format is removed {: #yeb } - -Support for the experimental `.yeb` easyconfig format has been removed (see [easybuild-framework PR #4237](https://github.com/easybuilders/easybuild-framework/pull/4237)). -This format allowed easyconfigs to be specified in YAML. However, there has been no recent development of this -format and little suggestion that anyone was using it. - ---- - -## Deprecated functionality in EasyBuild v5.0 {: #deprecated_v5 } - -Some functionality is being deprecated in EasyBuild v5.0, and will no longer be supported in EasyBuild v6.0: - -* [`easybuild.tools.py2vs3` module][py2vs3] - -If you trigger any deprecated functionality, a warning message will be printed. - ---- - -### `easybuild.tools.py2vs3` module {: #py2vs3 } - -[easybuild-framework PR #4229](https://github.com/easybuilders/easybuild-framework/pull/4229) - -The following table lists the changes required to replace imports from the the `py2vs3` module. - -| `from easybuild.tools.py2vs3 import ...` | Replacement | -|--|--| -| `ascii_letters` | `from string import ascii_letters` | -| `ascii_lowercase` | `from string import ascii_lowercase` | -| `build_opener` | `from urllib.request import build_opener` | -| `ConfigParser` | `from configparser import ConfigParser` | -| `configparser` | `import configparser` | -| `create_base_metaclass` | `from easybuild.base.wrapper import create_base_metaclass` | -| `extract_method_name` | No import required. Replace `extract_method_name(method)` with `'_'.join(method.__code__.co_names)` | -| `HTMLParser` | `from html.parser import HTMLParser` | -| `HTTPError` | `from urllib.request import HTTPError` | -| `HTTPSHandler` | `from urllib.request import HTTPSHandler` | -| `json_loads` | `from json import loads` and rename `json_loads` to `loads` | -| `Mapping` | `from collections.abc import Mapping` | -| `mk_wrapper_baseclass` | `from easybuild.base.wrapper import mk_wrapper_baseclass` | -| `OrderedDict` | `from collections import OrderedDict` | -| `raise_with_traceback` | No import required. Replace `raise_with_traceback(exception, message, tb)` with `raise exception(message).with_traceback(tb)` | -| `reload` | `from importlib import reload` | -| `Request` | `from urllib.request import Request` | -| `string_type` | No import required. Use `str` directly. | -| `StringIO` | `from io import StringIO` | -| `std_urllib` | `import urllib.request as std_urllib` | -| `subprocess_popen_text` | `from easybuild.tools.run import subprocess_popen_text` | -| `subprocess_terminate` | `from easybuild.tools.run import subprocess_terminate` | -| `urlencode` | `from urllib.parse import urlencode` | -| `URLError` | `from urllib.request import URLError` | -| `urlopen` | `from urllib.request import urlopen` | + This section covers the major planned changes. + It is not intended to be a *complete* list of all changes that are planned for EasyBuild v5.0. diff --git a/docs/easybuild-v5/policies.md b/docs/easybuild-v5/policies.md new file mode 100644 index 000000000..773f922c4 --- /dev/null +++ b/docs/easybuild-v5/policies.md @@ -0,0 +1,5 @@ +# Policies related to EasyBuild v5.0 + +*(for a full overview of changes in EasyBuild v5.0, see [here](overview-of-changes.md))* + +Starting with EasyBuild v5.0, we will enforce the new [toolchain support policy](../policies/toolchains.md). diff --git a/docs/easybuild-v5/proposed-changes.md b/docs/easybuild-v5/proposed-changes.md new file mode 100644 index 000000000..d7af4c5df --- /dev/null +++ b/docs/easybuild-v5/proposed-changes.md @@ -0,0 +1,9 @@ +# Proposed changes for EasyBuild v5.0 + +*(for a full overview of changes in EasyBuild v5.0, see [here](overview-of-changes.md))* + +There are several proposed changes where the EasyBuild maintainers are seeking community feedback. +If you wish to provide feedback then please comment in the GitHub issue for the proposal. + +* [Minimum supported Lmod version](https://github.com/easybuilders/easybuild/issues/871) +* [Toolchain support policy](https://github.com/easybuilders/easybuild/issues/872) diff --git a/docs/easybuild-v5/run_shell_cmd.md b/docs/easybuild-v5/run_shell_cmd.md new file mode 100644 index 000000000..279dac9ec --- /dev/null +++ b/docs/easybuild-v5/run_shell_cmd.md @@ -0,0 +1,27 @@ +# `run_shell_cmd` function + +*(for a full overview of changes in EasyBuild v5.0, see [here](overview-of-changes.md))* + +In EasyBuild v5.0, a new function named `run_shell_cmd` is introduced to run shell commands. + +This function replaces both the `run_cmd` and `run_cmd_qa` functions, which will be deprecated in EasyBuild v5.0. + +## Motivation + +... + +## High-level overview + +... + +## Use cases + +... + +## Transitioning from `run_cmd` to `run_shell_cmd` + +... + +## Transitioning from `run_cmd_qa` to `run_shell_cmd` + +... diff --git a/docs/easybuild-v5/run_shell_cmd.md.bk b/docs/easybuild-v5/run_shell_cmd.md.bk new file mode 100644 index 000000000..3e03d96ef --- /dev/null +++ b/docs/easybuild-v5/run_shell_cmd.md.bk @@ -0,0 +1,54 @@ +# `run_shell_cmd` function + +In EasyBuild v5.0, a new function named `run_shell_cmd` is introduced to run shell commands. + +This function replaces both the `run_cmd` and `run_cmd_qa` functions, which will be deprecated in EasyBuild v5.0. + +## Motivation + +The motivation for this change is manifold: + +- The API of the `run_cmd` and `run_cmd_qa` was quite messy at best, with cryptic options like `log_all`, `log_ok`, etc. +- Dropping support for running EasyBuild on top of Python 2.7 implies an opportunity to revisit + the implementation of core functions, like the ones used for running shell commands. + Rather than trying to improve the existing `run_cmd` and `run_cmd_qa` functions, + it was significantly easier to start over with a fresh implementation, + in which we leverage features supported by Python 3.6+. +- The `run_cmd` function was not used to run Lmod commands, because it did not have support + for splitting the output produced via the stdout and stderr output channels. +- A cleaner API and fresh implementation allowed for designing the `run_shell_cmd` function + with additional features in mind, which would have been more difficult to implement + in the existing `run_cmd` and `run_shell_cmd` functions. + +## High-level overview + +### Return value + +```python +res = run_shell_cmd("echo example; echo oops >&2") +``` + + +## Use cases + +### Default usage + +```python +run_shell_cmd("echo example") +``` + +### Running "hidden" shell commands + +```python +run_shell_cmd("ulimit -a", hidden=True) +``` + +### Running a shell command that may fail + +```python +res = run_shell_cmd("russian_roulette", fail_on_error=False) +``` + +## Transitioning from `run_cmd` to `run_shell_cmd` + +## Transitioning from `run_cmd_qa` to `run_shell_cmd`