From 4ad68e8a3f5152ba0e39752c24e6d96af59c3f68 Mon Sep 17 00:00:00 2001 From: Bart Oldeman Date: Wed, 17 Jan 2024 14:20:20 -0500 Subject: [PATCH 1/5] Document how to transition from `run_cmd` to `run_shell_cmd` --- docs/easybuild-v5/run_shell_cmd.md | 41 +++++++++++++++++++++++++++++- 1 file changed, 40 insertions(+), 1 deletion(-) diff --git a/docs/easybuild-v5/run_shell_cmd.md b/docs/easybuild-v5/run_shell_cmd.md index 279dac9ec..94c6df7ba 100644 --- a/docs/easybuild-v5/run_shell_cmd.md +++ b/docs/easybuild-v5/run_shell_cmd.md @@ -20,7 +20,46 @@ This function replaces both the `run_cmd` and `run_cmd_qa` functions, which will ## Transitioning from `run_cmd` to `run_shell_cmd` -... +First of all `run_shell_cmd` has been designed so the defaults can be used for most situations. +In that case the main thing to watch out for is the return code, which changed from a tuple `(output, exit_code)` +to a named tuple with three fields: +- `output`: command output, `stdout`+`stderr` combined if `split_stderr` is disabled, only `stdout` otherwise +- `exit_code`: exit code of command (integer) +- `stderr`: stderr output if `split_stderr` is enabled, `None` otherwise + +A typical transition will then change +``` +(out, _) = run_cmd(cmd, log_all=True, simple=False) +``` +to +``` +res = run_shell_cmd(cmd) +out = res.output +``` + +For parameters in general, the following translation table can be used, where the default values are shown: + +| run_cmd parameter |run_shell_cmd parameter| meaning | +| ---------------------|-----------------------|---------| +| `cmd` | `cmd` | command to run | +| `log_all=False` | (removed) | always log command output and exit code (now always `True`) | +| `simple=False` | (removed) | if `True`, just return `True`/`False` to indicate success (obsolete) | +| `regexp=True` | (removed) | regex used to check the output for errors (obsolete) | +| `log_ok=True` | `fail_on_error=True` | fail on non-zero exit code | +| | `split_error=False` | split of stderr from stdout output | +| `inp=None` | `stdin=None` | the input given to the command via `stdin` | +| | `env=None` | environment to use to run command (if `None`, inherit current process environment) | +| `trace=True` | `hidden=False` | don't show command in terminal output with `--trace`, or `--extended-dry-run` / `-x`) | +| `force_in_dry_run=False`| `in_dry_run=False` | also run command in dry run mode | +| `verbose=True` | `verbose_dry_run=False` | show that command is run in dry run mode (overrules `--hidden`) | +| `path=None` | `work_dir=None` | working directory to run command in (current working directory if `None`) | +| `shell=None` | `use_bash=True` | execute command through bash shell (`run_cmd` enables this for `None`)| +| `log_output=False` | `output_file=True` | collect command output in temporary output file (changed default) | +| `stream_output=None` | `stream_output=None` | stream command output to stdout (auto-enabled with `--logtostdout` if `None`) | +| `asynchronous=False` | `asynchronous=False` | run command asynchronously (not yet implemented for `run_shell_cmd`)| +| `with_hooks=True` | `with_hooks=True` | trigger pre/post `run_cmd` or `run_shell_cmd` hooks | +| | `qa_patterns=None` | list of 2-tuples with patterns for questions + corresponding answers (not yet implemented for `run_shell_cmd`) | +| | `qa_wait_patterns=None`| list of 2-tuples with patterns for non-questions and number of iterations to allow these patterns to match with end out command output (not yet implemented for `run_shell_cmd`)| ## Transitioning from `run_cmd_qa` to `run_shell_cmd` From dc26eeb6a252106acb5c44870c833b08cddd24ab Mon Sep 17 00:00:00 2001 From: Bart Oldeman Date: Wed, 17 Jan 2024 14:24:37 -0500 Subject: [PATCH 2/5] Address markdown linting errors --- docs/easybuild-v5/run_shell_cmd.md | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/docs/easybuild-v5/run_shell_cmd.md b/docs/easybuild-v5/run_shell_cmd.md index 94c6df7ba..c1a1e786d 100644 --- a/docs/easybuild-v5/run_shell_cmd.md +++ b/docs/easybuild-v5/run_shell_cmd.md @@ -23,16 +23,20 @@ This function replaces both the `run_cmd` and `run_cmd_qa` functions, which will First of all `run_shell_cmd` has been designed so the defaults can be used for most situations. In that case the main thing to watch out for is the return code, which changed from a tuple `(output, exit_code)` to a named tuple with three fields: + - `output`: command output, `stdout`+`stderr` combined if `split_stderr` is disabled, only `stdout` otherwise - `exit_code`: exit code of command (integer) - `stderr`: stderr output if `split_stderr` is enabled, `None` otherwise A typical transition will then change -``` + +```python (out, _) = run_cmd(cmd, log_all=True, simple=False) ``` + to -``` + +```python res = run_shell_cmd(cmd) out = res.output ``` From 84f6e7db2e7aee8b2f13d75dbde0985d25f6c8ed Mon Sep 17 00:00:00 2001 From: Bart Oldeman Date: Wed, 17 Jan 2024 14:48:06 -0500 Subject: [PATCH 3/5] Further documentation for `run_shell_cmd` --- docs/easybuild-v5/run_shell_cmd.md | 37 ++++++++++++++++++++++-------- 1 file changed, 28 insertions(+), 9 deletions(-) diff --git a/docs/easybuild-v5/run_shell_cmd.md b/docs/easybuild-v5/run_shell_cmd.md index c1a1e786d..540692c91 100644 --- a/docs/easybuild-v5/run_shell_cmd.md +++ b/docs/easybuild-v5/run_shell_cmd.md @@ -8,18 +8,13 @@ This function replaces both the `run_cmd` and `run_cmd_qa` functions, which will ## Motivation -... +Over the years `run_cmd` and `run_cmd_qa` accumulated a lot of arguments which sometimes have misleading names. +Moreover they have two different kinds of return values (depending on whether called with `simple=True` or `simple=False`). +So a new command `run_shell_cmd` is introduced to replace both, using more direct and natural arguments, and as a bonus, +better error reporting. In line with `--trace` being set by default, `run_shell_cmd` is now also more verbose by default. ## High-level overview -... - -## Use cases - -... - -## Transitioning from `run_cmd` to `run_shell_cmd` - First of all `run_shell_cmd` has been designed so the defaults can be used for most situations. In that case the main thing to watch out for is the return code, which changed from a tuple `(output, exit_code)` to a named tuple with three fields: @@ -41,6 +36,28 @@ res = run_shell_cmd(cmd) out = res.output ``` +## Use cases + +Examples: + +- Basic use + + ```python + cmd = ' '.join([self.cfg['preinstallopts'], install_cmd, self.cfg['installopts']]) + run_shell_cmd(cmd) + ``` + +- Get error code for both failure and non-failure of the command, as otherwise `run_shell_cmd` will raise `RunShellCmdError`. Additionally, don't display this command in terminal output: + + ```python + cmd = "cmake --version" + res = run_shell_cmd(cmd, hidden=True, fail_on_error=False) + out = res.output + ec = res.code + ``` + +## Transitioning from `run_cmd` to `run_shell_cmd` + For parameters in general, the following translation table can be used, where the default values are shown: | run_cmd parameter |run_shell_cmd parameter| meaning | @@ -67,4 +84,6 @@ For parameters in general, the following translation table can be used, where th ## Transitioning from `run_cmd_qa` to `run_shell_cmd` +This is still to be implemented in `run_shell_cmd`. + ... From be3a149ece278644f42f88f0475c88f7f71e74b4 Mon Sep 17 00:00:00 2001 From: Kenneth Hoste Date: Mon, 22 Jan 2024 18:13:06 +0100 Subject: [PATCH 4/5] minor tweaks to documentation on migrating from run_cmd to run_shell_cmd --- docs/easybuild-v5/run_shell_cmd.md | 18 ++++++++---------- 1 file changed, 8 insertions(+), 10 deletions(-) diff --git a/docs/easybuild-v5/run_shell_cmd.md b/docs/easybuild-v5/run_shell_cmd.md index 540692c91..fc337b5f8 100644 --- a/docs/easybuild-v5/run_shell_cmd.md +++ b/docs/easybuild-v5/run_shell_cmd.md @@ -17,7 +17,7 @@ better error reporting. In line with `--trace` being set by default, `run_shell_ First of all `run_shell_cmd` has been designed so the defaults can be used for most situations. In that case the main thing to watch out for is the return code, which changed from a tuple `(output, exit_code)` -to a named tuple with three fields: +to a named tuple with multiple fields, the most prominent ones being: - `output`: command output, `stdout`+`stderr` combined if `split_stderr` is disabled, only `stdout` otherwise - `exit_code`: exit code of command (integer) @@ -40,15 +40,13 @@ out = res.output Examples: -- Basic use - +- Basic usage: ```python cmd = ' '.join([self.cfg['preinstallopts'], install_cmd, self.cfg['installopts']]) run_shell_cmd(cmd) ``` - Get error code for both failure and non-failure of the command, as otherwise `run_shell_cmd` will raise `RunShellCmdError`. Additionally, don't display this command in terminal output: - ```python cmd = "cmake --version" res = run_shell_cmd(cmd, hidden=True, fail_on_error=False) @@ -63,19 +61,19 @@ For parameters in general, the following translation table can be used, where th | run_cmd parameter |run_shell_cmd parameter| meaning | | ---------------------|-----------------------|---------| | `cmd` | `cmd` | command to run | -| `log_all=False` | (removed) | always log command output and exit code (now always `True`) | -| `simple=False` | (removed) | if `True`, just return `True`/`False` to indicate success (obsolete) | -| `regexp=True` | (removed) | regex used to check the output for errors (obsolete) | +| `log_all=False` | (removed) | always log command output and exit code *(now always `True`)* | +| `simple=False` | (removed) | if `True`, just return `True`/`False` to indicate success *(obsolete)* | +| `regexp=True` | (removed) | regex used to check the output for errors *(obsolete)* | | `log_ok=True` | `fail_on_error=True` | fail on non-zero exit code | -| | `split_error=False` | split of stderr from stdout output | +| | `split_error=False` | split of stderr from stdout output *(new feature)*| | `inp=None` | `stdin=None` | the input given to the command via `stdin` | -| | `env=None` | environment to use to run command (if `None`, inherit current process environment) | +| | `env=None` | environment to use to run command (if `None`, inherit current process environment) *(new feature)* | | `trace=True` | `hidden=False` | don't show command in terminal output with `--trace`, or `--extended-dry-run` / `-x`) | | `force_in_dry_run=False`| `in_dry_run=False` | also run command in dry run mode | | `verbose=True` | `verbose_dry_run=False` | show that command is run in dry run mode (overrules `--hidden`) | | `path=None` | `work_dir=None` | working directory to run command in (current working directory if `None`) | | `shell=None` | `use_bash=True` | execute command through bash shell (`run_cmd` enables this for `None`)| -| `log_output=False` | `output_file=True` | collect command output in temporary output file (changed default) | +| `log_output=False` | `output_file=True` | collect command output in temporary output file *(changed default)* | | `stream_output=None` | `stream_output=None` | stream command output to stdout (auto-enabled with `--logtostdout` if `None`) | | `asynchronous=False` | `asynchronous=False` | run command asynchronously (not yet implemented for `run_shell_cmd`)| | `with_hooks=True` | `with_hooks=True` | trigger pre/post `run_cmd` or `run_shell_cmd` hooks | From 4144b7ac2fb8a954b04cae0932afc31cb2eb7971 Mon Sep 17 00:00:00 2001 From: Kenneth Hoste Date: Mon, 22 Jan 2024 18:16:20 +0100 Subject: [PATCH 5/5] add back empty lines above code blocks to make MarkDown linter happy --- docs/easybuild-v5/run_shell_cmd.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/docs/easybuild-v5/run_shell_cmd.md b/docs/easybuild-v5/run_shell_cmd.md index fc337b5f8..d1df7bd7d 100644 --- a/docs/easybuild-v5/run_shell_cmd.md +++ b/docs/easybuild-v5/run_shell_cmd.md @@ -41,12 +41,14 @@ out = res.output Examples: - Basic usage: + ```python cmd = ' '.join([self.cfg['preinstallopts'], install_cmd, self.cfg['installopts']]) run_shell_cmd(cmd) ``` - Get error code for both failure and non-failure of the command, as otherwise `run_shell_cmd` will raise `RunShellCmdError`. Additionally, don't display this command in terminal output: + ```python cmd = "cmake --version" res = run_shell_cmd(cmd, hidden=True, fail_on_error=False)