From c46a74f292cfa04c54e1dd51bcf362c95a19eece Mon Sep 17 00:00:00 2001 From: John Gerrard Holland Date: Fri, 19 Jan 2024 12:21:21 -0500 Subject: [PATCH 1/6] docs: add overview page for CLI --- docs/cli/index.md | 34 ++++++++++++++++++++++++++++++++++ 1 file changed, 34 insertions(+) create mode 100644 docs/cli/index.md diff --git a/docs/cli/index.md b/docs/cli/index.md new file mode 100644 index 00000000..ffd89d88 --- /dev/null +++ b/docs/cli/index.md @@ -0,0 +1,34 @@ +# Command Line Interface + +Often, different parts of AutoRA experiments require very different computational resources. For instance: +- The theorist and experimentalist might require training or use of neural networks, and benefit from high + performance computing (HPC) resources for short bursts – minutes or hours. +- The experiment runner might post an experiment using a service like "Prolific" and poll every few minutes for + hours, days or week until the experimental data are gathered. + +Running the experiment runner with the same resources as the theorist and experimentalist in this case would be +wasteful, and may be prohibitively expensive. + +To solve this problem, AutoRA comes with a command line interface (CLI). This can be used with HPC schedulers like +[SLURM](https://slurm.schedmd.com/) to run different steps in the cycle with different resources. + +You can use the CLI if the following conditions are true: + +1. Every part of `s` can be successfully [pickled](https://docs.python.org/3/library/pickle.html). +2. You can write each step of your experiment as a single importable function which operates on a state and returns + a state: + ```python + from example.lib import initial_state, experimentalist, experiment_runner, theorist + s = initial_state() + for i in range(3): + s = experimentalist(s) + s = experiment_runner(s) + s = theorist(s) + ``` + +Contents of this section: +- ["Basic Usage"](./basic-usage): Basic usage of the CLI +- ["Usage with Cylc workflow manager"](./cylc-pip): Example using the Cylc workflow manager (which + handles cyclical processes) +- ["Usage with Cylc workflow manager and Slurm"](./cylc-slurm-pip): Example using the Cylc + workflow manager and the SLURM scheduler with different resources for each step in the cycle. From 9d331e994c42e9265dbd259d75f752afd8622b7f Mon Sep 17 00:00:00 2001 From: John Gerrard Holland Date: Fri, 19 Jan 2024 12:21:45 -0500 Subject: [PATCH 2/6] docs: add additional intro to basic usage --- docs/cli/basic-usage/README.md | 28 +++++++++++++++++++++++++++- 1 file changed, 27 insertions(+), 1 deletion(-) diff --git a/docs/cli/basic-usage/README.md b/docs/cli/basic-usage/README.md index 812bc651..b6cc5e9c 100644 --- a/docs/cli/basic-usage/README.md +++ b/docs/cli/basic-usage/README.md @@ -1,7 +1,33 @@ -# Command Line Interface Overview +# Command Line Interface Basic Usage The command line interface allows us to load and save `States` and run arbitrary functions on them. +You can use the command line interface if the following conditions are true: + +1. Every part of `s` can be successfully [pickled](https://docs.python.org/3/library/pickle.html). +2. You can write each step of your experiment as a single importable function which operates on a state and returns + a state: + ```python + from example.lib import initial_state, experimentalist, experiment_runner, theorist + s = initial_state() + for i in range(3): + s = experimentalist(s) + s = experiment_runner(s) + s = theorist(s) + ``` + +Often, different parts of AutoRA experiments require very different computational resources. For instance: +- The theorist and experimentalist might require training or use of neural networks, and benefit from high + performance computing (HPC) resources for short bursts – minutes or hours. +- The experiment runner might post an experiment using a service like "Prolific" and poll every few minutes for + hours, days or week until the experimental data are gathered. + +Running the experiment runner with the same resources as the theorist and experimentalist in this case would be +wasteful, and may be prohibitively expensive. + +To solve this problem, AutoRA comes with a command line interface. This can be used with HPC schedulers like [SLURM] +(https://slurm.schedmd.com/) to run different steps in the cycle with different resources. + ## Setup To use the command line, we first define a package `example` containing the functions we want to run on the State: From e54c5a6c4bbe04aa3bcbc2d2e884a1fa6193f696 Mon Sep 17 00:00:00 2001 From: John Gerrard Holland Date: Fri, 19 Jan 2024 12:21:51 -0500 Subject: [PATCH 3/6] Revert "docs: add additional intro to basic usage" This reverts commit 9d331e994c42e9265dbd259d75f752afd8622b7f. --- docs/cli/basic-usage/README.md | 28 +--------------------------- 1 file changed, 1 insertion(+), 27 deletions(-) diff --git a/docs/cli/basic-usage/README.md b/docs/cli/basic-usage/README.md index b6cc5e9c..812bc651 100644 --- a/docs/cli/basic-usage/README.md +++ b/docs/cli/basic-usage/README.md @@ -1,33 +1,7 @@ -# Command Line Interface Basic Usage +# Command Line Interface Overview The command line interface allows us to load and save `States` and run arbitrary functions on them. -You can use the command line interface if the following conditions are true: - -1. Every part of `s` can be successfully [pickled](https://docs.python.org/3/library/pickle.html). -2. You can write each step of your experiment as a single importable function which operates on a state and returns - a state: - ```python - from example.lib import initial_state, experimentalist, experiment_runner, theorist - s = initial_state() - for i in range(3): - s = experimentalist(s) - s = experiment_runner(s) - s = theorist(s) - ``` - -Often, different parts of AutoRA experiments require very different computational resources. For instance: -- The theorist and experimentalist might require training or use of neural networks, and benefit from high - performance computing (HPC) resources for short bursts – minutes or hours. -- The experiment runner might post an experiment using a service like "Prolific" and poll every few minutes for - hours, days or week until the experimental data are gathered. - -Running the experiment runner with the same resources as the theorist and experimentalist in this case would be -wasteful, and may be prohibitively expensive. - -To solve this problem, AutoRA comes with a command line interface. This can be used with HPC schedulers like [SLURM] -(https://slurm.schedmd.com/) to run different steps in the cycle with different resources. - ## Setup To use the command line, we first define a package `example` containing the functions we want to run on the State: From ab497d8a17d5ed936bd850df1a2e19a71016122e Mon Sep 17 00:00:00 2001 From: John Gerrard Holland Date: Fri, 19 Jan 2024 12:26:04 -0500 Subject: [PATCH 4/6] docs: add intro page to nav --- mkdocs.yml | 1 + 1 file changed, 1 insertion(+) diff --git a/mkdocs.yml b/mkdocs.yml index 71011cd2..015b2368 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -30,6 +30,7 @@ nav: - Cycle: - Functional: 'cycle/Linear and Cyclical Workflows using Functions and States.ipynb' - "Command Line Interface": + - Home: "cli/index.md" - "Basic Usage": "cli/basic-usage/README.md" - "Using Cylc": "cli/cylc-pip/README.md" - "Using Cylc and Slurm": "cli/cylc-slurm-pip/README.md" From 2cd99ff1ecfbccf4cf9dbc7c6c52bc6b98581118 Mon Sep 17 00:00:00 2001 From: John Gerrard Holland Date: Fri, 19 Jan 2024 12:34:36 -0500 Subject: [PATCH 5/6] docs: update formatting of bullet points --- docs/cli/index.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/docs/cli/index.md b/docs/cli/index.md index ffd89d88..9ad3b07e 100644 --- a/docs/cli/index.md +++ b/docs/cli/index.md @@ -1,6 +1,7 @@ # Command Line Interface -Often, different parts of AutoRA experiments require very different computational resources. For instance: +Different parts of AutoRA experiments can require very different computational resources. For instance: + - The theorist and experimentalist might require training or use of neural networks, and benefit from high performance computing (HPC) resources for short bursts – minutes or hours. - The experiment runner might post an experiment using a service like "Prolific" and poll every few minutes for From f536e482e70ec434e0eb3f8df62e8bad95780b92 Mon Sep 17 00:00:00 2001 From: John Gerrard Holland Date: Fri, 19 Jan 2024 12:37:00 -0500 Subject: [PATCH 6/6] docs: update bullet point formatting --- docs/cli/index.md | 1 + 1 file changed, 1 insertion(+) diff --git a/docs/cli/index.md b/docs/cli/index.md index 9ad3b07e..6404655d 100644 --- a/docs/cli/index.md +++ b/docs/cli/index.md @@ -28,6 +28,7 @@ You can use the CLI if the following conditions are true: ``` Contents of this section: + - ["Basic Usage"](./basic-usage): Basic usage of the CLI - ["Usage with Cylc workflow manager"](./cylc-pip): Example using the Cylc workflow manager (which handles cyclical processes)