Merge pull request #151 from MetOffice/documentation-restructure

Documentation restructure

docs/source/background/index.rst

@@ -0,0 +1,10 @@
+Background Information
+======================
+
+This section contains explanations and discussions to give background and
+context to parts of CSET, such as why something is the way it is.
+
+.. toctree::
+    :maxdepth: 2
+
+    why-cset

docs/source/background/why-cset.rst

@@ -0,0 +1,22 @@
+Why use CSET?
+=============
+
+When evaluating weather models you are trying to understand the processes that
+lead to biases in our model configurations, and how that compares to other
+models and observations. This can be a very iterative process, and each step of
+evaluation unveils more questions that need investigations.
+
+CSET aids in this by providing a flexible way to interrogate model data, using
+diagnostics that can be quickly created by the combination of operators in
+:doc:`/usage/operator-recipes`. Common operations such as reading, writing, and
+regridding are provided to reduce duplication of effort.
+
+CSET is a centralised place for custom diagnostics to live, with well defined
+:doc:`/contributing/index` to ensure that they stay maintained. By contributing
+diagnostics to CSET you ensure that they outlive the paper they were written
+for, and benefit the entire modelling community. Furthermore, any operators that
+the community finds especially useful will be contributed back to
+METplus_, so they can serve an
+even wider community.
+
+.. _METplus: https://dtcenter.org/community-code/metplus

docs/source/conf.py

@@ -20,18 +20,14 @@
     "sphinx.ext.doctest",
 ]
 
-templates_path = ["_templates"]
-exclude_patterns = []
-
 # -- Options for HTML output -------------------------------------------------
 # https://www.sphinx-doc.org/en/master/usage/configuration.html#options-for-html-output
 
 html_theme = "furo"
-html_static_path = ["_static"]
 
 # -- LaTeX output configuration ----------------------------------------------
 # https://www.sphinx-doc.org/en/master/latex.html
 
-# latex_engine = 'xelatex'
+latex_engine = "xelatex"
 latex_elements = {"papersize": "a4paper"}
-# latex_show_urls = 'footnote'
+latex_show_urls = "footnote"

docs/source/working-practices/documentation.rst → docs/source/contributing/documentation.rst

@@ -10,6 +10,9 @@ maintain documentation within the version control system, and keep it up to
 date. The `Sphinx website`_ has a useful primer to using reStructuredText for
 documentation.
 
+The documentation is organised intro sections following the `Divio documentation
+system`_.
+
 You can build the documentation of CSET with the following command:
 
 .. code-block:: bash
@@ -18,6 +21,8 @@ You can build the documentation of CSET with the following command:
 
 The documentation will be output to the docs/build directory.
 
+.. _Divio documentation system: https://documentation.divio.com/
+
 Docstrings
 ----------
 

docs/source/working-practices/index.rst → docs/source/contributing/getting-started.rst

@@ -1,28 +1,12 @@
-Working Practices
-=================
-
-To aid in the development of CSET we are adopting some working practices so
-everyone involved knows what to expect. On this page is everything you need to
-get started, and the links below go into more detail on specific topics.
-
-.. toctree::
-    :maxdepth: 2
-
-    git
-    testing
-    documentation
-    code-review
-    releases
-
 .. _working_practices_getting_started:
 
 Getting Started
----------------
+===============
 
 Before you can get to coding, there are a few steps you need to do.
 
 Getting the code
-~~~~~~~~~~~~~~~~
+----------------
 
 Git is a version control software, which makes collaborating on software much
 easier. If you are new to git have a look at the :doc:`git` page to get started.
@@ -44,7 +28,7 @@ of its history onto your computer. From there we need to setup a few things
 before we can develop.
 
 Setting up tooling
-~~~~~~~~~~~~~~~~~~
+------------------
 
 To be able to run the tests or build the documentation you need some
 prerequisite software. The easiest way to get this is with `conda_`. The

docs/source/contributing/index.rst

@@ -0,0 +1,16 @@
+Contributing to CSET
+====================
+
+To aid in the development of CSET we are adopting some working practices so
+everyone involved knows what to expect. Linked from this page is everything you
+need to get started, and the links below go into more detail on specific topics.
+
+.. toctree::
+    :maxdepth: 2
+
+    getting-started
+    git
+    testing
+    documentation
+    code-review
+    releases

docs/source/getting-started/index.rst

@@ -0,0 +1,10 @@
+Getting Started
+===============
+
+This section contains tutorials suitable for someone getting started with CSET.
+By following them you should learn the basics of using CSET.
+
+.. toctree::
+    :maxdepth: 2
+
+    installation

docs/source/getting-started/installation.rst

@@ -0,0 +1,15 @@
+Installation
+============
+
+.. Tutorial saying how to install CSET. For edge cases should link elsewhere.
+
+The recommended way to install CSET is via conda. It is packaged on
+`conda-forge`_ and can be installed with a simple ``conda install -c conda-forge
+cset``.
+
+If you instead want to run a development version that has yet to be released,
+the easiest way is via an editable install. You can learn how to do this in the
+:ref:`working_practices_getting_started` section of the
+:doc:`/contributing/index`.
+
+.. _conda-forge: https://anaconda.org/conda-forge/cset

docs/source/index.rst

@@ -4,44 +4,32 @@ CSET Documentation
 .. toctree::
    :hidden:
 
+   self
+   getting-started/index
    usage/index
-   working-practices/index
-   operators
-   genindex
-   GitHub <https://github.com/MetOffice/CSET>
-
-:abbr:`CSET (Convective Scale Evaluation Tool)` is a tool to aid in verifying
-and evaluating convective-scale and turbulence-scale (regional and increasingly
-global) model configurations. It aims to replace the RMED RES and Toolbox and
-the collections of bespoke scripts littering people's home directories, reducing
-effort wasted on duplicating already existing code. This centralisation of
-diagnostics should also make evaluations more consistent, reproducible and
-comparable. Development takes place in the `CSET repository on GitHub`_.
+   reference/index
+   background/index
+   contributing/index
+
+:abbr:`CSET (Acronym meaning contested)` is a tool to aid in verifying and
+evaluating :abbr:`ESM (Earth System Model)` configurations. It aims to replace
+the :abbr:`RES (Regional Evaluation Suite)`, :abbr:`RMED (Regional Model
+Evaluation & Development)` Toolbox and the collections of bespoke scripts
+littering people's home directories. By centralising these in CSET the effort
+spent on duplicate code can be reduced, and give a chance for code review to
+catch any bugs. This will make evaluations more consistent, reproducible and
+comparable. For more, see :doc:`/background/why-cset`
 
 Use the side bar to the left to access other pages of the documentation.
 
-Why use CSET?
--------------
+Useful Links
+------------
 
-When evaluating weather models you are trying to understand the processes that
-lead to biases in our model configurations, and how that compares to other
-models and observations. This can be a very iterative process, and each step of
-evaluation unveils more questions that need investigations.
+`Source Code`_ | `Issue Tracker`_ | Releases_
 
-CSET aids in this by providing a flexible way to interrogate model data, using
-diagnostics that can be quickly created by the combination of :doc:`operators`
-in an :doc:`usage/operator-recipes`. Common operations such as reading, writing,
-and regridding are provided to reduce duplication of effort.
-
-CSET is a centralised place for custom diagnostics to live, with well defined
-:doc:`working-practices/index` to ensure that they stay maintained. By
-contributing diagnostics to CSET you ensure that they outlive the paper they
-were written for, and benefit the entire modelling community. Furthermore, any
-operators that the community finds especially useful will be contributed back to
-[METplus](https://dtcenter.org/community-code/metplus), so they can serve an
-even wider community.
-
-.. _CSET repository on GitHub: https://github.com/MetOffice/CSET
+.. _Source Code: https://github.com/MetOffice/CSET
+.. _Issue Tracker: https://github.com/MetOffice/CSET/issues
+.. _Releases: https://github.com/MetOffice/CSET/releases
 
 Licence
 -------

docs/source/reference/cli.rst

@@ -0,0 +1,71 @@
+CLI Usage
+=========
+
+.. _cset-bake-command:
+
+cset bake
+~~~~~~~~~
+
+Used to run a recipe on some data. The recipe, input file, and a path to an
+output file must be provided. Additional help is available with the ``--help``
+option. The recipe format is described on the :doc:`/usage/operator-recipes` page. If
+recipe_file is omitted from the command line, the recipe will be read from the
+``CSET_RECIPE`` environment variable instead. This environment variable should
+contain the YAML text that you want executed.
+
+.. code-block:: text
+
+    usage: cset bake [-h] input_file output_file [recipe_file]
+
+    positional arguments:
+    input_file   input file to read
+    output_file  output file to write
+    recipe_file  recipe file to execute. If omitted reads from CSET_RECIPE environment variable
+
+    options:
+    -h, --help   show this help message and exit
+
+.. _cset-cookbook-command:
+
+cset cookbook
+~~~~~~~~~~~~~
+
+Saves the included recipe files to a local directory. This allows access to
+pre-created recipes for many common tasks. See [LINK TO DOCUMENTATION PAGE ON
+RECIPES] for descriptions of available recipes.
+
+.. code-block:: text
+
+    usage: cset cookbook [-h] [recipe_dir]
+
+    positional arguments:
+    recipe_dir  directory to save recipes. If omitted uses $PWD/recipes
+
+    options:
+    -h, --help  show this help message and exit
+
+.. _cset-graph-command:
+
+cset graph
+~~~~~~~~~~
+
+Visualise a recipe. This renders a graph representing the recipe, and saves it
+to a temporary file, which is automatically opened. If the ``--detailed`` option
+is specified it also includes operator arguments in the output, so you can see
+exactly what ran.
+
+.. image:: recipe-graph.svg
+    :alt: A graph visualising a recipe. The nodes are linked with directed edges showing the flow of data.
+
+.. code-block:: text
+
+    usage: cset graph [-h] [-o [OUTPUT_PATH]] [-d] [recipe]
+
+    positional arguments:
+    recipe                recipe file to read. If omitted reads from CSET_RECIPE environment variable
+
+    options:
+    -h, --help            show this help message and exit
+    -o [OUTPUT_PATH], --output_path [OUTPUT_PATH]
+                            file in which to save the graph image, otherwise uses a temporary file. When specified the file is not automatically opened
+    -d, --detailed        include operator arguments in output

docs/source/reference/index.rst

@@ -0,0 +1,12 @@
+Technical Reference
+===================
+
+This section contains comprehensive descriptions of CSET, and its various
+components.
+
+.. toctree::
+    :maxdepth: 2
+
+    cli
+    operators
+    recipe-format

docs/source/operators.rst → docs/source/reference/operators.rst

@@ -3,7 +3,7 @@ CSET Operators
 
 This page details the operators contained within CSET. It is automatically
 generated from the code and its docstrings. Operators should be used via
-:doc:`usage/operator-recipes`.
+:doc:`/usage/operator-recipes`.
 
 CSET.operators.constraints
 --------------------------
@@ -17,7 +17,6 @@ CSET.operators.filters
 .. automodule:: CSET.operators.filters
    :members:
 
-
 CSET.operators.misc
 -------------------
 

docs/source/reference/recipe-format.rst

@@ -0,0 +1,60 @@
+Recipe Format
+=============
+
+The recipes are text files written in `YAML 1.2`_, a configuration language that
+is widely used. They are saved with the ``.yaml`` extension. Below is a
+commented example recipe:
+
+.. code-block:: yaml
+
+    name: Name of recipe
+    description: |
+      Extended description that can
+      go across multiple lines.
+
+    steps:
+      # Specify the operator to run in each step.
+      - operator: read.read_cubes
+
+    - operator: filters.filter_cubes
+      # Can specify extra keyword arguments as sub-maps.
+      constraint:
+        # Can nest in another operator to use its output as an argument.
+        operator: generate_constraints.generate_stash_constraints
+        # Input implicitly taken from the previous step, but can be overridden
+        # by using the appropriate keyword argument.
+        stash: m01s03i236
+
+    - operator: write.write_cube_to_nc
+      # Specify the name of the argument, and its value.
+      file_path: CSET_OUTPUT_PATH
+      # "CSET_OUTPUT_PATH" is special and becomes the runtime output file path.
+
+The name and description keys provide a human readable description of what the
+recipe does. They are currently not used anywhere, but it is good practice to
+include them for future users of the recipe.
+
+The steps keys specifies a list of processing steps. The steps are run from top
+to bottom, with each step specifying an operator to run, and optionally any
+additional inputs to that operator. A step is denoted by a ``-`` under the
+``steps:`` key. The operators are specified on the operator key. Its value
+should be a string of the form ``file.function``. For additional inputs the key
+should be the name of the argument.
+
+The below code block shows how you can nest operators multiple levels deep. For
+details of the specific operators involved, and the arguments that can take, see
+the :doc:`/reference/operators` page.
+
+.. code-block:: yaml
+
+  - operator: filters.filter_cubes
+    constraint:
+      operator: constraints.combine_constraints
+      constraint:
+        operator: constraints.generate_stash_constraint
+        stash: m01s03i236
+      cell_methods_constraint:
+        operator: constraints.generate_cell_methods_constraint
+        cell_methods: []
+
+.. _YAML 1.2: https://yaml.org/