From 1ac62f8d1a73651049f8d7051d2b41b5d9df01e2 Mon Sep 17 00:00:00 2001 From: pciturri Date: Thu, 26 Sep 2024 14:28:14 +0200 Subject: [PATCH] docs: completed Model Configuration section, --- docs/guide/experiment_config.rst | 46 ++-- docs/guide/model_config.rst | 388 +++++++++++++++++++++++++++- docs/guide/reproduce_experiment.rst | 6 + docs/guide/run_experiment.rst | 6 + docs/index.rst | 17 +- docs/tutorials/case_h.rst | 2 +- 6 files changed, 439 insertions(+), 26 deletions(-) create mode 100644 docs/guide/reproduce_experiment.rst create mode 100644 docs/guide/run_experiment.rst diff --git a/docs/guide/experiment_config.rst b/docs/guide/experiment_config.rst index a62a3e5..6ade1b7 100644 --- a/docs/guide/experiment_config.rst +++ b/docs/guide/experiment_config.rst @@ -1,16 +1,18 @@ +.. _experiment_config: + Experiment Configuration ======================== -**floatCSEP** provides a standardized workflow for forecasting experiments, whose instructions can be set in a configuration file. Here, we need to define the Experiments' temporal settings, geographic region, seismic catalogs, forecasting models, evaluation tests and post-process. +**floatCSEP** provides a standardized workflow for forecasting experiments, the instructions of which can be set in a configuration file. Here, we need to define the Experiments' temporal settings, geographic region, seismic catalogs, forecasting models, evaluation tests and any post-process options. Configuration Structure ----------------------- -Configuration files are written in ``YAML`` format and is divided into different aspects of the Experiment setup: +Configuration files are written in ``YAML`` format and are divided into different aspects of the experiment setup: -1. **Experiment Metadata**: Experiment's information such as its ``name``, ``authors``, ``doi``, ``URL``, etc. +1. **Metadata**: Experiment's information such as its ``name``, ``authors``, ``doi``, ``URL``, etc. 2. **Temporal Configuration** (``time_config``): Temporal characteristics of the experiment, such as the start and end dates, experiment class (time-independent or time-dependent), the testing intervals, etc. -3. **Spatial and Magnitude Configuration** (``region_config``): Describes testing region, such as its geographic bounds, magnitude ranges, and depth ranges. +3. **Spatial and Magnitude Configuration** (``region_config``): Describes the testing region, such as its geographic bounds, magnitude ranges, and depth ranges. 4. **Seismic Catalog** (``catalog``): Defines the seismicity data source to test the models. It can either link to a seismic network API, or an existing file in the system. 5. **Models** (``models``): Configuration of forecasting models. It can direct to an additional configuration file with ``model_config`` for readability. See :ref:`model_config`. 6. **Evaluation Tests** (``tests``): Configuration of the statistical tests to evaluate the models. It can direct to an additional configuration file with ``test_config`` for readability. See :ref:`evaluation_config`. @@ -18,7 +20,7 @@ Configuration files are written in ``YAML`` format and is divided into different .. note:: - YAML (Yet Another Markup Language) is a human-readable format used for configuration files. It uses **key: value** pairs to define settings, and indentation to represent nested structures. Lists are denoted by hyphens (`-`). + `YAML` (Yet Another Markup Language) is a human-readable format used for configuration files. It uses **key: value** pairs to define settings, and indentation to represent nested structures. Lists are denoted by hyphens (`-`). **Example Basic Configuration** (``config.yml``): @@ -74,7 +76,7 @@ Any non-parsed parameter (e.g., not specified in the documentation) will be stor Temporal Definition ------------------- -Configuring the temporal definition of the experiment is indicated with the ``time_config`` option, followed by an indented block of admissible parameters. The purpose of this configuration section is to set a testing **time-window**, or a sequence of **time-windows**. Each time-window is then defined by ``datetime`` strings representing its lower and upper edges. +Configuring the experiment temporal definition is indicated with the ``time_config`` option, followed by an indented block of admissible parameters. The purpose of this configuration section is to set a testing **time-window**, or a sequence of **time-windows**. Each time-window is defined by two ``datetime`` strings representing its lower and upper edges. Time-windows can be defined from different combination of the following parameters: @@ -86,13 +88,13 @@ Time-windows can be defined from different combination of the following paramete * - **end_date** - The end date of the experiment in UTC and ISO8601 format (``%Y-%m-%dT%H:%M:%S``) * - **intervals** - - An integer amount of testing intervals (time windows). + - An integer amount of testing intervals (time windows). If **horizon** is given, each time-window has a length equal to **horizon**. Else, the range between **start_date** and **end_date** is equally divided into the amount of **intervals**. * - **horizon** - Indicates the time windows `length`. It is written as a number, followed by a hyphen (`-`) and a time unit (``days``, ``weeks``, ``months``, ``years``). e.g.: ``1-days``, ``2.5-years``. - * - **offset** - - Offset between consecutive time-windows. If none given or ``offset=horizon``, time-windows are non-overlapping. It is written as a number, followed by a hyphen (`-`) and a time unit (``days``, ``weeks``, ``months``, ``years``). e.g.: ``1-days``, ``2.5-years``. * - **growth** - How to discretize the time-windows between ``start_date`` and ``end_date``. Options are: **incremental** (The end of a time window matches the beginning of the next) or **cumulative** (All time-windows have a start at the experiment ``start_date``). + * - **offset** + - Offset between consecutive time-windows. If none given or ``offset=horizon``, time-windows are non-overlapping. It is written as a number, followed by a hyphen (`-`) and a time unit (``days``, ``weeks``, ``months``, ``years``). e.g.: ``1-days``, ``2.5-years``. * - **exp_class** - Experiment temporal class. Options are: **ti** (default): Time-Independent; **td**: Time-Dependent. @@ -151,7 +153,7 @@ Some example of parameter combinations: | exp_class: td | | +------------------------------------------------+----------------------------------------------------------+ -Alternatively, time windows can be defined explicitly as a **list** of timewindow (which are a **list** of ``datetimes``): +Alternatively, time windows can be defined explicitly as a **list** of time-windows (each of which is a **list** of ``datetimes``): .. code-block:: yaml @@ -165,7 +167,7 @@ Alternatively, time windows can be defined explicitly as a **list** of timewindo Spatial and Magnitude Definition -------------------------------- -Configuring the spatial and magnitude definitions is done through the ``region_config`` option, followed by an indented block of admissible parameters. Here, we need to define the spatial region (check the `Region `_ concept from **pyCSEP**), the magnitude `bins` (i.e., discretization) and the `depth` extent (used mostly to filter out seismicity outside this range): +Configuring the spatial and magnitude definitions is done through the ``region_config`` option, followed by an indented block of admissible parameters. Here, we need to define the spatial region (check the `Region `_ documentation from **pyCSEP**), the magnitude `bins` (i.e., discretization) and the `depth` extent. .. list-table:: :widths: 20 80 @@ -177,7 +179,7 @@ Configuring the spatial and magnitude definitions is done through the ``region_c * - **mag_max** - The maximum magnitude of the experiment. * - **mag_bin** - - The size of a magnitude bin. + - The size of the magnitude bin. * - **depth_min** - The minimum depth (in `km`) of the experiment. * - **depth_max** @@ -187,8 +189,14 @@ Configuring the spatial and magnitude definitions is done through the ``region_c 1. The ``region`` parameter can be defined from: - * A **CSEP** region: This correspond to pre-established testing regions for highly seismic areas. This parameter is linked to **pyCSEP** functions, and can be either ``italy_csep_region``, ``nz_csep_region``, ``california_relm_region`` or ``global_region``. - * A `txt` file with the spatial cells collection , where each cell is defined by its origin (e.g., the x (lon) and y (lat) of the lower-left corner). See the **pyCSEP** `documentation `_ on Regions, the class :class:`~csep.core.regions.CartesianGrid2D` and its method :meth:`~csep.core.regions.CartesianGrid2D.from_origins`. For example, for a region consisting of three cells, their origins can be written as: + * A **CSEP** region: They correspond to pre-established testing regions for seismic areas. This parameter is linked to **pyCSEP** functions, and can be one of the following values: + + * ``italy_csep_region`` + * ``nz_csep_region`` + * ``california_relm_region`` + * ``global_region``. + + * A text file with the spatial cells collection. Each cell is defined by its origin (e.g., the x (lon) and y (lat) of the lower-left corner). For example, for a region consisting of three cells, their origins can be written as: .. code-block:: @@ -196,6 +204,7 @@ Configuring the spatial and magnitude definitions is done through the ``region_c 10.0 40.1 10.1 40.0 + See the **pyCSEP** `Region documentation `_, the class :class:`~csep.core.regions.CartesianGrid2D` and its method :meth:`~csep.core.regions.CartesianGrid2D.from_origins` for more info. 2. Magnitude definition: We need to define a magnitude discretization or `bins`. The parameters **mag_min**, **mag_max**, **mag_bin** allows to create an uniformly distributed set of bins. For example, the command: .. code-block:: yaml @@ -204,7 +213,7 @@ Configuring the spatial and magnitude definitions is done through the ``region_c mag_max: 5.0 mag_bin: 0.5 - would result in two magnitude bins with ranges ``[4.0, 4.5)`` and ``[4.5, 5.0)``. Alternatively, magnitudes can be written explicitly by their bin edges. For example: + would result in two magnitude bins with ranges ``[4.0, 4.5)`` and ``[4.5, 5.0)``. Alternatively, magnitudes can be written explicitly by their bin `left` edge. For example: .. code-block:: yaml @@ -213,7 +222,8 @@ Configuring the spatial and magnitude definitions is done through the ``region_c - 4.1 - 4.2 - resulting in the ``[4.0, 4.1)`` and ``[4.1, 4.2)`` + resulting in the ``[4.0, 4.1)``, ``[4.1, 4.2)`` and ``[4.2, 4.3)``. + 3. Depths: The minimum and maximum depths are just required to filter out seismicity outside those ranges. @@ -253,8 +263,8 @@ The seismicity catalog can be defined with the ``catalog`` parameter. It represe **floatCSEP** can retrieve the catalog from a seismic network API. The possible options are: - - ``query_gcmt``: Global Centroid Moment Tensor Catalog (https://www.globalcmt.org/) - via the API of the ISC (https://www.isc.ac.uk/) - - ``query_comcat``: ANSS Comprehensive Earthquake Catalog ComCat (https://earthquake.usgs.gov/data/comcat/) + - ``query_gcmt``: Global Centroid Moment Tensor Catalog (https://www.globalcmt.org/), obtained via ISC (https://www.isc.ac.uk/) + - ``query_comcat``: ANSS ComCat (https://earthquake.usgs.gov/data/comcat/) - ``query_bsi``: Bollettino Sismico Italiano (https://bsi.ingv.it/) - ``query_gns``: GNS GeoNet New Zealand Catalog (https://www.geonet.org.nz/) diff --git a/docs/guide/model_config.rst b/docs/guide/model_config.rst index b3faccc..5678855 100644 --- a/docs/guide/model_config.rst +++ b/docs/guide/model_config.rst @@ -1,6 +1,392 @@ .. _model_config: + Models Configuration ==================== -TBI \ No newline at end of file +**floatCSEP** can integrate **source-code** models or just **forecast files**. Depending on the model type, configuration can be as simple as specifying a file path or as complex as defining the computational environment, run commands and model arguments. In the case of source-codes, the **Model Integration** section covers the environment management, executing the model code, and input/output dataflow. + +In the experiment ``config.yml`` file (See :ref:`experiment_config`), the parameter ``model_config`` can point to a **model configuration** file, also in ``YAML`` format, with the generic structure: + +**Example**: + + .. code-block:: yaml + + - MODEL_1 NAME: + parameter_1: value + parameter_2: value + ... + - MODEL_2 NAME: + parameter_1: value + parameter_2: value + ... + ... + +Model names are used to identify models in the system, and spaces are replaced by underscores `_`. + + +Time-Independent Models +----------------------- + +A **Time-Independent** model is usually represented by a single-file forecast, whose statistical description does not change over time. +Thus, the model configuration needs only to point to the **path** of the file relative to the ``model_config`` file. + +**Example**: + +.. code-block:: yaml + + - GEAR: + path: models/gear.xml + forecast_unit: 1 + +``forecast_unit`` represents the time frame upon which the forecast rates are defined (Defaults to 1). In time-independent forecasts, ``forecast_unit`` is in decimal **years**. Forecasts are scaled to the testing time-window if its length is different to the one of the forecast. + + + +Time-Dependent Models +--------------------- + +**Time-Dependent** models are composed by forecasts issued for multiple time windows. These models can be either a **collection** of forecast files or a **source-code** that generate such collection. + + +1. **Forecast Collection**: + + In this case, the ``path`` must point to a model **directory**. To standardize with the directory structure of **source-code** models, forecasts should be contained in a folder named **forecasts** inside the model's ``path``. + + **Example**: + + .. code-block:: yaml + + - ETAS: + path: models/etas + forecast_unit: 3 + n_sims: 10000 + + * Forecasts must be contained in a folder ``models/etas/forecasts``, relative to the ``model_config`` file. + * The ``forecast_unit`` is defined in **days** for Time-Dependent models. + * ``n_sims`` represents the total number of simulations from a catalog-based forecast (usually simulations with no events are not written, so the total amount of catalogs must be explicit). + + .. important:: + + Forecast files are automatically detected. The standard way the model source should name a forecast is : + + .. code-block:: + + {model_name}_{start}_{end}.csv + + where ``start`` and ``end`` follow either the ``%Y-%m-%dT%H:%M:%S`` - ISO8601 format, or the short date version ``%Y-%m-%d`` if the windows are set by UTC midnight. + + See the **pyCSEP** `Documentation `_ to see how forecast files should be written. See the :ref:`model_integration` section for details about how a model source-code should be designed or adapted to be integrated with **floatCSEP** + +1. **Source-Code**: + + **floatCSEP** interacts with a model's source code by (i) creating a running environment, (ii) placing the input data (e.g., training catalog) within the model's directory structure, (iii) executing an specified run command and (iv) retrieving forecasts from the model directory structure. These actions will be detailed in the :ref:`model_integration` section. + + The basic parameters of the configuration are: + + * ``path`` refers to the source-code directory. + * The ``build`` parameter defines the environment type (e.g., ``conda``, ``venv``, or ``docker``) and ensures the model runs in isolation with the necessary dependencies. + * ``func`` is a `shell` command (**entrypoint**) with which the source-code is executed inside the environment. + * The ``forecast_unit`` is defined in **days** for Time-Dependent models. + + **Example**: + + .. code-block:: yaml + + - STEP: + path: models/step + build: docker + func: etas-run + forecast_unit: 1 + +Repository Download +------------------- + +A model file(s) or source code can be accessed from a code or data repository (i.e., `GitHub `_ or `Zenodo `_). + +.. code-block:: yaml + + - etas: + giturl: https://git.gfz-potsdam.de/csep/it_experiment/models/vetas.git + repo_hash: v3.2 + +where ``repo_hash`` refers to a given **release**, **tag** or **branch**. Alternatively, a model can be retrieved from a Zenodo repository by specifying its ID: + +.. code-block:: yaml + + - wheel: + zenodo_id: 6255575 + + + +Configuration Parameters +------------------------ + +Here you can find a comprehensive list of parameters used to configure models + +.. list-table:: + :widths: 20 20 60 + :header-rows: 1 + + * - **Name** + - **Type** + - **Description** + * - **path** (required) + - All + - Path to the model’s (i) **forecast file** for a time-independent class, or (ii) **model's directory** for time-dependent class + * - **build** + - TD + - Specifies the environment type in which the model will be built (e.g., ``conda``, ``venv``, ``docker``). + * - **zenodo_id** + - All + - Zenodo record ID for downloading the model's data. + * - **giturl** + - All + - Git repository URL for the model’s source code. + * - **repo_hash** + - All + - Specifies the commit, branch, or tag to be checked out from the repository. + * - **args_file** (required) + - TD + - Path to the input arguments file for the model, relative to ``path``. In here, the forecast start_date and end_date will be dynamically written before each forecast creation. Defaults to ``input/args.txt``. + * - **func** + - TD + - The command to execute the model (i.e., **entrypoint**) in a terminal. Examples of ``func`` are: ``run``, ``etas-run``, ``python run_script.py``, ``Rscript script.r``. + * - **func_kwargs** (optional) + - TD + - Additional arguments for the model execution, passed via the arguments file. + * - **forecast_unit** (required) + - All + - Specifies the time unit for the forecast. Use **years** for time-independent models and **days** for time-dependent models. + * - **store_db** (optional) + - All + - If the model consists on only files, this is a boolean (true/false) specifying whether to store the forecast in a database (HDF5). + * - **flavours** (optional) + - All + - A set of parameter variations to generate multiple model variants (e.g., different settings for the same model). + * - **prefix** (optional) + - TD + - The prefix used for the model to name its forecast (The default is the Model's name) + * - **input_cat** (optional) + - TD + - Specifies the input catalog path used by the model, relative to the model's ``path``. Defaults to ``input/catalog.csv``. + * - **force_stage** (optional) + - All + - Forces the entire staging of the model (e.g., downloading data, database preparation, environment creation, installation of dependencies and source-code build) + * - **force_build** (optional) + - All + - Forces the build of the model's environment (e.g., creation, dependencies installation and source-code build) + + + +.. _model_integration: + +Model Integration +----------------- + +The integration of external model source-codes into **floatCSEP** requires: + +* Follow (loosely) a directory structure to allow the dataflow (input/output) between the model and **pyCSEP**. +* Define a environment/container manager. +* Provide source-code build instructions. +* Set up an entrypoint (terminal command) to run the model and create a forecast. + +.. note:: + + To integrate a broader range of model classes and code complexities, we opted in **floatCSEP** for a simple interface design rather than specifying a complex model API. Therefore, the integration will have sometimes strict requirements, or customizable options and sometimes undefined aspects. We encourage any feedback from modelers (and hopefully their contributions) through our GitHub, to encompass the majority of model implementations possible. + +Directory Structure +~~~~~~~~~~~~~~~~~~~ + +The repository should contain, at the least, the following structure: + +.. code-block:: none + + model_name/ + ├── /forecasts # Forecast outputs should be stored here (Required) + ├── /input # Input data will be placed here dynamically by **floatCSEP** (Required) + │ ├── {input_catalog} # Input catalog file provided by the testing center + │ └── {args_file} # Contains the input arguments for model execution + ├── /{source} # [optional] Where to store all the source code of the model + │ └── ... + ├── /state # [optional] State files (e.g., data to be persisted throughout consistent simulations) + ├── README.md # [optional] Basic information of the model and instructions to run it. + ├── {run_script} # [optional] Script to generate forecasts. Can be either located here, or in the environment PATH (e.g., a binary entrypoint for python) + ├── Dockerfile # Docker environment setup file + ├── environment.yml # Instructions to build a conda environment. + └── setup.py # Script to build the code with "pip install . ". Can also be `project.toml` or `setup.cfg` + + +* The name of the files ``input_catalog`` (default: `catalog.csv`) and ``args_file`` (default: `args.txt`) can be controlled within ``model_config``. +* It is required (for this integration protocol) that the folders ``input`` and ``forecasts`` exists in the model directory. The latter could be created during the first model run. + +.. important:: + The directory structure should remain unchanged during the experiment run, except for the dynamic modification of the `input/`, `forecasts/` and `state/` contents. All of the source-code file management routines should point to these folders (e.g., routines to read input catalogs, read input arguments, to write forecasts, etc.). + + +Environment Management +~~~~~~~~~~~~~~~~~~~~~~ + +The `build` parameter in the model configuration specifies the environment type (e.g., `conda`, `venv`, `docker`). Models should be defined in an isolated environment to ensure reproducibility and prevent conflicts with system dependencies. + +1. **venv**: A Python virtual environment (`venv`) setup is specified. The source code will be built by running the command ``pip install .`` within the virtual sub-environment (an environment within the one **floatCSEP** is run, but isolated from it), pointing to a ``setup.py``, ``setup.cfg`` or ``project.toml`` (See the `Packaging guide `_) + +2. **conda**: The model sub-environment is managed via a `conda` environment file (``environment.yml``). The model source-code will still be built using ``pip``. + +3. **docker**: A Docker container is created based on a provided `Dockerfile` that contains the instruction to build the source-code within.(`Writing a Dockerfile `_). If python, the model source-code will still be built using ``pip`` inside a virtual environment. + +.. note:: + All the environment names will be handled internally by **floatCSEP**. + +**Example setup.cfg** + + +.. code-block:: cfg + + [metadata] + name = cookie_model + description = Just another model + author = Monster, Cookie + + [options] + packages = + cookie_model + install_requires = + numpy + python_requires = >=3.9 + + [options.entry_points] + console_scripts = + cookie-run = cookie_model.main:run + +This build configuration installs the dependencies (``numpy``), the module ``cookie_model`` (i.e., the ``{source}`` folder) and creates an entrypoint command (see the :ref:`model_execution` section). + + + +**Example Dockerfile** + +.. code-block:: dockerfile + + # Use a specific Python version from a trusted source + FROM python:3.9.20 + + # Set up user and permissions + ARG USERNAME=modeler + ARG USER_UID=1100 + RUN useradd -u $USER_UID -m -s /bin/sh $USERNAME + + # Set work directory + WORKDIR /usr/src/ + + # Copy repository contents to the container + COPY --chown=$USERNAME cookie_model ./cookie_model/ + COPY --chown=$USERNAME setup.cfg ./ + + # Install the Python package and upgrade pip + RUN pip install --no-cache-dir --upgrade pip && pip install . + + # Set the default user + USER $USERNAME + + +This Dockerfile will install the python package inside a container, but the concept can be applied also for other programming languages. The ``func`` parameter will be used identically as done for ``conda`` and ``venv`` options, but now **floatCSEP** will handle the container execution and the entrypoint. + + +.. _model_execution: + +Model Entrypoint +~~~~~~~~~~~~~~~~ + +A model should be executed always with a shell command through a terminal. This provides flexibility to the modeler to abstract their model as convenient. +The **func** parameter in the model configuration defines the shell command used to execute the model. This command is invoked within the environment set up by **floatCSEP**, and will be run from ``model_path`` or the entrypoint defined in the ``Dockerfile``. + +Example ``func`` commands: + +.. code-block:: console + + $ cookie-run + $ python run.py + $ Rscript run.R + $ sh run.sh + +The ``cookie-run`` was a binary python entrypoint defined in the previous **Example setup.cfg**. It allows to execute the command ``cookie-run`` from the terminal, which itself will run the `python` function :func:`cookie_model.main.run` from the file ``cookie_model/main.py``. + +.. note:: + + This entrypoint function should contain the high-level logic of the model workflow (e.g, reading input, parsing arguments, calling core routines, write forecasts, etc.). An example pseudo-code of a model's workflow is: + + .. code-block:: R + + start, end, args = read_input(args_path) + training_catalog = read_catalog(input_cat) + parameters = fit(training_catalog) + forecast = create_forecast(start, end, args, parameters) + write(forecast) + + + +Input/Output Dataflow +~~~~~~~~~~~~~~~~~~~~~ + +The input to run a model will be placed into the ``model_path/input/`` directory dynamically by the testing system before each model execution. The model should be able to read these files from this directory. Similarly, after each model execution, the resulting forecast should be stored in a ``model_path/forecasts/`` directory + +We distinguish **input data** versus **input arguments**. The input data is given to a model without control of the modeler (e.g. authoritative input catalog, region), whereas input arguments (as in *function* arguments) can be the forecast specifications (e.g. time-window, target magnitudes) or hyper-parameters (e.g. declustering algorithm, optimization time-windows, cutoff magnitude) that control the model. + + +1. **Input Arguments**: The input arguments are the forecast specifications (e.g. time-window, target magnitudes) and hyper-parameters (e.g. declustering algorithm, optimization time-windows, cutoff magnitude) that will control the model. The input arguments will be written in the ``args_file`` (default ``args.txt``) always located in the input folder. A model requires at minimum one set of modifiable arguments: ``start_date`` and ``end_date`` (in ISO8601), but it is possible to include additional arguments. + + Example content of ``args.txt``: + + .. code-block:: yaml + + start_date: 2023-01-01T00:00:00 + end_date: 2023-01-02T00:00:00 + seed: 23 + nsims: 1000 + + Therefore, the model source-code should be at least able to dynamically read the obligatory arguments (simply the time window of the issued forecast) + +2. **Input Data**: Correspond to any data source outside the control of the modeler (e.g., authoritative input catalog, testing region). For now, **floatCSEP** just handles an input **catalog**, which are all the events within the **main catalog** until the forecast **start_date**. The catalog is written by default in ``model_path/input/catalog.csv`` in the CSEP ascii format (see :doc:`pycsep:concepts/catalogs`) as: + + .. code-block:: none + + longitude, latitude, magnitude, time_string, depth, event_id + + - **longitude**: Decimal degrees of the forecasted event location. + - **latitude**: Decimal degrees of the forecasted event location. + - **magnitude**: Magnitude of the forecasted event. + - **time_string**: Timestamp in UTC following the ISO8601 format (`%Y-%m-%dT%H:%M:%S`). + - **depth**: Depth of the event in kilometers. + - **event_id**: The event ID in case is necessary to map the event to an additional table. + + +3. **Output Forecasts**: After execution, forecast files should be written to the `forecasts/` folder. The forecast output must follow the filename convention: + + .. code-block:: none + + {model_name}_{start-date}_{end-date}.csv + + ``model_name`` can be replaced in the model configuration with the parameter ``prefix``, such that: + + .. code-block:: none + + {prefix}_{start-date}_{end-date}.csv + + + This ensures that forecast files are easily identified and retrieved by **floatCSEP** for further evaluation. + + + .. important:: + + The forecast files should adhere to the **pyCSEP** format. In summary, each forecast file should be a ``.csv`` file containing rows for each forecasted event, whose columns are: + + .. code-block:: none + + longitude, latitude, magnitude, time_string, depth, catalog_id, event_id + + where catalog_id represents the a single simulation of the stochastic catalog collection. This format ensures compatibility with the **pyCSEP** testing framework (See the `Catalog-based forecasts `_ documentation for further information). + + + + + diff --git a/docs/guide/reproduce_experiment.rst b/docs/guide/reproduce_experiment.rst new file mode 100644 index 0000000..a194d0c --- /dev/null +++ b/docs/guide/reproduce_experiment.rst @@ -0,0 +1,6 @@ +.. _reproduce: + +Reproducing an Experiment +========================= + +TBI \ No newline at end of file diff --git a/docs/guide/run_experiment.rst b/docs/guide/run_experiment.rst new file mode 100644 index 0000000..6b5a655 --- /dev/null +++ b/docs/guide/run_experiment.rst @@ -0,0 +1,6 @@ +.. _running: + +Running an Experiment +===================== + +TBI \ No newline at end of file diff --git a/docs/index.rst b/docs/index.rst index 8d36a03..0f08e9f 100644 --- a/docs/index.rst +++ b/docs/index.rst @@ -152,12 +152,6 @@ Collaborators tutorials/case_g.rst tutorials/case_h.rst -.. toctree:: - :hidden: - :maxdepth: 0 - :caption: Help & Reference - - reference/api_reference .. toctree:: :maxdepth: 2 @@ -168,6 +162,17 @@ Collaborators guide/model_config.rst guide/evaluation_config.rst guide/postprocess_config.rst + guide/run_experiment.rst + guide/reproduce_experiment.rst + + +.. toctree:: + :hidden: + :maxdepth: 0 + :caption: Help & Reference + + reference/api_reference + .. toctree:: :hidden: diff --git a/docs/tutorials/case_h.rst b/docs/tutorials/case_h.rst index c9e6637..667c083 100644 --- a/docs/tutorials/case_h.rst +++ b/docs/tutorials/case_h.rst @@ -1,6 +1,6 @@ .. _case_h: -H - A time-dependent experiment +H - A Time-Dependent Experiment =============================== Here, we run an experiment that access, containerize and execute multiple **time-dependent models**, and then proceeds to evaluate the forecasts once they are created.