Add your own diagnostic to the C ESM EP_v2

The C-ESM-EP offers the possibility to add your own personal diagnostic (either written with CliMAF or from a personal script) to the existing battery of diagnostics.

This way you can create your own html atlas that fits your personal needs, taking advantage of the C-ESM-EP framework (looping on datasets, easy pre-treatments, building an html page…).

The minimum you need to do to add your own diagnostic in C-ESM-EP_v2

In C-ESM-EP_v2, you have an example of a simple diagnostic (using CliMAF) in standard_comparison/MyOwnDiag. We will use MyOwnDiag as an illustration.

To build a new atlas (new html page), you need to go through the following steps:

• In your comparison directory (say, toy_comparison), create a new subdirectory for your atlas (ex: AtlasEx in standard_comparison in C-ESM-EP_v2) using standard_comparison/MyOwnDiag as a basis. Copy MyOwnDiag to create your own atlas with an explicit name (i.e. NorthAtlanticConvection, SST_SSS_MLD...) or you use MyOwnDiag as it is (as we will do here). The role of the atlas directory is two-fold:
  • It contains the parameter and diagnostic files of the atlas (mandatory to run an atlas)
  • It will receive the output files of the execution of your atlas when you submit a job with run_C-ESM-EP.py
• in the atlas subdirectory, you need to have:
  • a diagnostic file named like: MyOwnDiag/diagnostics_MyOwnDiag.py ; this diagnostic file contains the code of your diagnostics and builds the html page (using CliMAF functions like start_line(), cell()...)
  • a parameter file named like: MyOwnDiag/params_MyOwnDiag.py that can contain external instructions for the diagnostics ; it is mandatory to have a parameter file in the directory even if it is empty (and it can be empty if you control everything from the diagnostics file)
• Set the title of your page either in the diagnostics file or the params file with the variable atlas_head_title (will also be used as text for the link on C-ESM-EP frontage)
• Develop the code of your diagnostic in the diagnostic file: follow the example in standard_comparison/MyOwnDiag/diagnostics_MyOwnDiag.py
• Run the C-ESM-EP (interactively or with run_C-ESM-EP.py) to correct what needs to be corrected

And lastly, enjoy using it routinely!

Note that the links to your custom atlases automatically appear as the last links (at the bottom of the list of links) on the C-ESM-EP frontpage.

Eventually your can move your diagnostic file in share/cesmep_diagnostics if you want to use only one version for different comparisons as for the standard atlases like Atmosphere_Surface, AtlasExplorer, etc. The C-ESM-EP_v2 will first try to get the diagnostic file from the atlas directory (highest priority). If there is no diagnostic file in the atlas directory, it will get the shared diagnostic file that is (should be) in share/cesmep_diagnostics. The C-ESM-EP will return an error if no diagnostic file exist either in the atlas subdirectory nor in share/cesmep_diagnostic. Having your diagnostic file in share/cesmep_diagnostics avoids maintaining multiple copies of the same diagnostics in different comparison directories.

Illustrated example to create a new atlas:

In your comparison directory (say, toy_comparison), create a new directory for your atlas; to do this, copy an existing atlas directory (MyOwnDiag for instance) and name it after your own choice:

cd standard_comparison
cp –r MyOwnDiag my_own_atlas

The name of the parameter and diagnostic files have to follow the name of the directory:

# ${my_own_atlas}/params_${my_own_atlas}.py
# ${my_own_atlas}/diagnostics_${my_own_atlas}.py
mv my_own_atlas/params_MyOwnDiag.py my_own_atlas/params_my_own_atlas.py
mv my_own_atlas/diagnostics_MyOwnDiag.py my_own_atlas/diagnostics_my_own_atlas.py

Note on the instruction variable to activate/deactivate the execution of the diagnostics In C-ESM-EP_v1, we used ‘instruction variables’ to activate/authorize the execution of the code in main_C-ESM-EP.py . Those instructions are not needed in C-ESM-EP_v2, but can you still use them for convenience.

Develop your diagnostic using the CliMAF functionalities in diagnostics_my_own_atlas.py You can add all the variables, python dictionaries or CliMAF settings (calias, project definition, derive...) direcyly in the diagnostic file.

The simplest and most straightforward way to develop your own diagnostic is to use CliMAF operators (see the CliMAF documentation), and do the plot with the CliMAF plotting operators (‘plot’ and ‘curves’).

The example in MyOwnDiag/diagnostic_MyOwnDiag.py will give you a basis to start with:

• How to use the CliMAF function to write the html index
• Copy the ‘models’ list to do a ‘Wmodels’ list that you will work on
• Loop on variables
• Loop on models
• Apply the frequency and period manager
• Do my own analysis
• Plot the result
• And add the plot to the html page

Looping on the models is actually not mandatory but it is one of the interests of the C-ESM-EP

Adding your own script

CliMAF offers an easy way to plug your own script of diagnostic/plot if the existing CliMAF operators/functions do not cover your needs.

!! Example missing!!

You can plug any script (i.e. make a CliMAF operator of it) with ‘cscript’ if this script verifies the following constraints:

• You can execute it in one command line taking arguments
• The name of the input netcdf and the output netcdf file or figure have to be explicitely part of the script arguments (see the cscript documentation for more details: )

The advantages of plugging your script are (notably):

• using the climaf cache
• be fully part of the workflow (better in terms of coding
• Be part of the C-ESM-EP sustainably

You will find a lot of examples in ${CLIMAF}/climaf/standard_operators.py of the use of cscript to plug external scripts or command lines (cdo and nco).

Note that it is possible to call a script in a compound command line, separated with ‘; ‘. This allows for more flexibility.

Although it is not the cleanest way to do it, you can also take advantage of intermediate or auxillary files in the command line that are hard coded. In other words: better to plug a script in a not perfect fashion than not plugging it!