Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Momentum TeX #11

Closed
wants to merge 2 commits into from
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
18 changes: 13 additions & 5 deletions .github/workflows/buildMomentumSpec.yml
Original file line number Diff line number Diff line change
Expand Up @@ -11,13 +11,21 @@ jobs:
- uses: actions/checkout@v2
- name: Install dependencies
run: |
echo 'sudo apt install'
- name: build content #toy content
sudo apt update
sudo apt -yq install texlive texlive-latex-extra inkscape
- name: build content
run: |
date > thisdate
export apath=./specifications/momentum
export TEX=$apath/Momentum-netCDF
TEXINPUTS=$apath/: pdflatex -shell-escape -halt-on-error -interaction=nonstopmode -output-directory=$apath $TEX
TEXINPUTS=$apath/: BIBINPUTS=$apath/: bibtex $TEX
TEXINPUTS=$apath/: pdflatex -output-directory=$apath -shell-escape $TEX
TEXINPUTS=$apath/: pdflatex -output-directory=$apath -shell-escape $TEX
cp $TEX.pdf $apath/publish
ls -l $apath/publish
# Upload artifact containing Momentum-netCDF.pdf
- name: Upload cf-conventions doc preview
uses: actions/upload-artifact@v2
with:
name: thisdate
path: ./
name: Momentum-netCDF
path: ./specifications/momentum/publish/Momentum-netCDF.pdf
7 changes: 7 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
*~
*.aux
*.bbl
*.bcf
*.blg
*.log
*.out
40 changes: 40 additions & 0 deletions specifications/momentum/Momentum-netCDF.tex
Original file line number Diff line number Diff line change
@@ -0,0 +1,40 @@
\documentclass{article}
\usepackage[a4paper,includeheadfoot,margin=2.54cm]{geometry}
\usepackage{hyperref}
\usepackage{graphicx}
\usepackage{amsmath}
\usepackage{newclude}
\usepackage{lscape}
\usepackage{enumitem}

\numberwithin{enumi}{section}

% \titlehead{\includegraphics[width=6cm]{images/momentumLogo.png}}
\title{Momentum File Formats}

\author{Owner: Sam Adams\\Editors: mark Hedley\\version 0.1-draft}

\begin{document}

\maketitle

\include*{ch01-Intro}

\include*{ch02-Components}

\include*{ch03-Diagnostics}

\include*{ch04-DumpFiles}

\include*{ch05-CheckpointRestart}

\include*{ch06-FramesLBC}

\include*{ch07-AncillaryFiles}


\bibliographystyle{ieeetr}
\bibliography{references}


\end{document}
23 changes: 23 additions & 0 deletions specifications/momentum/ch01-Intro.tex
Original file line number Diff line number Diff line change
@@ -0,0 +1,23 @@
\section{Introduction}

This document describes the format of the data files used within the Momentum forecast/climate model system, the scientific desktop analysis environment, and the MASS archive. The note describes the main format of file in use: the UGRID-NetCDF and variations thereof that are used for specific purposes.

\subsection{Layout of this document}

This document provides a description of the data formats used within Momentum. Chapter 2 gives a detailed
list of the metadata and data fields that are in use, chapters 3-7 describe in more detail each variant file format
and where it is used

\subsection{The UGRID-NetCDF file format}


From its initiation in 2017, the Next Generation Modelling Systems (NGMS) Programme has introduced fundamental changes in the Met Office weather and climate systems and their underpinning infrastructure, resulting in the delivery of Momentum, the Unified Earth Environment Prediction Framework.
// Add official link when we have one
A full description of NGMS is out of scope for this document, but more information can be found in the Programme SharePoint site \cite{moex}. The underlying grid and data model changed significantly and consequently this has resulted in changes to file formats. The adoption of a semi-structured cubed-sphere grid and a change to mixed-finite element formulation for the dynamical core motivated the change to an unstructured file format. The specific choice of file format was influenced by several things: the need to support an unstructured data layout that uses indirect addressing, the desire to avoid bespoke file formats, and the capabilities of existing IO frameworks to write and read an appropriate format. The NetCDF file format \cite{nc} is well-known in the weather and climate community and to the Met Office as are the CF conventions for metadata to describe weather and climate data \cite{cf}. The UGRID conventions \cite{ugrid} are an extension to CF and allow for unstructured representations. Within UGRID there are several different options for unstructured representations including 1D, 2D, Layered 3D and fully 3D. Currently the Met Office uses 1D and layered 3D variants of UGRID-NetCDF and these will be described more fully in the sections that follow.



\subsection{Additions and Changes to the file formats}

Occasionally model developments or new data products are made that cannot be supported by the current file formats. When considering how to accommodate any new data products into the file format please consult widely. Any changes you make will have potential downstream impact in other applications including MASS archiving, data analysis and visualisation, post-processing, verification and the operational and product suites.
% I think it is worthwhile keeping this section, but it needs a point of contact for those who are planning changes. Presumably would be the owner of this document or the owner of the file formats?
251 changes: 251 additions & 0 deletions specifications/momentum/ch02-Components.tex
Original file line number Diff line number Diff line change
@@ -0,0 +1,251 @@
\section{List of data set components}

% Generic list of metadata and data fields we use for ll file formats. Don't repeat full UGRID conventions, just state what we use and our conventions, and any extensions we use

% Note this is taken from the current output from the NWP suite. This applies some post-processing of metadata and data to the raw model output

This section lists all variables and attributes that are used in the Momentum UGRID-NetCDF file formats with a brief explanation of what they are used for. Refer to the CF and UGRID conventions for more information.

\subsection{Global Attributes}

These are metadata that the model writes as standard and generally not changed.

% Do we need other things in here - e.g. that the output is from the Met Office Momentum model?

\begin{verbatim}
description
interval_write
title
Conventions
\end{verbatim}

\subsection{Dimensions}

Each NetCDF dataset is characterised by a set of dimensions which contains the full range of sizes and
shapes necessary to describe the data of every variable in the dataset. For each 2D mesh present in the dataset there are three dimensions which correspond to the number of nodes, edges and faces on the mesh.

.Standard Topology dimensions
\begin{verbatim}
nMesh_id_node
nMesh_id_edge
nMesh_id_face
Mesh_id_face_N_nodes
Mesh_id_edge_N_nodes
Mesh_id_face_N_edges
Mesh_id_face_N_faces
Mesh_id_edge_N_faces
\end{verbatim}

Standard Time Dimensions

Time is represented as an umlimited NetCDF dimension, meaning that new data can be appended easily to that dimension for time-series capability.

% check unlimited, is this necessary for HDF5 encoded files?

\begin{verbatim}
time_instant
time_centered
\end{verbatim}

% Does this imply time averages will be labelled with the central time? This is I think a departure from um/pp approach. Perhaps a comment would be useful.

Other Miscellaneous Dimensions

There are application-specific dimensions of several kinds. Vertical dimensions (e.g. model levels, pressure levels,
depth); Non-spatial dimensions (e.g. tiles or pseudo-levels)

Finally, there are some additional dimensions: bnds which contains the number of bounds for a 1D quantity (i.e. 2) and dim1 for the number of bounds for a face (i.e. 4).

% These are named horribly - maybe they could be changed to something more informative?

\begin{verbatim}
boundary_layer_type
bnds
dim1
depth
pressure
\end{verbatim}

\subsection{Mesh and Topology Variables and Attributes}

These use standard topology dimensions as defined in section 2.2.

\begin{verbatim}
int Mesh_id
Mesh_id:cf_role
Mesh_id:topology_dimension
Mesh_id:long_name
Mesh_id:node_coordinates
Mesh_id:edge_coordinates
Mesh_id:face_coordinates
Mesh_id:face_node_connectivity
Mesh_id:edge_node_connectivity
Mesh_id:face_edge_connectivity
Mesh_id:face_face_connectivity
Mesh_id:edge_face_connectivity
\end{verbatim}

\begin{verbatim}
float Mesh_id_node_x(nMesh_id_node)
Mesh_id_node_x:units
Mesh_id_node_x:standard_name
Mesh_id_node_x:long_name
float Mesh_id_node_y(nMesh_id_node)
Mesh_id_node_y:units
Mesh_id_node_y:standard_name
Mesh_id_node_y:long_name
float Mesh_id_edge_x(nMesh_id_edge)
Mesh_id_edge_x:units
Mesh_id_edge_x:standard_name
Mesh_id_edge_x:long_name
float Mesh_id_edge_y(nMesh_id_edge)
Mesh_id_edge_y:units
Mesh_id_edge_y:standard_name
Mesh_id_edge_y:long_name
float Mesh_id_face_x(nMesh_id_face)
Mesh_id_face_x:units
Mesh_id_face_x:standard_name
Mesh_id_face_x:long_name
float Mesh_id_face_y(nMesh_id_face)
Mesh_id_face_y:units
Mesh_id_face_y:standard_name
Mesh_id_face_y:long_name
int Mesh_id_face_nodes(nMesh_id_face, Mesh_id_face_N_nodes)
Mesh_id_face_nodes:long_name
Mesh_id_face_nodes:cf_role
Mesh_id_face_nodes:start_index
int Mesh_id_edge_nodes(nMesh_id_edge, Mesh_id_edge_N_nodes)
Mesh_id_edge_nodes:long_name
Mesh_id_edge_nodes:cf_role
Mesh_id_edge_nodes:start_index
int Mesh_id_face_edges(nMesh_id_face, Mesh_id_face_N_edges)
Mesh_id_face_edges:long_name
Mesh_id_face_edges:cf_role
Mesh_id_face_edges:start_index
int Mesh_id_face_links(nMesh_id_face, Mesh_id_face_N_faces)
Mesh_id_face_links:long_name
Mesh_id_face_links:flag_meanings
Mesh_id_face_links:flag_values
Mesh_id_face_links:cf_role
Mesh_id_face_links:start_index
int Mesh_id_edge_face_links(nMesh_id_edge, Mesh_id_edge_N_faces)
Mesh_id_edge_face_links:long_name
Mesh_id_edge_face_links:comment
Mesh_id_edge_face_links:cf_role
Mesh_id_edge_face_links:start_index
\end{verbatim}

\subsection{Time Variables and Attributes}

% Validity time required?

These use standard time dimensions as defined in section 2.2.

\begin{verbatim}
double time_instant(time_instant)
time_instant:axis
time_instant:units
time_instant:standard_name
time_instant:long_name
time_instant:calendar
time_instant:time_origin

int64 forecast_period(time_instant)
forecast_period:units
forecast_period:standard_name

int64 forecast_reference_time
forecast_reference_time:units
forecast_reference_time:standard_name
forecast_reference_time:calendar

double time_centered(time_centered) ;
time_centered:axis
time_centered:bounds
time_centered:units
time_centered:standard_name
time_centered:long_name
time_centered:calendar
time_centered:time_origin

double time_centered_bnds(time_centered, bnds)

int64 forecast_period_0(time_centred)
forecast_period_0:bounds
forecast_period_0:units
forecast_period_0:standard_name

int64 forecast_period_0_bnds(time_centered, bnds)
\end{verbatim}
% I do not know what this is. If it is standard it should be named more informatively

\subsection{Data Variables and Attributes}

These use standard topology and time dimensions as well as miscellaneous dimensions as
defined in section 2.2.

\begin{verbatim}
double data_var_surface([time dim], [mesh dim]) ;
data_var_surface:long_name
data_var_surface:units
data_var_surface:interval_operation
data_var_surface:online_operation
data_var_surface:cell_methods
data_var_surface:mesh
data_var_surface:location
data_var_surface:coordinates


type data_var_layer([time dim], [layer dim], [mesh dim]) ;
data_var_layer:long_name
data_var_layer:units
data_var_layer:interval_operation
data_var_layer:online_operation
data_var_layer:cell_methods
data_var_layer:mesh
data_var_layer:location
data_var_layer:coordinates

type data_var_vector([dim]):
data_var_vector:axis
data_var_vector:bounds
data_var_vector:units
data_var_vector:long_name
data_var_vector:positive

type data_var_vector_bnds([dim], [dim])


type data_var_scalar ;
scalar_var:units
scalar_var:long_name
\end{verbatim}


\subsection{Constants}

% I don't think this is needed as much of it covered my metadata dimensions. Might be needed for any data constants we use?

\subsubsection{Integer}
\subsubsection{Real}
\subsubsection{Level Dependent}
\subsubsection{Row Dependent}
\subsubsection{Column Dependent}


\subsection{ADDITIONAL PARAMETERS}

% Is this needed?

This section was formerly known as field-dependent constants.
The data is stored as a 1D array. There are two dimension values, only the first of these is used. This section is used to store stochastic physics coefficient arrays.

% enumerate list including section numbering, e.g. 2.7.1

\begin{enumerate}
\item coeffc: spherical harmonic cosine coefficients for SPT scheme. size=stph n2 * (stph n2+1)
\item coeffs: spherical harmonic sine coefficients for SPT scheme. size=stph n2 * (stph n2+1)
\item dpsidtc: 2D version of d(psi)/d(t) COS coeffs in Fourier for SKEB scheme. size=(stph n2-stph n1) * (stph - n2+1)
\item dpsidts: 2D version of d(psi)/d(t) SIN coeffs in Fourier. size=(stph n2-stph n1) * (stph n2+1)
\item rp coef: Array of parameters for RP scheme. size=rp max (currently 25)
\end{enumerate}
28 changes: 28 additions & 0 deletions specifications/momentum/ch03-Diagnostics.tex
Original file line number Diff line number Diff line change
@@ -0,0 +1,28 @@
\section{Diagnostic Output Files}

% For each file format we use describe the specifics of what it is used for and anything different from conventions

% Some links from LFRic wiki that might be useful to populate this chapter.

% https://code.metoffice.gov.uk/trac/lfric/wiki/LFRic/IO/FileFormat

% https://code.metoffice.gov.uk/trac/lfric/wiki/LFRic/IO/FileFormatRequirements

% https://code.metoffice.gov.uk/trac/lfric/wiki/LFRic/DatesAndTimes

% https://code.metoffice.gov.uk/trac/lfric/wiki/LFRic/Diagnostics/LFRic/Diagnostics/MetaDataDesign
% https://code.metoffice.gov.uk/trac/lfric/wiki/LFRic/Diagnostics/DiagnosticSystem


The diagnostic output uses the 3D layered UGRID representation \cite{ug3d} where fields are represented as a series
of 2D unstructured "slices" of data, with each slice representing a vertical layer or pseudo-level. As this is not a full 3D representation, data that has a vertical dimension within a cell is collapsed to a lower-dimensional representation (i.e. the 3D cell volume has been collapsed into a 2D face). Scalar quantities
are held on 2D faces (e.g. temperature, pressure) and vector quantities that would naturally be held on
vertical faces are held on the edges of horizontal faces (e.g. u and v winds).

% Add a diagram?

% Note - that in the NWP suite output, winds are on faces
% I've seen output with winds also on edges, but the whole vector, not just one cpt as in UM (to allow westerly-southerly representation). Useful if want to get closer to LFRic internal representation but hard to use.

% Add a sample txt output here and /or a link to standard sample data?

1 change: 1 addition & 0 deletions specifications/momentum/ch04-DumpFiles.tex
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
\section{Dump Files}
1 change: 1 addition & 0 deletions specifications/momentum/ch05-CheckpointRestart.tex
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
\section{Checkpoint / Restart Files}
2 changes: 2 additions & 0 deletions specifications/momentum/ch06-FramesLBC.tex
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
\section{Frames and Lateral Boundary Condition Files}

2 changes: 2 additions & 0 deletions specifications/momentum/ch07-AncillaryFiles.tex
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
\section{Ancillary Files}

Binary file added specifications/momentum/images/momentumLogo.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
1 change: 1 addition & 0 deletions specifications/momentum/publish/readme.md
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
directory for published built content
Loading
Loading