diff --git a/docs/conversion/ccpp-conversion-guide.md b/docs/conversion/ccpp-conversion-guide.md deleted file mode 100644 index 5307b0a..0000000 --- a/docs/conversion/ccpp-conversion-guide.md +++ /dev/null @@ -1,82 +0,0 @@ -# CAM Parameterization to CCPP - -## Background - -### Running jobs in CAM and CAM-SIMA primer (experienced users can jump to the next section) - -To make runs in either CAM or CAM-SIMA the commands are identical: -```bash -./create_newcase… #(the commands will differ and are specified later in this document) -cd YourCaseDirectory -./xmlchange… #(specific commands are specified later in this document) -./case.setup -./case.build -./case.submit -``` - -Depending on which machine you are on, you may prefer to run the ./case.build command on a compute node instead of the login node due to user resource utilization limits on the login nodes. - -For more detailed information on case creation and building, see https://ncar.github.io/CAM/doc/build/html/users_guide/building-and-running-cam.html - -## Prep Work - -### Conversion Spreadsheet -Put the parameterization that you are going to convert into the conversion spreadsheet -https://docs.google.com/spreadsheets/d/1_1TTpnejam5jfrDqAORCCZtfkNhMRcu7cul37YTr_WM/edit#gid=0 - -### Create Github Issues -Create a Github Issue in the [ESCOMP/CAM](https://github.com/ESCOMP/CAM) repo that states which physics parameterization you are planning to convert to the CCPP framework. Then create another issue in the [ESCOMP atmospheric physics](https://github.com/NCAR/atmospheric_physics) repo describing the same physics parameterization that you are now planning to add to the collection of NCAR CCPP physics suites. Doing this allows the software engineers to keep track of which physics routines are being worked on, and which still need to be assigned. The goal of converting the physics parameterization is to ultimately have the CCPP-ized physics package reside in [ESCOMP atmospheric physics](https://github.com/NCAR/atmospheric_physics) and be removed from [ESCOMP/CAM](https://github.com/ESCOMP/CAM). - -### Setting up your sandbox - -Make sure you have github forks for both ESCOMP/CAM-SIMA and ESCOMP/atmospheric_physics. If needed see https://github.com/ESCOMP/Cam/wiki/CAM-Development-Workflow-in-GitHub#how-to-makestore-revisions-to-your-personal-cam-repository-github-fork - - -To begin, fork ESCOMP/CAM-SIMA: -![text](fork-cam-sima.png "Forking CAM-SIMA") - -And select the `Create new fork` option. This will bring you to the "Create new fork" screen: -![text](fork-cam-sima-2.png "Forking CAM-SIMA") - -!!! warning "Uncheck the "Copy the `main` branch only" option" - - Failure to uncheck this will prevent you from pulling in updates from the `development` branch easily. - -As you make changes and want to commit them to your github repos, you will be managing two separate repos. When you issue git commands, be aware of where you are in your code tree. If you want to see changes in CAM-SIMA, you can issue a `git status` in the main CAM-SIMA directory. If you want to see changes in the atmospheric_physics repo, make sure you are in `src/physics/ncar_ccpp` before you issue the `git status` command. All other git commands will be relative to your current working directory as well. - -### Opttional: pre-split tthe module - -Many CAM schemes have more than one `run` function contained within them. To seperate them into seperate files and test them: - - In the copied atmospheric physics directory, create a seperate submodule for each parameterization which has `run` method. - - An easy way to see what routines need to be seperated out is to look at the `use` statement(s) for your parameterization. If more than one routine is listed, you most likely will need to seperate these out. -- If there is shared module level data or shared subroutines which are called internally, put these all in a `_common.F90` module. - -## CCPP-ization - -### Convert the code in the "portable" parameterization layer - -SCHEMES = the base level routines which are currently called bby the CAM interface routines. - -1. Convert the original routines in the file you copied over to `src/physics/ncar_ccpp//.F90` to one or more of the 5 following subroutines. These will be called by CCPP in CAM-SIMA and in CAM6. - -!!! note "`` should be the full name of your module" - - For example, if you are converting the `tj2016` `precip_tend` function, then `` would be `tj2016_precip_tend`. - -Parameterizations may not need all of the routines listed below and do not need to supply them if they are not needed. However, all subroutine input/output arguments need to have an `intent` label (you may refer to `src/atmos_phys/kessler` and `src/atmos_phys/held_suarez` for specific examples). - -- `_init` - - Add all code that is run only during the first time step (`nstep=0`). Typically fold `register` and `init` routines in current CAM into this routine. - -- `_timestep_init` - - Add all pre-processing code needed by the scheme at the start of each timestep. This may contain code from the CAM interface routine which prepares data for the run routine at each timestep. - -- `_run` - - This is the workhorse routine which is run at each timestep. The bulk of your ported code will likely be here. - -- `_timestep_final` - - Add all post-processing code needed by the scheme at the end of each timestep. This may contain code from the CAM interface routine which manipulates data after the run routine at each timestep. - -- `_final` - - Most current CAM routines do not have code in this category. This code is run once at the very end of the job. - diff --git a/docs/conversion/conversion-background.md b/docs/conversion/conversion-background.md new file mode 100644 index 0000000..1e9ee06 --- /dev/null +++ b/docs/conversion/conversion-background.md @@ -0,0 +1,104 @@ +# 0 - Background & Prep work + +## Background + +### Running jobs in CAM and CAM-SIMA +See [this](../usage/creating-a-case.md) section for how to run CAM-SIMA and CAM. + +Depending on which machine you are on, you may prefer to run the ./case.build command on a compute node instead of the login node due to user resource utilization limits on the login nodes. + +## Prep Work + +### Conversion Spreadsheet +Put the parameterization that you are going to convert into the [conversion spreadsheet](https://docs.google.com/spreadsheets/d/1_1TTpnejam5jfrDqAORCCZtfkNhMRcu7cul37YTr_WM/edit#gid=0). + +### Create Github Issues +1. Create a Github Issue in the [ESCOMP/CAM](https://github.com/ESCOMP/CAM) repo that states which physics parameterization you are planning to convert to the CCPP framework. +1. Create another issue in the [ESCOMP/atmospheric physics](https://github.com/NCAR/atmospheric_physics) repo describing the same physics parameterization that you are now planning to add to the collection of NCAR CCPP physics suites. Doing this allows the software engineers to keep track of which physics routines are being worked on, and which still need to be assigned. The goal of converting the physics parameterization is to ultimately have the CCPP-ized physics package reside in [ESCOMP atmospheric physics](https://github.com/NCAR/atmospheric_physics) and be removed from [ESCOMP/CAM](https://github.com/ESCOMP/CAM). + +### Setting up your sandbox + +Make sure you have github forks for both [ESCOMP/CAM-SIMA](https://github.com/ESCOMP/CAM-SIMA) and [ESCOMP/atmospheric_physics](https://github.com/ESCOMP/atmospheric_physics). If needed see [Working with git and GitHub](../development/git-basics.md) + + +To begin, fork ESCOMP/CAM-SIMA: +![text](figures/fork-cam-sima.png "Forking CAM-SIMA") + +And select the `Create new fork` option. This will bring you to the "Create new fork" screen: +![text](figures/fork-cam-sima-2.png "Forking CAM-SIMA") + +!!! warning "Uncheck the "Copy the `main` branch only" option" + + Failure to uncheck this will prevent you from pulling in updates from the `development` branch easily. + +#### Set up local clones and branches + +- On the machine and in the directory in which you would like to develop, clone CAM-SIMA (the CCPP-ized version of CAM) with your fork as the origin: +``` +git clone -o https://github.com//CAM-SIMA CAM-SIMA +``` + +- Navigate into the directory you just created: +``` +cd CAM-SIMA +``` + +- Add the `ESCOMP` remote so you can keep your fork up to date: +``` +git remote add ESCOMP https://github.com/ESCOMP/CAM-SIMA +``` + +- Fetch the upstream ESCOMP tags and branches from GitHub: +``` +git fetch --tags ESCOMP +``` + +- Create and checkout a new branch off of the `development` branch of the ESCOMP remote: +``` +git branch ESCOMP/development +git checkout +``` + +- Set up the upstream (on GitHub) version of your branch: +``` +git push -u +``` + +- Populate your externals: +``` +bin/git-fleximod update +``` + +- Make a new directory for your parameterization within the atmospheric_physics submodule directory +``` +mkdir src/physics/ncar_ccpp/ +``` + +- Navigate into the atmospheric_physics submodule directory +``` +cd src/physics/ncar_ccpp +``` + +- Add your fork of atmospheric_physics as a remote +``` +git remote add https://github.com//atmospheric_physics +``` + +- Fetch the upstream branches and tags +``` +git fetch +``` + +- Create, checkout, and push a new branch upstream +``` +git checkout -b +git push -u +``` +!!! Warning "Multiple repositories!" + As you make changes and want to commit them to your github repos, you’ll be managing two separate repos. When you issue git commands, be aware of where you are in your code tree. + +- If you want to see changes in CAM-SIMA, you can issue a “git status” in the main CAM-SIMA directory. +- If you want to see changes in the atmospheric_physics repo, make sure you are in src/physics/ncar_ccpp before you issue the “git status” command. +- All other git commands will be relative to your current working directory as well. + +Congratulations! You've set up your sandbox! Proceed to [1 - Convert the "portable" layer](./step1.md) \ No newline at end of file diff --git a/docs/conversion/figures/atmos-phys-issue.PNG b/docs/conversion/figures/atmos-phys-issue.PNG new file mode 100644 index 0000000..1f73e8b Binary files /dev/null and b/docs/conversion/figures/atmos-phys-issue.PNG differ diff --git a/docs/conversion/figures/cam-issue.PNG b/docs/conversion/figures/cam-issue.PNG new file mode 100644 index 0000000..5b39a84 Binary files /dev/null and b/docs/conversion/figures/cam-issue.PNG differ diff --git a/docs/conversion/fork-cam-sima-2.png b/docs/conversion/figures/fork-cam-sima-2.png similarity index 100% rename from docs/conversion/fork-cam-sima-2.png rename to docs/conversion/figures/fork-cam-sima-2.png diff --git a/docs/conversion/fork-cam-sima.png b/docs/conversion/figures/fork-cam-sima.png similarity index 100% rename from docs/conversion/fork-cam-sima.png rename to docs/conversion/figures/fork-cam-sima.png diff --git a/docs/conversion/figures/spreadsheet-change-highlighted.PNG b/docs/conversion/figures/spreadsheet-change-highlighted.PNG new file mode 100644 index 0000000..ee11819 Binary files /dev/null and b/docs/conversion/figures/spreadsheet-change-highlighted.PNG differ diff --git a/docs/conversion/figures/spreadsheet-change.PNG b/docs/conversion/figures/spreadsheet-change.PNG new file mode 100644 index 0000000..4d50e8d Binary files /dev/null and b/docs/conversion/figures/spreadsheet-change.PNG differ diff --git a/docs/conversion/step1.md b/docs/conversion/step1.md new file mode 100644 index 0000000..ca786f8 --- /dev/null +++ b/docs/conversion/step1.md @@ -0,0 +1,170 @@ +# 1 - Convert the "portable" layer + +## Move the portable layer to atmospheric_physics + +The *portable* parameterization layer is the module that contains the core physics code that is called by the CAM interface. + +The portable module will most likely live in `$CAM/src/physics/cam` or `$CAM/src/physics/cam7` + +The driver code in each physics directory is `physpkg.F90`, which typically calls the CAM interface to the portable layer (often called `_cam.F90`), but also sometimes calls portable layer directly. +Once you locate the core code that you will be converting, copy it into the new directory you created in the atmospheric_physics submodule directory: +``` +cp $CAM/src/physics//.F90 $CAM-SIMA/src/physics/ncar_ccpp/ +``` + +## Optional: pre-split the module +Many CAM schemes have more than one "run" or "tend" method contained within them. To split them into separate files and test them, do the following: + +- In the atmospheric physics directory (`ncar_ccpp`), create a separate module for each piece which has a "run" or "tend" method. + - An easy way to see what routines need to be separated out, is to look at the "use" statement(s) in `physpkg.F90` for your parametrization's module. If more than one routine is listed, you most likely will need to separate these out. +- If there is shared, module-level data or shared subroutines which are called internally, put these all in a _common.F90 module. + +## 1a - Change routine names +Convert the original routines (except readnl - we'll get to that [later](step4.md)!) in the file(s) you copied over to $CAM-SIMA/src/physics/ncar_ccpp to one or more of the following 5 subroutines: +!!! note "`` should be the full name of your module" + + For example, if you are converting the `tj2016` `precip_tend` function, then `` would be `tj2016_precip_tend`. + +- **_init** + - Add all code that is run only during the first timestep (nstep=0). Typically, fold register and init routines in current CAM into this routine. +- **_timestep_init** + - Add all pre-processing code needed by the scheme at the start of each timestep. This may contain code from the CAM interface (`physpkg.F90`) routine which prepares data for the run routine at each timestep +- **_run** + - This is the workhorse routine which is run at each timestep. The bulk of your ported code will likely be here. + - Often, the workhorse routine in CAM is called `_tend` +- **_timestep_final** + - Add all post-processing code needed by the scheme at the end of each timestep. This may contain code from the CAM interface routine (in `physpkg.F90`) which manipulates data after the run routine at each timestep +- **_final** + - Most current CAM routines do not have code in this category. This code is run once at the very end of the CAM model execution. + +!!! note + + You may not need all of the routines listed, nor do you need to supply them if they are not needed; however, all subroutine input/output (dummy) arguments need to have an "intent" label. + + +## 1b - Add required `\htmlinclude` lines + +Add two lines before each of the up to 5 phases of subroutines (skip `readnl`) +``` +!> \section arg_table__ Argument Table +!! \htmlinclude _.html +``` + +## 1c - Clean up dummy argument dimensions +Make sure no input/output variables have named dimensions in their declaration inside the routines. + +1. Replace named dimensions with “:” + - for example: `real(r8), intent(in) :: zm(pcols, pver)` will become `real(r8), intent(in) :: zm(:,:)` +1. If a named dimension is no longer used, it may be removed from the calling list (i.e. pcols, etc) +1. On the CAM interface side, any variables dimensioned by `pcols` in CAM will need to be subsetted to `1:ncol` in the call to the CCPP-ized routine so that only the active columns are passed +1. All `intent(out)` variables which are dimensioned pcols` should be initialized to zero right before being called in the main CAM physics calling routine to prevent extraneous values from existing in a pcols-dimensioned array. + - Put `!REMOVECAM/!REMOVECAM_END` labels around these initializations as any which remain after CAM is retired no longer need this precautionary step. +1. Repeat #1-4 with any routines which are called internally +1. Search through the code and make sure that all locations which call these updated routines have been modified correctly + - Use the subsetted 1:ncol arrays when calling the updated routine + - Initialize all intent(out) arrays to zero before making the call + +## 1d - Use kind_phys instead of r8 +1. Remove `use shr_kind_mod, only: r8=> shr_kind_r8` +1. Replace with `use ccpp_kinds, only: kind_phys` +1. Do a find and replace for `r8` with `kind_phys` in the module (and any dependencies) + +## 1e - Remove use statements +Remove all “use” statements and have the data appear in the calling list (the input and output variables to the routine). The only exception to this rule is for use statements to routines contained within the parameterization package or dependencies. + +- If a use statement is bringing in a variable from another module (e.g. something from `physconst`), add the variable to the calling list of the subroutine(s) that use it (and remove the use statement) +- If a use statement is bringing in an external routine (not part of the parameterization package), you have a few options (consult with the other CAM SEs on how to proceed): + 1. If the routine already exists in the core CAM-SIMA code tree (not in `ncar_ccpp`), ask another CAM SE about how and where to call the routine within the CAM-SIMA run loop and set a variable to be passed into the physics (variable will need to be added to the registry) + 1. CCPP-ize the external routine and/or module and add it to the suite definition file before or after the parameterization + 1. Postpone CCPP-zing the external routine and add it as a dependency (we'll revisit this in [create metadata](step3.md)) +- Comment out `outfld`, `addfld` use statements for now + - Also comment out the `addfld` and `outfld` calls within the module(s) + +## 1f - Add error variables +Add `errmsg, errflg` to the end of your calling list. Set `errflg` to non-zero if an error is encountered in your routine and set `errmsg` with an appropriate text indicating the error. + +The declarations for these variables are: +``` + character(len=512), intent(out) :: errmsg + integer, intent(out) :: errflg +``` + +At the top of your routine initialize the variables as follows: +``` + errmsg = ‘ ‘ + errflg = 0 +``` + +## 1g - Replace state and ptend variables in calling list +`state` and `ptend` variables will not be passed in as full objects, but instead as the individual state and ptend fields that are used in the routine. + +If `state` appears in the calling list: + +1. Search the routine for "state%" and, for each of the distinct variables found (`state%`), add the variable (``) to the calling list +1. Replace all calls to `state%` with `` +1. Remove `state` from the calling list + +If `ptend` appers in the calling list: + +1. Search the routine for "ptend%" and, for each of the distinct variables found (`ptend%`), add the variable to the calling list + - the variable name is up to you, some suggestions: + 1. `_tend` + 1. `ddt` +1. Replace all calls to `state%` with the new name +1. Remove `ptend` from the calling list + +## 1h - Remove pbuf variables +pbuf variables will not be in CCPP code. Instead, pass required individual variables as `intent(in)`, `intent(out)` or `intent(inout)` depending on whether they are being used or set in the interstitial. + +For example, if the following line of code exists in a routine: + +``` +call pbuf_get_field(pbuf, taubljx_idx, taubljx) +``` + +You will want to search around in the routine (and routines that it calls) to determine if `taubljx` is read from, written to, or both: + +1. If the variable is read from only (e.g. is only on the right-hand side of equations), the variable will be `intent(in)` +1. If the variable is written to only (e.g. is only on the left-hand side of equations), the variable will be `intent(out)` +1. If the variable is both read from and written to (e.g. it appears on both sides of equations), the variable will be `intent(inout)` + +In this example, `taubljx` is `intent(out)`, so we'd remove the `pbuf_get_field` call and add `taubljx` to the calling list as an ouput variable. + +## 1i - Mark variables as initialized +- If you add new variables that do not need to be read in from a file but come directly from the host model/registry and are uninitialized by CAM-SIMA, use the `mark_as_initialized` subroutine in the `_init` phase. +- Also, if you are setting a (non-namelist) public module level variable(a variable above the contains statement) in CAM-SIMA that will be used by a physics scheme, then also set the mark_as_initialized for that variable (also in the `_init` phase) + +``` +use phys_vars_init_check, only: mark_as_initialized +``` +``` +call mark_as_initialized('') +``` + +## 1j - Initial standard name check +Do a preliminary look at the variables on the calling lists and make sure that CCPP standard names and units exist for all of them, by checking for them in CAM Standard Names Spreadsheet. + +- If variables are not filled out in the list, highlight the entire line with yellow. +- If they don't exist at all in the spreadsheet, enter them into the bottom of the sheet and highlight them with yellow. Note that physconst and diagnostic-only variables *may* not reside in the spreadsheet. +- Let Jesse, Courtney or Cheryl know if you've encountered missing CCPP standardnames and/or units and added them to the list, so they can be discussed at an upcoming meeting. + +!!! note + This process can normally take 2-4 weeks, so a preliminary look is advisable. + +## 1k - Update CAM interface call(s) +Update the calls in the CAM interface code to reflect any changes to the routine names and calling lists. + +## 1l - OPTIONAL: Make a CAM tag +Make NCAR/atmospheric_physics and ESCOMP/CAM tags if substantial changes have been made up to this point. This is also a good step to take if there is potential for others to be making modifications to the same physics module. This will allow them to make changes which can be merged via git commands. + +- Do at the very least a sanity compilation and run using the ESCOMP/CAM code base + - Delete the original module from src/physics/cam + - Use your pulled apart modules in atmos_phys/. + - In bld/configure in the ESCOMP/CAM source code, find the section "Add the CCPP'ized subdirectories". Add following line: +``` +print $fh "$camsrcdir/src/atmos_phys/\n"; +``` +- Modify code as needed until this modified code compiles and runs properly. +- If you want to benchmark your development to this point, you will need to open PRs to both NCAR/atmospheric_physics and ESCOMP/CAM. + +Proceed to [2 - Create snapshots of CAM](step2.md) \ No newline at end of file diff --git a/docs/conversion/step2.md b/docs/conversion/step2.md new file mode 100644 index 0000000..7dbdf6b --- /dev/null +++ b/docs/conversion/step2.md @@ -0,0 +1,123 @@ +# 2 - Create snapshots of CAM + +## Configure and set-up CAM snapshot +Make a normal CAM run using cam_snapshot to capture before and after files using a compset which exercises the parameterization being converted. + +- The cam_snapshot feature dumps the entire state/tend/pbuf/cnst arrays to netCDF files. + - The variables `cam_snapshot_before_num` and `cam_snapshot_after_num` are the output file numbers (offset by 1 as they are in CAM). + - The `cam_take_snapshot_before` and `cam_take_snapshot_after` are usually set to the same parameterization to capture the variables before and after the named parameterization. + - To find the snapshot name, look in your source code checkout in `bld/namelist_files/namelist_definition.xml`. + - Inside this file, find the “cam_take_snapshot_before” definition. In this definition, it lists all possible parameterizations which can have a snapshot file created around. + - If your parameterization is there, that will be your `cam_take_snapshot_before` and `_after` values in your `user_nl_cam` + - If it's not there, you'll add `user_set` snapshot calls to `physpkg.F90` in the following way: + +For `tphysbc` in `physics/cam7/physpkg.F90`: +``` +if (trim(cam_take_snapshot_before) == "user_set") then + call cam_snapshot_all_outfld_tphysbc(cam_snapshot_before_num, state, tend, cam_in, cam_out, pbuf, & + cmfmc, cmfcme, zdu, rliq, rice, dlf, dlf2, rliq2, net_flx) +end if + + + +if ( (trim(cam_take_snapshot_after) == "user_set") .and. & + (trim(cam_take_snapshot_before) == trim(cam_take_snapshot_after))) then + call cam_snapshot_ptend_outfld(ptend, lchnk) +end if + + + +if (trim(cam_take_snapshot_after) == "user_set") then + call cam_snapshot_all_outfld_tphysbc(cam_snapshot_after_num, state, tend, cam_in, cam_out, pbuf, & + cmfmc, cmfcme, zdu, rliq, rice, dlf, dlf2, rliq2, net_flx) +end if +``` + +For `tphysac`: +``` +if (trim(cam_take_snapshot_before) == "user_set") then + call cam_snapshot_all_outfld_tphysac(cam_snapshot_before_num, state, tend, cam_in, cam_out, pbuf, & + fh2o, surfric, obklen, flx_heat, cmfmc, dlf, det_s, det_ice, net_flx) +end if + + + +if ( (trim(cam_take_snapshot_after) == "user_set") .and. & + (trim(cam_take_snapshot_before) == trim(cam_take_snapshot_after))) then + call cam_snapshot_ptend_outfld(ptend, lchnk) +end if + + + +if (trim(cam_take_snapshot_after) == "user_set") then + call cam_snapshot_all_outfld_tphysac(cam_snapshot_after_num, state, tend, cam_in, cam_out, pbuf, & + fh2o, surfric, obklen, flx_heat, cmfmc, dlf, det_s, det_ice, net_flx) +end if +``` +!!! Note + The above calls may vary if your physpkg.F90 exists in a different directory within `$CAM/src/physics` (not `cam7`) + + +- To enable cam_snapshot, add the following to your user_nl_cam file: +``` +cam_snapshot_before_num=6 +cam_snapshot_after_num=7 +cam_take_snapshot_before='_tend' ! or 'user_set' +cam_take_snapshot_after='_tend' ! or 'user_set' +nhtfrq = 0,0,0,0,0,1,1 +ndens = 2,2,2,2,2,1,1 +``` + +- The “before” file will be used for input for testing; “after” file will be used for comparisons +- Make sure to avoid using tape number one (h0) for `cam_snapshot_before_num` and `cam_snapshot_after_num` +- Check your namelist for any extra “fincl” settings added by use cases, as those will add variables that can conflict with the snapshot output. + - To check this, before you modify the `user_nl_cam` file, run `./case.setup`, run `./preview_namelists` and then look at the resulting `atm_in` file in the run directory. Look for `finclX` where X is your possible “before” and “after” numbers. Choose a number that is not used. + - You most likely will see fincl1 which is why it is not an option. +- Set nhtfrq to 1 so that data is written out on every time step. Note that the nhtfrq and ndens variables are arrays based off of the “tape” number specified in `cam_snapshot_before` and `cam_snapshot_after`. (i.e. 6 = the sixth number in these two arrays) +- Finally, as cam_snapshot writes out all of state, pbuf and other variables at every time step, it is suggested that a limited number of time steps be made in a cam_snapshot run. + - The run must be at least 2 time steps in order to work in subsequent steps. We recommend 3 to at most 9 time steps. (Note a 3 timestep run using a 2 degree FV grid can produce snapshot files with sizes over 4 Gb each). + - To change the number of timesteps to 3 use `./xmlchange STOP_OPTION=nsteps` and `./xmlchange STOP_N=3` + +!!! Warning + With short NUOPC runs, you may need to change ROF_NCPL and/or GLC_NCPL to the same size at ATM_NCPL to get them to run properly (`./xmlchange ROF_NCPL=\$ATM_NCPL` and `./xmlchange GLC_NCPL=\$ATM_NCPL`) + +## Create CAM snapshots +Run CAM normally with +``` +./case.setup +./case.build +./case.submit +``` +Upon a successful completion, you will see a `cam.h5i.xxx.nc` and `cam.h6i.xxx.nc` in your run directory (assuming you used before and after of 6 and 7) + +## Save snapshot files +The generated snapshot files should be saved on **derecho** here: +``` +/glade/campaign/cesm/community/amwg/sima_baselines/cam_sima_test_snapshots +``` + +with the naming convention `cam___snapshot_derecho__before_cYYYYMMDD.nc` and `cam___snapshot_derecho__after_cYYYYMMDD.nc` + +Be sure to run +``` +chmod u=rw,g=r,o=r +``` +or +``` +chmod 644 +``` +on every file you copy over so that it is readable by everyone + +On **izumi**, save your converted snapshot files in: +``` +/project/amp02/cam_snapshot_files/ +``` + +## CAM snapshot run tips + +1. Try to use the lowest resolution grid possible in order to keep the snapshot file size small (e.g. `ne3pg3_ne3pg3_mg37`). +1. We generally recommend running with debug flags on (`./xmlchange DEBUG=True`), as this way errors that can occur when you bring the CCPP-ized code back into CAM are easier to catch. + - However, if you decide not to, then do make sure that all of your other CAM runs also have debug flags off, as otherwise you can get differences in the results that are solely due to compiler settings, and not your code changes. +1. You should generate at least one set of snapshot files using GNU on derecho with DEBUG=TRUE, as these will be saved for use in the CAM-SIMA physics testbed to prevent unexpected answer changes. + +Once you have created your snapshot files, proceed to [3 - Create metadata](step3.md) \ No newline at end of file diff --git a/docs/conversion/step3.md b/docs/conversion/step3.md new file mode 100644 index 0000000..f8a3ca9 --- /dev/null +++ b/docs/conversion/step3.md @@ -0,0 +1,72 @@ +# 3 - Create metadata + +## Create template +To create a template metadata file based on the parameterization, run the following command in the directory with the routine you are converting (.F90): +``` +python $CAM-SIMA/ccpp_framework/scripts/ccpp_fortran_to_metadata.py .F90 +``` + +If you get errors: + +1. Confirm you included the [argtable lines](step1.md#1b-add-required-htmlinclude-lines) above each routine +1. If you see a message "Missing local variables" and the variable is there, it may be missing its intent attribute +1. Hopefully other errors will be easy to address; consult other CAM SEs if there's any remaining confusion/errors + +Completion of `ccpp_fortran_to_metadata.py` will result in the creation of `.meta` + +## Complete metadata + +Replace all `enter_*` sections with appropriate information (standard_name, units, dimensions, long_name) + +!!! Warning "Named dimension in metadata template" + If you have a dimension in the metadata field which is NOT of the form `enter_standard_name_X:enter_standard_name_Y` but is rather a single dimension without a :, this means that you have a named dimension in your converted routine and you should remove the name and replace it with ":" + +- *standard_name*: the mapping between CAM variables and standard_names can be found [here](https://docs.google.com/spreadsheets/d/1vpQ_xDZk00Z-_3SpW5N2EF3_FY6K7opNN4cqtSMlbwU/edit?gid=0#gid=0) + - You may have to trace the variable back through the CAM code to find the `Snapshot or Local name` to look for + - Officially accepted standard names can be found in this [repository](https://github.com/ESCOMP/CCPPStandardNames/blob/main/Metadata-standard-names.md), but the CAM Standard Names Spreadsheet should include all of the CAM variables. + - If a standard name is not found: + - CCPP names are trying to adhere to CF names, so you may find names on the CF web page: https://cfconventions.org/Data/cf-standard-names/76/build/cf-standard-name-table.html + - If you need to create your own standard name, the current proposal for creating standard names can be found here: https://github.com/ESCOMP/CCPPStandardNames/blob/main/StandardNamesRules.rst + - May need to consult with WRF / MPAS scientists and other SIMA groups to coordinate StandardName usage. + - Keep a list of StandardNames that you introduce and give them to a CAM SE for incorporation into an upcoming CCPPStandardNames PR + + +- *units*: the units can be found in the standard names spreadsheet as well (`Snapshot Units` column) +- *dimensions*: Refer to the table below + - If you have module level variables (variables above the "contains" line in a module file) which are declared with dimension `pver` or `pverp`, then you need to make this variable be allocatable and allocate the variable in the init routine. This is due to the fact that pver is not known at compilation time, but is rather a run-time option in CAM-SIMA + - Note for pcols inside parameterizations: As only active columns will be passed to the parameterizations, there is no need for a maximum size variable anymore + - In the template, you can convert `enter_standard_nameXX:enter_standard_nameYY` to just use the appropriate dimension_name or do `1:dimension_name` (both work) + +
+| CAM dimension | CCPP phase | standard_name | +|:--------------|------------|---------------| +| pver | all | vertical_layer_dimension | +| pverp (or pver+1) | all | vertical_interface_dimension | +| ncols or pcols| non-run phase | horizontal_dimension | +| ncols or pcols| run phase | horizonal_loop_extent OR horizontal_loop_begin:horizontal_loop_end| +| pcnst | all | number_of_ccpp_constituents | +
+ +- *optional arguments*: If there are optional arguments, consult with a CAM SE. Optional attributes are an unsupported configuration, but are being incorporated into the framework. That said, there may be a workaround to avoid the use of optional arguments in your scheme +- Note: Try to avoid module level allocations and/or assignments even in the init phases of the code. If you believe you need to do this, speak with a Jesse and/or Cheryl for guidance +- error variables: see below for error variable metadata +``` +[ errmsg ] + standard_name = ccpp_error_message + long_name = Error message for error handling in CCPP + units = none + type = character | kind = len=512 + dimensions = () + intent = out +[ errflg ] + standard_name = ccpp_error_flag + long_name = Error flag for error handling in CCPP + units = flag + type = integer + dimensions = () + intent = out +``` +- constituents: see [constituent usage](../design/constituents.md/#constituent-usage) + - you can tell a variable is a constituent if it is in the `state%q` array in CAM (may have to trace back in the code a bit) + +Once you have created your metadata, proceed to [4 - Create namelist XML file](step4.md) \ No newline at end of file diff --git a/docs/conversion/step4.md b/docs/conversion/step4.md new file mode 100644 index 0000000..9e01ccf --- /dev/null +++ b/docs/conversion/step4.md @@ -0,0 +1,59 @@ +# 4 - Create namelist XML file +If your scheme has a `readnl` scheme, create an .xml file with the same name as the scheme which uses the namelist (place it in the same location as the parameterization in `ncar_ccpp`. + +- The filename should be `_namelist.xml` +- If you have metadata entries for these namelist entries from your previous conversion steps, it may be useful to copy the metadata entry and edit it to be xml. If you do this, you will delete the dimensions and intent lines and add category, group, desc and values/value. + +The CCPP Framework will autogenerate the namelist reader based on the elements in the namelist XML file associated with the parameterization. Each namelist element (``) will have the following fields: `` ``, ``, ``, ``, ``, `` and ``. Note that all variables in the namelist should appear in the parameterization’s `_init` subroutine interface. The description of each field is as follows: + +- *id*: The namelist variable's name. The name must be lower case. The module converts all namelist variable names to lower case since Fortran is case insensitive. +- *type*: An abbreviation of the fortran declaration for the variable. Any of these types may be followed by a comma separated list of integers enclosed in parentheses to indicate an array. The current namelist validation code only distinguishes between string and non-string types. Valid declarations are: + - char*n + - integer + - logical + - real +- *kind*: Only needed if type is real (if type is real, 'kind' will be 'kind_phys') +- *input_pathname* (optional): Only include this attribute to indicate that the variable contains the pathname of an input dataset that resides in the CESM inputdata directory tree. Note that the variables containing the names of restart files that are used in branch runs don't reside in the inputdata tree and should not be given this attribute. The recognized values are: + - "abs" to indicate that an absolute pathname is required, or + - "rel:var_name" to indicate that the pathname is relative and that the namelist variable "var_name" contains the absolute root directory +- *category*: A category assigned for organized the documentation. This can be found by copying from the `bld/namelist_files/namelist_definition.xml` file in CAM +- *group*: The namelist group that the variable is declared in (found in `_readnl` in CAM) +- *standard_name*: The CCPP standard name +- *units*: Units for the variable. Must adhere to CCPP units convention (link?) +- *desc*: The description for this namelist variable. This can be found by copying from the `bld/namelist_files/namelist_definition.xml` file in CAM. This is parsed and used in the creation of the CAM namelist web page. Description should be adequate for this purpose (**Maybe not? This is implemented in SIMA yet...). +- *value*: Initial value for the variable. This can be a valid default value or an out-of-range value to allow for trapping in the code. +- *valid_values* (optional): Attribute that is mainly useful for variables that have only a small number of allowed values. + +**Example Namelist XML file** +``` + + + + + + + + + + + integer + rayleigh_friction + rayleigh_friction_nl + rayleigh_friction_center_vertical_index + 1 + + Variable to specify the vertical index at which the + Rayleigh friction term is centered (the peak value). + + + 2 + + + + + +``` +!!!Note "namelist values" + When populating the value fields, you don't need to port all of them at this time. You need to put in the value that was used in making the snapshot file, but the rest will be ported once all schemes have been ported. + +Once you have made your namelist file, proceed to [5 - Interstitials](step5.md) \ No newline at end of file diff --git a/docs/conversion/step5.md b/docs/conversion/step5.md new file mode 100644 index 0000000..be18b3d --- /dev/null +++ b/docs/conversion/step5.md @@ -0,0 +1,73 @@ +# 5 - Interstitials +## Overview +An **interstitial** is a scheme that does calculations or variable modifications to connect what the host model has to what the scheme requires and vice versa. + +- These schemes are placed before or after the core parameterization scheme in the suite definition file (SDF) to prep for the scheme or take what the scheme produces and translate it back to what the host model (CAM-SIMA) expects +- In CAM, interstitial code will appear as either function calls or calculations/modifications in the CAM interface just before or just after the call to the parameterization + - If the interstitial code is a function call, that function/module should be CCPP-ized (if not already) and that scheme will be included in the SDF + - It the interstitial code is loose in the CAM interface, it is recommended that, when possible, that code be moved into the scheme (either the beginning or end of the scheme subroutine) + - UNLESS: it's a common (across multiple parameterizations) calculation/translation/modification. In which case, a new scheme should be created to do that calculation (scheme will live in `$CAM-SIMA/src/physics/ncar_ccpp/utilities`) + - If it is absolutely necessary to create a scheme-specific interstitial, that scheme should be called `_pre` or `_post` (depending on where in the SDF it will be placed) and will live in the `ncar_ccpp/` directory + +This section covers a few interstitial scenarios you are likely to face. + +## Diagnostics interstitial +All `addfld`/`outfld` calls will go in a `_diagnostic.F90` interstitial: + +- `addfld` calls will go into the `_diagnostic_init` subroutine +- `outfld` calls will go into the `_diagnostic_run` subroutine +- The interstitial itself will reside after the `` line in the SDF + - See: [History Usage](../usage/history.md/#adding-a-diagnostic-field-to-the-cam-sima-source-code) for the specifications of the CAM-SIMA-versions of `addfld` and `outfld` calls + +## Utilities +As mentioned, there are some calculations/conversions/translations that are performed often throughout the physics code in CAM. These schemes are available for use in the `$CAM-SIMA/src/physics/ncar_ccpp/utilities` directory and include: + +- **state_converters.F90**: contains common conversions/calculations of [state variables](../design/ccpp-in-cam-sima.md/#state-and-tendency-variables), including these schemes: + +| Scheme name | Description | Output variable | Input variables | +|:------------|-------------|-----------------|-----------------| +| temp_to_potential_temp | convert temperature to potential temperature | air_potential_temperature | air_temperature
inverse_exner_function | +| potential_temp_to_temp | convert potential temperature to temperature | air_temperature |air_potential_temperature
inverse_exner_function | +| calc_dry_air_ideal_gas_density | Calculate density from equation of state/ideal gas law | dry_air_density |composition_dependent_gas_constant_of_dry_air
air_pressure_of_dry_air
air_temperature | +| calc_exner | calculate dimensionless exner function | dimensionless_exner_function |composition_dependent_specific_heat_of_dry_air_at_constant_pressure
composition_dependent_gas_constant_of_dry_air
surface_reference_pressure
air_pressure | +| wet_to_dry_water_vapor | convert water vapor from wet to dry mixing ratio | water_vapor_mixing_ratio_wrt_dry_air |air_pressure_thickness
air_pressure_thickness_of_dry_air
water_vapor_mixing_ratio_wrt_moist_air_and_condensed_water | +| dry_to_wet_water_vapor | convert water vapor from dry to wet mixing ratio | water_vapor_mixing_ratio_wrt_moist_air_and_condensed_water | air_pressure_thickness
air_pressure_thickness_of_dry_air
water_vapor_mixing_ratio_wrt_dry_air | +| wet_to_dry_cloud_liquid_water | convert cloud liquid from wet to dry mixing ratio | cloud_liquid_water_mixing_ratio_wrt_dry_air | air_pressure_thickness
air_pressure_thickness_of_dry_air
cloud_liquid_water_mixing_ratio_wrt_moist_air_and_condensed_water | +| dry_to_wet_cloud_liquid_water | convert cloud liquid from dry to wet mixing ratio | cloud_liquid_water_mixing_ratio_wrt_moist_air_and_condensed_water | air_pressure_thickness
air_pressure_thickness_of_dry_air
cloud_liquid_water_mixing_ratio_wrt_dry_air | +| wet_to_dry_rain | convert rain from wet to dry mixing ratio | rain_mixing_ratio_wrt_dry_air | air_pressure_thickness
air_pressure_thickness_of_dry_air
rain_mixing_ratio_wrt_moist_air_and_condensed_water | +| dry_to_wet_rain | convert rain from dry to wet mixing ratio | rain_mixing_ratio_wrt_moist_air_and_condensed_water |air_pressure_thickness
air_pressure_thickness_of_dry_air
rain_mixing_ratio_wrt_dry_air | + +- **geopotential_temp.F90**: + - **geopotential_temp**: compute geopotential height (`geopotential_height_wrt_surface`) and geopotential height at interfaces (`geopotential_height_wrt_surface_at_interface`) from temperature (`air_temperature`) +- **static_energy.F90** + - **update_dry_static_energy**: calculate dry static energy (`dry_static_energy`) +- **qneg.F90**: + - **qneg**: Set values for constituent variables that are less than the minimum value to the minimum value (and print out what it's doing - configurable!) +- **physics_tendency_updaters.F90**: apply tendencies output by physics to state variables + +| Scheme name | Description | Inout variable | Input variable | +|:------------|-------------|----------------|----------------| +| apply_tendency_of_
eastward_wind | Apply the eastward wind tendency calculated in the previous physics scheme(s) to the `eastward_wind` state variable | eastward_wind | tendency_of_eastward_wind_
due_to_model_physics
timestep_for_physics | +| apply_tendency_of_
northward_wind| Apply the northward wind tendency calculated in the previous physics scheme(s) to the `northward_wind` state variable | northward_wind | tendency_of_northward_wind_
due_to_model_physics
timestep_for_physics | +| apply_heating_rate | Apply the heating rate (`tendency_of_dry_air_enthalpy_at_constant_pressure`) to the temperature tendency and temperature state variable | air_temperature
tendency_of_air_temperature_
due_to_model_physics | tendency_of_dry_air_enthalpy_
  at_constant_pressure
composition_dependent_specific_heat_
  of_dry_air_at_constant_pressure
timestep_for_physics | +| apply_tendency_of_
air_temperature | Apply the temperature tendency calculated in the previous physics scheme(s) to the `air_temperature` state variable | air_temperature | tendency_of_air_temperature_
due_to_model_physics
timestep_for_physics | + +## Temporary constituent handling +For now, we don't have tendency updaters for the constituents. As a result, if your parameterization includes one or more constituent, you'll have to do a little finagling. + +!!! Note "Problem overview" + CAM outputs a tendency and then passes that tendency to physics_update to be added to state%q, but CAM-SIMA (currently) updates the constituent array directly + +1. Write the scheme to update the constituent data directly (constituent variable passed in with `intent(inout)`) +1. In CAM, pass in a temporary array for the constituent rather than the actual `state%q(:,:,index)` +1. Then, back out the tendency after calling scheme_run and reassign that to `state%q(:,:,index)` + +An example of how this is done can be found ~line 276 in: +``` +$CAM/src/physics/simple/kessler_cam.F90 +``` + +Proceed to [6 - Create an SDF](step6.md). + +!!! Warning + You may have to revisit this step as you debug your code. \ No newline at end of file diff --git a/docs/conversion/step6.md b/docs/conversion/step6.md new file mode 100644 index 0000000..67f7544 --- /dev/null +++ b/docs/conversion/step6.md @@ -0,0 +1,30 @@ +# 6 - Create an SDF +The **Suite Definition File (SDF)** tells the CCPP-Framework which schemes will be run in what order. For more, see [CCPP in CAM-SIMA](../design/ccpp-in-cam-sima.md) + +- In `$CAM-SIMA/src/physics/ncar_ccpp`, create `suite_.xml`. This is your SDF! +- See the template below. You will need to select either `physics_before_coupler` or `physics_after_coupler` as your group name. +``` + + + + + + + +``` + +
+| CAM physpkg routine | CAM-SIMA SDF Group | Description | +|:--------------------|------------------------|-------------| +| tphysbc | physics_before_coupler | Physics schemes run before the surface coupler is called | +| tphysac | physics_after_coupler | Physics schemes run after the surface coupler is called | +
+ +- Add any interstitials which you’ve created or are using in the appropriate location relative to your parameterization + +For example: if your parameterization requires dry water vapor, you'll want to add: + +- `wet_to_dry_water_vapor` before your core parameterization scheme to convert water vapor to a dry mixing ratio (CAM-SIMA has "wet") +- `dry_to_wet_water_vapor` after your parameterization scheme to convert back to a wet mixing ratio to return to CAM-SIMA + +Once you have created your SDF, proceed to [7 - Check metadata](step7.md) \ No newline at end of file diff --git a/docs/conversion/step7.md b/docs/conversion/step7.md new file mode 100644 index 0000000..3239c6a --- /dev/null +++ b/docs/conversion/step7.md @@ -0,0 +1,131 @@ +# 7 - Check metadata + +- Create and configure a short test case in CAM-SIMA that exercises your new scheme. You may find it easiest to use the FPHYStest compset (using the null dycore). For more on setting up a case, see [Create a case](../usage/creating-a-case.md) + +``` +./create_newcase --case --compset FPHYStest --res ne3pg3_ne3pg3_mg37 --compiler --run-unsupported +cd +./xmlchange CAM_CONFIG_OPTS="--dyn none --physics-suites " +./xmlchange STOP_OPTION="nsteps" +./xmlchange STOP_N= +./xmlchange DOUT_S=False +``` + +- Run `./preview_namelists` to see if the framework will generate the caps. A few possible outcomes: + - There are no errors and you are somehow a perfect person. Move on to the [next step](step8.md) and ultimately run the model + - You get an error + - Determine what the course of action should be based on the table below. Once you think you have addressed the issue, rerun `./preview_namelists` and repeat! + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
ErrorSituationAction
`Input agument for _, , not found`Standard name incorrect when compared against the spreadsheetCorrect standard name
Variable traced back to "use" statement in CAMFollow [procedure 1](#procedure-1) below
Variable is a constituent (can be traced back to the `state%q` array in CAM)Add `advected = true` to the variable's metadata in your parameterization
Variable can be traced back to a calculation in the CAM interface codeMake an [interstitial](step5.md)
None of the above criteria are metFollow [procedure 2](#procedure-2) to add the variable to the registry
`parse_source.ParseSyntaxError: Unsupported unit conversion, '' to '' for ''`Units do not match and there is not automatic unit converter for the conversionFollow [procedure 3](#procedure-3)
`parse_source.CCPPError: Invalid 'standard_name' property value`Standard name is not parsing correctlyLook for a missing "_" or a typo in the standard name
`CCPPError: Could not find dimension, :, in `Dimension is either invalid or not implemented in CAMConfirm that the dimension is valid and that you selected the correct one (e.g. `vertical_layer_dimension` vs `vertical_interface_dimension`. If it's valid and still missing, consult with the other CAM SEs
Some other misc errorYou've run into either a bug or just a weird scenarioIf the error message is unhelpful, ask Courtney
+ +Once you have validated your metadata, proceed to [8 - Run CAM-SIMA](step8.md) + +## Procedure 1 +If the CAM variable can be traced back to a use statement, you may have to create or amend a SIMA-side metadata file. + +- Try to find the equivalent module and the "used" variable within it in CAM-SIMA + - If you find it the module: + - It already has a metadata file, move the variable in the Fortran to be with the other variables that have metadata. Then, add your variable to the appropriate place in the metadata file. + - It doesn't have a metadata file, create one and add your variable. You'll also have to add the ["html" tags](step1.md#1b-add-required-htmlinclude-lines) and potentially rearrange module-level variables so the only variable(s) under the tags are the ones you're providing metadata for. + - Then add the path to the metadata file to the `metadata_file` section of `$CAM-SIMA/src/data/registry.xml` + - If you can't find the module, consult with the other CAM SEs + +## Procedure 2 +If the variable in question cannot be tracked back to a “use” statement or a calculated value in the CAM interface code, it likely needs to be provided by the host model. To enable this, you’ll have to add the variable to the registry (`$CAM-SIMA/src/data/registry.xml`). Here is the structure of how that will look: + +``` + + horizontal_dimension + phis state_phis + + +``` + +- If the variable you are adding is a state or ptend variable, be sure to also add the variable standard name to the relevant ddt in the registry. +- `dimensions` is a space-separated list (in order) of the standard names of the dimensions for the variable +- `ic_file_input_names` is a space-separated list of possible names on the snapshot/inputdata file + +!!! Note - registry variables + Variables in the registry are read in from the initial data file by default. + +There are two options for registry variables: + +1. Variable read in from the initial data file: no action required +1. Variable not read in from the initial data file, and instead intialized during CAM-SIMA initialization. You will need to add code to do this: + - Determine the best place to initialize this variable (has to occur at some point during initialization - see [CAM-SIMA run](../design/cam-run-process.md/#cam_init) + - In the location you have chosen, add the use statements below to import the registry variable from the auto-generated fortran AND the mark_as_initialized routine + - After adding the code to initialize the variable, add the `mark_as_initialized` call below to indicate that that variable should NOT be read in from the file. + +``` +use physics_types, only +use phys_vars_init_check, only: mark_as_initialized +``` + +``` + = +call mark_as_initialized('') +``` + +## Procedure 3 +The units you are using for a variable in your converted scheme has a different unit than the host model (or possibly different from elsewhere in your suite). + +- Do a grep in $CAM-SIMA/src for the standard name to see what the discrepancy is. +- Double-check that the units for your scheme match the units in the spreadsheet +- If it seems like the automatic unit converter should be able to handle the unit conversion, talk to Courtney +- If it’s more of a one-off conversion, two options remain: + 1. Write an interstitial for the unit conversion + 2. Include the conversion in the scheme code (you will also need to update the CAM interface to make sure it’s passing in the same units as the CAM-SIMA host has) \ No newline at end of file diff --git a/docs/conversion/step8.md b/docs/conversion/step8.md new file mode 100644 index 0000000..93fba2d --- /dev/null +++ b/docs/conversion/step8.md @@ -0,0 +1,46 @@ +# 8 - Run CAM-SIMA + +It's time to try to run CAM-SIMA! + +- Either navigate to the case you created when you were validating your metadata or [create a new case](../usage/creating-a-case.md) +- Your `user_nl_cam` needs to contain the following (to bring in the cam_snapshot for input and to activate the validation tool): +``` +ncdata='/FullPathName/your_before_snapshot_file.nc' +ncdata_check='/FullPathName/your_output_filename.nc' +debug_output = 0 !Turns off debug messages which interfere with the validation tool output +``` +- If the number of vertical levels in your snapshot file is something other than 30, then you will also need to add the following to your user_nl_cam: + + ``` + pver = N + ``` + + where “N” is the number of vertical levels in your snapshot file (likely 32 levels for CAM6) + +- Run `./case.build` and `./case.submit` + - See [debugging tips](../development/debugging.md) for help if you're getting errors + +- Once the model completes without error, the results from the validation tool will appear in the `atm_log` file for each timestep. + - The message “no differences found” should be logged for each timestep. + - If there are differences found, that indicates there are issues with your ported code + - See the section on [Tips for uncovering unexpected answer changes](#tips-for-uncovering-unexpected-answer-changes) for debugging strategies + +- Make at least one run using the NAG compiler in debug mode and specifying “-nan” as an additional compilation option. This option initializes all variables to NaN so that if something is uninitialized and is used, it will be trapped. To enable this flag: + - After executing the ./create_newcase and ./case.setup commands and BEFORE executing the ./case.build, edit the file in your case directory `cmake_macros/nag.cmake` (- If you want to change the file so that it runs every time you execute a `./create_newcase`, you can find the file at `ccs_config/machines/cmake_macros/nag.cmake`). + - Change the line Append line so that it contains ‘-nan’ and it should look like this: +``` +if (DEBUG) + string(APPEND FFLAGS " -nan -C=all -g -time -f2003 -ieee=stop") +endif() +``` + +Once you have successfully run CAM-SIMA and all answer changes have either accepted or fixed, proceed to [9 - Bring back into CAM](step9.md) + +## Tips for uncovering unexpected answer changes + +- Check all of your standard_names. CAM-SIMA will run successfully, but if a standard_name is incorrect, you may be running with an incorrect variation of the variable that you intended +- If comparing printed values on the physics grid between CAM and CAM-SIMA, then it might be good to turn-off chunking in CAM, which can be done by setting “-pcols” in “CAM_CONFIG_OPTS” to be a large number (e.g. >500 for the ne3 grid). + - You can also utilize the `tools/find_max_nonzero_index.F90` tool to (more) easily compare values between CAM and CAM-SIMA at a specific rank and index that is known to be non-zero +- Note that the way latitude in radians is calculated for the null dycore in CAM-SIMA is different (at a round-off level) to the way it is calculated in the SE dycore in CAM. This means if you are validating a physics scheme that has latitude (in radians) as an input, then in CAM you’ll need to change the scheme interface to read in latitude in degrees, and then convert to radians by multiplying by pi/180 (which you’ll only want to do when creating the snapshot file, not when validating CAM itself). +- Sometimes CAM-SIMA will set a state variable directly, while CAM will only set a tendency variable and then use the “physics_update” routine to update the state using the tendency. This can result in round-off level differences. If this happens then simply try setting the state variable directly in CAM when creating the snapshot files for CAM-SIMA testing. +- If you run into a "tip" that should be added here, please confer with the other CAM SEs to get it added to the documentation!! \ No newline at end of file diff --git a/docs/conversion/step9.md b/docs/conversion/step9.md new file mode 100644 index 0000000..ce3504f --- /dev/null +++ b/docs/conversion/step9.md @@ -0,0 +1,47 @@ +# 9 - Bring back into CAM + +## Code modifications +Bring the CCPP version back into a new CAM clone or branch + +- Make a new clone or branch of CAM6 (personal preference) +- Remove the non-CCPP version from your CAM source directory +- Make a new directory in the atmosperhic_physics submodule of CAM (make sure the directory name is the one you used in step one of the conversion to CCPP): +``` +mkdir src/atmos_phys/ +``` +- Copy your new CCPP version files from CAM-SIMA into this new directory +- Update the routine names to match the CCPP-ized subroutine names where the subroutines are called +- Update the calling lists if they were changed when the CCPP-ized subroutines were made +- If the CCPP-ized module is in a new subdirectory, you will need to add it to `bld/configure`: +``` +print $fh "$camsrcdir/src/atmos_phys/\n"; + +## Testing +``` +- To quickly test that your code is working during development, you can create two new model cases with a compset that involves your physics scheme, and the following XML changes: + +``` +./xmlchange --force ROF_NCPL=\$ATM_NCPL +./xmlchange DOUT_S=FALSE,STOP_OPTION=nsteps,STOP_N=9,DEBUG=TRUE +``` + +!!! Note - test cases + The first case will be CAM without your changes, and the second will be CAM with your new physics scheme code changes. Both simulations should produce a `.cam.rh0.XXX` file, which you can compare by using the cprnc tool, which can be found (and used) on derecho below. Your new physics code is working correctly if the printed output from cprnc states that there are no differences in the files. + +``` +/glade/campaign/cesm/cesmdata/tools/cprnc/cprnc +cprnc +``` +- Optional step: if want to have additional testing... Once you believe it is working, repeat the [snapshot](step2.md) section to run CAM with this modified clone, generating snapshot files. + - Use the cprnc tool to compare the resulting CAM6 “after” snapshot file with your original CAM6 “after” snapshot file (from before the port). If they differ and you changed the internal calculations during your port, you will need to determine if these changes are expected and correct. + - If they are different (and the changes are correct), then you will need to repeat the steps listed in the [Run CAM-SIMA](step8.md) section with the newly generated snapshot files from the modified CAM clone. + +## Committing your changes + +- Open a PR in [ESCOMP/atmospheric_physics](https://github.com/ESCOMP/atmospheric_physics) for your parameterization and any interstitial code (including diagnostics) using the code in your sandbox + - Make sure you include a ChangeLog and NamesNotInDictionary.txt file as specified [here](https://github.com/ESCOMP/atmospheric_physics/wiki/Development-workflow#changelog-and-namesnotindictionarytxt) +- Open a PR in [ESCOMP/CAM](https://github.com/ESCOMP/CAM) for your changes in CAM (if calling lists changed) or it may as simple as a PR to update the atmospheric_physics tag in `.gitmodules`. Don't forget to make a ChangeLog. +- If any changes were made to CAM-SIMA (more than likely, at least an update to the atmospheric_physics submodule will probably be needed), open a PR in [ESCOMP/CAM-SIMA](https://github.com/ESCOMP/CAM-SIMA). +- Once all PRs are merged and tagged, check off the scheme as "Converted" in the [spreadsheet](https://docs.google.com/spreadsheets/d/1_1TTpnejam5jfrDqAORCCZtfkNhMRcu7cul37YTr_WM/edit?gid=0#gid=0) + +[Details on opening PR’s](https://github.com/ESCOMP/CAM/wiki/CAM-Development-Workflow-in-GitHub#how-to-submit-code-changes-to-be-included-in-escompcam) \ No newline at end of file diff --git a/docs/conversion/walkthrough.md b/docs/conversion/walkthrough.md new file mode 100644 index 0000000..6bbfc9b --- /dev/null +++ b/docs/conversion/walkthrough.md @@ -0,0 +1,691 @@ +# Walkthrough Example + +GASP! Our favorite imaginary software engineer **Bug E. Code** has been tasked with CCPP-izing lunar_tides*! + +*Scheme doesn't do anything outside of WACCM; chosen because it won't be CCPP-ized for real any time soon (so you can see what the existing code looks like) + +*Here's how Bug completes this assignment...* + +## 0 - Background & prep work +1. Bug updates the [conversion spreadsheet](https://docs.google.com/spreadsheets/d/1_1TTpnejam5jfrDqAORCCZtfkNhMRcu7cul37YTr_WM/edit#gid=0) with his new assignment. + + ![text](figures/spreadsheet-change-highlighted.PNG "Spreadsheet update") + +1. Bug opens issues in ESCOMP/CAM and ESCOMP/atmospheric_physics + + ![text](figures/cam-issue.PNG "CAM issue") + + ![text](figures/atmos-phys-issue.PNG "atmospheric_physics issue") + +1. He sets up his sandbox (with his preexisting forks) and copies the portable layer via the following commands: +``` +git clone -o bugsy9236 https://github.com/bugsy9236/CAM CAM +cd CAM +git fetch --tags ESCOMP +git branch ccppize-lunar-tides ESCOMP/cam_development +git checkout ccppize-lunar-tides +git push -u bugsy9236 ccppize-lunar-tides +bin/git-fleximod update +cd .. +git clone -o bugsy9236 https://github.com/bugsy9236/CAM-SIMA CAM-SIMA +cd CAM-SIMA +git remote add ESCOMP https://github.com/ESCOMP/CAM-SIMA +git fetch --tags ESCOMP +git branch ccppize-lunar-tides ESCOMP/development +git checkout ccppize-lunar-tides +git push -u bugsy9236 ccppize-lunar-tides +bin/git-fleximod update +mkdir src/physics/ncar_ccpp/lunar_tides +cp ../CAM/src/physics/cam/lunar_tides.F90 src/physics/ncar_ccpp/lunar_tides/ +cd src/physics/ncar_ccpp +git remote add bugsy9236 https://github.com/bugsy9236/atmospheric_physics +git fetch bugsy9236 +git checkout -b ccppize-lunar-tides +git push -u bugsy9236 ccppize-lunar-tides + +``` + +## 1 - Convert the portable layer + +### 1a - Change routine names +Bug notes that there are three routines in `lunar_tides.F90`: + +``` +public :: lunar_tides_readnl +public :: lunar_tides_init +public :: lunar_tides_tend +``` + +1. He ignores `lunar_tides_readnl` for now (just leaves it as is) +1. He's *pretty sure* that `lunar_tides_init` is an init routine, but confirms this by checking where it is called in `physpkg.F90`. It is indeed called by `phys_init`, so he can comfortably keep the `lunar_tides_init` routine name as is. No action needed here yet. +1. Bug determines that `lunar_tides_tend` is called in `tphysac` (called every timestep after the coupler), so he renames `lunar_tides_tend` to `lunar_tides_run` throughout the module and proceeds + +End result: +``` +public :: lunar_tides_readnl +public :: lunar_tides_init +public :: lunar_tides_run +``` + +### 1b - Add required `\htmlinclude` lines +Bug adds the necessary two lines above both the `lunar_tides_init` and `lunar_tides_run` routines. + +``` +!> \section arg_table_lunar_tides_init Argument Table +!! \htmlinclude lunar_tides_init.html + subroutine lunar_tides_init() + use cam_history, only: addfld + use time_manager,only: timemgr_get_calendar_cf + + if (apply_lunar_tides) then + if (timemgr_get_calendar_cf().ne.'gregorian') then + call endrun('lunar_tides_init: calendar must be gregorian') + endif + call addfld('UT_LUNAR', (/ 'lev' /), 'A','m/s2','Zonal wind tendency due to lunar tides') + call addfld('VT_LUNAR', (/ 'lev' /), 'A','m/s2','Meridional wind tendency due to lunar tides') + end if + + end subroutine lunar_tides_init + + !========================================================================== + !========================================================================== +!> \section arg_table_lunar_tides_run Argument Table +!! \htmlinclude lunar_tides_run.html + subroutine lunar_tides_run( state, ptend ) + use time_manager, only: get_curr_date, get_julday + use physconst, only: pi, rearth + use ppgrid, only: pver + use cam_history, only: outfld + + type(physics_state), intent(in) :: state + type(physics_ptend), intent(out):: ptend + ... +``` + +### 1c - Clean up dummy argument dimensions +Bug makes sure no input/output variables have named dimensions in their declaration inside the routines. In this case, there are no dimensioned variables in the argument lists. So hooray! + +### 1d - Use kind_phys instead of r8 +Bug changes the use statement +``` +use shr_kind_mod, only: r8=>shr_kind_r8 +``` +to +``` +use ccpp_kinds, only: kind_phys +``` +And does a find-and-replace for `r8` with `kind_phys` + +`lunar_tides_run` change (old -> new): + +
+``` +subroutine lunar_tides_run( state, ptend ) + use time_manager, only: get_curr_date, get_julday + use physconst, only: pi, rearth + use ppgrid, only: pver + use cam_history, only: outfld + + type(physics_state), intent(in) :: state + type(physics_ptend), intent(out):: ptend + + integer :: tod,yr,mm,dd + real(r8) :: jd,nu,lt,lun_lt + + integer :: i, k + + real(r8), parameter :: deg2hrs = 1._r8/15._r8 + real(r8), parameter :: rad2deg = 180._r8/pi + real(r8), parameter :: rad2hrs = rad2deg*deg2hrs + real(r8), parameter :: tod2hrs = 24._r8/86400._r8 + real(r8), parameter :: hrs2rad = 1._r8/rad2hrs +``` +``` +subroutine lunar_tides_run( state, ptend ) + use time_manager, only: get_curr_date, get_julday + use physconst, only: pi, rearth + use ppgrid, only: pver + use cam_history, only: outfld + + type(physics_state), intent(in) :: state + type(physics_ptend), intent(out):: ptend + + integer :: tod,yr,mm,dd + real(kind_phys) :: jd,nu,lt,lun_lt + + integer :: i, k + + real(kind_phys), parameter :: deg2hrs = 1._kind_phys/15._kind_phys + real(kind_phys), parameter :: rad2deg = 180._kind_phys/pi + real(kind_phys), parameter :: rad2hrs = rad2deg*deg2hrs + real(kind_phys), parameter :: tod2hrs = 24._kind_phys/86400._kind_phys + real(kind_phys), parameter :: hrs2rad = 1._kind_phys/rad2hrs +``` +
+### 1e - Remove use statements +Bug removes the `use` statements (except for `addfld` and `outfld` calls) and adds the relevant variables to the calling list. He also comments out any use statements from cam_history as well as any addfld and outfld calls. He also moves the namelist variable to module-level (above the `CONTAINS` statement) and passes it in to the calling list for _init + +For `lunar_tides_init`: + +
+``` +subroutine lunar_tides_init() + use cam_history, only: addfld + use time_manager,only: timemgr_get_calendar_cf + + if (apply_lunar_tides) then + if (timemgr_get_calendar_cf().ne.'gregorian') then + call endrun('lunar_tides_init: calendar must be gregorian') + endif + call addfld('UT_LUNAR', (/ 'lev' /), 'A','m/s2','Zonal wind tendency due to lunar tides') + call addfld('VT_LUNAR', (/ 'lev' /), 'A','m/s2','Meridional wind tendency due to lunar tides') + end if + + end subroutine lunar_tides_init +``` +``` +subroutine lunar_tides_init(calendar, apply_lunar_tides_in) + !use cam_history, only: addfld + character(len=32), intent(in) :: calendar + logical, intent(in) :: apply_lunar_tides_in + + apply_lunar_tides = apply_lunar_tides_in + + if (apply_lunar_tides) then + if (calendar.ne.'gregorian') then + call endrun('lunar_tides_init: calendar must be gregorian') + endif + !call addfld('UT_LUNAR', (/ 'lev' /), 'A','m/s2','Zonal wind tendency due to lunar tides') + !call addfld('VT_LUNAR', (/ 'lev' /), 'A','m/s2','Meridional wind tendency due to lunar tides') + end if + + end subroutine lunar_tides_init +``` +
+ +For `lunar_tides_run`: +
+``` +subroutine lunar_tides_run( state, ptend ) + use time_manager, only: get_curr_date, get_julday + use physconst, only: pi, rearth + use ppgrid, only: pver + use cam_history, only: outfld + + type(physics_state), intent(in) :: state + type(physics_ptend), intent(out):: ptend +``` +``` +subroutine lunar_tides_run(state, ptend, curr_date_yr, & + curr_date_mm, curr_date_dd, curr_date_tod, julday, & + pi, rearth, pver) + !use cam_history, only: outfld + + type(physics_state), intent(in) :: state + type(physics_ptend), intent(out):: ptend + integer, intent(in) :: curr_date_yr + integer, intent(in) :: curr_date_mm + integer, intent(in) :: curr_date_dd + integer, intent(in) :: curr_date_tod + real(kind_phys), intent(in) :: julday + real(kind_phys), intent(in) :: pi + real(kind_phys), intent(in) :: rearth + integer, intent(in) :: pver + +``` +
+ +### 1f - Add error variables +Bug adds the `errmsg` and `errflg` variables to the end of the calling list and initializes them. +``` +subroutine lunar_tides_run(state, ptend, curr_date_yr, & + curr_date_mm, curr_date_dd, curr_date_tod, julday, & + pi, rearth, pver, errmsg, errflg) + !use cam_history, only: outfld + + type(physics_state), intent(in) :: state + type(physics_ptend), intent(out):: ptend + integer, intent(in) :: curr_date_yr + integer, intent(in) :: curr_date_mm + integer, intent(in) :: curr_date_dd + integer, intent(in) :: curr_date_tod + real(kind_phys), intent(in) :: julday + real(kind_phys), intent(in) :: pi + real(kind_phys), intent(in) :: rearth + integer, intent(in) :: pver + character(len=512), intent(out):: errmsg + integer, intent(out):: errflg + + ... + + errflg = 0 + errmsg = '' +``` + +### 1g - Replace state and ptend variables in calling list +`lunar_tides_run` includes both `state` and `ptend`! Bug starts with `state` and figures out which state variables are used in the routine + +- He determines that the routine uses `state%lat`, `state%lon`, `state%ncols`, and `state%zm` +- He passes each in individually and ends up with: +``` +subroutine lunar_tides_run(ncols, lat, lon, zm, ptend, & + curr_date_yr, curr_date_mm, curr_date_dd, curr_date_tod, & + julday, pi, rearth, pver, errmsg, errflg) + !use cam_history, only: outfld + + integer, intent(in) :: ncols + real(kind_phys), intent(in) :: lat(:) + real(kind_phys), intent(in) :: lon(:) + real(kind_phys), intent(in) :: zm(:,:) + type(physics_ptend), intent(out):: ptend + integer, intent(in) :: curr_date_yr + integer, intent(in) :: curr_date_mm + integer, intent(in) :: curr_date_dd + integer, intent(in) :: curr_date_tod + real(kind_phys), intent(in) :: julday + real(kind_phys), intent(in) :: pi + real(kind_phys), intent(in) :: rearth + integer, intent(in) :: pver + character(len=512), intent(out):: errmsg + integer, intent(out):: errflg + +``` +Now he does the same with `ptend` (the routine uses `ptend%u` and `ptend%v`) and ends up with: +``` +subroutine lunar_tides_run(ncols, lat, lon, zm, dudt, dvdt, & + curr_date_yr, curr_date_mm, curr_date_dd, curr_date_tod, & + julday, pi, rearth, pver, errmsg, errflg) + !use cam_history, only: outfld + + integer, intent(in) :: ncols + real(kind_phys), intent(in) :: lat(:) + real(kind_phys), intent(in) :: lon(:) + real(kind_phys), intent(in) :: zm(:,:) + real(kind_phys), intent(out):: dudt(:,:) + real(kind_phys), intent(out):: dvdt(:,;) + integer, intent(in) :: curr_date_yr + integer, intent(in) :: curr_date_mm + integer, intent(in) :: curr_date_dd + integer, intent(in) :: curr_date_tod + real(kind_phys), intent(in) :: julday + real(kind_phys), intent(in) :: pi + real(kind_phys), intent(in) :: rearth + integer, intent(in) :: pver + character(len=512), intent(out):: errmsg + integer, intent(out):: errflg +``` + +As part of removing `ptend`, Bug removes the call to `physics_ptend_init` in `lunar_tides_run` + +Replacing all the relevant variables in the routine results in... +``` +subroutine lunar_tides_run(ncols, lat, lon, zm, dudt, dvdt, & + curr_date_yr, curr_date_mm, curr_date_dd, curr_date_tod, & + julday, pi, rearth, pver, errmsg, errflg) + !use cam_history, only: outfld + + integer, intent(in) :: ncols + real(kind_phys), intent(in) :: lat(:) + real(kind_phys), intent(in) :: lon(:) + real(kind_phys), intent(in) :: zm(:,:) + real(kind_phys), intent(out):: dudt(:,:) + real(kind_phys), intent(out):: dvdt(:,;) + integer, intent(in) :: curr_date_yr + integer, intent(in) :: curr_date_mm + integer, intent(in) :: curr_date_dd + integer, intent(in) :: curr_date_tod + real(kind_phys), intent(in) :: julday + real(kind_phys), intent(in) :: pi + real(kind_phys), intent(in) :: rearth + integer, intent(in) :: pver + character(len=512), intent(out):: errmsg + integer, intent(out):: errflg + + real(kind_phys) :: nu,lt,lun_lt + + integer :: i, k + + real(kind_phys), parameter :: deg2hrs = 1._kind_phys/15._kind_phys + real(kind_phys), parameter :: rad2deg = 180._kind_phys/pi + real(kind_phys), parameter :: rad2hrs = rad2deg*deg2hrs + real(kind_phys), parameter :: tod2hrs = 24._kind_phys/86400._kind_phys + real(kind_phys), parameter :: hrs2rad = 1._kind_phys/rad2hrs + + if (apply_lunar_tides) then + + ! calculation relies on time from noon on December 31, 1899, so + ! subtract 2415020, which corresponds to the Julian date for Dec. 31 1899. + julday = julday - 2415020._kind_phys + julday = julday / 36525._kind_phys ! convert to julian centuries + + ! Calculate the lunar local time (nu) based on the the time + ! in Julian centuries using the formula given in Chapman and Lindzen (1970) + nu = -9.26009_kind_phys + 445267.12165_kind_phys*julday+0.00168_kind_phys*julday*julday !nu in degrees + + do i=1,ncol + ! solar local time (hours) + lt = real(curr_tod,kind=kind_phys)*tod2hrs + lon(i)*rad2hrs + + ! lunar local time + lun_lt = lt - nu*deg2hrs ! hours + lun_lt = lun_lt*hrs2rad ! radians + + do k=1,pver + ! Calculate the M2 lunar tide forcing in the zonal and meridional directions. + ! The forcing is calculated based on the gradient of the M2 tidal + ! potential, which is given in Chapman and Lindzen (1970). + ! Additional details on the derivation of the forcing are in + ! Pedatella, Liu, and Richmond (2012) + dudt(i,k) = (-1._kind_phys/((zm(i,k)+rearth)*cos(lat(i))))*2.456_kind_phys*3._kind_phys * & + ((zm(i,k)+rearth)/rearth)**2*cos(lat(i))*cos(lat(i))*2._kind_phys*sin(2._kind_phys*lun_lt) + dvdt(i,k) = (1._kind_phys/(zm(i,k)+rearth))*2.456_kind_phys*3._kind_phys * & + ((zm(i,k)+rearth)/rearth)**2*cos(2._kind_phys*lun_lt)*2._kind_phys*cos(lat(i))*sin(lat(i)) + end do + end do + + !call outfld('UT_LUNAR', ptend%u(:state%ncol,:), state%ncol, state%lchnk) + !call outfld('VT_LUNAR', ptend%v(:state%ncol,:), state%ncol, state%lchnk) + + end if + + end subroutine lunar_tides_run + +``` + +### 1h - Remove pbuf variables +Bug discovers that there are no `pbuf` variables in lunar_tides! Yay! + +### 1i - Mark variables as initialized +There are no variables that Bug needs to mark as initialized. At least, so he thinks. He may revisit this! + +### 1j - Initial standard name check +Bug does an initial check of the [standard names spreadsheet](https://docs.google.com/spreadsheets/d/1vpQ_xDZk00Z-_3SpW5N2EF3_FY6K7opNN4cqtSMlbwU/edit?gid=0#gid=0). He is able to find standard anmes for all of the variables except the calendar type, current date and julian date info. He adds those to the spreadsheet and sends an email to the other CAM SEs about needing to decide on a standard name for these variables. + +### 1k - Update CAM interface call(s) +Bug updates the call in `physpkg.F90`: + +- He renames the subroutine `lunar_tides_run` +- He moves the use statements that he removed from `lunar_tides_run` to `tphysac` and adds calls to `get_curr_date` and `get_julday` to just before the call to `lunar_tides_run`. He passes in all the new variables. +- He moves the use statement that he removed from `lunar_tides_init` to `phys_init`, adds a call, and passes the `calendar` variable into `lunar_tides_init` + +### 1l - OPTIONAL: Make a CAM tag +Bug opts not to make a CAM tag at this time. + +## 2 - Create snapshots of CAM +Bug discovers that there are no snapshot calls around `lunar_tides_run` in `physpkg.F90`. So he uses `user_set` snapshot calls as such: +``` +! Lunar tides + if (trim(cam_take_snapshot_before) == "user_set") then + call cam_snapshot_all_outfld_tphysac(cam_snapshot_before_num, state, tend, cam_in, cam_out, pbuf,& + fh2o, surfric, obklen, flx_heat) + end if + call lunar_tides_tend( state, ptend ) + if ( (trim(cam_take_snapshot_after) == "user_set") .and. & + (trim(cam_take_snapshot_before) == trim(cam_take_snapshot_after))) then + call cam_snapshot_ptend_outfld(ptend, lchnk) + end if + if ( ptend%lu ) then + call outfld( 'UTEND_LUNART', ptend%u, pcols, lchnk) + end if + if ( ptend%lv ) then + call outfld( 'VTEND_LUNART', ptend%v, pcols, lchnk) + end if + call physics_update(state, ptend, ztodt, tend) + if (trim(cam_take_snapshot_after) == "user_set") then + call cam_snapshot_all_outfld_tphysac(cam_snapshot_after_num, state, tend, cam_in, cam_out, pbuf,& + fh2o, surfric, obklen, flx_heat) + end if +``` + +Our main man Bug creates a test case with resolution `ne3pg3_ne3pg3_mg37` with DEBUG=True and the following `user_nl_cam`: +``` +apply_lunar_tides = .true. +cam_snapshot_before_num=6 +cam_snapshot_after_num=7 +cam_take_snapshot_before='user_set' +cam_take_snapshot_after='user_set' +nhtfrq = 0,0,0,0,0,1,1 +ndens = 2,2,2,2,2,1,1 +``` + +and runs the model for 7 timesteps. He saves the snapshot files in `/glade/campaign/cesm/community/amwg/sima_baselines/cam_sima_test_snapshots` with the names `cam_ne3pg3_lunar_tides_snapshot_derecho_gnu_before_c20240801.nc` and `cam_ne3pg3_lunar_tides_snapshot_derecho_gnu_after_c20240801.nc` + +## 3 - Create metadata file +To generate his metadata file, Bug runs: +``` +python ../../../../ccpp_framework/scripts/ccpp_fortran_to_metadata.py lunar_tides.F90 +``` +He then populates the generated metadata template (`lunar_tides.meta`) with the standard names, dimensions, and units. He ends up with: +``` +[ccpp-table-properties] + name = lunar_tides + type = scheme + +[ccpp-arg-table] + name = lunar_tides_init + type = scheme +[ calendar ] + standard_name = enter_standard_name_1 + units = none + type = character | kind = len=32 + dimensions = () + intent = in +[ apply_lunar_tides_in ] + standard_name = do_apply_lunar_tides + units = none + type = logical + dimensions = () + intent = in +[ errmsg ] + standard_name = ccpp_error_message + units = none + type = character | kind = len=512 + dimensions = () + intent = out +[ errflg ] + standard_name = ccpp_error_code + units = 1 + type = integer + dimensions = () + intent = out + +[ccpp-arg-table] + name = lunar_tides_run + type = scheme +[ ncols ] + standard_name = horizontal_loop_extent + units = count + type = integer + dimensions = () + intent = in +[ lat ] + standard_name = latitude + units = degree_north + type = real | kind = kind_phys + dimensions = (horizontal_loop_extent) + intent = in +[ lon ] + standard_name = longitude + units = degree_east + type = real | kind = kind_phys + dimensions = (horizontal_loop_extent) + intent = in +[ zm ] + standard_name = geopotential_height_wrt_surface + units = m + type = real | kind = kind_phys + dimensions = (horizontal_loop_extent, vertical_layer_dimension) + intent = in +[ dudt ] + standard_name = tendency_of_eastward_wind + units = m s-2 + type = real | kind = kind_phys + dimensions = (horizontal_loop_extent, vertical_layer_dimension) + intent = out +[ dvdt ] + standard_name = tendency_of_northward_wind + units = m s-2 + type = real | kind = kind_phys + dimensions = (horizontal_loop_extent, vertical_layer_dimension) + intent = out +[ curr_date_yr ] + standard_name = enter_standard_name_9 + units = count + type = integer + dimensions = () + intent = in +[ curr_date_mm ] + standard_name = enter_standard_name_10 + units = count + type = integer + dimensions = () + intent = in +[ curr_date_dd ] + standard_name = enter_standard_name_11 + units = count + type = integer + dimensions = () + intent = in +[ curr_date_tod ] + standard_name = enter_standard_name_12 + units = count + type = integer + dimensions = () + intent = in +[ julday ] + standard_name = enter_standard_name_13 + units = none + type = real | kind = kind_phys + dimensions = () + intent = in +[ pi ] + standard_iname = pi_constant + units = 1 + type = real | kind = kind_phys + dimensions = () + intent = in +[ rearth ] + standard_name = radius_of_earth + units = m + type = real | kind = kind_phys + dimensions = () + intent = in +[ pver ] + standard_name = vertical_layer_dimension + units = count + type = integer + dimensions = () + intent = in +[ errmsg ] + standard_name = ccpp_error_message + units = none + type = character | kind = len=512 + dimensions = () + intent = out +[ errflg ] + standard_name = ccpp_error_code + units = 1 + type = integer + dimensions = () + intent = out +``` +Bug will revisit the missing standard names when he hears back from the other CAM SEs/scientists about what the names should be. + +## 4 - Create namelist XML file +Bug creates the following namelist xml file (called `lunar_tides_namelist.xml`) in the `ncar_ccpp/lunar_tides` directory. It contains one entry - for `apply_lunar_tides` +``` + + + + + + + logical + waccm_phys + lunar_tides_opts + control_for_lunar_tides + none + + + Switch to apply lunar tidal tendencies to neutral winds. Default: FALSE + + + .false. + + + + +``` + +He then removes `lunar_tides_readnl` from `lunar_tides.F90`. This routine and it's metadata will be auto-generated by CAM-SIMA. + +## 5 - Interstitials +Bug determines that he does not need any interstitials. The only candidates were calculations of the current time and julian year, but, after consulting with other CAM SEs, it was determined that CAM-SIMA will be updated to provide these values. + +## 6 - Create an SDF +Bug creates the following Suite Definition File (including state updaters): +``` + + + + lunar_tides + apply_tendency_of_eastward_wind + apply_tendency_of_northward_wind + + +``` + +## 7 - Check metadata +Once he hears back about the missing standard names and populates them in the metadata, Bug creates a new case and runs `./preview_namelists`. + +He runs into the following error: +``` +Input argument for lunar_tides_init, calendar_type, not found. +``` + +This is one of the new standard names, but it isn't something read in from a file. He'll need to add it to the registry and then initialize the variable within CAM-SIMA. +He adds this entry to the registry (`src/data/registry.xml`): + +``` + + calendar type + +``` + +Then, in `cam_comp.F90`, he adds these use statements to the top of the module: +``` +use time_manager, only: timemgr_get_calendar_cf +use physics_types, only: calendar_type +use phys_vars_init_check, only: mark_as_initialized +``` + +Then adds these calls after the call to `timemgr_init`: +``` +calendar_type = timemgr_get_calendar_cf() +mark_as_initialized('calendar_type') +``` + +Bug then works through any remaining errors until the `preview_namelists` command completes. + +## 8 - Run CAM-SIMA +Bug builds and runs the model with `user_nl_cam` pointing to the before/after snapshots. + +``` +ncdata = '/glade/campaign/cesm/community/amwg/sima_baselines/cam_sima_test_snapshots/cam_ne3pg3_lunar_tides_snapshot_derecho_gnu_before_c20240801.nc' +ncdata_check = '/glade/campaign/cesm/community/amwg/sima_baselines/cam_sima_test_snapshots/cam_ne3pg3_lunar_tides_snapshot_derecho_gnu_after_c20240801.nc' +``` + +Bug gets extremely lucky (or maybe he's just really skilled) and there are no differences found in the atm.log file! + +## 9 - Bring back into CAM +Bug moves the CCPP-ized version of lunar_tides.F90 into the CAM source tree, updates `configure`, and runs CAM. He confirms that it runs and answers haven't changed. + +Bug commits his changes to his fork/branch of the three repos: CAM, CAM-SIMA, atmospheric_physics. + +- He opens a PR into atmospheric_physics (target: `development` branch), goes through the review process, adds a Changelog entry, and then commits the PR when approvals are received. He then makes a tag (incrementing the minor version). +- Once he has a tag, he opens PRs into CAM (target: `cam_development` branch) and CAM-SIMA (target: `development` branch) with the updated tag in `.gitmodules` (and code changes needed). When approvals are in and he gets the go-ahead to make a tag, he follows the procedures to make a CAM or CAM-SIMA tag. +- Once all tags are made, he checks off "lunar tides" as completed in the [spreadsheet](https://docs.google.com/spreadsheets/d/1_1TTpnejam5jfrDqAORCCZtfkNhMRcu7cul37YTr_WM/edit#gid=0). + +A single tear of joy navigates its way down Bug's weathered face. He has prevailed. He solemnly closes his computer and walks off, disappearing into the horizon. diff --git a/docs/design/ccpp-in-cam-sima.md b/docs/design/ccpp-in-cam-sima.md index 4802132..f1137ba 100644 --- a/docs/design/ccpp-in-cam-sima.md +++ b/docs/design/ccpp-in-cam-sima.md @@ -107,7 +107,7 @@ SDFs are located in the root directory of the repository and scheme source code The `diagnostics` directory contains all diagnostic schemes (the global ones used for state and tendency output, as well as the scheme-specific diagnostic schemes). -The `to_be_ccppized` directory contains physics schemes and utilties that have not been [CCPP-ized](../conversion/ccpp-conversion-guide.md), but were needed by an CCPP-ized scheme. +The `to_be_ccppized` directory contains physics schemes and utilties that have not been [CCPP-ized](../conversion/conversion-background.md), but were needed by an CCPP-ized scheme. The `utilities` directory contains schemes that are used regularly, such as tendency applicators and state converters. See [below](#state-and-tendency-variables) for more.