From a43c2cf9f4d34061a1760959268d66002ed13e29 Mon Sep 17 00:00:00 2001 From: Marlee Smith Date: Tue, 5 Nov 2024 09:50:26 -0700 Subject: [PATCH 1/9] Adding info about getting restarts from the gdex; removing info about script to copy restarts --- models/POP/readme.rst | 54 ++++--------------------------------------- 1 file changed, 4 insertions(+), 50 deletions(-) diff --git a/models/POP/readme.rst b/models/POP/readme.rst index 6ece91e7fa..e0fd9b4f34 100644 --- a/models/POP/readme.rst +++ b/models/POP/readme.rst @@ -83,6 +83,8 @@ Summary To use DART and CESM POP2 on NSF NCAR's supercomputer, you will need to complete the following steps. +#. Download an intial ensemble of POP2 restart files from the `NSF NCAR Geoscience + Data Exchange `_ #. Configure the scripts for your specific experiment by editing ``DART_params.csh``. #. Stage your initial ensemble using ``copy_POP_JRA_restarts.py``. @@ -114,53 +116,6 @@ in subdirectories that correspond releases of CESM. For example: contains scripts that should be used with CESM releases 2.1.0-2.1.3. -copy_POP_JRA_restarts.py -~~~~~~~~~~~~~~~~~~~~~~~~ - -This script stages an intial ensemble of POP2 restart files by copying files -from a prior experiment run by *Who Kim*. Thanks Who! - -These restart files can be used as an initial ensemble of model -states. The files are kept in a directory on GLADE that is owned by the Climate -and Global Dynamics (CGD) Ocean Section: - -.. code-block:: - - /glade/campaign/cgd/oce/people/whokim/csm/g210.G_JRA.v14.gx1v7.01 - -Unless you're already a member of the CGD Ocean Section, you must be granted -access to this directory by CISL. Use the `Service Desk -`_ to request permission. If -you're unable to get permission, contact DAReS staff for assistance by emailing -dart@ucar.edu. - -Filepaths beginning with ``/glade/campaign/*`` can't be accessed from NSF NCAR's -supercomputer nodes. You must log on to NSF NCAR's data visualization computer to -copy files from ``/glade/campaign/*``. - -This python script was created by *Dan Amrhein*. Thanks Dan! - -+-------------------------------+-----------------------------------------------------------+ -| Script name | Description | -+===============================+===========================================================+ -| ``copy_POP_JRA_restarts.py`` | This script copies restart files from the | -| | g210.G_JRA.v14.gx1v7.01 experiment that are saved in | -| | campaign storage. You must be granted access to the CGD | -| | Ocean Section campaign storage directory and be logged on | -| | to NSF NCAR's data visualization computer in order to run | -| | this script. The assignment of the ``stagedir`` variable | -| | in this script should match the assignment of the | -| | ``stagedir`` variable in ``DART_params.csh``. | -+-------------------------------+-----------------------------------------------------------+ - -In order to use this script, log in to NSF NCAR's data visualization computer and -use python to run the script. For example: - -.. code-block:: - - $ cd DART/models/POP/shell_scripts/cesm2_1 - $ python copy_POP_JRA_restarts.py - DART_params.csh ~~~~~~~~~~~~~~~ @@ -175,9 +130,8 @@ It is run by the setup scripts. | | that you need to set in order to build and run cases. You | | | must read this file carefully and configure the variables | | | to match your needs. The assignment of the ``stagedir`` | -| | variable in this script should match the assignment of | -| | the ``stagedir`` variable in | -| | ``copy_POP_JRA_restarts.py``. | +| | variable in this script should match the directory path | +| | where the restarts from the GDEX were downloaded/stored. | +---------------------+-----------------------------------------------------------+ Setup scripts From 8b62eac688dca6ed25c619fa5036b2086fe44471 Mon Sep 17 00:00:00 2001 From: Marlee Smith Date: Tue, 5 Nov 2024 09:59:37 -0700 Subject: [PATCH 2/9] Adding info about downloading and using SourceMods --- models/POP/readme.rst | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/models/POP/readme.rst b/models/POP/readme.rst index e0fd9b4f34..690122fdd7 100644 --- a/models/POP/readme.rst +++ b/models/POP/readme.rst @@ -134,6 +134,13 @@ It is run by the setup scripts. | | where the restarts from the GDEX were downloaded/stored. | +---------------------+-----------------------------------------------------------+ +One set of variables to make sure are set correctly in this script are related to +the SourceMods enable POP to recompute the barotropic velocity to prevent the +barotropic solver from crashing. You can find and download the SourceMods available +for POP-DART `here `_. Ensure that the +environment variable use_SourceMods is set to TRUE and the environment variable +SourceModDir matches the directory path where the SourceMods were downloaded/stored. + Setup scripts ~~~~~~~~~~~~~ From 74a241d6e28b8cad18e020e294016e902167bc5a Mon Sep 17 00:00:00 2001 From: Marlee Smith Date: Tue, 5 Nov 2024 10:15:29 -0700 Subject: [PATCH 3/9] Adding doc reference --- models/POP/readme.rst | 2 ++ 1 file changed, 2 insertions(+) diff --git a/models/POP/readme.rst b/models/POP/readme.rst index 690122fdd7..614b30f461 100644 --- a/models/POP/readme.rst +++ b/models/POP/readme.rst @@ -1,3 +1,5 @@ +.. _POP: + POP === From 7648e5a88fb1ba32b38a5ab389f1dc76f929463a Mon Sep 17 00:00:00 2001 From: Marlee Smith Date: Tue, 5 Nov 2024 11:38:21 -0700 Subject: [PATCH 4/9] Removing additional bit about copy_POP_JRA_restarts.py --- models/POP/readme.rst | 16 +--------------- 1 file changed, 1 insertion(+), 15 deletions(-) diff --git a/models/POP/readme.rst b/models/POP/readme.rst index 614b30f461..f050f6082e 100644 --- a/models/POP/readme.rst +++ b/models/POP/readme.rst @@ -89,7 +89,6 @@ the following steps. Data Exchange `_ #. Configure the scripts for your specific experiment by editing ``DART_params.csh``. -#. Stage your initial ensemble using ``copy_POP_JRA_restarts.py``. #. Run the appropriate DART setup script to create and build the CESM case. If the DART setup script runs to completion, it will print instructions to the @@ -137,7 +136,7 @@ It is run by the setup scripts. +---------------------+-----------------------------------------------------------+ One set of variables to make sure are set correctly in this script are related to -the SourceMods enable POP to recompute the barotropic velocity to prevent the +the SourceMods, which enable POP to recompute the barotropic velocity to prevent the barotropic solver from crashing. You can find and download the SourceMods available for POP-DART `here `_. Ensure that the environment variable use_SourceMods is set to TRUE and the environment variable @@ -213,19 +212,6 @@ integration. Other files needed for assimilation ----------------------------------- -Creating an initial ensemble -~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -Karspeck et al. (2013) [3]_ find that an ensemble of 1 January model states -selected from a multi-decade free-running integration of POP2 can be used as an -initial ensemble. - -If you have access to CGD's Ocean Section directory on ``/glade/campaign`` you -can use the `copy_POP_JRA_restarts.py`_ script to stage a collection of POP -restart files from Who Kim's multi-century ``g210.G_JRA.v14.gx1v7.01`` -experiment to serve as an initial ensemble. This experiment uses the JRA-55 -dataset for atmospheric forcing (Tsujino et al. 2018 [4]_). - Observation sequence files ~~~~~~~~~~~~~~~~~~~~~~~~~~ From ca2238d83fd580ab6b47fbe4c658b0536560626a Mon Sep 17 00:00:00 2001 From: Marlena Smith <44214771+mjs2369@users.noreply.github.com> Date: Wed, 6 Nov 2024 13:10:49 -0700 Subject: [PATCH 5/9] Update models/POP/readme.rst to have better description of SourceMods Co-authored-by: Helen Kershaw <20047007+hkershaw-brown@users.noreply.github.com> --- models/POP/readme.rst | 11 ++++++----- 1 file changed, 6 insertions(+), 5 deletions(-) diff --git a/models/POP/readme.rst b/models/POP/readme.rst index f050f6082e..6d8e2c87cc 100644 --- a/models/POP/readme.rst +++ b/models/POP/readme.rst @@ -135,12 +135,13 @@ It is run by the setup scripts. | | where the restarts from the GDEX were downloaded/stored. | +---------------------+-----------------------------------------------------------+ -One set of variables to make sure are set correctly in this script are related to -the SourceMods, which enable POP to recompute the barotropic velocity to prevent the +SourceMods are required to enable POP to recompute the barotropic velocity to prevent the barotropic solver from crashing. You can find and download the SourceMods available -for POP-DART `here `_. Ensure that the -environment variable use_SourceMods is set to TRUE and the environment variable -SourceModDir matches the directory path where the SourceMods were downloaded/stored. +for POP-DART `here `_. + +Put the SourceMods in your home directory: + +~/${cesmtag}/SourceMods Setup scripts ~~~~~~~~~~~~~ From 2e3de8fd452a6cb4ccf33ac3683d60869727581a Mon Sep 17 00:00:00 2001 From: Marlee Smith Date: Wed, 6 Nov 2024 13:57:48 -0700 Subject: [PATCH 6/9] Re-including the reference to Karspeck's paper regarding generating an initial ensemble --- models/POP/readme.rst | 12 ++++++++++++ 1 file changed, 12 insertions(+) diff --git a/models/POP/readme.rst b/models/POP/readme.rst index 6d8e2c87cc..1c56c5dc7e 100644 --- a/models/POP/readme.rst +++ b/models/POP/readme.rst @@ -213,6 +213,18 @@ integration. Other files needed for assimilation ----------------------------------- +Creating an initial ensemble +~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Karspeck et al. (2013) [3]_ finds that an ensemble of 1 January model states +selected from a multi-decade free-running integration of POP2 can be used as an +initial ensemble. + +You can access a collection of POP restart files from Who Kim's multi-century +``g210.G_JRA.v14.gx1v7.01`` experiment to serve as an initial ensemble in the +`NSF NCAR Geoscience Data Exchange `_. This +experiment uses the JRA-55 dataset for atmospheric forcing (Tsujino et al. 2018 [4]_). + Observation sequence files ~~~~~~~~~~~~~~~~~~~~~~~~~~ From 8f906bd15cf10725b053a747ea198fb9ea9f0c10 Mon Sep 17 00:00:00 2001 From: Marlee Smith Date: Thu, 7 Nov 2024 13:34:16 -0700 Subject: [PATCH 7/9] Removing now redundant script to copy the initial ensemble from a specific directory on glade (ensemble now available on GDEX) --- .../cesm2_1/copy_POP_JRA_restarts.py | 148 ------------------ 1 file changed, 148 deletions(-) delete mode 100644 models/POP/shell_scripts/cesm2_1/copy_POP_JRA_restarts.py diff --git a/models/POP/shell_scripts/cesm2_1/copy_POP_JRA_restarts.py b/models/POP/shell_scripts/cesm2_1/copy_POP_JRA_restarts.py deleted file mode 100644 index d0667833a2..0000000000 --- a/models/POP/shell_scripts/cesm2_1/copy_POP_JRA_restarts.py +++ /dev/null @@ -1,148 +0,0 @@ -from shutil import copyfile, copyfileobj -import getpass -import os -import sys -import glob -from pathlib import Path - -# -----------------------------------------------------# -# Define needed function # -# -----------------------------------------------------# - - -def unzipNsave(old, new): - ''' - Saves *.gz file in old as unzipped file in new - ''' - import gzip - - with gzip.open(old, 'rb') as f_in: - with open(new, 'wb') as f_out: - copyfileobj(f_in, f_out) - - -# -----------------------------------------------------# -# Main body of the script # -# -----------------------------------------------------# - -# Get username # -# -----------------------------------------------------# - -USER = getpass.getuser() - -# Create stage directory if it doesn't exist # -# -----------------------------------------------------# - -case_name = 'g210.G_JRA.v14.gx1v7.01' -stagedir = '/glade/scratch/' + USER + '/' + case_name + '/rest/2010-01-01-00000/' -Path("stagedir").mkdir(parents=True, exist_ok=True) - -# First load from the directory of MONTHLY saves # -# -----------------------------------------------------# - -load_path = '/glade/campaign/cgd/oce/people/whokim/csm/' + case_name + '/rest/' - -years = list(range(41, 202, 10))+list(range(209, 251)) -# For debugging: -# years = list(range(31,32)) -inds = list(range(1, len(years)+1)) - -print('Copying from ' + load_path + ' to ' + stagedir) - -for ii, year in enumerate(years): - - # Zero pad the year and index string so that there are 4 digits - zero_filled_year = str(year).zfill(4) - zero_filled_ii = str(inds[ii]).zfill(4) - - print('Copying year ' + str(year) + ' to ensemble number ' - + zero_filled_ii) - - yearpath = load_path + '/' + zero_filled_year + '-01-01-00000/' - - # Define file paths - popold = yearpath + case_name + '.pop.r.' + zero_filled_year \ - + '-01-01-00000.nc' - ovfold = yearpath + case_name + '.pop.ro.' + zero_filled_year \ - + '-01-01-00000' - ciceold = yearpath + case_name + '.cice.r.' + zero_filled_year \ - + '-01-01-00000.nc' - - popnew = stagedir + case_name + '.pop_' + zero_filled_ii \ - + '.r.2010-01-01-00000.nc' - ovfnew = stagedir + case_name + '.pop_' + zero_filled_ii \ - + '.ro.2010-01-01-00000' - cicenew = stagedir + case_name + '.cice_' + zero_filled_ii \ - + '.r.2010-01-01-00000.nc' - - # Copy over - copyfile(popold, popnew) - copyfile(ovfold, ovfnew) - copyfile(ciceold, cicenew) - - -# Then load the rest from DAILY saves -# Some years (251-262) must be decompressed -# -----------------------------------------------------# - -case_name_dr = 'g210.G_JRA.v14.gx1v7.01.dr' -load_path = '/glade/campaign/cgd/oce/people/whokim/csm/' + case_name_dr + '/rest' - -ly = len(years) -years2 = list(range(251, 272)) - -# Start from where we left off with the daily saves with inds -inds = list(range(ly+1, ly+len(years2)+1)) - -print('Copying from ' + load_path + ' to ' + stagedir) - -for ii, year in enumerate(years2): - - # Zero pad the year and index string so that there are 4 digits - zero_filled_year = str(year).zfill(4) - zero_filled_ii = str(inds[ii]).zfill(4) - - yearpath = load_path + '/' + zero_filled_year + '-01-01-00000/' - - # Define file paths - popold = glob.glob(yearpath + case_name_dr + '.pop.r.' + zero_filled_year - + '-01-01-00000.nc*')[0] - ovfold = glob.glob(yearpath + case_name_dr + '.pop.ro.' + zero_filled_year - + '-01-01-00000*')[0] - ciceold = glob.glob(yearpath + case_name_dr + '.cice.r.' + zero_filled_year - + '-01-01-00000.nc*')[0] - - popnew = stagedir + case_name + '.pop_' + zero_filled_ii \ - + '.r.2010-01-01-00000.nc' - ovfnew = stagedir + case_name + '.pop_' + zero_filled_ii \ - + '.ro.2010-01-01-00000' - cicenew = stagedir + case_name + '.cice_' + zero_filled_ii \ - + '.r.2010-01-01-00000.nc' - - # Copy over, decompressing as necessary - - if popold[-3:] == '.gz': - print('Decompressing and copying year ' + str(year) - + ' to ensemble number ' + zero_filled_ii) - else: - print('Copying year ' + str(year) + ' to ensemble number ' - + zero_filled_ii) - - if popold[-3:] == '.gz': - unzipNsave(popold, popnew) - else: - copyfile(popold, popnew) - - # OVF - if ovfold[-3:] == '.gz': - unzipNsave(ovfold, ovfnew) - else: - copyfile(ovfold, ovfnew) - - # CICE - if ciceold[-3:] == '.gz': - unzipNsave(ciceold, cicenew) - else: - copyfile(ciceold, cicenew) - -print('Completed') From 55a30c2b3d69857e405a7d7d8465d700dac82c10 Mon Sep 17 00:00:00 2001 From: Helen Kershaw Date: Fri, 8 Nov 2024 12:38:30 -0500 Subject: [PATCH 8/9] typo --- models/POP/readme.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/models/POP/readme.rst b/models/POP/readme.rst index 1c56c5dc7e..3726297166 100644 --- a/models/POP/readme.rst +++ b/models/POP/readme.rst @@ -216,7 +216,7 @@ Other files needed for assimilation Creating an initial ensemble ~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -Karspeck et al. (2013) [3]_ finds that an ensemble of 1 January model states +Karspeck et al. (2013) [3]_ find that an ensemble of 1 January model states selected from a multi-decade free-running integration of POP2 can be used as an initial ensemble. From 9159690c2ca0a2cf40596bef1737e2b9cc181d8f Mon Sep 17 00:00:00 2001 From: Helen Kershaw Date: Fri, 8 Nov 2024 13:13:13 -0500 Subject: [PATCH 9/9] bump conf.py and changelog for release --- CHANGELOG.rst | 13 +++++++++++++ conf.py | 2 +- 2 files changed, 14 insertions(+), 1 deletion(-) diff --git a/CHANGELOG.rst b/CHANGELOG.rst index b15c62115b..899a28cba6 100644 --- a/CHANGELOG.rst +++ b/CHANGELOG.rst @@ -22,6 +22,19 @@ individual files. The changes are now listed with the most recent at the top. +**November 8 2024 :: POP initial ensemble available from GDEX. Tag v11.8.4** + +Documentation update: + + - POP initial ensemble available from GDEX. + +Bug-fixes: + + - Removed unnecessary loops around calendar types in time_manager_mod. + - Removed unused routine from normal_distribution_mod which case giving compilation warnings. + - Replaced broadcast_minmax calls with all_reduce_min_max. + + **November 7 2024 :: MPI window memory reduction. Tag v11.8.3** - Removes unnecessary copy of state into mpi window. diff --git a/conf.py b/conf.py index 23c29119d6..60e5f6f4cd 100644 --- a/conf.py +++ b/conf.py @@ -21,7 +21,7 @@ author = 'Data Assimilation Research Section' # The full version, including alpha/beta/rc tags -release = '11.8.3' +release = '11.8.4' root_doc = 'index' # -- General configuration ---------------------------------------------------