CTSM5.3.0: New raw pft/lai/glc-behavior/soilcolor/fire datasets

[slevis-lmwg]

Description of changes

Updating gen_mksurfdata_namelist.xml, initially with temporary paths to the new raw data.

New surface datasets because of updates in PFT/LAI/soil-color datasets, as well as two fire datasets (AG fire, pearland and the glacier behavior dataset. Also bring in an updated "PPE informed, hand-tuned" parameter file. And turn on Li2024 fire method. Also updates to allow creating transient landuse timeseries files going back to 1700. Fix an important bug on soil fields that was there since ctsm5.2.0. This results in mkaurfdata_esmf giving identical answers with a change in number of processors. Add in creation of ne0np4.POLARCAP.ne30x4 surface datasets.

Asides: Remove VIC fields from surface datasets and testing. Add version to the surface datasets. Update the fire emissions factor dataset for the fire emissions testing in CTSM. Remove the --hires_pft option from mksurfdata_esmf as we don't have the datasets for it.

TODOs:

• 1700 fsurdat for TRENDY2024 spinup and
• 1700-2023 landuse file for TRENDY2024 historical
• Run ctsm5.2 spinup and historical as the control and
• Compare ctsm5.3 (aka TRENDY2024) to ctsm5.2
• Bring in new dataset from Bill (see below)
• @samsrabin @lifang0209: new peatland and AG fire peak month datasets to @slevis-lmwg (will move to streams later on in the ctsm6.0.0 cycle)
• @adrifoster: Table new population stream file for now. Adrianna explains, "it's formatted oddly and would be a pain to deal with"
• Update gen_mksurfdata_namelist.xml accordingly
• Bring in new code changes from Bill (See below)
• Before making any ssp landuse/fsurdat files, confirm that @lawrencepj1 has updated the raw ssp data
• Remove the hires_pft option (will add back in in ctsm6.1 with hires datasets)
• Remove the -vic option and special handling in Makefile, as well as remove vic tests (see Deprecate and remove the VIC-Hydro option #2597)
• Existing "make all" seems to have worked after a few corrections; took 6 hrs to run but did not include ultra high res in this iteration
• Get new PtVg (noAnthro) raw data file from Peter
• Move all new files and update temporary file paths to their final locations, then ./rimport:
  • new param file(s), new GlacierRegion, fire, NOT population
  • raw lake and urban files 1700-1849
  • raw urban 2016-2023
  • raw lai, soil color
  • raw pft and PtVg
• Table Sam R.'s tweaked mesh file; the hypothesis did not work for now fsurdat: PCT_SAND, PCT_CLAY, ORGANIC differ with different PE layouts on derecho #2502
• Resolve all of the ctsm5.3.0 milestone issues (https://github.com/ESCOMP/CTSM/milestone/19)
• @ekluzek to merge Add version to srf datasets slevis-lmwg/ctsm#8 to this PR
• Assess if we need other 1979 surface datasets -- Erik: No, since 1979 is only used for FHIST compsets so only need a landuse.timeseries file that includes 1979.
• make all the datasets with a tag (branch_tags/ctsm5.3.n02_ctsm5.2.027) from this branch
  • Since PLUMBER2 is on main now, we'll also need to recreate these surface datasets; @slevis-lmwg tested neon_surf_wrapper and plumber2_surf_warpper manually, and they worked. Open new issue: Via make all I get this error because we need to define an abs. path to the following file:
    No such file or directory: 'neon_sites_dompft.csv'
  • Make polarcap surface datasets in ctsm5.3.0, remove VR 1850 datasets #2720 Could also be completed after the 5.3 tag but ultimately we do want this in make all
  • Ability to run TRENDY type simulations on mainline CTSM  #2675 Same comment: could complete after 5.3 but want in make all
• @slevis-lmwg copy to final locations and ./rimport new fsurdat/landuse files (finidat done)
  • ./rimport -list ctsm5.3.plumber2.list
  • ./rimport -list ctsm5.3.neon.list
  • ./rimport -list ctsm5.3.synth.list
  • ./rimport -list ctsm5.3.list (after removing four VR grid 1850 fsurdat files because not needed); there's a list of files that failed due to large size and a corresponding issue New filesize limit for svn inputdata that restricts our larger ctsm5.3.0 files being rimported #2751
  • copy to izumi's /inputdata area the files that failed to rimport (See New filesize limit for svn inputdata that restricts our larger ctsm5.3.0 files being rimported #2751)
  • @slevis-lmwg ./rimport new fsurdat/landuse that were generated Sep 12 and later
• @samsrabin updates hillslope datasets Update hillslope testing fsurdat to 5.3 slevis-lmwg/ctsm#12
• @ekluzek updates namelist defaults of fsurdat/landuse/finidat files for all resolutions, compsets, etc to their final location (this includes adding the new default grids that appear in the [spreadsheet] (https://docs.google.com/spreadsheets/d/1Osq56e423CF107zhoNQ0VS7-iH_JXLF9AtCvBdXyfJ4/edit?gid=584342654#gid=584342654): Update surface datasets in namelist defaults slevis-lmwg/ctsm#10
  Date-stamps needing replacement in namelist_defaults:
  • fsurdat c240206 (NEON)
  • fsurdat/landuse c240216 (except the LUH2 file)
  • fsurdat/landuse c240221
  • fsurdat/landuse c240425 (T42, VR)
  • fsurdat c240613 (cidadinho) (Move this and Smallville to synthetic data directory?) Move Smallville data to ...clm2/testdata/ #2515
  • finidat: same list of finidats changed recently here
  • hillslope testmods Update hillslope testing fsurdat to 5.3 slevis-lmwg/ctsm#12
• @slevis-lmwg try clm_pymods: did not need to update datasets in /python/ctsm/test/testinputs
• @ekluzek fix testlist because aux_clm fails
• Erik fix fire-emis with Sam R.'s suggestion
• 2 sparse-matrix fails
  • @slevis-lmwg investigate
  • @ekluzek will mark mimics-matrix test as expected fail / create issue for it New CN matrix fails with single point sites with the new ctsm5.3 datasets. #2780
  • @ekluzek will retest the other fail with an update to namelist_defaults (now works SMS_Ld10_D_Mmpi-serial.CLM_USRDAT.I1PtClm60Fates.derecho_intel.clm-FatesFireLightningPopDens--clm-NEON-FATES-NIWO)
• @slevis-lmwg investigate/fix LII tests (izumi OK, repeat on derecho)
  • 2 new finidats to namelist_defaults
  • @slevis-lmwg ./rimport 2 new finidats
• @slevis-lmwg run aux_clm on derecho, address failures, including generating new finidat files with use_init_interp as needed
• @ekluzek fix the namelist tester fail
• @slevis-lmwg run aux_clm, ctsm_sci, fates, mosart, rtm on derecho
• @slevis-lmwg run aux_clm, fates, mosart on izumi
• See the checklist from 5.2, make sure we covered everything here too.
• @ekluzek Updates to the ctsm5.3.0 ChangeLog and README files slevis-lmwg/ctsm#11 @slevis-lmwg start reviewing it (needs more work)
  • Erik will add a new file WhatsNewInCTSM5.3.md with a short description of what was updated since ctsm5.2
• @ekluzek Remove mkprocdata_map slevis-lmwg/ctsm#14
  • @slevis-lmwg can review/approve (it's ready)
• Final check of procedures for minor version update: (https://github.com/ESCOMP/CTSM/wiki/Minor-Version-Update-Protocols)

Specific notes

Contributors other than yourself, if any:
@lawrencepj1 @ekluzek @wwieder @adrifoster @samsrabin @billsacks @lifang0209 @linniahawkins @olyson

CTSM Issues Fixed (include github issue #):
Fixes #2723
Fixes #2720
Partly addresses #2752
Partly addresses #2672
Fixes #2452
Fixes #2570
Fixes #2618
Fixes #423
Fixes #2734
Fixes #2744
Fixes #2502
Resolves #2748

Are answers expected to change (and if so in what way)?
Yes, in generated fsurdat/landuse files.

Any User Interface Changes (namelist or namelist defaults changes)?
gen_mksurfdata_namelist.xml points to new datasets.

Testing performed, if any: will do release testing
In the first iteration generated new fsurdat/landuse files and ran the model (NCAR/LMWG_dev#59).
Second iteration in progress.
Now have updated datasets and verified with the build-namelist tester

[slevis-lmwg]

@lawrencepj1 I requested a review from you, but we could go over this together at our next "pft/landuse tools" meeting.

[slevis-lmwg]

UPDATE appears in #2570.

Also address #2617 with this PR.

[slevis-lmwg]

@slevis-lmwg @lawrencepj1 @samsrabin @ekluzek @linniahawkins discussed next steps for ctsm53.
@samsrabin took notes here:
https://docs.google.com/document/d/1-2LEjIhbs4j4NoythdLqdJ9A1pcNt4UJ_tyda2rrQWc

[slevis-lmwg]

Reviewing my own PR to include some reminders.

[lawrencepj1]

Hi @slevis-lmwg

Just to let you know that the raw PFT data for the SSPs rebased to TRENDY 2024 starting in year 2015 are available at:

/glade/campaign/cesm/development/lmwg/landuse_source_data/CTSM53RawData
globalctsm53TRSSP119Deg025_240728
globalctsm53TRSSP126Deg025_240728
globalctsm53TRSSP245Deg025_240728
globalctsm53TRSSP370Deg025_240728
globalctsm53TRSSP460Deg025_240728
globalctsm53TRSSP585Deg025_240728

I have checked them and everything looks good.
Peter

[slevis-lmwg]

Thank you @lawrencepj1

[ekluzek]

• @billsacks will give @slevis-lmwg a dataset for glacier regions by the end of this week.
• @billsacks will get code changes to @slevis-lmwg mid next week he will push directly to the branch

[slevis-lmwg]

Latest testing

izumi
PASS mosart
OK fates
OK aux_clm

derecho
PASS make black and make lint
PASS ./run_ctsm_py_tests -s and ./run_ctsm_py_tests -u
PASS ./build-namelist_test.pl test
OK mosart
OK rtm
OK aux_clm
OK fates
OK ctsm_sci (now in tests_0924-173947de, see notes below), RXCR* test in progress before baseline will be ready

I use this to check for failures in aux_clm:
./cs.status.fails | grep -v PASS | grep -v NLCOM | grep -v '28: DIF' | grep -v "fireemis' does not exist"

[ekluzek]

Based on the previous work and the commit I made last night. I expect almost all testing will be as expected, other than possibly a couple 1979 tests. We can resubmit those 1979 tests that fail. If we have any other failing tests we can create an issue and move on.

The other thing is to finish the change files.

In talking with @briandobbins and @wwieder they would like us to make a tag today.

[slevis-lmwg]

Failures that we need to open issues for.

1. DONE @slevis-lmwg will open issue for aux_clm tests that fail in the RUN phase because they need c13 in finidat

SMS_Lm1.f10_f10_mg37.I1850Clm50BgcCropCmip6waccm.derecho_gnu.clm-basic
ERS_Ly3.f10_f10_mg37.I1850Clm50BgcCropCmip6.derecho_intel.clm-basic
SMS_Ld2_D_PS.f09_g17.I1850Clm50BgcCropCmip6.derecho_intel.clm-basic_interp

2. DONE @ekluzek will open issue for ctsm_sci tests that give this error CLMBuildNamelist::setup_cmdl_fire_light_res() : fire_method is being set while use_cn and use_fates are both false in the SHAREDLIB_BUILD phase:

SMS_Ld5.f09_g17.IHistClm50Sp.derecho_intel.clm-nofire
SMS_Ld5.f09_g17.IHistClm60Sp.derecho_intel.clm-nofire
SMS_Ld5.f19_g17.IHistClm50Sp.derecho_intel.clm-nofire
SMS_Ld5.f19_g17.IHistClm60Sp.derecho_intel.clm-nofire

3. DONE @ekluzek will open issues for fates test that failed with error build-namelist failed rc=255, err=Unknown option: i-clm_demand, Unknown option: flanduse_timeseries in the SHAREDLIB_BUILD phase:

ERS_D_Ld30.f45_f45_mg37.I2000Clm50FatesCruRsGs.derecho_intel.clm-FatesColdLandUse
ERS_D_Ld30.f45_f45_mg37.I2000Clm50FatesCruRsGs.izumi_nag.clm-FatesColdLandUse

4. DONE @slevis-lmwg will correct typo in Expected Failures to Ld30 for this fates test that fails in the RUN phase for an unknown reproducible reason BUT passes on derecho:
   ERS_D_Ld30.f45_f45_mg37.I2000Clm50FatesCruRsGs.izumi_nag.clm-FatesColdLUH2
5. DONE @slevis-lmwg will resubmit izumi tests not finding finidat

SMS_D.f10_f10_mg37.I2000Clm60BgcCrop.izumi_gnu.clm-crop
SMS_D.f10_f10_mg37.I2000Clm60BgcCrop.izumi_intel.clm-crop
SMS_D.f10_f10_mg37.I2000Clm60BgcCrop.izumi_nag.clm-crop
SMS_D_Ld3.f10_f10_mg37.I2000Clm60Bgc.izumi_nag.clm-HillslopeD
SMS_D_Ld65.f10_f10_mg37.I2000Clm60BgcCrop.izumi_nag.clm-FireLi2024GSWP
PEM_Ld1.f10_f10_mg37.I2000Clm60BgcCrop.izumi_intel.clm-till--clm-matrixcnOn
PEM_Ld1.f10_f10_mg37.I2000Clm60BgcCrop.izumi_intel.clm-till
ERP_D_P48x1.f10_f10_mg37.IHistClm60Bgc.izumi_nag.clm-decStart--clm-matrixcnOn_ignore_warnings
ERP_D_P48x1.f10_f10_mg37.IHistClm60Bgc.izumi_nag.clm-decStart
SMS_Ld5_D_P48x1.f10_f10_mg37.IHistClm60Bgc.izumi_nag.clm-decStart
SMS_Ld10_D_Mmpi-serial.CLM_USRDAT.I1PtClm60Bgc.izumi_nag.clm-default--clm-NEON-HARV

[ekluzek]

I figure I spent around a week to 10 days of engineering time here on ctsm5.3.0.