Skip to content

2023 Software Engineering meeting notes

Jump to bottom Edit New page
Gregory Lemieux edited this page Nov 13, 2023 · 245 revisions

November 13, 2023

Upcoming meetings and events

  • Meetings, conferences, seminars, etc.
  • Machine outages
    • CISL: none planned
    • NERSC: 16 Nov, perlmutter
    • OLCF: none planned

PR board status

Latest issues and discussions

Calibration progress update

Two-stream update

Walk-on Topics

November 06, 2023

Upcoming meetings and events

  • Meetings, conferences, seminars, etc.
  • Machine outages
    • CISL: none planned
    • NERSC: 16 Nov, perlmutter
    • OLCF: 7 Nov, summit

PR board status

  • Updates
    • Notice: Greg added upcoming API plan to the User's Guide HLM table
      • Developers can use this to see what the integration plan is a little more cleanly than the PR board.
    • Het resp fix (1114): Reviewed and approved. CTSM-side API update testing complete. E3SM-side testing underway.
      • Issue is that the bc_out%ema_npp was not getting updated in time for the hi frequency output as the model restart and the fates update of ema_npp was not in sync (due to ema_npp update only happening daily).
      • Note this is being rolled into E3SM-side seed dispersal PR.
  • New

Latest issues and discussions

  • None to discuss

Calibration progress update

  • Jessie: tried pulling in Rosie's crop updates (PR 817), two-stream, and Medlyn parameters. Got a bunch of failures that she's working on fixing before comfortable with the runs.
    • Rosie believes she was using Ball-Berry and not Medlyn. Adrianna running default Medlyn parameters?
      • Adrianna: not using default. Using Charlie's recommended from another reference. Also saw same crashes that Jessie is seeing.
        • Discussion of low GPP in Adrianna's recent results.
    • Note: changes in landuse v2 reformulates the no-comp initialization logic

Two-stream update

  • Jenny triggered an error: check on solver results.
    • One particular pft was causing the issue
    • Ryan looked into LAPACK to understand what could be causing the issue (i.e. what is acceptable precision for matrix inversion?)
      • Issue turned out to be if two differencing terms (beam and diffusion) in eq 26 of Ryan's manuscript is near zero, values become huge. Ryan to introduce check and small perturbation to avoid this state.

Walk-on Topics

  • Adrianna: fire model refactoring discussion. Proposing creating a patch fuel data class.
    • Would take in litter and other states and calculate/store fuel fraction, bulk density, etc.
    • Has unit tests and functional tests :D!
    • Consensus is yes.

October 30, 2023

Upcoming meetings and events

PR board status

  • Integrated

    • Cross grid seed dispersal is in!

  • Bottlenecks:

    • Demography initialized gridded testing would require:
      • #1093 (allometry updates)
      • #1104 (cleanup initialization file format)
      • #1040 (v1 landuse change?)

  • Updates

    • Solar Radiation Unit Tests #1034
    • Maybe adding bleaf allometry updates to #1093 ?
    • Active crown fire #1108
    • Long-run restarts #1098

  • New

Latest issues and discussions

Calibration progress update

Walk-on Topics

  • Rosie: spectral element grids continued?
  • Adrianna: LAI difference continued?
  • Jennifer: FATES overview at Biogeosciences ESA?

October 23, 2023

Upcoming meetings and events

  • Meetings, conferences, seminars, etc.
  • Machine outages
    • CISL: Oct 23 - 27, all machines
    • NERSC: None planned
    • OLCF: None planned

PR board status

  • Integrated
  • Updates
    • Two stream: Marcos and Adrianna providing reviews.
      • Aside: Rosie noted that Mariana is looking to do some performance testing of fates. Rosie suggested she waited until this one was integrated.
        • Ryan wants to create a PR (HLM-side) to add namelist variable to tailor the set of history variable "classes" (i.e. site versus muliplexed). This would avoid calling the subroutines that populate the particular history variables.
    • tree pfts update (1093): Jessie reviewed the impact of fates_allom_blca_expnt_diff. Including this, which adjusts LAI wrt size, improves ILAMB results. Let move this in.
  • New
    • clean inventory init: removes extra columns and adds ed2 to fates python tool
      • Put c_height back in.
        • Aside: Marcos noted that dbh -> height isn't perfectly reversible
      • Consensus is that we should just remove the ED2 compatible format instead of keeping both FATES-only and ED compatible.
      • We should try running this globally to get a sense of the common assumption that this is going to be very slow.

Latest issues and discussions

Calibration progress update (Adrianna, Jessica, Rosie)

  • Adrianna: submitted min/max one-at-a-time run. Some of them crashed. Investigating.

Walk-on Topics

  • Rosie: spectral element grids. Does anyone know if CLM run on different grids.
  • Adrianna: LAI difference
    • Charlie: bigleaf and fates have differences in where they are being run for antarctic, greenland, and some other minor area. FATES is nan in the ice sheets.
  • Charlie: fates_max_nocomppft_landuse_vector discussion: we need to manage that we may have more pfts than we have patches available for a given landuse type. This new parameter would allow user to control this.
    • We define a maximum number of patches by landuse type. The new parameter would tell fates how many of the top pfts to prioritize to meet this limit.

October 16, 2023

Upcoming meetings and events

  • Meetings, conferences, seminars, etc.
  • Machine outages
    • CISL: Oct 23 - 27, all machines
    • NERSC: None planned
    • OLCF: None planned

PR board status

  • Integrated
  • New
    • ERS fix (1098): needs review, fast track fix. Ask Matvey/Mariana to confirm?
      • ERS fix API update (2199): updates ed_update_site call, to pass a restart flag
        • Ryan wants to use this API update to add the first test that uses an initiliazation file to provide structure
    • param I/O (1096): Is John ready for review?
      • Adrianna: Yes. This is coming in a first in a series of changes.
  • Updated
    • Two-stream (1036): needs review
      • Reach out to Anthony
    • tree pfts update (1093): under review
      • Ryan: should we notify the community?
      • Aside: Rosie suggested that we add an agenda item once a monthly to pull together items for our "fates digest" monthly email. This would be a good candidate for this.
        • This was a previous discussion that we had regarding how to update fates users of software updates
    • LUH2 v1 (1040): reverted back to original terminate_patches method
    • cross-grid seed (1005): API-side test changes need update per CTSM#2151

Latest issues and discussions

Calibration progress update (Adrianna, Jessica, Rosie)

  • Adrianna: struggling with getting sparse grid to work with NUOPC. Sam Levis helping out.
    • Ryan to apply two-stream to another branch to help Adrianna to move forward around the NUOPC issue.

Walk-on Topics

  • Rosie: Low LAI in tropics discussion. What are our next steps from here (assuming GPP and carbon use efficiency is in the ballpark)
    • Jessie/Rosie: Why doesn't increasing root longevity improve this?
    • Charlie: should we try decoupling root/leaf optimization (in trimming)?

October 9, 2023

Upcoming meetings and events

  • Meetings, conferences, seminars, etc.
    • FATES modeling meeting: Thursday October 12 - Ryan Knox and Adrianna Foster, Two-stream update and ILAMB results
  • Machine outages
    • CISL: Oct 23 - 27, all machines
    • NERSC: Tue Oct 10, perlmutter, Wednesday Oct 4th, NERSC jupyter hub
    • OLCF: None planned

PR board status

  • New
    • 1093 (tree default allometry update)
      • Charlie: maybe we need to be running with more patches to to smooth out
      • Consensus is that this should become the default param values (based on data, no worse than existing set)
      • Erik: does FATES plan on holding multiple parameter files aside from the default on main?
        • Consensus seems to be our goal of building param files on the fly would cover regional or run-mode specific variations of the param files
  • Updated

Latest issues and discussions

Calibration progress update (Adrianna, Jessica, Rosie)

  • See above PR 1093 discussion
  • Discussion of next steps wrt turnover times

Walk-on Topics

  • Rosie: individual-based version of fates? Ratju working on detailed fire model and curious about if this is a possibility with a future fates version. How difficult would this be?
    • Charlie: what would they do about spatial ambiguity?
    • Marcos: having an individual based model option would be helpful in implementing new models in fates

October 2, 2023

Upcoming meetings and events

  • Meetings, conferences, seminars, etc.
    • FATES modeling meeting: Thursday October 12 - Ryan Knox and Adrianna Foster, Two-stream update and ILAMB results
    • Jessie and Greg starting up regular meetings for the upcoming fates tutorial
  • Machine outages
    • CISL: Oct 23 - 27, all machines
    • NERSC: Tue Oct 10, perlmutter, Wednesday Oct 4th, NERSC jupyter hub
    • OLCF: Tue Oct 3, frontier

PR board status

Latest issues and discussions

Calibration progress update (Adrianna, Jessica, Rosie)

  • Jessie: has a boreal forest that looks good now. Currently has a run spinning up that has good ILAMB results so far. Can share results next week.
    • How do we decide what species are representative of a particular pft?
      • Looking at extra tropical trees, it suggests we may want more pfts (e.g. late-succesional deciduous, early-succesional deciduous).
      • As first pass, update since this is an improvement (provinence-wise and results-wise)
        • Potential future work: Weighting by abundance or iding species that are representative for the given pft?
  • Adrianna: kicking off one at a time min/max case with two-stream branch.

Walk-on Topics

September 25, 2023

Upcoming meetings and events

  • Meetings, conferences, seminars, etc.
  • Machine outages
    • CISL: Oct 23 - 27, all machines
    • NERSC: Thu September 28, 6:00 AM - 22:00 PDT, perlmutter
    • OLCF: none planned

PR board status

  • Integrated
    • fates-docs integrations and configuration yaml file
  • Updated
    • Greg to close out ctsm-side seed dispersal updates for Adrianna to review
    • Ryan to conduct review of LandUse v1
    • Jessie's PRs to be tested this week
    • Marcos getting back to review of 2-stream

Latest issues and discussions

Calibration progress update (Adrianna, Jessica, Rosie)

  • Jessie: Getting boreal forest with new parameter file, too much boreal now
    • Also tested BAD allometric updates
    • Seeing carbon balance error 42 years into run

Walk-on Topics

  • Rosie: What needs to be done to bring ctsm-fates nutrients up to parity with elm-fates?
    • Ryan: Root uptake and acquisition
      • hoping for someone on the ctsm-side to this as a way to develop familiarity and experience with ctsm-fates nutrients
    • Rosie to look at making a ctsm issue if one does not already exist

September 18, 2023

Check-in on format of this meeting

  • General balance of time between reporting and discussing
    • Adrianna: all for optimizing meeting time and reducing total hours.
    • Jennifer: Thinking about wants from an E3SM liaison standpoint.
      • Maybe including an ELM update for FATES team?
    • Marcos: as we bring new stuff on, rotate things across weeks as opposed to trying to put into a single meeting each week
  • How to format CPU going forward?
    • Jennifer: would like to keep there as an optional check-in.
  • PR Board Status plan to optimize
  • How to create opportunities for everyone to guide topics (more open forum time, check-in at beginning, query email?)
  • Should we have different formats? Long-term planning meeting once a month?

Upcoming meetings and events

  • Meetings, conferences, seminars, etc.

    • ILMF shareable modules webinar: Thursday September 21, 08:00 AM PDT (15:00 UTC)
    • FATES modeling meeting: Thursday September 28 - Chaopeng Shen and Doaa Aboelyazeed
      • Adrianna will reach out to solicit a talk in late Oct/early Nov on
      • Ryan and Adrianna to team up on a two-stream talk for October
      • Ask Rosie to reach out to Eva or Lasse about presenting Land Site Platform talk

  • Machine outages

    • CISL: none planned
    • NERSC:
      • Perlmutter: Wed September 20, 6:00 AM - 22:00 PDT
      • Perlmutter: Thu September 28, 6:00 AM - 22:00 PDT
    • OLCF: none planned

PR board status

  • Total number - 14 open - 6 stale WIPs - 4 are active but not ready (2Stream,Site-refactor,rad unit tests,Ryan’s gnu ctsm debug fix) - 4 are moving through integration pipeline ( Jessie’s no-comp error fix, luh2 data tool, LU v1, and seed dispersal)
    • Big items - LUH2, seed dispersal, 2-stream, unit tests
      • Ryan and Adrianna to meet to discuss 2-stream unit test
    • Refactors - site refactors
    • Bug Fixes - Jessie no comp init fix
    • Order - Jessie’s, Cross grid and LUH2
  • Author feedback needs or a hold-up
  • Reviewer feedback needs hold-up - none
  • Bottlenecks - none
  • Long in queue - site refactor?
    • Adrianna is planning on closing this out and create a new one in the future. LUH2 and seed dispersal creates lots of conflicts and expectation is that it will likely be easier to start anew.
  • Who needs help?
  • Updated

Latest issues and discussions

Calibration progress update (Adrianna, Jessica, Rosie)

  • Jessie: seeing issues with boreal trees. Using Rosie's parameter file
  • Arianna: latest ILAMB updates with two stream. Encouraging results. Albedo bias is better. Discussion of next steps. Medlyn parameter update. Use atkins.
    • Jennifer to create PR with daylight parameter code.
  • [ ]

Walk-on Topics

September 11, 2023

Upcoming meetings and events

  • Meetings, conferences, seminars, etc.

  • Machine outages

Landuse x PFT static file discussion (Erik, Greg, Charlie)

  • Discussed potential switch to streams infrastructure usage in the long term. For now, use simple scheme to allow fates Land Use v2 development to move forward.

PR board status

Latest issues and discussions

Calibration progress update (Adrianna, Jessica, Rosie)

  • Deferred

Walk-on Topics

  • None

September 4, 2023

No meeting due to federal holiday

August 28, 2023

Upcoming meetings and events

PR board status

Latest issues and discussions

E3SM restart issues (Jessie)

  • E3SM commit 0777e39115 on April 11th onwards introduces EAMXX. From that point onwards Jessie was finding restarts were not working.
    • Removing ./xmlchange DOUT_S=TRUE (short term archiving enabled) from script solved the restart issue

FATES paleo (Adrianna)

  • Sophia Macarewich running CLM-FATES Amazon single site runs in three different scenarios including 1850, last glacial maximum and paleo
  • Greg suggested this might be good for a future FATES modeling meeting topic. Adrianna suggests maybe November. She is providing support to Sophia.
  • Charlie noted that Jenny has paleo background and might be a good person to connect with.

'Outside' contributions to CTSM-FATES API (Adrianna, Greg)

  • Discussion about having John Alex contribute improvements through refactors for FATES API
  • Team is comfortable with John contributing

Calibration progress update (Adrianna, Jessica, Rosie)

  • No major updates

Walk-on Topics

  • No topics

August 21, 2023

Upcoming meetings and events

PR board status

  • Integrated

    • 999 (CLM BGC compatibility)
      • Ryan would like to assess impact of reordering of root fraction calculations on the elm-fates results.
        • This would ideally be before the elm-side luh2 and seed dispersal pull requests (but after the refactor).

  • Updated

    • E3SM#5849 (refactor) tested and waiting on e3sm approval/integration
      • Greg to check in with Gautum and Peter S.
    • 1055 (too much canopy): added final regression testing forthcoming
    • 1040 (LUH2): updated patch insertion and fusion procedures
      • Fusion routine updated to fuse very small patches into largest patch of different landuse type if not possible to merge into same type
        • Need to assess if landuse transition rates could create small patches below threshold

  • Next tags

    • Given BGC-reorder check and refactor merge on e3sm side, move forward with seed dispersal and landuse on ctsm side.
      • Find reviewers for #1040. Retest seed dispersal and move this ahead of landuse in the ctsm queue.

Latest issues and discussions

Calibration progress update (Adrianna, Jessica, Rosie)

  • Adrianna: Ryan to call into calibration meeting this week, to talk about getting 2-stream ready for PPE.
  • Adrianna: Sparse grid for dominant pfts. What threshold should we use for "dominant?"

Walk-on Topics

  • Greg to create an issue for assessment of sp mode area tolerances
    • Rosie noted that the genesis for this was due to needing canopy to fully fill error (or we see downstread rad errors).
      • Perhaps "canopy shyness" gap mechanism would help with rad error? Ryan agrees that 2-stream will help enable this if desired.

August 14, 2023

Upcoming meetings and events

  • Meetings

  • Conferences

  • Machine outages

    • CISL: None scheduled
    • NERSC: Perlmutter, Wed 16 August 0500-1400 PDT
    • OLCF: None scheduled

PR board status

Latest issues and discussions

Calibration progress update (Adrianna, Jessica, Rosie)

  • Adrianna: CLM5 PPE
    • fff has large impact on FATES GPP
    • Working on getting relative % change in parameter affect (as opposed to absolute change)
    • Charlie: looking at spatial distribution would be interesting. He suspects it'll be split between snow/no snow areas
    • Next steps:
      • Bring in Ryan's two-stream branch to see the affect on the albedo bias
      • Another PPE with just fates params and then maybe clm + fates
      • Latin hyper-cube set (on for albedo tuning and then decision on next calibration steps)
    • Charlie: What about elm/clm param divergence? Maybe to investigate: calibration of fates with just fates versus fates + hlm params?

Walk-on Topics

  • Jessie: ELM-FATES training next summer
    • Geared towards ForestGeo community and potentially run at their sites. Inventory init since no met data available.
    • Still need to talk with Stuart to see if we're going to hold it concurrently with existing planned ForestGeo workshop or be standalone.
    • Adrianna: could we improve inventory initialization file (considering compatibility with ED)

August 7, 2023

Upcoming meetings and events

PR board status

Latest issues and discussions

  • Rosie: Status of NoComp ERS issue?
    • Greg: still working on this. Has fallen down priority behind BGC and refactor PRs.

Calibration progress update (Adrianna, Jessica, Rosie)

  • Adrianna: creating parameter files to be able to run PPE with FATES

FATES August 3 modeling meeting notes discussion (Ryan)

Walk-on Topics

  • Charlie: Pft areas by landuse type file. Have this as a separate file our put into the LUH2 concatenated file.
    • Could allow us to decouple somewhat from the surface data set. Would allow us to reword nocomp mode.
  • Charlie to add Chongang on this meeting.
  • Adrianna: there is a proposal to use ChatGPT to refactor clm fortran code to another language to faciliate ML work.

July 31, 2023

Upcoming meetings and events

PR board status

Latest issues and discussions

  • No new issues or updates

Calibration progress update (Adrianna, Jessica, Rosie)

  • Deferred

Walk-on Topics

  • Discussing Ryan CNP reviewer response and diurnal eddy flux comparison findings.

July 24, 2023

Upcoming meetings

  • FATES modeling meeting: Thursday August 3 - TBD
    • We will reach out to Zachary ang Chongang. They both presented RFA updates at the NGEE-T all hands today.

PR board status

Integrated

Updated

  • 1005 (seed dispersal): fixes to SeedUpdate and minor refactor. Need to deconflict E3SM#5555.
  • CTSM#1959 (FATES BGC update): two unresolved issues and then testing.
  • 1032 (LUH2 data tool):
    • Refactored to use decode_times=False
    • ncks -v failing on secmb and range. Possible insight into runtime error seen?
      • Possibly due to corrupted download? If not, Charlie suggests try converting to a different NetCDF file type.

Latest issues and discussions

Closed

Updated

Calibration progress update (Adrianna, Jessica, Rosie)

  • Adrianna: Still working towards PPE run to assess impact of clm parameters on fates. Working on making PPE scripts a little more user friendly.
    • If PPE is cheap, we could run with Ryan and Atkins each.

Walk-on Topics

  • init_patches refactor, bareground, and natpft/landuse mapping discussion
    • Charlie: created a bc_in variable for bareground fraction to remove indexing pfts by zero. He will put this into a refactor PR.

July 17, 2023

Upcoming meetings

  • FATES modeling meeting: Thursday July 21 - Adam Hanbury-Brown
  • Distinguished Scientist Seminar Series, Sonia I. Seneviratne: Thursday, July 20th, 2023, 10-11:30 AM PDT
    • Jennifer hosting

PR board status

Integrated

New

Latest issues and discussions

New

Updated

  • 845 (sp mode too much canopy): patch fusion and resizing during init questions
    • Ryan: possibly zero LAI due to high resolution or crop?
    • Try running single point at a failing lat-lon
      • Could this be problematic based on how hlm handles single-point averaging for a specific lat-lon?
      • Adrianna: we could also take a look at the failing lat-lon to see what the lai is at those cells
        • mismatch between patch area and lai area?

Global variable refactor PR preview (Adrianna)

Local branch: https://github.com/adrifoster/fates/tree/hlmparams_refactor/api

  • Greg: we should point to a reference on get/set being recommended object oriented practice that we can point to for the developer's style guide section

Quarterly Highlights Request for E3SM-FATES (Ryan)

This request came from NGEE-Tropics project management (Sandy, Daniel)

  • Ryan and Jen to crib from Charlie's E3SM presentation
  • Highlighting our communication: E3SM Phase 3 meeting and E3SM-Infrastructure regular meeting participation

Calibration progress update (Adrianna, Jessica, Rosie)

  • Jessie: vertical scaling scheme cases are running. Results forthcoming.
    • Caught temperature sensitivity error. Give Rosie and Charlie a heads up on this. See PR#1053 for fix.
  • Adrianna: pulling together the scripts for PPE. Goal is to generate clm parameters that affect fates-sp mode output.
    • Discussion of elm vs clm parameter differences and potential impact

Walk-on Topics

  • Greg: schedule fates documentation triage meeting
  • Greg: ask Charlie about fates-largescale-tools
    • Jessie: new repository for fates tools to do large regional. Will likely have some overlap with calibration repository.

July 10, 2023

Upcoming meetings

  • FATES modeling meeting: Thursday July 21 - Adam Hanbury-Brown
  • Distinguished Scientist Seminar Series, Sonia I. Seneviratne: Thursday, July 20th, 2023, 10-11:30 AM PDT
    • Jennifer hosting

PR board status

Integrated

Updated

New

  • fates-doc#34 (reduced complexity modes): merged during the meeting.
    • Related: Created new issue in fates-users-guide repo to add section explaining how to update the documentation.
    • Related: Marcos and Jessie to work on add new sections.

Latest issues and discussions

New

Calibration progress update (Adrianna, Jessica, Rosie)

  • No major items to report

Walk-on Topics

July 03, 2023

Upcoming meetings

  • FATES modeling meeting: Thursday July 06 - TBD
    • Global carbon budget meeting is this day
  • CTSM bug squash week: July 10-14
  • Distinguished Scientist Seminar Series, Sonia I. Seneviratne: Thursday, July 20th, 2023, 10-11:30 AM PDT
    • Jennifer hosting

PR board status

Integrated

Updated

  • 999 (CLM BGC compatibility): Ryan testing
  • 1021 (Tree recruitment): Deconflicted. Initial testing revealed initialization error. Greg working on solution.
    • Look to see where we could add checks against regeneration_model to avoid allocation when not needed.
  • 1041 (EMA NPP fix): Needs debug mode test. Land developer tests are all non-debug.

Latest issues and discussions

Closed

New

Calibration progress update (Adrianna, Jessica, Rosie)

  • Jessie: Vertical scaling of respiration progress. Still need to look into low LAI issue (trimming, uptake)
  • Rosie: freezing tolerance testing
  • Rosie: adding list of biases to correct in minimum shippable fates document
    • Rosie: will share current parameterization file that addresses many of these

Walk-on Topics

  • Ryan: should we consider adding a forest size structure approximation in SP mode (for calibration and otherwise)?
    • Improving biomass representation and rooting depth for SP+Hydro.
    • Charlie: this will be useful when Gordon's multilayer canopy atmospheric model comes in to clm.

June 26, 2023

Upcoming meetings

PR board status

Updates

  • 958 (drought deciduous): Ryan ID'd possible ERS fix in issue #1046 (see below)
  • 1021 (Tree recruitment): Testing forthcoming. Marcos to re-review Adam's updates.
  • 1032 (LUH2 data tool): Testing on cheyenne forthcoming. Discussing integration into ctsm5.2 data pipeline.
  • 1041 (EMA NPP fix): Initial testing looks good. This needs an ELM-side parameter file update. Suggestion to update E3SM#5699 to update param file and point to this fix is pending feedback.

New

Latest issues and discussions

Issues

Discussions

Calibration progress update (Adrianna, Jessica, Rosie)

  • Adrianna: Met with Katie Dagon and Daniel Kennedy to discuss PPE division of labor. Can we just split up by fates/clm or are there clm params fates relies on?
    • CLM soil and ice albedo parameters would be important for fates PPE
    • Note PPE will not be run on latest fates commit
  • Rosie: calibration update
    • Rosie will stop here current results given that Ryan's work and Adrianna's PPE are forthcoming.
    • Carbon use efficiency (NPP/GPP) is very high in temperate and boreal post GPP calibration (using Ryan et al resp)
    • LAI is looking low (consistent with what Jessie is seeing as well in the tropics)
    • Charlie: Jessie to take up work synthesizing GEM data in conjuntction with Yadvinder Malhi to help solve this problem

Two-stream update (Ryan)

  • no update

LUH2 update (Charlie, Greg)

  • CTSM-FATES API ported from ELM-FATES successfully last week.
  • LUH2 data tooling: discussing integration into ctsm5.2 data workflow
    • Refactoring may be necessary for integration. Testing on cheyenne is next step.
    • CTSM software meeting folks agreed to add xesmf to ctsm_pylib (see 6-22-2023 meeting notes)

Walk-on Topics

  • Adrianna: CESM workshop discussion of minimum shippable fates
    • Check on updating fates to use LAI streams
    • Getting 2-stream in for this is high priority

June 19, 2023

No meeting due to federal holiday

June 12, 2023

Upcoming meetings

  • FATES modeling meeting: Thursday June 22 - TBD
    • Charlie to ask Zachary about presenting
    • Charlie suggested making the fates_developers google group the distribution list for the software meeting
  • CESM 2023 Workshop: June 12-14
  • E3SM 2023 All-Hands: June 26-28, Denver Airport Convention Center
  • CTSM bug squash week: July 10-14

PR board status

Integrated

New

Updates

Latest issues and discussions

Issues

Discussions

Calibration progress update (Adrianna, Jessica, Rosie)

  • Jessie: report back from demographic benchmarking model intercomparison project (D-BEN)
  • Jessie: continuing to review LAI bias

LUH2 update (Charlie, Greg)

  • There has been an issue with the model crashing 8 months into a global run due to BalanceCheck
    • Issue appears to have been a problem with max_patch_per_col (hlm-side). Updating this with maxpatch_total alleviates the issue.

Walk-on Topics

  • Land Sites Platform paper published: https://onlinelibrary.wiley.com/doi/10.1111/gcb.16808
  • Roughness length issue:
    • Rosie noted in an email that roughness length looks like its an order of mag different
    • Jessie: fates is passing roughness length back to elm, but it might not be getting updated
      • Might have been using bareground version of this (z0mg)

June 5, 2023

Upcoming meetings

PR board status

Integrated

New

Updates

Latest issues and discussions

  • No new issues of note

Calibration progress update (Adrianna, Jessica, Rosie)

  • Calibrating for NEON sites (Adrianna)
    • Questions: use default pft or specific NEON set? Joint calibration across all sites or individual site calibration?
    • Marcos and Adrianna have been talking about creating specific pfts for the site similar to his work in tropics
    • Making the LAI on surface data sets match site data

Two-stream progress update (Ryan)

  • Previewed draft PR #1036
    • Discussion of history output changes and unit test pr

LUH2 update (Charlie, Greg)

  • Run testing on Perlmutter on going
    • Greg investigating indexing by age issue. Caused by disturbance rate being larger than one, but not sure why this isn't being scaled as expected.

Walk-on Topics

  • Publishing discussion: model description paper suggestion (Marcos)
    • Convert technote?

May 29, 2023

No meeting due to federal holiday

May 22, 2023

Upcoming meetings

  • FATES modeling meeting: Thursday May 25 - Julia Maria Chacon-Labella
  • JuliaHub call (Greg & Ryan): Friday May 26, 8:30 AM PDT
    • JuliaHub reached out to Greg. Others welcome to attend if so desired.
  • CESM 2023 Workshop: July 12-14
    • Charlie to attend in person
  • E3SM 2023 All-Hands: June 26-28, Denver Airport Convention Center
    • Ryan, Charlie, Greg will be attending virtually. Jessica will be on site.
  • CTSM bug squash week: July 10-14

PR board status

Integrated

Updates

New

Latest issues and discussions

New

Calibration progress update (Adrianna, Jessica, Rosie)

  • Adrianna: calibration cascade at NEON sites. She is working on updating surface data to have LAI reflect actual observations at the site.
    • Marcos and Adrianna to talk about common approaches to this challenge. Comparison of ML techniques.

Two-stream progress update (Ryan)

  • Currently grappling with two-stream resulting in lower than expected LAI. Presented diagnostic case without trimming and N-scaling of dark respiration.
    • Potential: could we use observed data to help benchmark if the results are actually as expected (i.e. instead of comparing against Norman). Ryan and Adrianna to look into data at BCI and NEON, respectively. Marcos noted that Manaus has some old data.
    • LAPACK seems to be pulling its weight wrt the 2-stream matrix inversions: speed difference between Norman and 2-stream doesn't seem very apparent.

LUH2 update (Charlie, Greg)

  • See LUH2 data tool PR above. Greg working on discrepency handling between where LUH2 data says there isn't data, but HLM says there is.

Walk-on Topics

  • Jessica has an intern coming in that will be partnering with Margeret Torn's intern to run fates at Blogit (sp?) Experimental Site
  • Adrianna and Marocs' postdoc will try and meet up at the NEON meeting

May 15, 2023

Upcoming meetings

  • FATES modeling meeting: May 25 - Julia Maria Chacon-Labella
  • ESS-PI meeting: May 16-17
  • E3SM All-Hands: June 26-28

PR board status

Integrated

Updates

Latest issues and discussions

Calibration progress update (Adrianna, Jessica, Rosie)

Two-stream progress update (Ryan)

May 8, 2023

Upcoming meetings

PR board status

New

Latest issues and discussions

New

  • Photosynthesis being applied on mean pft-layer-level leaf areas, not cohort-actual Link
    • Consensus is to move forward with Ryan's idea
  • Need better protections for negative storage termination flux and negative germination flux to the litter pool Link
    • Ryan to continue to investigate this and start a github discussion

Calibration progress update (Adrianna, Jessica, Rosie)

  • Rosie: Improvements made in calibrating out the bias compared against CLM5.1. Working on the ILAMB comparison currently against CERES data. Started investigating GPP & ET calibration.
    • Jennifer has a branch with daylength addition to FATES that ELM/CLM both have. Rosie noted this is something she had to remove from CLM to aid comparison.
    • Which to work on first? What is the lowest hanging fruit? Daylength? Acclimation?

Two-stream progress update (Ryan)

  • Deferred

LUH2 update (Charlie, Greg)

  • Deferred

May 1, 2023

Upcoming meetings

  • FATES modeling meeting: May 11 - TBD
    • We will decide on office hours or cancellation next week
  • ESS-PI meeting: May 16-17

PR board status

Updates

Latest issues and discussions

  • No new or updated discussions or issues

Calibration progress update (Adrianna, Jessica, Rosie)

  • Jessica: Working on carbon budget tracking through all organs. To be a PR soon.
    • Assessing what TRENDY output requirements are already incorporated into fates.
  • Adrianna: Continuing to work on unit testing. Continuing ahead with short term solution to read in parameter values. Not pursuing longer term fates pio option at this time.
    • IO discussion recap

Two-stream progress update (Ryan)

  • Marcos: discussion of clumping index methodology
    • Current method is each cohort as its own scattering element

LUH2 update (Charlie, Greg)

  • Greg: SCORPIO divergence from PIO on variable dimension check is causing a read crash. Check in with the commit author about the definition of "extra dimensions".
    • Related to Adrianna's IO exploration: ask about general history of divergence between PIO and SCORPIO

Walk on topics

April 24, 2023

Upcoming meetings

PR board status

Updated

New

Latest issues and discussions

Update

Calibration progress update (Adrianna, Jessica, Rosie)

Two-stream progress update (Ryan)

FATES history compatibility with MIPs (Jessica)

Walk on topics

April 17, 2023

Upcoming meetings

PR board status

Integrated

Updates

  • 958 (Deciduous phenology)
    • Elongation factors added as arguments to the allometry. Scientific testing looks very similar.
    • Crown damage needed to be integrated. Jessie and Charlie to review.
    • Ready to deconflict and regression test soon.
  • CTSM#1959 (FATES BGC update)
    • Erik reviewed. Some changes requested. Also recommending review from Will.
  • 1008 (biomass partition): Zachary agreed to review. Rosie confirmed review as well.

Latest issues and discussions

Issue

Discussions

  • 1007 (hydro output error): Tested re-removing. This is still an issue (#261).
    • Ryan wants to priorize this.
  • 1015 (modify param file help): Ryan providing support
    • Ryan spun out part of this into a new discussion (#1020) about hydro calibration and noted that he asked for others to suggest additional answers.

Calibration progress update

  • Adrianna: focus will be limited to cohort refactoring to avoid complicating the forthcoming LUH2 PR from Charlie.
    • Type bound procedures doesn't increase the memory load appreciably.
  • Rosie: albedo bias structure bewtween CERES and GEWEX looks different based on looking at impact of reducing rhonir and rhovis.
    • Rosie is going to do some comparison to clm5 biases, but then move on from this due to shift to 2-stream
  • Jessie: first round or results submitted to DBEN. Moving on to trying to get the transitions to be represented.

Two-stream progress update

  • Presentation slide deck - deffered
  • Update: Marcos and Ryan math deriviation are consistent (and match previous efforts).
    • Working on figuring out ways to make reductions in the size of necessary calculations.
  • Q: is the current trimming method going to be compatible with 2-stream?

Walk on topics

April 10, 2023

Upcoming meetings

  • FATES modeling meeting: Thursday April 13 - Lingcheng Li
  • ESS-PI meeting: May 16-17
    • Jennifer and Charlie coordinating on a request from Beth Drewniak

PR board status

Integrated

Updates

Latest issues and discussions

Discussions

Issues

Calibration progress update (5-10 min)

  • Adrianna: refactoring work in progress. Testing for b4bness as work progresses.
    • Question: should we refactor standalone procedures as a method?
      • Ryan: what's the impact on memory?
      • Greg: suggestion creating a discussion with a table of the function to method conversion
    • Question: splitting NaNs out of zeroing routines? Yes.
  • Jessie: FBG simulations running in sparse grid mode. Amsterdam simulations nearly done. Waiting on perlmutter coming back up and more allocation hours.

Walk on topics

April 03, 2023

Upcoming meetings

  • FATES modeling meeting: Thursday April 13 - TBD
    • We came up with a number of potential options. Adrianna confirmed with Jackie that Julia could reschedule.

PR board status

Updated

New

  • 1009 (line continuation): Greg adding other b4b issue fixes. Needs to be integrated before CTSM#1962 can be retested.
  • 1008 (biomass partitioning): needs reviewers
    • Ryan to ask Zachary. Rosie will also review.
    • Jennifer noted that this has been discussed from a science perspective in the CA fates meeting.

Latest issues and discussions

No updates

Calibration progress update (5-10 min)

  • Jessie is seeing total balance check error cropping up again as was seen in issue 961
  • Rosie noted that she has a post-doc working on implementing the CLIMA calibration method

Two stream update

Refactoring our core types

  • Adrianna asked if the team was comfortable with refactoring patch and cohort types into their own modules
    • This would also include creating methods to the types (e.g. init) instead of having seperate subroutines
    • She will incrementally test to ensure b4b-ness
    • Team approves

Walk on topics

  • Rosie: Caldararu Silvia contacted Rosie to talk about QUINCY model and FATES

March 27, 2023

Attendence was low so we deferred a number of these items to next week

Github RSA update

Github replaced their RSA SSH host key Friday March 24. You may see the following warning:

@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@ WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED! @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
Someone could be eavesdropping on you right now (man-in-the-middle attack)!
It is also possible that a host key has just been changed.
The fingerprint for the RSA key sent by the remote host is
SHA256:uNiVztksCsDhcc0u9e8BujQXVUpKZIDTMczCvj3tD2s.
Please contact your system administrator.
Add correct host key in ~/.ssh/known_hosts to get rid of this message.
Host key for github.com has changed and you have requested strict checking.
Host key verification failed.

See the github blog post for information and fix.

Upcoming meetings

Discussion deferred

PR board status

Discussion deferred

Integrated

Updated

Latest issues and discussions

Discussion deferred

Calibration progress update (5-10 min)

Discussion deferred

Two stream update

Discussion deferred

Walk on topics

  • Ryan to create a new E3SM branch to bring some nutrients updates so others don't have to keep tracking his branch.
  • Jessie: Got the elm sparse grid working (on cori) and learned how to make custom sparse grids (Keith Oleson provided the scripts)
  • Jennifer: soil organic carbon discussion. She will diagnose and create a new issue/discussion.

March 20, 2023

Upcoming meetings

  • FATES modeling meeting: Thursday March 30 - Carvalhais
    • Chacon-Labella reschedule: Jackie to reach out to Julia to reschedule for April.
    • Ideas for April meetings?
      • Ryan suggested quick 10 minute presentations
        • Jessie offered Atkin comparison update
      • Adrianna to reach out to Will about NIWOT clm-fates comparisons

PR board status

Integrated

Updated

New

  • 1005 (seed dispersal): Yanlan opened recently. Greg to create e3sm-side PR soon.
    • Ryan suggested Eva. Marcos recommended Rachel Ward. Jackie suggested Adrianna.
  • 998 (issue triage): small PR to address recent issue triage

Latest issues and discussions

New

Calibration progress update (5-10 min)

  • Adrianna rehased the ensemble update from last Thursday fates meeting
  • Jessie reviewed how elm and trendy handles radiation partitioning; they are very different!
    • Adrianna helped Jessie determine how to modify the case scripts to make sure elm read in the multiple stream data correctly
    • Aside: TRENDY is an intercomparison project

Walk on topics

  • Unit testing radiation (Adrianna)
    • Adrianna has started working on unit testing framework for radiation. Discussion about how we might utilize it for other fates modules.
  • Integrate active crown fire (Jackie)
    • Jackie is leaving NCAR; date TBD (likely April). She is finalizing her manuscript. She is going to resurrect the integrate active crown fire PR.
  • European CLM group started up by Rosie; first meeting well attended (44 people). She will share the references and links with us: https://github.com/ciceroOslo/CLM-CET
    • Idea is to help create communication connections for Europeans working with CLM and CLM-FATES.

March 13, 2023

Upcoming meetings

PR board status

Integrated

Updated

  • E3SM#5369 (elm-fates fire): IBM specific complier issue resolved. Testing complete. Awaiting integration.
  • Soil moisture increase PRs:
    • CTSM#1962: Discussions ongoing (via email and ctsm software meeting)
    • E3SM#5508: Tested. Approved by Ryan, but not sure if Peter will want to discuss.
  • 991 (reduced restart memory): Reviewed. Ryan to add clarifying comments. To be tested.
  • 958 (drought deciduous phenology): Review in process. Check in with Chongang feedback?
    • Ryan will reach out to Chongang if necessary, but doesn't appear like it will be.
  • 984 (photosynth temp acclimation): fates_leaf_photo_temp_acclim_thome_time added to paramter file. Update xmlpatch and test.
    • New parameter variable is using years. Is there a strong feeling about whether or not to make the units consistent with its related parameter? No strong preference; keep as is.
  • 996 (NaN logging bugfix): Review in process. Fixes are straight forward. Should be able to test soon.
  • 999 (CLM BGC compatibility): Ryan testing; seeing some failures
    • Ryan and Adrianna meeting to discuss tomorrow. Charlie would like to see some explanation for the differences.

Check in

Latest issues

Charlie, Ryan and Greg conducted another issue triage review last week with multiple updates and closures.

Updated

Calibration progress update (5-10 min)

  • Adrianna working on 300 member ensemble to address albedo bias
    • Half failed due to various balance check failures.
    • Reviewing the tests that failed, only fates_rad_lead_xl parameter appears to be relatively well correlated to failing tests.
      • Adrianna to test ensemble without varying this to see how much this alleviates failures.
    • Review the logs to collect the balance check failure type
  • Jessie successfully ran simulations with TRENDY data (per feedback from Daniel Kennedy)
    • Combining the seperated diffuse and direct radiation in TRENDY data allowed elm to run. Dan said clm can handle this, but maybe elm can't?
      • Charlie recommended reviewing output to compare the way the model seperates the two, versus the TRENDY data split.

Walk on topic

  • Jennifer: Started part-time, temporary position at a carbon market startup (similar to Polly). Company is Cultivo.
  • Charlie: LUH2 python script prototype. xESMF issue feedback. Pushback is more using packages with deep dependencies.
  • Ryan: Unit testing on the Norman scheme to see where that component breaks. He feels like this is something worth elevating in priority.
    • Adrianna has been testing collapsing fates information into ctsm radiation scheme to see how this affects albedo and assess the differences.
    • Adrianna relayed that Keith has concerns that Norman is unstable for certain input conditions.

March 06, 2023

Upcoming meetings

  • FATES modeling meeting: Thursday March 16 - Chacon-Labella
    • Adrianna to lead as Charlie and Jackie will be out (Jackie to ask Adrianna)

PR board status

Update

  • 931 (Atkin respiration model): Implemented variable name and C4 updates
    • Ask Charlie if qualitative testing remains
      • Test if its b4b again
      • Jessie to run most recent changes to see how C4
    • Updates to the naming of parameters has been executed
      • Aside: parameter file update
        • Jessie would like the allometry for the parameter file to be updated
          • Consensus is that this should be its own thing
        • Acclimation PR will have a new parameter that is hardcoded. Charlie would like to make it a new parameter.
  • E3SM#5369 (elm-fates fire): Namelist defaults and build update. Final testing.
    • New tests are forthcoming pending discussion with E3SM dev team. Greg is rebasing and working on some namelist cleanup
  • 984 (Acclimation): will come after Atkin, Ryan will add XML parameter change content
    • Charlie will convert hardcoded parameter to file and add code to read in the new variable
  • 958 (Drought Phenology): Check in with Marcos, looks good, check in about grasses with Chonggang?
  • 995 (Large tree init)
    • Aside: Ryan wondering if this is recommended for beyond nocomp? Jessie and Charlie think this is a good question

To Do:

  • Create PRs to E3SM and CTSM that use higher soil moisture in cold start for ALL fates, not just fates hydro
    • Jessie: Where does the code exist in the CLM side? Charlie will create the ctsm-side PR. Jessie will create the E3SM PR.
      • Charlie wondering if there should be further discussion of why the hlms are using this low hydro init value. He will tag relevant people in the ctsm pr.

Latest issues

  • New
  • Updated
    • 935 (longwave down issue): investigations restarting
    • 845 (too much canopy): Adrianna and Jessie noting this is coming up in SP mode calibration work. This could also be causing issue #986.
      • This seems to be popping up a lot; we probably should spend some more time investigating this.

Calibration progress update (5-10 min)

  • Adrianna out today

Demographic benchmarking project update (Jessie)

  • Jessie having difficulty using the TRENDY data. Crash timing depends on how the ordering of the file streams.
    • Marcos wondering if TRENDY data has leap year and if that is causing difficulties in this case. He was seeing crashes in Feb.
    • Potentially something with the metadata? Maybe it's reading the time variable metadata? Jessie renamed the filename to a different year, but the metadata is still the original year.
    • Jessie will post a discussion/issue.

Discussion on Github reviews (Ryan)

  • Jackie: hand-holding is good, but don't transcribe offline review for the reviewer. Ask them to do this so they get practice.A
  • Scope of reviewer pool: Consensus is that we should include people outside of the core modeling team
    • Rosie suggests that we should bring this up regularly at the modeling call and include early career researchers more.
    • We should create a list of people to ask for review. Ryan and Rosie thinks this should be private to avoid perception of exclusivity.
      • Include the "carrot" of review contributions in next DOI
      • Charlie: Include both elm and clm communities to improve cross-pollination and exposure to the code.
  • Can we find a link to an existing github review tutorial?

February 27, 2023

Upcoming meetings

PR board status

Update

Latest issues

Calibration progress update (5-10 min)

February 13, 2023

Upcoming meetings

  • FATES modeling meeting: Thursday February 16, TBD
    • Consensus was to cancel this given all the meetings that have been happening recently.

PR board status

Latest issues

Calibration progress update (5-10 min)

  • Adrianna, Jessie, and Rosie to start meeting regularly this week on Wednesdays
  • GPP run disparity between Jessie (low) and Adrianna's (high) runs
    • Run through the same ILAMB intercomparison
    • Different forcing data sets (what is the E3SM default?). To do: create a common protocol document to make sure we are all working from the same settings, data, process, etc.

CLM-FATES Dynamic Land Units report back

Walk on topics

  • Sharing proposals using FATES that are in the works
  • MIMICS+ (Rosie): Ryan to share the development path document for FATES and MIMICS coupling

February 6, 2023

Canceled due to Land Model Working Group Meeting

January 30, 2023

Upcoming meetings

PR board status

Latest issues

Calibration progress update (5-10 min)

January 23, 2023

Upcoming meetings

PR board status

  • Update
    • E3SM#5106 (c-based harvest)
      • Holding for issue 973 fix. PR 976 drafted with fix.
    • 769 (leaf memory)
      • scientific testing ongoing, but there are concerns. Discussion set for next week.
    • 931 (Atkins)
      • Can we bring this in as a non-default option even given discussion around C4 grass handling?
        • Yes, although Charlie wants to clean up first.
    • 851 (hydro stability)
      • Marcos found what may have been a water-balance bug which was showing up in elm-fates

Latest issues

  • New
    • CTSM#1932 (NEON fates)
      • Adrianna to announce the capability on the FATES github discussion announcments page
    • 973 (tveg24 debug issue)
      • fix: set patchno for fbg + nocomp runs (not just sp mode runs). See PR 976 above.
    • 975 (nuopc issue): ctsm-side issue 1937 created.
      • OP has it working, but the fix is a little odd.
      • Greg to close this on the fates side and Erik to ask if this is ok to close on the ctsm side.

Calibration progress update (5-10 min)

  • Discussion of ILAMB script progress
  • Calibration team to set up time to bring Adrianna back up to speed (post-LMWG meeting) and meet weekly
  • Given bias with Norman, 2-stream is the long term goal
    • Most canned 2-stream algorithms assume stacked leaf-layers. Definitely possible, but needs to account for parallel leaf-layers.
    • Ryan has python prototype that does this that we could port.
    • We should also consider future proofing radiative transfer scheme (they are on to multi-streams)

January 9, 2023

Upcoming meetings

  • FATES modeling meeting: Thursday January 19, topic TBD
  • Land Model Working Group Meeting (Feb 6-8)
    • Coordinating a clm-fates development discussion
      • When Ryan is back we should coordinate a meeting to discuss LULCC
      • Will also floated the idea of scoping out more fates training with cesm containers
        • Josh Rady was interested in what containerization offerings where out there
  • E3SM Spring All-hands meeting: TBD

PR board status

Latest issues

Calibration progress update (5-10 min)

  • Getting Adrianna up to speed on recent work
    • More broadly: re-onboarding Adrianna to the updates from the last few months
  • Jessie continuing work on Atkins discepencies

CTSM PR expectations and NCAR resource allocation

  • PR expectations discussion
    • Rewording 2a so as to not suggest precluding turn FATES on by default
    • 2b example is hillslope-hydrology
    • Ryan noted that we should add a refactor/code cleanup/bug fixes/performance improvement item (1c?)
      • (Marcos) 1a concerns about correcting bug or improving model to better align with data but making something else worse
      • (Will) Agreed. Example LUNA fixes degraded the performance, but were necessary
  • Resource allocation discussion
    • Dedicated FATES resources at NCAR: Jackie and Adrianna
    • How to handle concern of where to focus development (big-leaf versus fates)
    • CESM3 and FATES
      • Ideally there would be CESM FATES runs for TRENDY and CMIP
        • LULCC is limiting factor -> resource allocation

January 2, 2023

No meeting due to lab holidays

Add a custom footer

Documentation

FATES technical documentation

FATES User's Guide

FATES Developer's Guide

FATES Activity Planning Board

Tutorials

2024 FATES Tutorial @ LBNL

FATES User's Guide Tutorial

Meetings

Software Engineering

2024 Software Engineering Meeting Notes

2023 Software Engineering Meeting Notes

2022 Software Engineering Meeting Notes

2021 Software Engineering Meeting Notes

FATES modeling

Upcoming schedule

2020 notes

Host land model links

FATES API and Host Land Model compatibility table

E3SM Github Repository

CTSM Github Repository

CTSM documentation

CIME documentation

References

Relevant References page (User's Guide)

Moorcroft et al. 2001. Ecological Monographs, 74:557-586.

Hurtt et al. 2002. PNAS, 99:1389-1394.

Fisher et al. 2015. GMD, Geosci. Model Dev., 8:3593-3619.

Clone this wiki locally