Skip to content

2024 Software Engineering meeting notes

Jump to bottom Edit New page
Gregory Lemieux edited this page Dec 16, 2024 · 238 revisions

December 16, 2024

Wins and Whimsy 🎉

  • Former tutorial student requested to use fates tutorial for Ghent University vegetation moduleing course
    • Jessie and Greg to reply. Add Ryan and Adrianna to the reply as they are interested in the course content as well.

Machine status and notices 🖥️

  • CISL: none planned
  • NERSC: Wednesday 18 December

Upcoming meetings and events 🗓️

PR board status update

  • Greg and Ryan to schedule longer outlook planning session for PR board in the new year

Integrated

Update

New

  • parameter switch migration
    • To be coordinated with ESCOMP/CTSM#2904
      • CTSM PRs waiting on high priority PR ahead of this
    • E3SM-side pull request TBD
    • CK: Should we update defaults at the same time? E.g. Medlyn.
      • RK: Jenny has update to default.
      • GL: We can, although there is an opportunity to bring this in on the ctsm b4b-dev branch if we don't
        • That said, doing so is dependent on some other ctsm queue factors
    • CK: how should we handle real number parameters that are switch dependent? How to make the user aware?
      • Differences between defaults for the host land models will eventually exist in parameter file
    • RK: This is a short-term solution. Should we accelerate moving to migrating parameter file to xml?
      • This would allow us to nest and check parameters based on the switch
      • Consensus is moves this forward as it solves current testing issues. Conduct
  • Don't zero patch%fuel%frac_burnt: needs review
    • Fixes issue #1301
    • CK: Fuel amount conditional on patch age. Adrianna to review.
  • Fix cmake for functional tests
    • AF: note that if you are running with latest ctsm tag, functional tests will break without this PR

Latest issues and discussions of note

Calibration progress update

  • deferred for walk on topic

Walk-on topic

  • Rosie: previewed FATES diagnostic work
    • TVEG issue discussion: sp mode looks ok, nocomp is too low.
      • RF: running with default parameters
      • RK: hlm_hio_ignore_val should not be averaged in
      • CK: make a map of equivalent host land model to see how it compares. Should align with fates.

December 09, 2024

Cancelled due to AGU

December 02, 2024

Wins and Whimsy 🎉

Machine status and notices 🖥️

  • CISL: none planned
  • NERSC: Wednesday 18 December

Upcoming meetings and events 🗓️

PR board status update

Integrated

Update

Latest issues and discussions of note

  • inventory restart question (fates-tutorial#64)
    • Greg and Ryan to ask clarifying questions
    • Suggest toggling the hard-coded switch to get more diagnostics
  • SP mode + finidat case failure (Adrianna)
    • Height coming from hlm is zero for some reason in sp mode when running a case copied from a previous case
      • Restarting from original case does not have this problem
    • Adrianna to create an issue (if necessary)

Calibration progress update

  • Adrianna to send python code to Jessie for sparse grid mapping

Patch allocation and boundary arrays discussion (Ryan)

Walk-on topic

November 25, 2024

Wins and Whimsy 🎉

Machine status and notices 🖥️

  • CISL: none planned
  • NERSC: Wednesday 18 December

Upcoming meetings and events 🗓️

PR board status update

Update

New

Latest issues and discussions of note

Calibration progress update

  • Adrianna: Discussion of recent results

Walk-on topic

November 18, 2024

Wins and Whimsy 🎉

  • Adrianna's first FATES merge and tag.
  • NorESM merged MEGAN-FATES update to their ctsm branch. Will be merged back to ESCOMP ctsm.

Machine status and notices 🖥️

  • CISL: none planned
  • NERSC: Wednesday 18 December

Upcoming meetings and events 🗓️

Activity planning board

  • Initial thoughts
    • Advertise board at modelling meeting and in newletter
    • Inviting people to contribute?
      • Greg to generate google form for people to have their project added
  • Archiving and converting items to issues? Consensus: yes

PR board status update

Integrated

Updates

New

Latest issues and discussions of note

Calibration progress update

  • Rosie: making updates to ensemble

Walk-on topic

November 11, 2024

Wins and Whimsy 🎉

Machine status and notices 🖥️

  • CISL: none planned
  • NERSC: Wednesday 13 November

Upcoming meetings and events 🗓️

Forward looking project board (Charlie)

  • This came out of a discussion between Charlie and Adrianna.
  • Both forward-looking (planning) and forward-facing (outreach)
    • A lot of our boards are for current efforts (e.g. PR board, Landuse board)
    • We want community to know what we are doing and know how to reach us for collabs
    • Jessie suggested reviewing this at the modeling meeting to encourage used
  • Allowing people to add their own items
  • Greg to sketch out the board which we'll review in this meeting next Monday

PR board status update

Updates

New

Latest issues and discussions of note

Calibration progress update

  • No updates

November 04, 2024

Wins and Whimsy 🎉

Machine status and notices 🖥️

  • CISL: Tuesday 5 November
  • NERSC: Wednesday 13 November

Upcoming meetings and events 🗓️

PR board status update

Integrated

Updates

Latest issues and discussions of note

  • Adrianna: imaginary quadratic solve issue
  • Rosie: finidat issue
  • Rosie: testing + compset concerns from Marianna

Calibration progress update

Walk-on topics

October 28, 2024

Wins and Celebrations 🎉

Machine status and notices 🖥️

  • CISL: Tuesday 5 November
  • NERSC: Wednesday 13 November

Upcoming meetings and events 🗓️

PR board status update

Integrated

Updates

New

Latest issues and discussions of note

  • sapflow bug (#1266): Discrepency between FATES_SAPFLOW_SZPF and FATES_SAPFLOW
    • Charlie/Marcos: Units are incorrect
  • Branch runs failure (#1271): Sam tested and found that branch runs fail for FATES. Tested using ERI prefix, for which no fates test exists.

Calibration progress update

  • Adrianna: GPP improvement globally (GBAF, FLUXCOM, WECAN). Slightly under calculating in the amazon now. Much better everywhere else including Africa.

Walk-on topics

October 21, 2024

Wins and Celebrations 🎉

Machine status and notices 🖥️

Upcoming meetings and events 🗓️

PR board status update

Integrated

Updates

Low relative humidity discontinuity discussion

  • Discovered via Demo of Leaf Biophys Unit Tests
    • Are the discontinuities due to singularities in stomatal conductance calculation?
    • Marcos has seen this in ED2 as well and addressed it.
      • Dynamically make sure that the initial guesses don't straddle the singularity.
  • Only an issue with Ball-Berry. Move to medlyn as the default?
    • Adrianna: Melyn has a correction that doesn't allow humidity to get that low (cap on VPDs between leaf and atm, not atm itself).
      • CLM apparently fixed the issue and removed the cap.
        • Ryan removed the caps for medyln and didn't see the issue reappear.
      • That said Adrianna is seeing convergence errors.
    • Ryan and Marcos to setup a meeting to discuss

Latest issues and discussions of note

  • Grass sapwood allometry hydro issue (#1254): Xiulin to create PR to address this soon.
  • Biomass outputs updating for non-new cohort only
    • Ryan: if its a state should be included, but if its a flux it shouldn't be used. We need to do an audit of Sam's list and what is/isn't a flux.
      • Marcos: Issue is more if its per plant rather than per area.
    • Future refactor: splitting history interface mode between fluxes and states
    • Immediate work: audit the list and create PR to correct. This should be based on Sam's PR and come in afterwards.
  • FATES simluation in desert environs: Rosie answered. Cross-post low relative humidity issue here as well.

Calibration progress update

  • Adrianna: getting results for co-dominant

Walk-on topics

  • Sam: distinction between subroutine hist dyn 1 and 2?
    • Adrianna: suggests renaming the subroutines to be more descriptive

October 14, 2024

Wins and Celebrations 🎉

Machine status and notices 🖥️

  • CISL: Tuesday 5 November
  • NERSC: Wednesday 16 Oct

Upcoming meetings and events 🗓️

PR board status update

Integrated

Updates

Latest issues and discussions of note

Calibration progress update

Walk-on topics

October 07, 2024

Wins and Celebrations 🎉

Machine status and notices 🖥️

  • CISL: none planned
  • NERSC: Oct 16, Perlmutter

Upcoming meetings and events 🗓️

PR board status update

Integrated

Updates

Latest issues and discussions of note

Grass sapwood allometry hydro issue (#1254)

Calibration progress update

  • Charlie: Seed rain impact on spin up discussion (wrt nocomp calibration)
    • Seeds may be supporting a population of plants that are constantly dying
    • Should we be making some metric/ecosystem benchmarks to make sure that certain pfts are being represented
      • It appears that die-off is mostly in grasses
      • Investigate using Xiulin's new grass allometry
      • Charlie to update with more detailed maps showing the differences
      • Maybe test seed dispersal

Walk-on topics

  • Rosie: FATES reading 78 pft issue
    • NorESM-FATES needs 16pft file for spectral element simulations as well
    • Discussion triggered by issue (#2810) testing showing that we needed a file for ctsm5.3.0.
      • Sam L and Mariana have created to address.

September 30, 2024

Upcoming meetings and events 🗓️

PR board status update

Updates

Latest issues and discussions of note

1250

Calibration progress update

Walk-on topics

September 23, 2024

Machine status and notices 🖥️

  • CISL: none planned
  • NERSC: Oct 16, Perlmutter

Upcoming meetings and events 🗓️

PR board status update

Updates

  • Arctic shrub PFT additions (#1236): queueing and regression testing
    • Jennifer: running an additional smoke test to conduct some difference mapping
    • updates are a result of observation data and FASST optimatization
    • Ryan: coordinating with Adrianna's sp mode update
      • Adrianna current work doesn't include bl ever extra trop, and broad decid extra trop
        • Since so much is changing with sp mode calibration inputs (e.g. gswp3 -> cru), might as well update this and redo sp calibration
    • Jennifer: Drive to get this in is to meet NGEE-A milestone
    • Greg: start regression testing to get ahead of any other issues since API updates are coming up soon in queue
      • Greg to start e3sm API update
  • Time integrated flux diag (#1217): under review
    • Jessie to add review later today
  • new patch insertion method (#1155): under review, Ryan may implement recommended changes for him
  • Spitfire fuel calc refactor (#1247)
    • Adrianna: question about historical context for zero fuel code comment
      • Adrianna will update this to allow for zero fuel during her refactor

Latest issues and discussions of note

No new issues or discussions of note

Newsletter feedback

  • Marcos: should we be advertizing FATES presenations as well?
    • Ryan to add call for people to provide highlights for a presentation and papers
    • Marcos to create google form to provide consistent input results
    • Goal: get your edits in by Friday. Ryan will send out then.

Calibration progress update

  • Jessie, Rosie, and Adrianna to meet on Wednesday
  • Linea and Adrianna to create repo for calibration

Walk-on topics

September 16, 2024

Wins and Celebrations 🎉

  • 2024 FATES Tutorial event was a success!

Machine status and notices 🖥️

  • CISL: Sep 16-19, Derecho
  • NERSC: Oct 16, Perlmutter

Upcoming meetings and events 🗓️

  • FATES calibration coordination meeting: CLM meeting, September 19
  • FATES modeling meeting: September 26, TBD
    • Ask Steven, who will be here at LBL, if he'd like to present in October
    • Bianca Rius for a model comparison. Marcos to reach out.
    • Reach out to Yanyan when Rosie is back

Tutorial Retrospective

  • Three attendees have signed up on the FATES model google group so far
  • Generated a couple issues and started up some PRs during the tutorial

Initial feedback

  • Just under half responded via survey so far: mostly positive feedback, some constructive critism
  • Software: Docker setup pre-tutorial could use improvement; more introductory material on working in the terminal
  • Presentations: Slow down speech cadence during presentations and spend a little more time during intro talks
  • Balance of time between hands-on practice and presentations: slight bias towards more presentations with examples of use

FATES team feedback

  • Ryan: Presentations as breaking up the day
  • Jessie: more presentations on different processes in fates (guided examples)
  • Ryan: make it more clear that they can take breaks when they want
  • Sam (via Jessie): have a section on debugging. E.g. where are the log files?

PR board status update

Updates

New

Latest issues and discussions of note

Calibration progress update

  • Adrianna: no major updates due to machine outage. Coordination meeting coming up this week.

Walk-on topics

  • Sam: refactoring history PR work
    • Having trouble reproducing result from the baseline. Needs a second pair of eyes to review.
      • Ryan to tag up with Sam.

September 9, 2024

Cancelled due to FATES Tutorial event

September 2, 2024

Cancelled due to Federal Holiday

August 26, 2024

Wins and Celebrations 🎉

Machine status and notices 🖥️

  • CISL: Sep 16-19, Derecho
  • NERSC: Sep 11, Perlmutter

Upcoming meetings and events 🗓️

  • FATES modeling meeting: September 26, return to regular cadence
    • Charlie added Yanyan Cheng recent paper. Ask Rosie about having them present paper.
    • Jessie schedule Anna Marie in November. Waiting for confirmation.
  • FATES calibration coordination meeting: CLM meeting, September 19
  • FATES software meeting: move to every other week cadence?
    • Informal poll: No strong feelings. Meeting is well run and meeting weekly works well.
      • Check in with Rosie and others about finding a better time for the future.

PR board status update

Updates

  • Fire Weather Refactor: Updates made and retested. Results as expected.
    • To be integrated next. No need to wait for CTSM API updates.
  • Arctic shrub PFT additions: Should this update the default or have a new cdl file? If not, how do we test this?
    • Making default cons: not tested pan-arcticly (one arctic site only), would complicated calibration efforts (?)
      • Adrianna: there have been multiple parameter file updates during calibration, so not that big of a concern for SP calibration work
      • Check in with Rosie about non-comp impact
    • Multiple parameter files? Consensus is no; either update default or just provide xml for this PR.
      • Testing with xml only option: we could test this using the parameter auto-gen workflow
      • TODO: run long global 4x5 smoke test. If results look good, make this default.
  • CTSM API 36.1 update: Waiting on higher priority ctsm-side PRs to meet CTSM5.3.0 tag

New

Latest issues and discussions of note

  • CanopyStructure third canopy layer
    • Added new clause is causing issues due to tolerance. Some version of check should be performed, but not sure what tolerance.
      • Ryan: imperfect plasticity would help this, but radiation scattering makes that difficult.
    • Charlie: leave open; needs more investigation.
  • EDPFTVarcon merging: Ryan will address this.
  • FATES inventory pft error: Greg to reach out to author to ask for files
  • Charlie's Restart Error: under investigation; Ryan reproduced error
    • Ryan: Not all of the params the drive 2-stream have reasonable values on the restart. Adding error codes to exit back into calling subroutine.

Calibration progress update

Adrianna: Shared recent results on broadleaf evergreen to get feedback

  • Atkin and stomatal conductance highly correlated.
    • Jessie: is this related to odd behavior between Atkin and stomatal conductance that Polly emailed about?
  • Workflow only took an hour

Tutorial update

Walk-on topics

August 19, 2024

Wins and Celebrations 🎉

Machine status and notices 🖥️

  • CISL: None scheduled
  • NERSC: Aug 27, Perlmutter

Upcoming meetings and events 🗓️

  • FATES modeling meeting: September 26, return to regular cadence
    • Stephenn deHertog to be here for a month after the tutorial. Marcos to invite him to talk on this date.
  • FATES calibration coordination meeting: CLM meeting, September 5
    • Goal: coordinate existing effort and timelines

PR board status update

Integrated

Updates

New

Future

  • Chonggang: Beetle code forthcoming
  • Chonggang: Water potential based on salinity just getting started
    • Rosie: Has done experimental water potential work with mangroves
  • Sam: Prioritizing escaped fire work
    • Sam to work on replicating Rosie's fire work (3 pfts)

Latest issues and discussions of note

  • 2-stream restart error: testing on main with Charlie's minimum script is passing
    • Charlie: Issue has to do with grazing. He's confirmed that removing grazing alleviates the issue he was seeing.
  • Charlie: canopy structure issue (to be )
    • There is a check missing that needs added to make sure that the canopy is full before adding to lower canopy layers
    • Immediate fix is to change order of promotion from lower to upper
    • Whats the reasonable tolerance level on check?
      • Rosie: check the largest error for crash
    • Prioritization of conservation values (dbh, mass, etc) resulting in conflicts
    • Marcos: ED2 doesn't conserve DBH and back-calculates. FATES is more complicated.

Calibration progress update

  • Adrianna: No major updates. Script for dominant and co-dominant grid working.
  • Greg: documentation of process and tools used? Maybe this should be a discussion point for the coordination meeting?

Tutorial update

  • Beta-testing started. Greg conducting OS testing on Windows. Jessie receiving inventory coordinates to generate surface/domain data subsets.

Walk-on topics

  • Rosie: AD spin-up use recently (in CLM-FATES)? Seeing crashes.
    • Charlie: should work, although had to comment out some phosphorous code in ELM for something specific
    • Adrianna suggested looking in CESM forum. She recalls seeing some issue posted there.
  • Chonggang: Topounits with FATES?
    • They should theoretically work together

August 12, 2024

Wins and Celebrations 🎉

Machine status and notices 🖥️

  • CISL: Derecho maintenance canceled
  • NERSC: Aug 27, Perlmutter

Upcoming meetings and events 🗓️

PR board status update

Latest issues and discussions of note

Calibration progress update

  • Rosie: objective SP mode calibration work shifting from derecho to betsy (NorESM machine)
  • Adrianna: workflow using PFT grids for SP mode targeted to be finished soon.

Tutorial update

  • Beta-testing scheduled to start this week. Content continuing to be expanded on.

Walk-on topics

  • Ryan: Working on refactoring photosythesis module and adding unit tests

August 5, 2024

Wins and Celebrations 🎉

Machine status and notices 🖥️

  • CISL: Derecho maintenance canceled
  • NERSC: Aug 27, Perlmutter

Upcoming meetings and events 🗓️

PR board status update

Latest issues and discussions of note

Calibration progress update

Tutorial update

August 5, 2024

Wins and Celebrations 🎉

Machine status and notices 🖥️

  • CISL: August 12-15
  • NERSC: July 31, Perlmutter

July 29, 2024

Wins and Celebrations 🎉

  • Land use v2 E3SM API integrated

Machine status and notices 🖥️

  • CISL: August 12-15
  • NERSC: July 31, Perlmutter

Lawrencium support

Lawrencium moved from CentOS to Rocky8 and the cime configuration to support this machine is broken

  • Do we want to prioritize an effort to fix this in the long term?

Upcoming meetings and events 🗓️

PR board status update

Use of fates synchronized branches with CTSM (not master)

  • Motivation: FATES development team can merge in PRs that are synchronized with CTSM on our own schedule, allows users to have a CTSM branch that is always compatible with FATES master
  • Questions:
    • Should we have a single branch (fates-latest)? Or a branch for each expected tag?
    • What frequency should we merge ctsm updates into this (these) branches?
      • Every ctsm tag/PR?
      • Only when the new branch is merged into ctsm master?
    • Should PRs be issued to fates-latest? Or ctsm's master?

Latest issues and discussions of note

Calibration progress update

Tutorial update

July 22, 2024

Wins and Celebrations 🎉

  • NGEE-T proposal review complete
  • Land use v2 integrated (E3SM API to be integrated soon)

Upcoming meetings and events 🗓️

  • FATES modeling meeting: August 15, Summer Office Hours
    • Pick up regular cadence starting after the tutorial in September?

Machine outages 🖥️

  • CISL: August 12-15
  • NERSC: July 31, Perlmutter

PR board status update

New

Updates

Latest issues and discussions of note

Calibration progress update

Tutorial update

  • Beta-testing scheduled for two-weeks starting Aug 12

July 15, 2024

Cancelled due to NGEE-Tropics proposal review

July 8th, 2024

Wins and Celebrations 🎉

Upcoming meetings and events 🗓️

Machine outages 🖥️

PR board status update

  • No PRs integrated this last week

New

Updates

Latest issues and discussions of note

  • Incorrect per-age-class outputs
    • Consensus is that the _AP should be weighted by the patch age class area, although we are not consistent currently
    • Sam also noting that #1170 looks like it may not have fully addressed the issue for FATES_LAI_AP.

Calibration progress update

  • nothing of note, focus is on preparing for NGEE-T proposal

Tutorial update

  • Greg shifting work post landuse v2 to tutorial infrastructure starting this week

July 1st, 2024

Wins and Celebrations 🎉

  • LU V2 seems to be passing tests, ready for integration soon?

Upcoming meetings and events 🗓️

Machine outages 🖥️

PR board status update

  • No PRs integrated this last week

New

Reviewers

Updates

Latest issues and discussions of note

No new issues or discussions

Calibration progress update

  • Rosie: Planning to check in with Charlie about handing off efforts to him wrt land use spin up.

Tutorial update

  • FATES tutorial will use FATES API 35

June 24, 2024

Wins and Celebrations 🎉

  • Carbon budget closing thanks to Ryan and Jessie's efforts. PR forthcoming.

Upcoming meetings and events 🗓️

Machine outages 🖥️

  • CISL: none planned
  • NERSC: June 26, Perlmutter

PR board status update

Integrated

Updates

  • Land use v2: final testing with ctsm and e3sm
  • Understory turnover rate: Approved with one minor comment. Needs testing. Has new parameters.
    • Plan is to have this be its own API update post land use API update.
    • Rosie would like to get this in soon for calibration efforts. As such, we will bring in #1206 and TBD nutrients switch default PR afterwards.
  • Dynamic patch arrays: Approved with one minor question. Needs testing.
  • Grass allometric equations: Check in with Xiulin. Needs testing. Updates param values.
  • Growth Resp to daily timestep: Ask for another reviewer. Needs build and run testing.
    • Jessie will review as well.

Latest issues and discussions of note

No new issues or discussions

Calibration progress update

  • Rosie: Planning to check in with Charlie about handing off efforts to him wrt land use spin up.

Tutorial update

  • FATES tutorial will use FATES API 35

June 17, 2024

Wins and Celebrations 🎉

Upcoming meetings and events 🗓️

  • FATES modeling meeting: June 20, Summer Office Hours
    • Graduate student intern visiting NCAR this summer will present briefly on pyrogenic carbon
    • Related: Rosie noted that there is a NorESM PR to bring couple trace gasses back to HLM
    • Adrianna to send out email
  • 2024 CESM workshop debrief
    • FATES presentations from Adrianna and Rosie

Machine outages 🖥️

  • CISL: none planned
  • NERSC: June 26, Perlmutter

PR board status update

Integrated: History flush and zero fixes

Updates

Latest issues and discussions of note

Calibration progress update

  • Discussion of determine the uncertainty of the parameters

Tutorial update

Walk on topics

  • Discussion of adapting damage model to fire processes
    • Damage model would likely slow down a global model quite a bit

June 10, 2024

Wins and Celebrations 🎉

Upcoming meetings and events 🗓️

Machine outages 🖥️

  • CISL: June 11, Derecho
  • NERSC: June 26, Perlmutter

PR board status update

Updates

Latest issues and discussions of note

Carbon balance closure issue

  • Marcos: attempt turning off reproduction to determine if that is the issue.
  • Ryan/Charlie: NPP minus seed decay should equal allocation
  • Ryan: we could also add another data structure to add this up over time

Calibration progress update

no updates

Tutorial update

  • Google group for attendees setup
  • Greg to send out email with "hello world" docker setup request

June 3, 2024

Wins and Celebrations 🎉

Upcoming meetings and events 🗓️

Machine outages 🖥️

  • CISL: none this week
  • NERSC: June 26, perlmutter

PR board status update

Updates

Latest issues and discussions of note

Issues

Discussions

Calibration progress update

Adrianna: updated emulator based on Jessie's suggestions which showed improvement in estimating average GPP.

Tutorial update

May 27, 2024

Cancelled due to Federal Holiday

May 20, 2024

Wins and Celebrations 🎉

Upcoming meetings and events 🗓️

Machine outages 🖥️

  • CISL: none this week
  • NERSC: June 26, perlmutter

PR board status update

Integrated: photosytem 2 named constant

Updates

  • Land use v2: addressing some run failures
    • Ryan: B4B expected
  • Growth Resp to daily timestep: review underway
  • Understood turnover rate: prioritization?
    • Rosie would like to see this in for calibration. Jessie has this on her calirbation work currently.
  • Dynamic patch arrays
    • Looking for clarification on the consensus from last week
    • Agreement that ncl of two is too small.
    • Next steps: test performance of ncl of three against dynamic allocation PR

Latest issues and discussions of note

Issues

Discussions

Calibration progress update

  • Rosie: need maintresp, allom_d2ca_coeff, allom_d2bl1 to be low to get LAI to observed
    • It's reasonable for this to be difficult for the model based on literature research
    • Ryan: is radiation model hampering things here? I.e. do we need some sort of explicit gap?
  • Adrianna: shared latest updates from variation on results based on number of gridcells included

Tutorial update

May 13, 2024

Wins and Celebrations 🎉

Jennifer on Star Talk

Upcoming meetings and events 🗓️

Machine outages 🖥️

  • CISL: none this week
  • NERSC: May 15, perlmutter

PR board status update

Integrated: Allometry unit test, Inventory input updates, e3sm unstructured grid tool

Updates

New

  • Dynamic patch arrays
    • team formulated to discuss and assess impact of setting nclmax
    • termination mortality proposal for cohorts beyond nclmax sounds good
  • Growth Resp to daily timestep: needs build testing
    • TL;DR: If plants aren't growing, they shouldn't be growth respiring. FATES only grows on daily timestep => only growth respiring on daily timestep
    • How does this compare to HLMs?
    • Don't use NEP for carbon budgeting; introduce a NBP flux for carbon budgeting.

Latest issues and discussions of note

  • no new issues or discussions of note

Calibration progress update

  • Adrianna: TRENDY plans discussion
    • TRENDY need: External pft-specific seed rain for extinct pfts (for spin up compensation)
    • Updated SP mode configurations forthcoming

Tutorial update

  • No new updates

May 6, 2024

Wins and Celebrations 🎉

New group using FATES: Tau Liu at Michigan Tech. Teaming up with Chonggang. Interested in calibration and regional runs.

Upcoming meetings and events 🗓️

Machine outages 🖥️

  • CISL: May 7-8, derecho
  • NERSC: May 15, perlmutter

PR board status update

Integrated: CCH water xfer functions, LAI-by-patchage

Updates

  • Small fire refactor: ready to test
  • Allometry unit testing: testing discussion held on Friday. Minor org refactor to be done.
  • Land use v2: Walkthrough held. Break out land use alignment from disturbance TBD.
    • CTSM API 36: drafted. run mode combo checks discussed and updated
    • How do we test spin up for land use?
      • Similar to the fact that we don't have AD testing
      • Sam Rabin knows how to add a new test in run_sys_test. Greg will tag up with him later.
      • Decision: make issue for testing spin up, but don't hold this up. Add standard regression tests for other landuse v2 modes.
    • Deadline: integrated Friday May 24

New

  • Land Energy Balance Solve
    • what is a good absolute case? Look at distribution of conductance based on different values and come up with reasonable delta.
    • Marcos: Stomata have minimum when close so what about using residual as the absolute tolerance?
    • Ryan: check in with Gautam for E3SM-side reviewer and feedback

Latest issues and discussions of note

  • no new issues or discussions of note

Calibration progress update

Tutorial update

  • No new updates

April 29, 2024

Wins and Celebrations 🎉

  • NGEET Phase 3 proposal submitted
  • First FATES newsletter sent out
  • ELM User's guide PR with connecting FATES documentation

Upcoming meetings and events 🗓️

Machine outages 🖥️

  • CISL: none planned
  • NERSC: May 15, perlmutter

PR board status update

Latest issues and discussions of note

  • spin-up speed-up
    • Dan Ricutto might now
    • E3SM specific. Loads stuff into memory.
  • growth respiration calc
    • Initial solution consideration (Marcos): calculate on a daily basis and apply with each timestep
      • Need to have code to account for edge cases given if function based on previous day's value
    • We want to get this done prior to the nocomp calibration release
    • How does this apply with HLMs (big leaf)?

Calibration progress update

  • Adrianna: Daniel and Lydia discussion about setting up emulator. How much do we need for a good set for each pft?
  • Rosie: Rerunning one at a time nocomp ensemble with latest main branch (daylength + alt vert scaling).
    • Charlie: High seed rain clarification: Rosie wants to dig into this a bit more, ideally with fully spun up case.
    • Is there a danger in including seed rain in an ensemble for global calibration?
    • Rosie will make global map of seed rain impact

Tutorial update

  • Logistics: Organizing travel

Walk-on topics

  • Should we re-review this meeting time?
    • Goverance meeting?

April 22, 2024

Wins and Celebrations 🎉

Upcoming meetings and events 🗓️

  • FATES modeling meeting: April 25, Huanyuan Zhang
    • Jessie got confirmation for this week
    • Marcos will need someone to host May 23

Newsletter discussion

  • What sort of topics would people like to see?
    • Add request for fates users
  • How to generate content?
    • Asynchoronous update + in meeting review
    • Dump "wins" from these meetings into the document live
  • Ryan to setup calendar reminder for sending it out every quarter
  • Rosie: Post-doc position - Rosie

Metrics questions

  • How many countries and institutions use FATES?
    • Rosie has a pretty good handle on this. She will send out slide with this information.
      • US universities is an area of uncertainty.
      • Tutorial application could be used for metrics in this regard.
  • How many members do we have?

Machine outages 🖥️

  • CISL: none planned
  • NERSC: May 15, perlmutter

PR board status update

  • API 35 integration and PR1128 is imminant
  • API 33 integration needs confirmation of developer tests passing
  • Landuse v2 walkthrough scheduling
    • Setup walkthrough meeting next Thursday in the timeslot for fates modeling
  • Ryan to reach out to hydro folks for final review
  • Adrianna looking for reviews for unit testing and fire refactor
    • Unit testing needs cime and shr
    • Greg and Charlie to review

Latest issues and discussions of note

  • Minor cleanup issue submitted by Sam Rabin

Calibration progress update

  • Rosie: shared parameter sensitivity results

Tutorial update

  • No new update

Walk-on topics

  • Carbon budget issue - DBEN - Jessie
  • Post-doc position - Rosie

April 15, 2024

Cancelled

April 8, 2024

Wins and Celebrations 🎉

  • Eclipse today! 🌞 🌑
  • Perlmutter speed up trick: DVS in read-only mode (see Jessie's email)
  • Jennifer's birthday

Upcoming meetings and events 🗓️

Machine outages 🖥️

  • CISL: none planned
  • NERSC: 17 Apr, perlmutter

PR board status update

  • New allometric modes: Greg testing to check b4b issues Ryan seeing. Test reverting code in EDCanopyStructureMod
  • CCH water xfer functions: CCH hydro improvements coming along. Ryan now tracking down issue with non-debug hydro resulting in carbon balance errors.

Latest issues and discussions of note

Calibration progress update

  • Rosie: Adrianna and Rosie discussing approach for next steps using Adrianna's pft files (one-at-a-time no comp simulations)
    • Rosie to integrate changes from Jessie's vert scaling PR as well
    • Workflow discussion
      • Jupyter notebooks are not great from a version control perspective, but nicely steps through ensembles updates

Tutorial update

  • No new update

Hydro / SP mode compatibility (Ryan)

  • Ryan noting that we need to add new features to make hydro and sp mode work together.
    • As such, we shouldn't consider failure in the mode to be a bug, but a feature not yet supported.
    • Greg to take the action to add graceful failure with both modes in API35 and API34 ctsm and e3sm PRs respectively.

Walk-on topics

  • Rosie: Fire BVOC emissions
    • Getting MEGAN to work with FATES: https://github.com/NorESMhub/CTSM/issues/36
      • Keeping fire emission factors in fates parameter file and push calculations in FATES to HLM
      • Rosie currently has a new parameter to map from FATES to MEGAN. Rosie doesn't like that this makes the param file not HLM agnostic.
  • Small conference rooms available in B74 (314 and 202)

April 1st, 2024

Wins and Celebrations 🎉

  • Fire trunk fuel bug fix
  • Allometry parameters are in!
  • Hydro restart bug fix
  • FATES tutorial application notices sent out. We've gotten confirmation back from most of them.

Upcoming meetings and events 🗓️

Machine outages 🖥️

  • CISL: 2-3 Apr, derecho
  • NERSC: 17 Apr, perlmutter

PR board status update

Latest issues and discussions of note

  • fire trunk issue: fixed per #1180
    • Discussion of impacts to current work.
      • Team is following up with those that were impacted.
    • Adrianna to run coarse global 100 year run to get some assessment of impact. She will update the issue with results and close it out.
    • Future refactor is to not include trunk loading in fuel class
  • SP restart crash: since there is no mort in SP mode, we will avoid these in fix

Calibration progress update

  • Jessie: high biomass issue may be alleviated by SP parameters from Adrianna's work. Also recent fire fix may help as well.
  • Adrianna: Planning on rerunning with new pft composition to address sparse grid limitations

Tutorial update

see wins

Walk-on topics

  • Adrianna: Testing stand structure faster (testing initialization)
  • SSH security heads up: backdoor found inxz-utils version 5.6.0
    • Fedora Linux 41 users should check security alert notice

March 25, 2024

Wins and Celebrations 🎉

Upcoming meetings and events 🗓️

Machine outages 🖥️

  • CISL: 2-3 Apr, derecho
  • NERSC: 17 Apr, perlmutter

PR board status update

Fire refactor overview

Latest issues and discussions of note

Improving PR integration workflow

  • Brainstorming ideas for reducing PR backlogs
  • Reviews
    • Hold more regular code walk-throughs for large PRs
    • Break-up and stage large PRs if possible
  • Testing
  • Other logistics
    • On-the-fly default parameter file build
    • parameter patch file auto-build/checking

Calibration progress update

deferred

Tutorial update

deferred

March 18, 2024

Cancelled

March 11, 2024

Wins and Celebrations 🎉

Upcoming meetings and events 🗓️

Machine outages 🖥️

  • CISL: none planned
  • NERSC: Wed 20 Mar, perlmutter
  • OLCF: Tue 19 Mar, center-wide

PR board status update

  • CCH water xfer functions: resolves #1163, but now seeing 1D hydro failure
    • Marcos/Ryan: capping psi_min and th_min discussion?
      • Discussion with Jenny last week (water balance conservation issue).
        • Ryan: Use linear model below zero similar to Van Genukten?
        • More discussion needed
    • Greg to track down regression test failure

Latest issues and discussions of note

Calibration progress update

  • Rosie: discussion of disconnect between tuning and diagnostics packages for land

Fire emission coupling hackathon (Rosie)

  • Goal: get fire emissions out of fates and into atmosphere model
  • Calculate biomass burned and pass to HLM to calculate emissions or fates calculate emissions directly
    • emissions are pft specific suggesting we should have fates calculate directly
      • would result in duplication of emissions factors from HLM
  • Charlie: is emission a function of fire "type" (i.e. anthro versus wildfire)
    • Rosie: yes, but pft is a proxy of this
  • Ryan: discussion or issue?

Hydaulic failure discussion (Chonggang)

  • Option: when leaf water potential high, zero stomatal conductance and not request water
    • May improve numerical stability
    • Chonggang hoping to attempt this at the end of the month
  • Ryan: preference is to implement linear model between
    • Alternate (short term?): run case at much shorter time steps to get through snags
  • Charlie: Develop dry condition test failures (e.g. single site at Atacama)
    • Ryan: functional unit testing
      • satphen + hydro?
    • Greg: being able to parameterize functional unit tests would be great

Tutorial update

  • deferred

March 4, 2024

Wins and Celebrations 🎉

Upcoming meetings and events 🗓️

Machine outages 🖥️

  • CISL: Tue 5 Mar, derecho
  • NERSC: Wed 20 Mar, perlmutter
  • OLCF: Tue 19 Mar, center-wide

PR board status update

  • Stuck:
  • Update:
  • Discussion:
    • Vertical scaling: Discuss email thread on clarifying scalar definitions and migration to parameter file
    • Action:
      • Ryan to create new issue for discussion about sla scaling in the biophyscial rates routine
      • Jessie: add parameters Ryan suggested to PR to tee up for future implementation without needing to worry about future API update.

Latest issues and discussions of note

  • No new issues or updates

Calibration progress update

  • No new issues or updates

Global stomatal conductances (Ryan)

  • Making coupling HLM-FATES faster. Honing in on the right stomatal conductance.
    • Noticing how wide range of values are globally. Why values in the sahara (full fates)?
    • Ryan will run an sp mode and share results

Tutorial update

Walk-on topics

  • Rosie: mini hackathon next Thursday/Friday to tackle the fire emissions coupling.
    • Add to discussion agenda for next Monday.

February 26, 2024

Upcoming meetings and events

FATES software meeting check-in

  • Is there interest in moving this meeting to an every-other week cadence?
    • Consensus: those present like the weekly cadence and the reporting back of PRs and issue status.
    • Greg: What about the notes format? Would people prefer to engage with the notes in another format?
      • Consensus: those present like the transparency of the wiki format and that the notes are co-located on the repo with everything else (issues, discussion, code, etc.)

Machine outages

  • CISL: none planned
  • NERSC: 20 Mar, perlmutter
  • OLCF: none planned

PR board status update

Updates:

Latest issues and discussions of note

  • #1163 and #1165 addressed by PRs #1164 and #1163, respectively

Calibration progress update

  • Jessie: finishing up respiration work; will turn into maniscript in near future.
    • Considering adoption of Adrianna's ML method for next iteration (post SP-mode calibration updates)
    • Ryan: stem respiration for calibration?
      • Jessie: Not currently. Kolby keen to investigate this for next project phase. Discussions about field-to-model integration details on going.

Tutorial update

  • Jennifer: what's the accessibility goal? Will we explain how to setup the container, etc?
    • Jessie: Ultimate goal is to make this available all online with a very simple BCI method.
      • Greg: We also have a tutorial setup guide repo in development to host a readthedocs style walkthrough of setting up on a laptop.
    • Greg: OLMT experience? We might use or compare the single site subsetting function for pre-processing.
      • Jennifer: Yes, two years ago. Current attempts to use were hampered however.
      • Ryan: Running pre-process all at once?
        • Jessie: yes, we could use the sparse grid NCL method.

February 12, 2024

Upcoming meetings and events

Machine outages

  • CISL: none planned
  • NERSC: 21 Feb, perlmutter
  • OLCF: none planned

PR board status update

Latest issues and discussions of note

FATES speed audit update (Ryan)

  • Can we avoid leaf layer loop: Trimming routine forces us to iterate over leaf layer depth
  • Current plan: investigate Ci iteration cap and threshold. Update as necessary along with remembering the previous value.
  • Initial iteration analysis update suggests that we often hit the iteration cap.
    • Initial iteration testing with ST3 mode (audit was using full dynamics however).
  • ED2 experience: What about trying Newton method using the derivitives from reduced analytical form? Review the ED2 approach and share with team.
  • Investigate impact of making photosynthesis loosely coupled?

Calibration progress update

  • Adrianna: CLM surface data does not seem to quite match ILAMB observation. This is causing issues with gridcell to gridcell comparison.
    • Ask Nate Collier for help.
  • Calibration team to review Adrianna's brute-force method and will review selected parameter ranges.

February 5, 2024

Upcoming meetings and events

Machine outages

  • CISL: 5-7 Feb, derecho
  • NERSC: 21 Feb, perlmutter
  • OLCF: 6 Feb, frontier

PR board status update

Latest issues and discussions of note

Calibration progress update

January 29, 2024

Upcoming meetings and events

Machine outages

  • CISL: Mon 5 - Wed 7 Feb, derecho
  • NERSC: none planned
  • OLCF: none planned

PR board status update

Latest issues and discussions of note

  • LU data tool fails on derecho: probably taking too much memory for login node. This will need to be addressed eventually for CLM LU build on the fly.
    • This came up in the context of checking ctsm python environment on derecho, due to a dependency issue for xesmf.
    • Erik/Charlie: test this on the develop queue

Calibration progress update

  • Adrianna: Ran Ball-Berry without temp acclimation. Observational mean is more easily achieved, but emulator does a worse job.
    • Temperture acclimation without very low vcmax
      • Charlie: Claire Zarackas is finding something similar in the absence of hydraulics. So there is a scientific rationale for excluding temp acclimation.
  • Jessie: met with Ryan to talk about carbon budget issue; add recruitment carbon flux. Carbon budget closes now.
    • Not sure if recruitments that die in the same timestep are showing up in the diagnostics

Walk-on topics

  • Rosie: Question: Matvey fates model focus? Take over Marius work? Where does it fall in the priority?
    • Jennifer: arctic-fire stuff coming up prior to arctic-hydro. Maybe hardening?
    • Greg: This is back on API 17, so bringing it up before it gets too stale would be good
    • Charlie/Ryan/Rosie: recent hydro issues do suggest that addressing hardening is a good idea.
    • Matvey: only calibrated to 2 local sites; will it work globally?

January 22, 2024

Upcoming meetings and events

Machine outages

  • CISL: Mon 5 - Wed 7 Feb, derecho
  • NERSC: none planned
  • OLCF: none planned

PR board status update

Building parameter files on the fly

  • Ryan to look at moving this forward this week

Latest issues and discussions of note

  • hydro dry region failure: Marcos has been working with Jenny on this. Recommendation is that she not trace the issue since its out of scope for her work.
    • Ryan proposing to add a fail safe for extreme outlier cases to allow fates run to complete. Still need to make sure this is actually an outlier.

Calibration progress update

  • Calibration branch
    • Jessie still needs this branch until 2-stream gets to E3SM (probably a few weeks at this point). The calibration branch is pretty close to FATES main for her purposes, so no need to merge main into calibration branch.

Walk-on topics

  • Erik asking about status of Marius' frost branch

January 15, 2024

No meeting due to lab holiday

January 8, 2024

Upcoming meetings and events

Machine outages

  • CISL: Mon 5 - Wed 7 Feb, derecho
  • NERSC: Wed 17 Jan, perlmutter
  • OLCF: none planned

PR board status update

Latest issues and discussions of note

Calibration progress update

  • Jessie presented latest results: GPP and biomass bias improvements

January 1, 2024

No meeting due to lab holiday

Add a custom footer

Documentation

FATES technical documentation

FATES User's Guide

FATES Developer's Guide

FATES Activity Planning Board

Tutorials

2024 FATES Tutorial @ LBNL

FATES User's Guide Tutorial

Meetings

Software Engineering

2024 Software Engineering Meeting Notes

2023 Software Engineering Meeting Notes

2022 Software Engineering Meeting Notes

2021 Software Engineering Meeting Notes

FATES modeling

Upcoming schedule

2020 notes

Host land model links

FATES API and Host Land Model compatibility table

E3SM Github Repository

CTSM Github Repository

CTSM documentation

CIME documentation

References

Relevant References page (User's Guide)

Moorcroft et al. 2001. Ecological Monographs, 74:557-586.

Hurtt et al. 2002. PNAS, 99:1389-1394.

Fisher et al. 2015. GMD, Geosci. Model Dev., 8:3593-3619.

Clone this wiki locally