Skip to content

2017 MARBL Dev team meetings

Michael Levy edited this page Dec 13, 2017 · 88 revisions

No Meetings

  • December 26, 2017 (lots of people out for Christmas / winter travel)
  • December 19, 2017 (CGD holiday party)

December 12, 2017

General discussion

  1. (POP listed before MARBL because MARBL updates relate to this new POP workflow)

POP Software Updates

  1. Proposed workflow for getting diagnostics from MARBL
    1. buildnml calls uses new MARBL_diagnostics_class (owned by MARBL) to create default diagnostic recommendations in MARBL-designed format; file will be Buildconf/popconf/marbl_diagnostics [NOTE: if SourceMods/src.pop/marbl_diagnostics exists, build_nml will copy it to popconf/ instead of running MARBL tools)
    2. ocn.ecosys.tavg.csh will be responsible for translating marbl_diagnostics into tavg_contents form and writing it to ecosys_tavg_contents (I will try to do this in C shell but may end up writing a new python file parse_MARBL_diagnostics.py to be called from ocn.ecosys.tavg.csh...)
  2. Issue with MARBL-generated marbl_in file during multi-instance cases (POP is looking for marbl_in but file is being renamed marbl_in_####)
    • I think this is the result of an incomplete bug-fix -- POP was previously using marbl_in on all tasks, which would lead to unexpected behavior if you tried to set up a multi-instance run with differing MARBL configurations. However, it seems like I missed one place where I needed to append the instance string to marbl_in (probably in build-namelist)
    • Taking a one-day break from diagnostics to fix this so Chris F can make the cesm2_0_beta08 tag
    • In addition to fixing this bug, I'll also add a multi-instance test to aux_pop_MARBL to make sure this doesn't creep back in later

MARBL Software Updates

  1. For now, we will treat the script to generate the MARBL recommended diagnostics differently than the settings. Namely, MARBL can read user_nl_marbl and override values in marbl_in accordingly... but there won't be an option to override the diagnostic defaults. To do that, the user needs to rely on the GCM.

  2. Current format of diagnostics output file (that a POP user would edit in SourceMods); note that variables are currently listed alphabetically:

    # This file contains list of all diagnostics MARBL can compute for a given configuration,
    # as well a recommended frequency for outputting each diagnostic. The format of this file is:
    #
    # DIAGNOSTIC_NAME : frequency
    #
    # And fields that should be output at multiple different frequencies will be comma-separated:
    #
    # DIAGNOSTIC_NAME : frequency1, frequency2, ..., frequencyN
    #
    ATM_CO2 : medium
    CO2STAR : medium
    DCO2STAR : medium
    DpCO2 : medium, high
    DUST_FLUX : never
    ECOSYS_ATM_PRESS : medium
    ECOSYS_IFRAC : medium, high
    ECOSYS_XKW : medium, high
    FG_ALT_CO2 : medium
    FG_CO2 : medium, high
    IRON_FLUX : medium
    NHx_SURFACE_EMIS : medium
    NHy_FLUX : medium
    NOx_FLUX : medium
    O2SAT : medium
    pCO2SURF : medium
    PH : medium
    PV_CO2 : never
    PV_O2 : never
    SCHMIDT_CO2 : medium
    SCHMIDT_O2 : medium
    STF_O2 : medium, high

December 5, 2017

General discussion

  1. Meeting next week? (AGU)

MARBL Software Updates

  1. marbl0.22.0 provides Python / JSON script for generating input settings file
  2. First bug found in marbl0.22.0: user_nl_marbl is case-sensitive when user_nl_pop wasn't (found because Jessica was setting lvariable_ptoc = .false. and the Python code expected lvariable_PtoC = .false. instead); fix available in #209
  3. Jessica also pointed out another issue: current computation of tracer count (used by POP as MARBL_NT) assumes PFT_defaults = 'CESM2' and doesn't play nice with PFT_defaults = 'user-specified'. This will be addressed after the CESM 2 code freeze, but prior to MARBL 1.0 release.
  4. Time for another release like 0.19.1? (Backlog of small fixes to make?)

POP Software Updates

  1. marbl_dev_n67: build-namelist is no longer responsible for marbl_in!
  2. marbl_dev_n68: Keith has an update to how we initialize radiocarbon for the 1850 control run. This tag will end up as next POP trunk (I don't want to merge marbl_dev_n67 onto trunk when I'm the only one who has run with it), likely in cesm2_0_beta10.

November 28, 2017

General discussion

MARBL Software Updates

  1. Jessica has a tool to generate size-structured PFT classes
  2. marbl0.21.2 fixes PAR threshold issue in surface layer
  3. marbl0.22.0 will provide python package for generating settings input file, computing tracer count, etc
  4. Keith found a new bug, #207: wrong denominator in bury coefficient computation

POP Software Updates

  1. marbl_dev_n61 brings in Keith's developments through marbl_dev_klindsay_n94 (phys cycle, refactoring forcing field initialization, CMIP forcings for CISO, etc)
  2. marbl_dev_n62 uses marbl0.21.2 (see above)
  3. marbl_dev_n63 brings in Keith's developments through marbl_dev_klindsay_n97 (split dust flux into fine and coarse; check for NaNs instead of blindly accepting MARBL output)
  4. marbl_dev_n64 and marbl_dev_n65 bring in latest POP trunk (Alper added namelist option for initial condition perturbation and then I added Keith's fix for reading r4 restart history files in DEBUG mode; plus two additional tags fixing bugs Keith and I introduced, which is pretty impressive given how small cesm_pop_2_1_20171115 was)
  5. cesm_pop_2_1_20171117b has marbl_dev_n65 merged in
  6. marbl_dev_n66 finishes the "zig-zag" merge (it's identical to marbl_dev_n65 and cesm_pop_2_1_20171117b)
  7. marbl_dev_n67 will have my updates to let MARBL generate its own input file (i.e. buildcpp and buildnml will call MARBL code to determine MARBL build-time settings / generate marbl_in instead of relying on build-namelist)

No Meeting

  • November 21 (canceled due to Thanksgiving)

November 14, 2017

General discussion

MARBL Software Updates

  1. user_nl_marbl (not user_settings_marbl) can go in $CASEROOT

POP Software Updates

  1. I need to set up baselines on yellowstone since it seems like cheyenne may be down for a while, but then I'll make a new marbl_dev tag with Keith's changes:
            This merge brings in
            1) support for phys_cycle
            2) refactoring of D14C atm forcing in abio and ciso
            3) introduction and usage of forcing_timeseries_mod
            4) new CMIP6 D14C and d13C atm forcing datasets
  2. Cecile uncovered a bug in reading single-precision restart history files containing data MARBL writes column-by-column; Keith has a fix that I'll get on the trunk (along with adding a test to aux_pop to prevent future regressions)
  3. I need to talk to Mariana or Jim about reading config_cache.xml in buildnml: want to compare MARBL_NT to what POP was build with and that seems like the best way (this will avoid issues between run-time configurability of MARBL tracer count and POP's requirement that it is known at build-time: e.g. you can not build POP and then change the value of ciso_on)
  4. I still need to remove the requirement that ciso is listed in OCN_TRACER_MODULES when setting ciso_on = .true.; hold up is matter of priority, not difficulty

November 7, 2017

General discussion

MARBL Software Updates

  1. YAML vs XML

    • PyYAML is not loaded by default in hobart system version of python, so before users can launch CESM scripts they need to run
    module load lang/python/2.7.11
    (CIME can not load this module because it is running inside a python environment already)
    • cheyenne compute nodes ignore the user's environment from the log-in node, so even if users load python/2.7.13 and can generate marbl_in from the head node, submitted jobs fail (unless env_mach_specific.xml is updated)

    SOLUTION: provide YAML -> JSON converter until CIME supports PyYAML (not CESM2, but sometime in CESM3 dev)

  2. Cheyenne testing uncovered an issue in how we initialize burial coefficients in marbl_particulate_share

    • Current workflow:
      1. if ladjust_bury_coeff == .false. and init_bury_coeff_opt == 'nml', then marbl_particulate_share%*_bury_coeff = parm_init_*_bury_coeff during initialization
      2. if ladjust_bury_coeff == .true. then marbl_particulate_share%*_bury_coeff is computed as a running mean; initial value for the running mean either comes from parm_init_*_bury_coeff (if init_bury_coeff_opt == 'nml') or from a POP running mean field written to a restart file (init_bury_coeff_opt == 'restfile') [note that eventually MARBL will compute the running mean itself, so this will come from a saved state field rather than being set by POP]
    • Issue: init_bury_coeff_opt is a MARBL setting, but its default needs to depend on both another MARBL setting (ladjust_bury_coeff) and the CESM state (use 'nml' for startup runs but 'restfile' for continuation runs (including hybrids / branches, basically any time the saved state field is available in restart file)
    • Potential solutions:
      1. Take init_bury_coeff_opt out of MARBL; if the running mean is available in the POP restart then overwrite the parm_init_*_bury_coeff initial value otherwise leave it unchanged. I think this will mimic the behavior we see once MARBL computes the running mean, because the GCM will provide the initial running mean via saved state if it exists in a restart file. [Downside: if for some reason running mean isn't in the restart file when user expects it to be, instead of an error MARBL will blindly use the settings file value.]
      2. (Temporary) Introduce lmarbl_rmeans_in_restfile in POP's &ecosys_driver_nml namelist; variable is .false. by default but set to .true. if RUN_TYPE != 'startup' and ladjust_bury_coeff = .true. (build-namelist already queries marbl_in for this value!); in the driver, if variable is true call put_settings to change init_bury_coeff_opt to 'restfile'. [Downside: seems awfully kludgy, we'll probably want to replace it once running means are part of MARBL and these variables are either in saved state (or not).]

POP Software Updates

  1. marbl_dev_levy update:
    • Using MARBL_tools.generate_settings_file() instead of POP's build-namelist utility to create marbl_in (Users edit $CASEROOT/user_settings_marbl)
    • Cleaned up a lot of MARBL-specific options from buildcpp
      1. OCN_TRACER_MODULES_OPT no longer contains CISO_NT, ECOSYS_BASE_NT, or VARIABLE_PTOC
      2. Users still add ciso to OCN_TRACER_MODULES, but only until MARBL can generate tavg_contents (or I add ciso to tracer module list in build-namelist if ciso_on = .true. in marbl_in)
  2. Up next: MARBL_NT set by MARBL (and removed from OCN_TRACER_MODULES_OPT)

October 31, 2017

General discussion

MARBL Software Updates

  1. Moving tool to generate input file into MARBL repository (lots of development in a one-off repo)

POP Software Updates

  1. Instead of just bringing in MARBL's src/ directory as POPDIR/source/marbl, POP gets full MARBL repo as POPDIR/externals/marbl
  2. Updates to scripts in cime_config/ to use input file generation tool
    • User specifies MARBL parameter changes in SourceMods/src.pop/marbl.parameters_input

October 24, 2017

General discussion

MARBL Software Updates

  1. YAML vs XML
    • PyYAML v3.11 is available on hobart with module load lang/python/2.7.11
    • PyYAML v3.12 is available on cheyenne with module load python/2.7.13
    • Also testing with PyYAML v3.12 on my laptop (appears to be latest version available)
  2. Grazing as part of zooplankton_type?
    • I haven't looked at the code in detail, but I don't really like the (max_grazing_prey_cnt, zooplankton_cnt) indices. (Jessica is currently trying to run with 4 zooplankton classes, 1 of which grazes on 5 PFTs but the others graze on 2, 3, or 4)

POP Software Updates

  1. New marbl_dev tag: marbl_dev_n60. Moved iron_frac_in_dust and iron_frac_in_bc from MARBL to POP.
  2. Jessica uncovered a bug and something that we probably need to improve
    • Issue 202: unexpected behavior if marbl_in has more than 512 lines
    • Should we add some namelist capability to setting tracer fallback values? She wants to easily test several different options for different PFT classes, and currently needs to rebuild just to make sure she is initializing tracers correctly

October 17, 2017

General discussion

MARBL Software Updates

  1. Still working on generating the input file
    • I think the python tool can now read an input file of its own
    • Some support for elements of a derived type depending on other elements (e.g. size of grazing%auto_ind(:) is grazing%auto_ind_cnt) but I need to generalize it in a better manner
    • Working on design document -- this was a little backwards, as I wrote the design document after coming up with an initial design, but I hope it helps clean up the python code now that I have a better sense of what the YAML needs to contain

POP Software Updates

  1. New marbl_dev tag: marbl_dev_n59. This brought in the latest POP trunk tag (from Oct 8), compatible with cesm2_0_beta07; C compsets are bit-for-bit with tests from marbl_dev_n58 and cesm2_0_alpha07b but updates to CICE mean that G compsets change answers.
  2. When should I make a new trunk tag? I'm thinking once we are happy with the input file generation tool. Two changes since last merge with the trunk (oldest at top):
    Tag Summary: Merge latest marbl_dev_levy (n119) into marbl_dev
    
                 This brings MARBL up to 0.20.0 (this version of MARBL improves
                 the initialization process in MARBL)
    
    Tag Summary: Merge latest marbl_dev_levy (n121) into marbl_dev
    
                 This adds more MARBL variables to marbl_in (only via user_nl_pop,
                 there are no new default values for them)

October 10, 2017

General discussion

MARBL Software Updates

  1. Getting closer to YAML / python combo for generating namelist
    • Have included derived types for autotrophs and zooplankton, but not grazing (yet)
    • Updated how arrays are initialized (using YAML's array notation rather than parsing a string with commas)
  2. Problems that still need solving
    • Need to know tracer count because that is dimension of tracer_restore_vars(:)
    • autotrophs_type and zooplankton_type both contain members that should NOT be set by user (for example, user sets autotrophs%alphaPI_per_day and then the Fortran code converts it to cgs units and stores it in autotrophs%alphaPI)
    • grazing_type contains two components that are allocatable arrays

POP Software Updates


October 3, 2017

General discussion

  1. Follow-up from discussion on how to use YAML to generate input file
    • Support for arrays (sort of)
    • Support for multiple defaults (resolution dependent, for example)

MARBL Software Updates

POP Software Updates

  1. Best approach for some refactoring Keith is doing
    • abio and ciso can both read d14c forcing from file (or set to constant); we want to ensure consistency between the two modules with both are active
    • abio and ecosys have same issue, but with atmospheric CO2. And do we want to force consistency or is it okay if, for example, ecosys gets value from coupler but abio reads from a file?

No Meeting

  • September 26 (canceled due to travel)

September 22, 2017

This is a special meeting to discuss the possibility of MARBL providing a tool to generate its own namelist in CESM rather than having it handled by POP's build-namelist script. The same tool would also provide ECOSYS_BASE_NT and CISO_NT to buildcpp rather than relying on the user to update OCN_TRACER_MODULES_OPT.

One possibility would be to use MARBL's stand-alone driver for this task. That would require building the Fortran tool before any other building is done in CESM [*]; one possible workflow would be:

  1. Will introduce a new python script in $POPROOT/cime_config/ that
    1. creates a place to build the MARBL driver ($CASEROOT/Tools/marbl_drv/ or $EXEROOT/ocn/marbl_drv?) if it doesn’t exist
    2. copies $POPROOT/source/marbl/{*.F90,makedep.py} and marbl driver F90 files to new directory
    3. replaces anything in new directory with version from SourceMods/src.pop/ (if applicable)
    4. Builds executable (will require yet another Makefile, should probably live in POP repo)
  2. buildcpp would run the get_tracer_cnt test
    1. requires tracer_cnt.nml in $POPROOT/marbl_tools
    2. uses user_nl_marbl as input file
  3. buildnml would run the gen_inputfile test
    1. requires gen_inputfile.nml in $POPROOT/marbl_tools
    2. uses user_nl_marbl as inputfile

[Minor] issues I've stumbled upon in first ten minutes of playing with the above proposal:

  1. CIME only copies user_nl_pop from $POPROOT/cime_config/
    • Solution: will need new CIME tag to copy user_nl_marbl as well
  2. POP needs $MARBL/tests/driver_src/ as an external
    • Solution: $POPROOT/marbl_tools/Makefile is in POP repo and $POPROOT/marbl_tools/driver comes from externals

I'm starting to play with this on hobart, and can maybe show some examples.

[*]

Buildtimes:

15s locally with gnu ; 25s locally with pgi ; 40s on hobart with nag ; 60s on cheyenne with intel


Sept 19, 2017

General discussion

  1. Visit from UCI folks led to generic MARBL interface project
    • Fortran interface module that links to full MARBL library, can be used from Matlab or Python
    • Still need to figure out best way to distribute / test it
  2. Build namelist

MARBL Software Updates

  1. MARBL 0.20.0: made MARBL more runtime-configurable, closed ~5 issue tickets
    • no more namelist, MARBL uses "input files" that it parses line by line instead
    • single init phase
    • PFT counts are all set via input file instead of namelist
    • PFT_defaults = 'CESM2' gives default PFTs, PFT_defaults = 'user-specified' requires user to set all PFT-related parameters via input file
    • merged autotroph_config_type and autotroph_parms_type back into autotroph_type (similar for zooplankton_type and grazing_type)

POP Software Updates

  1. marbl_dev_n57: updates required to move from MARBL 0.19.2 to 0.20.0
  2. marbl_dev_n58: added more autotroph_type objects to namelist_definitions_pop.xml to allow CESM users to use PFT_defaults = 'user-specified'

No Meetings

  • September 12 (canceled due to travel)
  • September 5 (canceled due to travel)

August 29, 2017

General discussion

MARBL Software Updates

  1. Status update for #189
    • Reworked initialization of PFTs -- default is still the same, but defaults when you change autotroph_cnt, zooplankton_cnt, and / or grazer_prey_cnt are coded differently (for one thing, no longer dependent on sname)
    • Created gen_inputfile test; Fortran-based way to produce input files

POP Software Updates


No Meeting

  • August 22 (canceled in favor of code review for #189)

August 15, 2017

General discussion

  1. Catch up on CESM 2.0 timeline
  2. POP initial condition question: gx1v6 and gx1v7 use ecosys_jan_IC_gx1v6_20150108.nc but ecosys_jan_IC_gx1v6_20161123.nc is in the repo and contains diatP, diazP, spP, and Lig (plus Fe is different). Should we switch, or is this an intermediate file with plans to introduce a new version before the 2.0 release?

MARBL Software Updates

  1. Update on #189
    • No more namelist!
    • No more build-time configuration!
    • Still need to clean up the code, but functionality is where we want it
    • Show documentation updates (still on branch)
  2. Merge Keith science changes before 189?
    • Initialization is very different, will be easier for me to handle conflicts than Keith
    • My changes are bit-for-bit, which will make testing more manageable

POP Software Updates


No Meetings

  • August 8 (travel)
  • August 1 (travel)

July 25, 2017

General discussion

  1. Retooling the initialization stage of MARBL... we need to separate init into distinct phases to allow us to define parameters and determine tracer count in a consistent manner, but instead of requiring GCMs to use put() calls between phases, could we pre-stage all the put calls and then process them between phases? We could use a logical flag to track which put calls had been applied, and then error out with variable not found if any statements had NOT been applied.
  2. The above point may render this moot, but how should we name the individual init phases? We are currently using a semantic naming convention (config, init, complete_config_and_init) but it's not entirely clear what each of those phases means. It would be nice to just use init_phase1, init_phase2, etc... but that doesn't give users a clear indication of what is happening in each phase. Two points of confusion with the current scheme:
    • We tend to think of init as "setting parameters in the parameterization sense of the word" and config as "variables that may affect tracer count, and also other logical variables that aren't really part of a parameterization" but I am working on a "pre-config" phase that would determine PFT counts (and possibly be a good place for configuring sinking detritus types if we add that feature) and suddenly this semantic description doesn't seem useful
    • I'd like to come up with a good rule of thumb for answering "I am adding a new step to initialization, where does it go?" Two possible answers are "initialize it as soon as everything else it depends on is locked" or "initialize it in the phase prior to when it is needed by other features." I have put together a quick list of every process that occurs in initialization, including which phase it occurs in and any dependencies it has.
  3. Another possibility is to keep all the init_phase#() routines public, but also provide a blank namelist on the interface so that developers can start with a single init() call and whatever default parameter values MARBL provides to get up and running, and then switch to the multi-phase calls with put() statements in between if necessary.
  4. CESM Tutorial -- anything MARBL-specific? Should we work something into the POP exercises?

MARBL Software Updates

  1. Working on #189, trying to rely less on build-time configuration options (which spawned the two discussion points above). This PR will let us remove marbl_sizes.F90 altogether, which is a huge step forward.

POP Software Updates


No Meeting

  • July 18 (canceled due to travel)

July 12, 2017

General discussion

  1. New tracer module to add: abio gasses

MARBL Software Updates

  1. New tags
    • 0.19.0: namelist size runtime configurable, consistent diagnostic names for autotroph Si Uptake, molw_Fe used in marbl_mod.F90
    • 0.19.1 & 0.19.2: lots of code clean-up (no interface changes, no answer changes). Two highlights: better error message when CO2 solver does not converge; consistent use of CaCO3_form throughout the code
  2. Currently working on making PFT counts runtime configurable, need smart way to ensure consistency between POP build-time settings and MARBL run-time settings
    • Use put() statements to override whatever is in MARBL namelist?
    • Have namelist defaults depend on POP build-time settings? For example MARBL_OPT = CMIP6_SETUP and several variables can depend on MARBL_OPT (will then get ported to MARBL when it builds its own namelist)

POP Software Updates

  1. Merged marbl_dev (n52) onto trunk, so MARBL 0.19.2 will be available in cesm2_0_alpha07a.

No Meetings

  • July 4 (National holiday)
  • June 27 (canceled due to travel)
  • June 20 (CESM workshop)

June 13, 2017

General discussion

  1. Gloss over topics that we didn't really get to last week
  2. Testing question: should we switch from T62_g16 to T62_g17? Or test a combination of the two resolutions? See #179.
  3. Unsorted tickets in the MARBL 1.0.0 project

MARBL Software Updates

  1. Lots of small updates (without interface changes) coming in marbl0.18.1
  2. Is now the right time to have MARBL produce list of diagnostics that POP converts into tavg_contents?

POP Software Updates

  1. New marbl_dev tag because I removed gcm_num_interior_forcing_elements from the interface (hardcoded to 1 until we update the code)

June 6, 2017

General discussion

  1. Try using github projects for MARBL 1.0.0 milestone?
    • Also added project page for Post-MARBL 1.0.0 milestone to better visualize (maybe we want to change some tickets to MARBL 1.0.0?)
  2. Three tickets without a milestone: should we try to get any (or all) of these into MARBL 1.0.0?
    • I think 22 needs to be done before CESM 2.0 or not at all -- another major release with an inconsistent tavg variable name will make it harder to correct afterwards

MARBL Software Updates

  1. 168 -- apparently PGI 17.4 is somewhat buggy, because for some reason a string being set wasn't always populated with the right value the first time it was accessed... but the workaround for it actually ties in to #167
  2. 169 -- easy way to use PGI compiler on local machines (--mach local-pgi); not urgent to accept, but useful for testing since PGI 17.4 is free and fixed previous PGI bugs that prevented MARBL from building / running (though, as noted above, it's still not perfect)
  3. 170 -- drop gcm_num_elements_interior_forcing from interface (has a related POP update)
  4. 171 -- found some bad use whitespace when examining log files
  5. 172 -- split thermodynamic_coefficients_type to separate species concentrations from actual coefficients.

POP Software Updates

  1. MARBL 0.17.1 is on the trunk (as part of marbl_dev_n46, which contains marbl_dev_klindsay_n73 and marbl_dev_levy_n91)
  2. Alper made a new trunk tag for Jim (cleaning up some XML info, no expected conflicts), so marbl_dev is a little behind

May 30th

General discussion

  1. Best tool for linking meeting agenda to github?
    • Using the github projects doesn't seem sustainable long-term
    • How about a wiki?
  2. Some discussion on MARBL 1.0.0 milestone
    • @mnlevy1981 will talk to @klindsay28 over coming weeks to prioritize open tickets
    • Will see where we are when CESM 2.0 is frozen

MARBL Software Updates

There are currently four open pull requests, that will be merged in the following order:

  1. 165 -- bit-for-bit, no interface changes
  2. 166 -- bit-for-bit unless ladjust_bury_coeff = .true., no interface changes
  3. 164 -- round-off level answer changes, no interface changes
  4. 157 -- big answer changes (default parameter values changed); changes namelist as well so there is corresponding GCM tag to make

First three should be merged and tagged by the end of May (as MARBL version 0.16.1); last one will be merged later in the week

POP Software Updates

Two recent marbl_dev tags:

  1. marbl_dev_n43 -- changed default PE layouts for better out-of-the-box throughput (with and without ecosystem) on NCAR machines. See this google spreadsheet for timing details
  2. marbl_dev_n44 -- branch is up-to-date with the current trunk (May 16th, 2017 tag)

Upcoming marbl_dev tags:

  1. All changes between marbl_dev_klindsay_n58 and marbl_dev_klindsay_n71 have been reviewed / are ready to merge
  2. marbl_dev_klindsay_n72 needs to be reviewed
  3. marbl_dev_klindsay_n73 and beyond are in progress, should be merged when ready / reviewed
Clone this wiki locally