Skip to content

JK HSP2 Log

Jump to bottom
Joey Kleiner edited this page Dec 13, 2022 · 61 revisions

Work Session 9/7/22:

  • def _hydr_(ui, ts, COLIND, OUTDGT, rowsFT, funct, Olabels, OVOLlabels, specactions):
  • def specl(uci, ts, step, specactions):
  • def _specl_(uci, ts, step, specactions):

Week of 9/12/22:

HSP2 flowchart (incomplete)

HSP2_flowchart_2

Questions:

  • I'm having trouble loading any modules on deq1 (deq3)
    • pandas, h5py etc.
    • spent time adjusting where python looks for the python/module installation
    • hsp2 was having no trouble loading them the last time I ran the model. Check the installation issue: #9
  • Where do the different model objects come in to the sequence? https://github.com/HARPgroup/HSPsquared/issues/7
    • We don't know yet. Objects may be an enhancement to the specl facility from the original model, but it may be more appropriate to make them a separate module but we are using this issue to scope that out.
  • When does the demand() function come into play? (located in HYDR.py)
    • demand() appears to get arguments pertaining to OUTDGT, hence, I think that anything that alters OUTDGT will influence the processing of demand(), but a primary purpose of this scoping is to determine precisely how that works.
  • Is OUTDGT different from the withdrawal parameter that will be in the specactions dictionary?
    • OUTDGT will be set by specl entities.
  • Where will the parser come into the sequence?
    • The parser is at the very beginning when we run hsp2 import_uci. Check the parser issue for detail. https://github.com/respec/HSPsquared/issues/99
    • it may be a good idea to merge the parser branch into your development branch. It doesn't work, but it dorsn't break anything and it does tie the different functions together.
  • Is the HARP hydr CSV the input to specactions = {}? https://github.com/HARPgroup/HARParchive/issues/536#issuecomment-1245729144
  • Would be good to have an easy way to visualize model results of test runs (using one of the HARP scripts): https://github.com/HARPgroup/HARParchive/issues/505
    • I project they will have the whole set of hydrology metrics integrated next week. :)

Work Session 9/14/22:

  • Running jk_tester.py
    • nano /usr/local/bin/hsp2
      • Top line in this file notes the python executable to be used (python 3 in this case): #!/usr/bin/python3
      • Note: "#" is not a comment in this context, it is a shebang line
      • Added this header to jk_tester.py
    • helpful commands:
      • python2 -m site
      • python3 -m site
    • setting binary user permissions
      • chmod 755 jk_tester.py
        • chmod [owner][group][others], 755 equivalent to rw rx rx
      • only have to execute this once, then file permissions are changed forever till you run chmod again
    • How to run
      • ./jk_tester.py (assuming jk_tester.py has the header #!/usr/bin/python3)
      • OR: python3 jk_tester.py (header not needed)
  • Code Updates
    • Updated specl() and _specl_() to take input parameter "ui" NOT "uci"
      • ui is the Numba-ized uci
    • Should send ts_save and ts
      • At the beginning of the simulation ts is zero
      • ts_save is our state
  • Where do the objects come in?
    • Ans: we don't know yet. One way of doing it is in special actions since that facility is in there. We know special actions can modify state variables, which is exactly what the objects will do
  • General HSP2 notes
    • The .uci file specifies which .wdm files get pulled in
    • import_uci 1) imports the uci & 2) imports all the wdm timeseries data
    • in the CBP model the demands come in through a wdm, timeseries of demands for every day in the simulation
    • test10 does not do withdrawals
    • Some of our questions:
      • How does OUTDGT operate? -> gets passed to _hydr_(), runs demand()
      • How does demand() operate? -> several calls to demand() but presumably it only executes once in the loop (lots of if statements)
    • the lowecase variable name is the pass by reference of the UPPERCASE variable in the current timestep
    • The withdrawals are represented as the diversions in the uci. example:
      • DIVR = non-ag
      • DIVA = agricultural
      *** DIVERSIONS
WDM3  3007 DIVR     ENGLZERO          SAME RCHRES   1     EXTNL  OUTDGT 1
WDM3  3008 DIVA     ENGLZERO          SAME RCHRES   1     EXTNL  OUTDGT 2
    • demand()
      • See od.sum() -> we're assuming this sums the multiple OUTDGTs in the wdm
        • the uci syas how many are summed, the above example has 2
    • We think OUTDGT is the way to do the withdrawals
    • We should probably run hours
  • Running tests
    • New directory created: /opt/model/HSPsquared/tests/testcbp
    • Had to do the following within this new directory
      • chmod g+w * -R aka change group to writable recursive (for all files and directories under that directory)
      • chgrp modelers * -R
    • Set up test for PL3_5250_0001 (Occoquan Main Dam watershed)
      • we know this one has pretty big values
      • uci and wdm files copied over
        • PL3_5250_0001.uci
        • met_A51059.wdm
        • prad_A51059.wdm
        • ps_sep_div_ams_hsp2_2022_PL3_5250_0001.wdm (This is where the withdrawals are, the avg is ~51 cfs or mgd?)
        • PL3_5250_0001.wdm
      • Modified the date range of the simulation at the top of PL3_5250_0001.uci to START 2001/01/01 END 2001/12/31

Work Session 9/21/22

  • Do a comparison where we manipulate the withdrawal by a lot, and one where we manipulate it zero
    • Then see what the data looks like in the .h5 file to see if this approach works
    • Set up the outdgt (the variables that represent withdrawals)
      • Because really we want to tell it what the withdrawal is and then have it do the management of the withdrawal
  • outdgt is an array - it supports 3 potential values
  • We think:
    • OUTDGT 1 is DIVR
    • OUTDGT 2 is DIVA
  • We’ll pass the equations and lookups into _specl_()
    • Within _specl_() the objects will do their thing and then it’ll log the value to outdgt
    • We don’t know where the objects will ultimately come into play BUT for our draft testing purposes, objects will be instantiated within specl
    • The objects will say “What value am I setting and how am I setting it”
  • In the old model it uses a broadcast to handle the withdrawal
    • We’ll alter that so the river object reads the withdrawal value for all the things contained within it and then sums them
  • River Withdrawal Model
  • Method
    • We will blank the outdgt and then set it ourselves
    • We will zero it even though we don’t really need to, because we’ll be setting it in the specl process anyway
    • Right now it’s just helpful to run with some outdgt in there for testing

Work Session 9/23/22

  • Check for changing values in the H5
    • Set OUTDGT 2 aka DIVA and see what happens
    • Get with HARP on these items as-needed
  • The RCHRES gets sent to hydr
  • steps rows, nexits columns
  • Strategy
    • We want to be modifying the ts
    • We want to choose the one that gets used in the equations, if we can just pass in ts and modify ts['OUTDGT'][step], then we’re good
    • Want to minimize passing in other variables if we can
  • See ~line 73 in HYDR.py: OUTDGT comes from ts
  • Way to print all keys in ts?
    • See print(names) ~ line 75 of HYDR.py
    • Example for printing all keys in a .h5:
   import h5py
   filename = "PL3_5250_0001.h5"
   
   with h5py.File(filename, "r") as f:
      # Print all root level object names (aka keys) 
      # these can be group or dataset names 
      print("Keys: %s" % f.keys())
  • In the uci, DIVA comes out at OUTDGT2 the other is OUTDGT1
  • Now in the hydr routine:
    • Did the code do anything with the lowercase outdgt?
    • Did it do anything with a manually set 99 value?
  • Note: we look at step-1 because the withdrawal stuff gets executed before the river channel stuff. SO the flow in the stream is 0 for the current time step, therefore you have to look at the previous time step
    • And that’s not unrealistic since withdrawers typically look at the gage reading from the previous day to base their withdrawals
  • Findings:
    • O and and RO are influenced by what outdgt is through the demand() function
    • See below ~line 477 in HYDR.py for where those things get set based on the lowercase vars at the end of the loop. See # HYDR
    • We found that modifying OUTDGT2 in ts does not automatically alter outdgt
  • One option:
    • Pass OUTDGT to the specl function
    • Another option is have a function that loads things from ts
  • One next step: want to spend time looking at those things that get set at the bottom of loop in HYDR.py
    • what do those look like in the .h5 file?

Week of 9/26/22:

  • outdgt() influences O and RO in the next time step through the demand() function
  • Current Question: Does pass by reference work for timeseries table ts?
  • Progress has been made on the parser
    • https://github.com/HARPgroup/HSPsquared/issues/24
    • psa branch pull request submitted to RSEPEC (once they merge it, I can pull it back into the specl branch)
    • See:
      • readUCI.py
        • see line d =parseD(line, parse[‘SPEC-ACTIONS’
        • uses ParseTable.csv
      • UCI.py: self.specactions defined
      • hdf.py
      • main.py: uci_obj = io_manager.read_uci() & specations = uci_objspecations
  • tip: use fgrep for investigating
  • Next steps:
    • wrapping up coding of withdrawals
    • then next step is using real withdrawals & flowby and/or releases
  • Investigations
    • developed testing script jk_tester.R
      • cd /opt/model/HSPsquared/tests/testcbp/HSP2results
      • Rscript jk_tester.R
    • Spent time getting oriented with the h5 structure
  • ro is the first value in OUTDGT aka OUTDGT1 
step:  431  of:  8760  steps
Trying specl
[73.66304779 -0.          0.        ] # print(OUTDGT[step, :])
73.66304779049824 # print(ro)
Made it to specl()
Made it to _specl_()
[-0.0, 99.0] # print([OUTDGT2_save, ts['OUTDGT2'][step - 1] ])
[73.66304779052734, 73.66304779052734] # print([OUTDGT1_save, ts['OUTDGT1'][step - 1] ])
[73.66304779 -0.          0.        ] #print(outdgt[:])

 step:  8759  of:  8760  steps
Trying specl
[83.55184937 -0.          0.        ]
83.55184936523438
Made it to specl()
Made it to _specl_()
[-0.0, 99.0]
[83.55184936523438, 83.55184936523438]
[83.55184937 -0.          0.        ]
  • Notice, ro is different for every step in the loop, even though its not explicitly called/set at the beginning of the loop
  • After manually setting ro to 99:
 step:  8759  of:  8760  steps
Trying specl
[83.55184937 -0.          0.        ]
96.18687265721383
Made it to specl()
Made it to _specl_()
[-0.0, 99.0]
[83.55184936523438, 83.55184936523438]
[83.55184937 -0.          0.        ]
  • Verified that the values in “RESULTS/RCHRES_R001/HYDR/table” do in fact change, but not quite as expected
    • See the output files HYDR_table_v1.csv vs HYDR_table_v2.csv (this one used the ro = 99)

Week of 10/3/22:

  • Added line to gitignore: tests/testcbp/HSP2results/*.csv
  • Currently assuming RO = actual withdrawals with availability considered
  • All names in ts: ['WIND', 'SOLRAD', 'PREC', 'POTEV', 'PLADFX2 1', 'PLADFX1 1', 'PKIF5', 'PKIF4', 'PKIF3', 'PKIF1', 'OXIF2', 'OXIF1', 'OUTDGT2', 'OUTDGT1', 'NUIF23 2', 'NUIF23 1', 'NUIF22 2', 'NUIF22 1', 'NUIF21 2', 'NUIF21 1', 'NUIF14', 'NUIF12', 'NUIF11', 'NUADFX3 1', 'NUADFX2 1', 'NUADFX1 1', 'IVOL', 'ISED3', 'ISED2', 'ISED1', 'IHEAT', 'GATMP', 'DEWTMP', 'CLOUD']
  • All names in ui: ['uunits', 'steps', 'nrows', 'nodfv', 'nexits', 'errlen', 'delt', 'VOL', 'VCONFG', 'STCOR', 'OUTDG5', 'OUTDG4', 'OUTDG3', 'OUTDG2', 'OUTDG1', 'ODGTF5', 'ODGTF4', 'ODGTF3', 'ODGTF2', 'ODGTF1', 'ODFVF5', 'ODFVF4', 'ODFVF3', 'ODFVF2', 'ODFVF1', 'NOCGT', 'NEXITS', 'LKFG', 'LEN', 'KS', 'IRMINV', 'IREXIT', 'ICAT', 'FUNCT5', 'FUNCT4', 'FUNCT3', 'FUNCT2', 'FUNCT1', 'FTBW', 'DELTH', 'DB50', 'CPRI5', 'CPRI4', 'CPRI3', 'CPREC7', 'CPREC6', 'CPREC5', 'CPREC4', 'CPREC3', 'CPREC2', 'CPREC1', 'COLIN5', 'COLIN4', 'COLIN3', 'COLIN2', 'COLIN1', 'CFVOL3', 'CFVOL2', 'CFVOL1', 'CFVOL', 'CFRAC7', 'CFRAC6', 'CFRAC5', 'CFRAC4', 'CFRAC3', 'CFRAC2', 'CFRAC1', 'CEXIT4', 'CEXIT3', 'CEXIT2', 'CEXIT1', 'CEVAP5', 'CEVAP4', 'CEVAP3', 'CEVAP2', 'CEVAP1', 'CDEM2', 'CDEM1', 'AUX3FG', 'AUX2FG', 'AUX1FG']
    • Recall, ui is passed into _hydr_. ui is a dictionary: ui = make_numba_dict(uci)
  • HYDR/table columns: "","index","DEP","IVOL","O1","O2","O3","OVOL1","OVOL2","OVOL3","PRSUPY","RO","ROVOL","SAREA","TAU","USTAR","VOL","VOLEV"
  • Test Runs:
    • Test 0: Raw, no changes:
step:  8759  of:  8760  steps
Trying specl
[83.55184937 -0.          0.        ]
83.55184936523438
Made it to specl()
Made it to _specl_()
[-0.0, -0.0]
[83.55184936523438, 83.55184936523438]
[83.55184937 -0.          0.        ]
83.55184936523438
[83.55184937 -0.          0.        ]
  • Test 1: Inside SPECL.py: ts['OUTDGT2'][step] = 99
    • Note: there was no observed change in output h5 HYDR_table
step:  8759  of:  8760  steps
Trying specl
[83.55184937 -0.          0.        ]
83.55184936523438
Made it to specl()
Made it to _specl_()
99.0
83.55184936523438
[83.55184937 -0.          0.        ]
83.55184936523438
[83.55184937 -0.          0.        ]
  • Test 2: In HYDR.py: OUTDGT[step, :] = [ts['OUTDGT1'][step], ts['OUTDGT2'][step], 0.0]
    • Now in output h5 HYDR_table, O2 = 99 for many timesteps
step:  8759  of:  8760  steps
Trying specl
OUTDGT[step, :] [83.55184937 -0.          0.        ]
ro 78.74899348493953
Made it to specl()
Made it to _specl_()
ts['OUTDGT2'][step] 99.0
ts['OUTDGT1'][step] 83.55184936523438
OUTDGT[step, :] [83.55184937 -0.          0.        ]
OUTDGT[step, :] [83.55184937 99.          0.        ]
roseff 78.74899348493953
outdgt[:] [83.55184937 99.          0.        ]
  • Findings in HYDR/table
    • RO = O1 + O2 + O3
    • ROVOL = OVOL1 + OVOL2 + OVOL3
  • Pass by reference vs. Pass by value
    • When you pass an object into a function, you're passing a reference (a reference to that data in memory, not a copy)
      • If you change the data of that object, you’re changing the original object
    • Verified that changing a value on the ts dict object within _specl_ automatically alters the ts object within hydr()
  • slicing np arrays makes a difference: https://github.com/HARPgroup/HSPsquared/issues/20
  • O3 could be a reservoir thing
  • They have exits - sometimes exits represent withdrawals, sometimes they represent releases
    • so those exits might get mapped over to an O
  • confirmed OVOL3 is the outflow from the reach to the next downstream segment
    • OVOL1 2 and 3 are all flows out of the reach
  • makes sense that’s overwriting it because that happens after specl
    • the Os are exits
    • O1 is exit 1, O2 is exit 2
    • RO:
    • ROVOL:
  • O2 goes from 99 to O, this can happen, the Os get modified if there’s not enough water in the reach
  • O1 + O2 + is the satisfied demand
    • It satisfies the withdrawal “outflow” before the flow out of the reach (makes sense)
  • OD is the outflow demand
    • either by way of the ftable geom
    • or the requested demands
  • Lowercase o in the code is an array of O1 + O2 + O3
  • Nexit in the uci represents the total number of exits (3 in this case)
    • it relates to OD1 OD2 OD3
  • irrigation withdrawals is a separate operation, CBP doesn’t use that, its there from the original HSPF developed in the 70s
    • recall CBP model != HSPF
  • Next:
    • Try setting OUTDGT in a way that allows us to modify in in ts (~ line 73 in HYDR)
  • If there’s *** anywhere on a line in HSPF, that whole line is a comment
    • SO line 51 are the only values that matter in the uci
  • Line 78 in HYDR.py is checking those flags
  • Then the next test is - does modifying this actually modify the value in ts

Week of 10/19/22:

  • develop is essentially respec's beta version branch
    • they do their development on branches, then pull request to develop branch, then pull request to master
  • Made "initial" pull request from HARPgroup/specl -> respec/develop: https://github.com/respec/HSPsquared/pull/104

11/2/22 (Tackling issue installing non-master branches of HSPsquared):

  • Last time Rob linked python 3.9
  • Python3 -V
  • Have to sudo when installing packages, otherwise it’ll do it in your local environment
  • sudo pip install . (Works from within the HSPsquared directory)
  • Rob IS able to install with the main branch BUT not the develop branch
    • I verified this is the same on my machine too
  • environment_dev.til calls updated versions of the packages
    • One thing to try is updating our packages
  • Hints for Linux
    • diff file name file name
    • This will show differences if there are any
    • similarly, git diff will show you all the changes you’ve made
    • remove a directory with rm -Rf testcbp
    • cp FROM TO
  • Unlikely our pushed updates would be causing this (the PSA branch and the SPECL)
    • This was the flow of branch merges we did
      • Specl -> psa -> develop
  • Git checkout (commit id)
    • Using this method we compared earlier commits and then tried to reinstall hsp2 till it worked
  • **configuration.py is where we should be loading the specl.py
  • Will keep tinkering
    • One option, take a copy of that good commit and sequentially put in our changes

Afternoon:

  • Create a new branch from an existing branch
    • git checkout -b [to_branch_name] [from_branch_name]
  • I made a new branch that is an exact copy of the master, and it compiles
    • specl_11.2.22

11/9/22

  • Next steps:
    • We want to modify ts dictionary values without resetting ts
    • Within SPECL, change value of outdgt2
    • Look at old branch, review progress, attempt to that within the specl function
      • RB will set up his equation function to do this
    • We know how to modify the ts variable, Now we need to know how to modify the state variable (outdgt)
  • See: https://github.com/HARPgroup/HSPsquared/issues/5#issuecomment-1309287801

11/28/22

Clone this wiki locally