Scoping Code Tasks:

[rburghol]

Scoping code goal: create the functions to call and execute a single, test withdrawal

• Understanding numba and the @njit decorator
  • Excerpt from RESPEC HSPsquared documentation "The reason for the two functions levels is that Numba does not compile all Python language constructs, any Pandas functions, and some Numpy library functions. So the top level function, snow() for example, can be arbitrary Python, Pandas and Numpy code which prepares data to be used by the second level code like _snow_( ). The second level code and all routines called by it must be compatible with Numba so that the code can be compiled. Any violation of Numba constraints will prohibit the compilation and force the code to run a Python speed."
  • https://github.com/respec/HSPsquared/wiki/HSP2_Design_Details#basic-information
• Create code hooks for specl() and _specl_()
  • See SPECL.py: https://github.com/HARPgroup/HSPsquared/blob/specl/HSP2/SPECL.py
  • specl() arguments: see pull request test passing the TS and modifying the last step value #23
    • should take the the domain specific ts[step-1], and all speclactions from the UCI
  • def specl(io_manager, siminfo, ui, ts, step, speclactions) where ts = ts[step], state = ts[step -1] (more details of this approach in specl(): Proposed function definition for SPEC-ACTIONS  #13)
    • had to change this to pass the whole ts, and the step since I couldn't figure out how to pass just a single ts[step] since that particular notation throws an error. But, once inside the specl() function we can do things like ts['VOL'][step - 1] = 1000.0 and that works nicely, sets back the variable in the calling function (i.e. ts is passed by reference)
    • because, OVOL[step -1] would be used by flowbys, but withdrawals will want to modify OUTDGT[step]
  • _specl_() should take the same parameters as specl()??: looks like these do NOT take identical params, since it causes errors with numpy due to numpy not liking all dictionaries (only numerical ones?)
• Put the functions specl() and _specl_() inside HYDR.py until we determine how to insure hsp2 code files are loaded with the model
• Insert testing code (specl() called at line 267 (or) just before code line that says if irexit >= 0: - see: https://github.com/respec/HSPsquared/blob/f9555f2e48cc3409f33c29d3c6504d9bdc0f91fb/HSP2/HYDR.py#L267
  • print message out if model is on any step of the run divisible by 100 (just an arbitrary time to print out and make sure that we don't flood the console with messages while testing).
  • What variables to send?
  • I think we just want to send the state variables, which is ts[step - 1]?
  • Code convention uses lower case variable names as state variables in the loop for upper case variable columns in the ts dictionary. Example, vol is the current time-step volume, which is equal to ts['VOL'][step], which is also settable by reference to the variable VOL[step]
  • But, these should not only modify the ts dictionary, but also, should modify the state variables.
  • So, ask if the specl() function sets ts['VOL'][step-1] = XXX, does this automatically make the variable vol = XXX? (see below, same question with outdgt variable which holds withdrawal information`
• Set up a scoping UCI dataset based on a river segment uci, with date range set really short to speed experiments up.
  • The Occoquan river reservoir at PL3_5250_0001 might be ideal, since we know that it has demands and is part of our set of test cases: http://deq1.bse.vt.edu:81/p532/tmp/uci/river/hsp2_2022/PL3_5250_0001.uci
• Hard code a testing set of withdrawals to (see Document location of withdrawal code in HSPF and hsp2 #11 )
  • Does pass by reference work for timeseries table ts? See Setting data in ts and ui, pass by reference in python functions #19
    • verify this by altering ts in a specl and then checking the hdf5 in R after the model runs to see if it got applied.
      • See investigations: https://github.com/HARPgroup/HSPsquared/wiki/JK-HSP2-Log#week-of-92622
      • Pass by reference vs. Pass by value
        • When you pass an object into a function, you're passing a reference (a reference to that data in memory, not a copy)
          • If you change the data of that object, you’re changing the original object
        • Verified that changing a value on the ts dict object within _specl_ automatically alters the ts object within hydr()
  • work with harp on Occoquan issue to investigate if HSP2 currently handles withdrawals in same way as HSPF. Occoquan (spec-act team): PL3_5250_0001 HARParchive#498
  • Successfully alter OUTDGT See Passing referenced OUTDGT (and others) into _hydr_() #31
  • What does demand() function do? Does it overwrite our OUTDGT work? (see lines 277, 279, 329, 337, etc )
  • Output to h5 and see if it works. Use export routines, and examine changes in O1, O2, O3 and RO see: Handle PS and WD (DIVR) in data harvest to _hydr.csv file HARParchive#541
    • Findings verified in HYDR/table:
      • RO = O1 + O2 + O3
      • ROVOL = OVOL1 + OVOL2 + OVOL3
• How do we acccess the specl UCI table? (assuming this is where our defs will go), do we add them to the ui/ts variables that are passed in to each function? See Write a parser for basic SPECL (just setting variables till we decide on our final approach) #24
  • See code in main.py, there are a set of things that get included in the ui and ts depending on which module/block is being executed, we can make specl objects passed in to all perhaps?
• Examine different data storage options re speed: Specl and Objects hdf5 data model #20

Notes From Scoping Session 8/26/22:

@rburghol

• SPEC-ACTIONS is how we propose to handing things
• HSPF considers everything to be in a sequential order, it'll simulate the flow for every timestep in the simulation, then it'll simulate pollutant transport, then it'll simulate algae, then DO and so on...
  • We don't need all of those pieces, but good to know how that is structured
• From what we can gather, the H5 file contains the data its acting
• main.py is what runs the model: https://github.com/HARPgroup/HSPsquared/blob/master/HSP2/main.py
  • It has a big for loop that goes through the UCIs
  • We’ll have to figure how to have the resegmented ones run
  • Special actions can run at the beginning of every time step
• HYDR is an activity inside the RCHRES operation
  • hydr() function defined in file HYDR.py: https://github.com/HARPgroup/HSPsquared/blob/master/HSP2/HYDR.py
• There are 4 variables that go into every activcity
  • function(io_manager, siminfo, ui, ts)
  • HYDR gets a fifth variable ftables
  • RQUAL has some other variables passed in as well
• ui is a dictionary
  • ui = uci[(operation, activity, segment)]
  • RCHRES is an operation
  • HYDR is the activity
• ts is a dictionary
  • ts variables are initialized in HYDR.py
  • the keys are days
  • ROVOL is the flow out variable
• pass by reference?
  • assumption is that ts is being modified by the _hydr_ function
    • anything that happens to ts in there modifies ts
  • So this ROVOL[step] = rovol / VFACT we believe actually modifies the value(s) for ROVOL in the ts dictionary
• HYDR iterates the entire timeseries
  • It does all of its calculations (ftables solving)
• OVOL = Qout
• First we need to know/understand how withdrawals components work, and then put into a specl call
• In HSPF, spec actions are done before the processing inside the loop
• The specl call will send the ts for that step
  • Question of how to get the current state?
    • get ts from prior step using step-1 ? see notes above
• VOL is the volume of water in the reach
• we DO want the spec acts to be able to refer back to and modify the original ts dictionary
  • But the question is how do we get it to modify the things that are set from those dictionary variables
    • i.e those lowercase variables rovol = ROVOL
• we believe the step is just an array of integers (0 to x or seconds since Unix epoch)

---

• Pump-stores add the most complexity
  • Harder b/c in addition to affecting runoff into the stream, water is pumped from the main segment into the pump store
  • What we do now:
    • When we run a rseg, it also runs the tributary(s) of that rseg
    • In OM those 2 segment are in diff containers that are handled by the simulation
  • If you have multiple tributaries in a single uci, it still runs them all separately
  • We could go with simulating all the local tributaries as SPEC-ACTIONS within the rseg
    • This would only be a problem if the land use was significantly different in the local tributary than the main rseg
    • Pass in unit area runoffs
    • Then use spec action to decrease IVOL by runit*trib area
      • IVOL[step] + runit*trib area?
    • Most tribs can be solved very economically (simple equation based), its the pump store ones that will involve more equations and thus slow down the simulation etc.
  • The ETM does all the landuse multiplying
    • Will be a big lift there, RB will work on (when we resegment, the landuse numbers change for the original segment etc.)

[rburghol]

Hey @gcambridge and @jdkleiner -- I just had a breakthrough. Got the hsp2 on deq1 functioning like the one on hsp2. And, here is the output that the model tells me (someone put a little message in):

I made it this far!
I made it this far!
I made it this far!
I made it this far!
I made it this far!
I made it this far!
2022-08-09 20:53:24.03         ADCALC
2022-08-09 20:53:25.90   Done; Run time is about 00:18.7 (mm:ss)

[jdkleiner]

Hey @gcambridge and @jdkleiner -- I just had a breakthrough. Got the hsp2 on deq1 functioning like the one on hsp2. And, here is the output that the model tells me (someone put a little message in):

I made it this far!
I made it this far!
I made it this far!
I made it this far!
I made it this far!
I made it this far!
2022-08-09 20:53:24.03         ADCALC
2022-08-09 20:53:25.90   Done; Run time is about 00:18.7 (mm:ss)

This is most excellent (glad the little message I put in there is functioning as expected!). What test did you run that resulted in that output? I want to be sure I can replicate your result, I've tried running the test10 but not getting the same output

[jdkleiner]

@rburghol disregard the above comment, the following does indeed work for me.

# convert the UCI fileto h5
hsp2 import_uci test10.uci test10.h5
# do the run
hsp2 run test10.h5

# result: 
...
I made it this far!
I made it this far!
I made it this far!
I made it this far!
I made it this far!
I made it this far!
I made it this far!
I made it this far!
I made it this far!
2022-08-10 19:11:33.55         ADCALC
2022-08-10 19:11:34.94         CONS
2022-08-10 19:11:36.44         HTRCH
2022-08-10 19:11:39.16         SEDTRN
2022-08-10 19:11:48.64         RQUAL

[jdkleiner]

@rburghol SPECL.py has been set up and initial proof of concept testing successful (preforms a simple calculation on an array and returns the modified array to HYDR.py)

https://github.com/HARPgroup/HSPsquared/blob/specl/HSP2/SPECL.py

[rburghol]

See here for example of how to view HDF5 data in R: #24 (comment)