Specl and Objects hdf5 data model

[rburghol]

Overview

All data paths are to be added to hdf5 structure.

• /SPECL: holds old-style special action definitions from parser
• /OBJECTS: holds new styled dynamic object simulation (may be used to replicate or provide backend for specl)
• /STATE: Use to store state, facilitates cross-domain (like RCHRES_001 to RCHRES_002) data transfer, and for storing the last state in the event of an exception that halts model execution.
  • See: Data Model for Shared STATE (supports model modularity) RFC respec/HSPsquared#112
  • To minimize bandwodth, only written to hdf5 file in the event of an exception
  • Stored in memory as Dict with name state_ix (see below, In Memory Runtime Variables)
  • Examples:
    • /STATE/RCHRES_R001/HYDR/IVOL - Holds state of IVOL variable from river reach with ID 1 in model.

See also:

• How data passes in python/hsp2 Data Passing/Inputs in hsp2/python #18
• Migrating OM Model connections Model linkages #30

In Memory Runtime Variables

• state_ix

As it stands, it appears that the entirety of the hsp2 simulation state data is stored in the hdf5 table, meaning there are no persistent object of any sort, but rather, functions that operate on the data that is present in the hdf5, passed in to the given function in the form of the ui and ts variables. SPECL or other dynamic objects may need to be stored in the hdf5 table? Or do we deviate from the standard process?

• Basic object parameterization structure (not log data)
  • Store each type of object as their own tree in the hdf5, referring to their parent, via entity_type and entity_id
    • /objects/equation/eq0001/ has columns entity_type, entity_id, equation, 'default`, ...
    • /objects/datamatrix/dm0001/ has columns entity_type, entity_id, matrix, lucol1, lutype1, 'default`, ...
• Data structure for complex OM objects LOG data
  • Decide, can we create hdf5 tables for each complex object at hsp2 import_uci step, then use them to log during simulation?
  • Can we store different state variables for different unique_object_id in the same table level?, like:
    • /timeseries/facility001/specl/Qintake has object_class, equation, default
    • /timeseries/facility001/specl/base_demand_pstatus_mgd has object_class, matrix, default, lucol1, lucol2, intmethod1,...
• Formulate data structure for storing specl see SPEC-ACTIONS in hsp2 respec/HSPsquared#90
• Is there a speed benefit or cost from using persistent objects or hdf5 state storage only (if we have objects that use the state variables, does instantiating them each time come at a cost?)
  • arrays can be sliced and these slices act as a pass-by-reference (see numpy array slices below)
  • See references:
    • basic array setting and references/views in numpy https://jakevdp.github.io/PythonDataScienceHandbook/02.00-introduction-to-numpy.html
    • Computations on np arrays: https://jakevdp.github.io/PythonDataScienceHandbook/02.03-computation-on-arrays-ufuncs.html
  • see also Data Passing/Inputs in hsp2/python #18
  • note: the HYDR.py routines use numba which "... is a just-in-time compiler for Python that works best on code that uses NumPy arrays and functions, and loops."
• Can we slices data from master np.arrays into hierarchically ordered executable arrays and then leverage numpy speed to do calculations on them?

numpy array slices

Note: np array slices are views (pass by references), but a simple extract of a value is NOT.

a = np.array([[1,2,3],[2,4,6]])
b = a[1,2] # extracts the 6
b
> 6
b = 77
b
> 77
a   # is unchanged
> array([[1, 2, 3],
>       [2, 4, 6]])

b = a[1:2,2:3] # extract a slice
b   # show the contents and it is an array, with 6 as value
> array([[6]])
# now, set the value of the extracted element
b[[0]] = 77
a    # now the final element of a has been set to 77
> array([[ 1,  2,  3],
>       [ 2,  4, 77]])

b = 99 # doing this OVERWRITES b, rather than setting the *view* element
a       # so final element of a is STILL 77
array([[ 1,  2,  3],
       [ 2,  4, 77]])

[rburghol]

Note. np array slices are views (pass by references), but a simple extract of a value is NOT.

a = np.array([[1,2,3],[2,4,6]])
b = a[1,2] # extracts the 6
b
> 6
b = 77
b
> 77
a   # is unchanged
> array([[1, 2, 3],
>       [2, 4, 6]])

b = a[1:2,2:3] # extract a slice
b   # show the contents and it is an array, with 6 as value
> array([[6]])
# now, set the value of the extracted element
b[[0]] = 77
a    # now the final element of a has been set to 77
> array([[ 1,  2,  3],
>       [ 2,  4, 77]])

b = 99 # doing this OVERWRITES b, rather than setting the *view* element
a       # so final element of a is STILL 77
array([[ 1,  2,  3],
       [ 2,  4, 77]])

[rburghol]

@jdkleiner -- just tagged you in this so we can review at least some of this in todays work session.

[jdkleiner]

@rburghol Below works as well, so it might not be just slicing that does it, setting b = a and then setting a value in the b array updates the a array

a = np.array([[1,2,3],[2,4,6]])
b = a
print(b)
[[1 2 3]
 [2 4 6]]

b[1,2] = 77
print(a)
[[ 1  2  3]
 [ 2  4 77]]