specl: ACTIONS (parser and code)

[rburghol]

Tasks

• Data Model
• Parser: see Psa #102
  • Code example for parse verification, action viewing in hdf5
• Outline Steps to integrate with State (see below: "Steps to integrate with STATE")
• Integrate with STATE Data Model for Shared STATE (supports model modularity) RFC #112
  • Operator Integer ID = 100
• Executable Code
• Unit Tests (see tests/test10specl)
  • Test basic function
  • Test start date (code is complete, but not tested yet)
  • Test number of repeats (code complete, not tested)
• Performance Tests

hdf5 Data Model

• hdf5 Path: /SPEC_ACTIONS/ACTIONS/table
• Example:

index	OPTYP	RANGE1	RANGE2	DC	DS	YR	MO	DA	HR	MN	D	T	VARI	S1	S2	AC	VALUE	TC	TS	NUM	CURLVL
0	PERLND	1		DY		1984	1	1	12		2	3	FNO3			+=	0.000000				1
1	PERLND	1		DY		1984	2	1	12		2	3	FNO3			+=	0.090044				1

HSPF UCI Data Model

Execution Code

• Fortran version SUBROUTINE PSPACT in 12.2

Runtime steps to integrate with STATE

1. Find integer key for source and destination data pointing to storage slot in state_ix
   • Use get_state(state_ix, state_paths, var_path); where var_path is the full hdf5 path to the data quantity
     • Note: if var_ix == False: var_ix = set_state(state_ix, state_paths, var_path, default_value)
     • This should be done at the beginning of model execution so that all entities that are used in state simulation exist. This is a fairly simple process, outlined in this PR: https://github.com/respec/HSPsquared/pull/123/files
2. Load source data for operation from STATE (get values from state_ix)
   • var_value = state_ix[var_ix]
3. Execute code for operation
   • var_value = some_function(var_value, some_other_operand)
4. Set value of target STATE var to result of state
   • state_ix[var_ix] = var_value

Manual STATE Example

# get the state index key for IVOL1 in Reach 001 with:

src_var_path = state_info['domain'] + "/IVOL1"; 
# note: src_var_path is by definition the "domain" + the variable name, so, 
src_var_ix = get_state(state_ix, state_paths, var_path)`
ivol1_val = state_ix[var_ix]
# Perform operation
ivol1_val = 1.1 * ivol1_val
# Store data (since ivol1 is source and target data we use same ix
state_ix[var_ix] = ivol1_val

Using ModelObject STATE Example

This outlines a process that use an object class to handle pre-run parsing of operation data to tokenization and runtime execution and state integration via the variable op_tokens. Details of this can be found in branch hydrocomp https://github.com/HARPgroup/HSPsquared/tree/hydrocomp

• Create a sub-class of the ModelObject
  • Ex: class SpeclAction(ModelObject):
  • Class handles all the basics of registration in state_paths and state_ix
  • Sub-class methods to customize parsing, validation and tokenization
• Initialize and populate OM model runtime Dicts with op codes and dependency info (op_codes, 'model_exec_list`)
  • Add support for new classes in om.py:model_class_loader
    • This is very simple, a line like this is all that is needed: model_object = [new class name](model_props.get('name'), container, model_props)
      • Where [new class name] is the class defined for the specact handler i.e. SpeclAction
      • It is possible that this can be done with a dynamic class name, so that there is no need to add a new line, similar to how the variable function is used in main.py to dynamically call model activity functions.
  • During setup, make calls to 3 functions:
    • model_loader_recursive(model_data, model_root_object)
    • model_path_loader(model_object_cache)
    • model_tokenizer_recursive(model_root_object, model_object_cache, model_exec_list)
• Handles tokenization in an integer Dict with individual integer values referring to the state_ix of source and destination variables, function types and operators (if a function has multiple modes).
• the final model_tokenizer_recursive() Handles hierarchical execution, sorts operators prior to runtime based on dependencies, stored in an ordered list model_exec_list

Supporting STATE inside an hsp2 njit function

• Non-jitted function (ex: hydr()) must allow argument state which contains all persistent state info as well as domain specific info, like the current function, operation, and segment.
• njit function (ex: _hydr()) must allow arguments: state_ix, state_paths, and state_info
  • state_info is Dict with character key and character value, containing domain info
    • domain is defined as state['domain'] = "/STATE/" + operation + "_" + segment + "/" + activity
    • This is obtained automatically in main.py via call to state_context_hsp2(state, operation, segment, activity)
    • domain is essentially the hdf5 path to the state variables for the given activity/segment and operation
  • state_ix is Dict with integer key and floating point value for containing actual runtime data state for scalar variables
  • state_info is Dict with character key (the hdf5 path of the variable) and integer value, with value being a pointer to the integer key of the variable in the state_ix Dict
• njit function must explicitly define which variables will be modifiable via the state facility.
  • Ex: hydr_ix = hydr_get_ix(state_ix, state_paths, state_info['domain'])
  • The function hydr_get_ix() returns a Dict with local variable character name as a key, and integer value of the key in state_ix
  • (EXAMPLE NOT YET COMPLETE) i.e., If state_info['domain'] = '/RCHRES/...

Testing / Debugging

Parse UCI

Parser now successfully handle simple "classic" actions.

• Example file https://github.com/HARPgroup/HSPsquared/blob/develop/tests/land_spec/hwmA51800.uci
  • UCI contains several lines like this:

  PERLND  1    DY  11984  1  1 12    2 3  FNO3           +=         0.         
  PERLND  1    DY  11984  2  1 12    2 3  FNO3           +=   0.090044

• hsp2 import_uci hwmA51800.uci hwmA51800.h5
  • the hdf5 now contains the correctly parsed data.

Verify ACTIONS parsing in R

• See UCI sample here:
• Read ACTIONS data from hdf5 file.

UCI 1: Excerpt of ACTION block

SPEC-ACTIONS
*** ACTIONS
***optyp range dc ds yr  mo da hr mn d t   vari  s1 s2 s3 ac  value    tc ts num
  <****><-><--><><-><--><-><-><-><-><><>  <----><-><-><-><-><--------> <> <-><->
  RCHRES  1    DY  11984  1  1 12    2 3  IVOL           +=         10.
END SPEC-ACTIONS

Code 1: View contents of uci parsed to hdf5 in R.

library("rhdf5")
# open the file in R
h5_file_name = "hwmA51800.h5" 
# open the file in R
h5f = H5Fopen(h5_file_name)

h5read(h5f , "/SPEC_ACTIONS/ACTIONS/table")[1:2,]

Outputs:

  index  OPTYP RANGE1 RANGE2 DC DS   YR MO DA HR MN D T VARI S1 S2 AC    VALUE
1     0 PERLND      1        DY    1984  1  1 12    2 3 FNO3       += 0.000000
2     1 PERLND      1        DY    1984  2  1 12    2 3 FNO3       += 0.090044
  TC TS NUM CURLVL
1                1
2                1

View contents of uci parsed to hdf5 in python.

Code 2: Print in python, and convert byte encoding.

import h5py
h5_file_name = "hwmA51800.h5"
f = h5py.File(h5_file_name,'r')
f['/SPEC_ACTIONS/ACTIONS/table']
f['/SPEC_ACTIONS/ACTIONS/table'][0:2,]
specla=f['/SPEC_ACTIONS/ACTIONS/table']
for idx, x in np.ndenumerate(specla):
   print(x[1].decode("utf-8"),x[2].decode("utf-8"), x[13].decode("utf-8"), x[16].decode("utf-8"), x[17])
   

Outputs:

array([(0, b'PERLND', b'1', b'', b'DY', b'', b'1984', b'1', b'1', b'12', b'', b'2', 3, b'FNO3', b'', b'', b'+=', 0.      , b'', b'', b'', 1),
       (1, b'PERLND', b'1', b'', b'DY', b'', b'1984', b'2', b'1', b'12', b'', b'2', 3, b'FNO3', b'', b'', b'+=', 0.090044, b'', b'', b'', 1)],
      dtype=[('index', '<i8'), ('OPTYP', 'S6'), ('RANGE1', 'S1'), ('RANGE2', 'S1'), ('DC', 'S2'), ('DS', 'S1'), ('YR', 'S4'), ('MO', 'S1'), ('DA', 'S1'), ('HR', 'S2'), ('MN', 'S1'), ('D', 'S1'), ('T', '<i8'), ('VARI', 'S4'), ('S1', 'S1'), ('S2', 'S1'), ('AC', 'S2'), ('VALUE', '<f8'), ('TC', 'S1'), ('TS', 'S1'), ('NUM', 'S1'), ('CURLVL', '<i8')])
...
PERLND 1 FNO3 += 0.0
PERLND 1 FNO3 += 0.090044
PERLND 1 FNO3 += 1.91746
PERLND 1 FNO3 += 2.733176
PERLND 1 FNO3 += 1.23e-07
PERLND 1 FNO3 += 6.302083
PERLND 1 FNO3 += 0.207089

[rburghol]

Making the actions work with STATE.

• Current call: specl(ui, ts, step, state_info, state_paths, state_ix, specactions)
• What is STATEID of action?
  • Any quantity to be modified has a numeric value with a unique integer key in the state_ix array
  • Any quantity that can be viewed or modified also its unique state_ix ID stored with a character key with it's full path (domain + variable name)
  • Finding the variable requires knowing the "domain" and variable name, i.e., the hdf5 path
    • Example: to access or modify O1 in HYDR:
      • the domain is '/STATE/RCHRES_R001/HYDR/'
      • The variable name is O1
      • The full path is /STATE/RCHRES_R001/HYDR/O1
    • And then state_paths[/STATE/RCHRES_R001/HYDR/O1] = integer key for value in state_ix
• Does domain have a match with Spec actions that have been defined?
  • Classic spec actions have 2 of the 3 elements needed to identify DOMAIN. The operation and segment, but the activity is not included.
  • Classic ACTIONS also include the variable name.
  • The variable name is globally unique (is this true everywhere?)
    • Therefore, multiple activity could modify the same variable?
    • For now we assume that we will keep all three in path, as there may be some non-unique variable names that need to be modified.
• Find ACTIONS with matching (or null) time factors
  • yr = state_ix[state_paths['STATE/timer/year']]
  • mo = state_ix[state_paths['STATE/timer/month']]
  • da = state_ix[state_paths['STATE/timer/day']]
  • hr = state_ix[state_paths['STATE/timer/hour']]
  • mi = state_ix[state_paths['STATE/timer/minute']]
  • se= state_ix[state_paths['STATE/timer/second']]
  • Query ACTIONS table for those modifying the given domain and time
• Find ACTIONS with OPTYP & VARI matching domain

Table 1: Example for class ACTIONS that increases IVOL by 1.0 on January 1st, 1984 .

index	OPTYP	RANGE1	RANGE2	DC	DS	YR	MO	DA	HR	MN	D	T	VARI	S1	S2	AC	VALUE	TC	TS	NUM	CURLVL
0	RCHRES	1		DY		1984	1	1	12		2	3	IVOL			+=	1.000000				1

• Approaches:
  • Loop thru each action, each time, and see if there is a domain match?
  • Pre-sort actions according to the domain in question when calling the base routine
    • i.e., the specactions Dict that gets sent in when executing HYDR for RCHRES1 has already been sorted thru,
    • presumably we would do this once at the very beginning of the model, then stash in a Dict
    • Or, we can do this at parse UCI time, and store the SPECL in the hdf5 already keyed by domain, and then load it when runtime occurs
    • Potentially add "stateid" as column to the /SPEC_ACTIONS/ACTIONS/table?
    • Or, better to have a full parsed ACTIONS table, with tokens?
    • ACTIONID, OP1
• Enabling STATE for a function in HSP2 main.py (other than HYDR)
  • Add fn to declare modifiable variables [function]_init_ix(state_ix, state_paths, domain)
  • Note: domain = "/STATE/" + operation + "_" + segment + "/" + activity ; per hsp2 convention in hdf5 (see domain setup in

    HSPsquared/HSP2/state.py

    Line 124 in 4431180

    state['domain'] = "/STATE/" + operation + "_" + segment + "/" + activity

    )
  • Call [function]_init_ix() at the beginning of the run through of the function for a given domain