Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Dynamic python code #65

Closed
rburghol opened this issue Jun 20, 2023 · 6 comments
Closed

Dynamic python code #65

rburghol opened this issue Jun 20, 2023 · 6 comments
Assignees

Comments

@rburghol
Copy link

rburghol commented Jun 20, 2023

Goal: have a set of functions defined by local code that are executable in @njit. see respec#113

  • Auto load files with name [riverseg].py - here: lines 342-359
  • Currently testing with branch state on deq1
    • test seg: PL3_5250_0001
    • '/home/rob/working/modeling/hsp2/river_wd/PL3_5250_0001.py'
    • If theres a PL3_5250_0001.py file in that directory, it'll load the file and the functioned defined within
      • Functions defined in that file
        • That way you can have custom equations for individual riversegs
        • And it allows faster development without making pull requests
    • Within PL3_5250_0001.py is the function om_test_fn()
  • @jdkleiner this is now working nicely.
  • Performance Notes: Performance lags a bit since it recompiles the dynamic code every time (adds 7-8 seconds), so maybe a necessary evil, or maybe only to be used for testing or when we simply can't make it another way. Lag quantification and potential ways of unerstanding this lag.
    • Performance comparison:
      • branch fixjup (develop with install fix for bad jupyter line) takes 7 seconds (:15 on first compile)
      • branch spects loads 20-30 OM components and takes 12 seconds (:22 on 1st compile)
      • branch state loads 0 OM, and takes 17 seconds each time (no diff w/1st compile), even if there is no dynamic code loaded, or empty dynamic code function.
      • branch statex used to disable, then sequentially enable things (run from 1984-2001):
        1. 9 seconds with state loaded and added as arg to hydr(), and does setting state_ix[o1_ix' ... each timestep, but no computations
        2. 9 seconds with state loaded, dynamic module loaded in main, and all state loading stuff in HYDR, but no njoi module loading, nor any code in hydr executed.
        3. 9.1 seconds with b + load dynamic module in HYDR
        4. 9.1 seconds d + pass state_info to _hydr_()
        5. 9.0 seconds e + pass state_paths
        6. 15.3 f + pass all other Dicts and state_step_hydr
        7. 9.0 seconds g + do not pass state_step_hydr
        8. 9.0 seconds h + added and set all state data in _hydr_()
    • Known and potential overhead:
      • Passing in the function accounts for over 6.x seconds
      • Executing the dynamic code accounts for 1.x seconds
      • 80% of performance hit involved with dynamic code loading is simply passing the dynamic function into _hydr_
      • Performance hit does not scale with increased time span, so it is a 1-time incursion each model run.
  • main.py
    • calls load_dynamics() (need to move this from main.py since the dynamic code must be loaded in the same module that it is called in, however, maybe we can also pass it in as part of a standard python dictionary, but then split it out as an individual function when passing t
  • hsp2_local_py = dynamic_module_import(fbase, local_path + "/" + fbase + ".py", "hsp2_local_py")
    • This imports the hsp2_local_py module
  • hsp2_local_py.state_step_hydr(model_exec_list, op_tokens, state_ix, dict_ix, ts_ix, hydr_ix, step)

Loading and Executing @njit code,

Since all functions called by njit code must be njit also, the dynamic functions must be defined before the first call of the larger routine, such as _hydr_().

  • njit function must be loaded prior to calling hydr()`

Example Withdrawal

  • Using test h5 file PL3_5250_0001.h5
  • Create a file with the following py code PL3_5250_0001.py:
import sys
from numba import int8, float32, njit, types, typed # import the types
print("Loaded a set of  HSP2 code!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!")

@njit
def state_step_hydr(state_ix, dict_ix, ts_ix, hydr_ix, step):
    fn_defined = False
    state_ix[hydr_ix['O1']] = 0.0 * 1.547
    if (step <= 1):
        print("Custom state_step_hydr() called")
        print("state at start", state_ix)
    if (step > 24*40*364):
        print("state at end", state_ix)
    return
  • Then run hsp2 and export hydr:
hsp2 hsp2 run PL3_5250_0001.h5
Rscript /opt/model/meta_model/scripts/h5/export_hsp_h5.R PL3_5250_0001.h5 hydr_10mgd.csv "//RESULTS/RCHRES_R001/HYDR/table"
  • Then edit PL3_5250_0001.py and set the withdrawa (O1)l to 0.0
  • Then run and export again:
hsp2 hsp2 run PL3_5250_0001.h5
Rscript /opt/model/meta_model/scripts/h5/export_hsp_h5.R PL3_5250_0001.h5 hydr.csv "//RESULTS/RCHRES_R001/HYDR/table"
@jdkleiner
Copy link
Member

jdkleiner commented Jun 21, 2023

Notes 6/21/23:

These notes have been incorporated into the body of this issue, and in a separate issue #67

@rburghol
Copy link
Author

rburghol commented Jun 21, 2023

This allows you to test manually at the command line. More illuminating is the code in the body of the issue.


from numba.typed import Dict
from numba import types
import numpy as np
os.chdir("/opt/model/HSPsquared")
from HSP2.utilities_specl import *
from HSP2.SPECL import specl, _specl_
from HSP2.om_model_object import *
from HSP2.om_equation import *
from HSP2.om_data_matrix import *
from HSP2.om_sim_timer import *
from HSP2.om_model_linkage import ModelLinkage, step_model_link
from HSP2.om_model_broadcast import *

hydr_ix = Dict.empty(key_type=types.unicode_type, value_type=types.int64)
op_tokens = Dict.empty(key_type=types.int64, value_type=types.i8[:])
state_paths = Dict.empty(key_type=types.unicode_type, value_type=types.int64)
state_ix = Dict.empty(key_type=types.int64, value_type=types.float64)
dict_ix = Dict.empty(key_type=types.int64, value_type=types.float64[:,:])
ts_ix = Dict.empty(key_type=types.int64, value_type=types.float64[:])

local_path = os.getcwd()
fbase="PL3_5250_0001"
numdict = numdict=np.asarray([],dtype="float32")
io_manager, siminfo, op_tokens, state_paths, state_ix, dict_ix, ts_ix, model_object_cache = numdict,numdict,numdict,numdict,numdict,numdict,numdict,numdict

hsp2_local_py = dynamic_module_import(fbase, local_path + "/" + fbase + ".py", "hsp2_local_py")

# just set some test values
hydr_ix['O1'] = 1
from hsp2_local_py import state_step_hydr
state_step_hydr(numdict, op_tokens, state_ix, dict_ix, ts_ix, hydr_ix, 1)
# now show the values
state_ix[hydr_ix['O1']]

Sample code to put into a model [riverseg].py file.

import sys

print("Loaded a dynamic HSP2 code file!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!")

def om_init_model(io_manager, siminfo, op_tokens, state_paths, state_ix, dict_ix, ts_ix, model_object_cache):
    print("Initialize om_init_model from code")

@njit
def state_step_hydr(model_exec_list, op_tokens, state_ix, dict_ix, ts_ix, hydr_ix, step):
    print("Do something")

print("Loaded state_step_hydr()")

@rburghol rburghol self-assigned this Jun 22, 2023
@jdkleiner
Copy link
Member

jdkleiner commented Jun 23, 2023

@rburghol Couple notes when running the test above:

  • Need to import os before the line os.chdir("/opt/model/HSPsquared") i.e. just needs a import os
  • When I try running I get NameError: name 'dynamic_module_import' is not defined, where does the function dynamic_module_import get loaded from?

@rburghol
Copy link
Author

Hey @jdkleiner -- sorry, the test code does suffer from the lack you describe :) -- dynamic_module_import() is in the HSP2/STATE.py file. However, you may find the actual embedded code in hsp2 more interesting. That is the code that is in the body of this issue at the top. Also, in the state branch now there is a sample file in tests/test10/test10.py that will function fully, and should run out of the box in that branch on your install, provided of course that there is no incompatibility that I did not foresee in a windows versus linux install?

@rburghol
Copy link
Author

@rburghol rburghol mentioned this issue Jun 1, 2023
6 tasks
@rburghol
Copy link
Author

rburghol commented Oct 3, 2023

Tested on:

  • Linux, Python 3.8.10
  • Windows Python 3.8.9
    • powershell works
    • git bash.exe works also

@rburghol rburghol closed this as completed Oct 3, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants