Set up dev environment and/or install paths and/or scripts for hsp2 tinkering

[rburghol]

Overview

• Goal: to be able to switch effortlessly between dev and production version of hsp2 to enable us to break things in specl development without screwing up other modelers project work.
• Approach: have a separate install on deq1 for testing, then deploy on deq2 for larger model testing once stability is assured.
• Note: installation of python can be tricky, since it is normally expected to be run by a single user, but in development mode, that is not practical. The default is to install in a users $HOME/.local/lib directory which then becomes the first thing that python checks when running the package. For info on overcoming this see: Install HSP2 Python on Linux #9
• See also Windows: Install & Run HSP2 Python on Windows #48

Trees and branches:

• /opt/model/HSPsquared: this is the sole place for development where we can run the model on deq2
  • branchdevelop: exact duplicate of upstream respec repo. This is the branch we should use as the master to run our models on for the time being since it has the IWAT fix (which fixes many uci defaults)
  • branch master: exact duplicate of upstream respec repo. Should be the one to run production on after they merge the IWAT fix to master.
  • branch specl: current special actions development branch. Consider it to be a "psuedo" main branch, that is, it is our place for the current functioning special actions code, and we should branch from it and do pull requests against it when we make code changes.
  • I resynched our master with the respec master, but managed to maintain all of our previous branches. I did kill the old specl branch to make it a copy of the respec master for starters.
• Process to resynch with upstream:
  • git remote add upstream https://github.com/respec/HSPsquared
• See also:
  • Initial creation of python hsp2 installer for Linux Create setup.py for pip install HSPsquared respec/HSPsquared#55
  • Our local install for deq detailed here: Install HSP2 Python on Linux #9

Options

• Use git to checkout the harp dev branch when running tests
• Because specl are defined in the UCI, or optionally in a separate file (we will code this), we can perhaps avoid much of this by simply not enabling spec-actions unless we are testing. In other words, our base scenario for modeling is now hsp2_2022, we can create a scenario called specl_2022.
• Use some sort of python environment to enable a custom hsp2 executable
  • See the tensor flow demo for entering an env 2022/01/10 Tensorflow hydrology #15
• Create a parallel executable via entry points:
  • Code 1 below shows how the current install in hsp2 uses pip and the concept of entry points to do this. (see below)
  • /usr/local/bin/hsp2 has this: load_entry_point('HSPsquared', 'console_scripts', 'hsp2'):
    • HSPsquared name of the package
    • console_scripts the hsp2 installer uses the console_scripts argument, which says that this should be a command line executable
    • hsp2 name of the executable to create
  • entry points are set up in the file setup.py
    • This in turn creates /opt/model/HSPsquared/HSPsquared.egg-info/entry_points.txt
    • Note: HSPsquared.egg-info is in .gitignore so it does not appear until after you run the pip setup (see Install HSP2 Python on Linux #9 )
  • Here is a potentially useful blog on entry points https://amir.rachum.com/blog/2017/07/28/python-entry-points/
• Set up dev testing environment on deq1

Code 1: Python install script created when we run pip install -e (see #9), which creates a script hsp2 that runs the model. This is stored in /usr/local/bin/hsp2 (so everyone can run it), and it's contents looks like:

#!/usr/bin/python3
# EASY-INSTALL-ENTRY-SCRIPT: 'HSPsquared','console_scripts','hsp2'
__requires__ = 'HSPsquared'
import re
import sys
from pkg_resources import load_entry_point

if __name__ == '__main__':
    sys.argv[0] = re.sub(r'(-script\.pyw?|\.exe)?$', '', sys.argv[0])
    sys.exit(
        load_entry_point('HSPsquared', 'console_scripts', 'hsp2')()
    )

Ideally, I think we'd have a script called hsp2dev that if called, would be able to seamlessly integrate with our existing hspf workflows, since the configuration files for our models we can specify the executable we want (hspf or hsp2, or hsp2dev for example) and it should work automatically. This will allow us to just set up a separate scenario for dev testing, indicate the executable we want, and blammo, break stuff to our hearts content.

[jdkleiner]

Looked into this a little bit, perhaps venv is the optimal method?

• Creation of virtual environments: https://docs.python.org/3/library/venv.html
• Setup in linux: https://cloud.google.com/python/docs/setup#linux_1

[rburghol]

@jdkleiner thanks for doingsome leg work on this. Do you have time to meet up and discuss this one sometime today?

[jdkleiner]

@rburghol sure thing, looking forward to diving into this

[rburghol]

@gcambridge at your earliest convenience, can you please push the branch that you have been developing on? We want to preserve the progress that you made and see the locations and prototype functions for us to build on. Thanks!

[jdkleiner]

Soln. use virtual environments: #55