Reinstate ogusa

[hdoupe]

This PR reinstates OG-USA on TaxBrain. In doing so, the input processing logic for the dynamic/views.py function dyanamic_input was changed so that it used the input processing logic introduced in PR #641. Further, the input processing that was done in DynamicCompute.submit_ogusa_calculation is now done in dynamic_input.

[hdoupe]

Below is the traceback for the error that I am getting now. The problem is that Tax-Calculator is looking for a puf.csv file but we swapped to the compressed file, puf.csv.gz. I unzipped the puf.csv.gz file and now everything seems to be running just fine. I'm going to dig into this a little further to see how the path to the puf file can be set to puf.csv.gz.

@jdebacker @rickecon @PeterDSteinberg

  Traceback (most recent call last):
File "/anaconda/envs/aei_dropq/lib/python2.7/site-packages/celery/app/trace.py", line 374, in trace_task
  R = retval = fun(*args, **kwargs)
File "/anaconda/envs/aei_dropq/lib/python2.7/site-packages/celery/app/trace.py", line 629, in __protected_call__
  return self.run(*args, **kwargs)
File "/Users/henrydoupe/Documents/PolicyBrain/deploy/taxbrain_server/celery_tasks.py", line 160, in ogusa_async
  guid=guid)
File "/Users/henrydoupe/Documents/PolicyBrain/deploy/taxbrain_server/run_ogusa.py", line 50, in run_micro_macro
  runner(**kwargs)
File "/anaconda/envs/aei_dropq/lib/python2.7/site-packages/ogusa/scripts/execute.py", line 48, in runner
  start_year=user_params['start_year'], reform=reform, guid=guid)
File "/anaconda/envs/aei_dropq/lib/python2.7/site-packages/ogusa/txfunc.py", line 1566, in get_tax_func_estimate
  analytical_mtrs, age_specific, reform)
File "/anaconda/envs/aei_dropq/lib/python2.7/site-packages/ogusa/txfunc.py", line 1039, in tax_func_estimate
  start_year=beg_yr, reform=reform)
File "/anaconda/envs/aei_dropq/lib/python2.7/site-packages/ogusa/get_micro_data.py", line 102, in get_data
  reform=reform)
File "/anaconda/envs/aei_dropq/lib/python2.7/site-packages/ogusa/get_micro_data.py", line 55, in get_calculator
  records1 = Records()
File "/anaconda/envs/aei_dropq/lib/python2.7/site-packages/taxcalc/records.py", line 120, in __init__
  self._read_data(data, exact_calculations)
File "/anaconda/envs/aei_dropq/lib/python2.7/site-packages/taxcalc/records.py", line 406, in _read_data
  taxdf = read_egg_csv(data)  # pragma: no cover
File "/anaconda/envs/aei_dropq/lib/python2.7/site-packages/taxcalc/utils.py", line 1248, in read_egg_csv
  raise ValueError('could not read {} data from egg'.format(fname))
ValueError: could not read puf.csv data from egg

[rickecon]

This looks good from my end. @hdoupe @jdebacker @MattHJensen

[brittainhard]

@hdoupe We need this PR merged to complete issues #481 and #482. Would be be possible to get it merged tomorrow so we can build off this code? We will have to change some things in #703 to get it to mesh with this one.

[hdoupe]

@brittainhard Most of the work here is done. One problem is that Tax-Calculator is looking for the puf file at taxbrain_server/puf.csv instead of taxbrain_server/puf.csv.gz. If I unzip puf.csv.gz then the model runs. But after it estimates the tax function parameters, I get this error:

Traceback (most recent call last):
  File "/anaconda/envs/aei_dropq/lib/python2.7/site-packages/celery/app/trace.py", line 374, in trace_task
    R = retval = fun(*args, **kwargs)
  File "/anaconda/envs/aei_dropq/lib/python2.7/site-packages/celery/app/trace.py", line 629, in __protected_call__
    return self.run(*args, **kwargs)
  File "/Users/henrydoupe/Documents/PolicyBrain/deploy/taxbrain_server/celery_tasks.py", line 165, in ogusa_async
    guid=guid)
  File "/Users/henrydoupe/Documents/PolicyBrain/deploy/taxbrain_server/run_ogusa.py", line 48, in run_micro_macro
    runner(**kwargs)
  File "/Users/henrydoupe/Documents/OG-USA/Python/ogusa/scripts/execute.py", line 50, in runner
    run_params = ogusa.parameters.get_parameters(test=test, baseline=baseline, guid=guid)
  File "/Users/henrydoupe/Documents/OG-USA/Python/ogusa/parameters.py", line 452, in get_parameters
    flag_graphs)
  File "/Users/henrydoupe/Documents/OG-USA/Python/ogusa/demographics.py", line 699, in get_pop_objs
    graph=False)
  File "/Users/henrydoupe/Documents/OG-USA/Python/ogusa/demographics.py", line 287, in get_mort
    mort_rates_sub = np.zeros(num_sub_bins * max_yr, dtype=np.float64)
TypeError: 'float' object cannot be interpreted as an index

I'm not sure if this is a numpy version problem or something else.

[brittainhard]

@hdoupe I know in the case of CCC it wants the extracted puf.csv. This could also be the case with OGUSA

[hdoupe]

@brittainhard this is the case for OG-USA, too. Where does B-Tax get the extracted puf.csv file?

[brittainhard]

Got a syntax error here @hdoupe

[hdoupe]

Thanks @brittainhard. I'll have a fix up shortly.

[hdoupe]

@brittainhard there are a couple other issues. Hang on

[hdoupe]

@brittainhard there are a couple other issues. Hang on

Should be good to go now.

[brittainhard]

@hdoupe i'm seeing the error you posted earlier, TypeError: ‘float’ object cannot be interpreted as an index. Did you figure out a solution to it?

[hdoupe]

@brittainhard No, I did not. The up-to-date (including OG-USA PRs merged after 0.5.9) OG-USA version I worked with in PSLmodels/OG-Core#316 worked fine. I need to dig in deeper to see why I was able to run the regression reforms, but now it doesn't work.

My hunches are:

1. This will be resolved once a new version of OG-USA is released that is more similar to what was used in Regression testing PSLmodels/OG-Core#316.
2. There is an issue with how PolicyBrain is interacting with OG-USA.

[hdoupe]

In the regression tests, I ran the reforms with age_specific set to True but in PolicyBrain it is set to False.

This could be causing a problem. @jdebacker what do you think?

I'm also going to try to run the run_ogusa.py script in the ospcdyn environment to see if there is an environment specific problem.

@brittainhard @PeterDSteinberg @rickecon

[jdebacker]

@hdoupe I do not this that is the issue. My guess is the error is a result of some issue with a version of the packages. OG-USA needs the environment set correctly to run without error. Is the environment set exactly as in the OG-USA environment.yml?

[hdoupe]

Ok, no I think it is running in the aei_dropq environment. I inserted this code snippet to get an idea of what was running:

def run_micro_macro(start_year, reform, user_params, guid):
    proc = sp.Popen("conda list", stdout=sp.PIPE, stderr=sp.STDOUT,
                    shell=True, cwd=os.path.abspath(os.curdir),
                    env=os.environ)
    with open('env.txt', 'w') as f:
        f.write(proc.stdout.read())

    start_time = time.time()

    REFORM_DIR = "./OUTPUT_REFORM_" + guid
    BASELINE_DIR = "./OUTPUT_BASELINE_" + guid

    with open("log_{}.log".format(guid), 'w') as f:
        f.write("guid: {}\n".format(guid))
        f.write("reform: {}\n".format(reform))
        f.write("user_params: {}\n".format(user_params))
    ...

and got:

# packages in environment at /anaconda/envs/aei_dropq:
#
alabaster                 0.7.10                   py27_0  
amqp                      2.2.2                     <pip>
argh                      0.26.2                    <pip>
argparse                  1.2.1                     <pip>
asn1crypto                0.23.0                    <pip>
babel                     2.5.0                    py27_0  
backports                 1.0                      py27_0  
backports-abc             0.5                       <pip>
backports_abc             0.5                      py27_0  
bcrypt                    3.1.4                     <pip>
beautifulsoup4            4.6.0                     <pip>
billiard                  3.5.0.3                   <pip>
bkcharts                  0.2                      py27_0  
bokeh                     0.12.7                   py27_0  
boto                      2.48.0                    <pip>
btax                      0.1.9                    py27_0    ospc
celery                    4.1.0                     <pip>
certifi                   2016.2.28                py27_0  
cffi                      1.11.2                    <pip>
click                     6.7                      py27_0  
cryptography              2.1.2                     <pip>
cycler                    0.10.0                   py27_0  
decorator                 4.1.2                     <pip>
diff-match-patch          20121119                  <pip>
dj-database-url           0.3.0                     <pip>
dj-static                 0.0.6                     <pip>
Django                    1.8.2                     <pip>
django-appconf            1.0.2                     <pip>
django-debug-toolbar      1.2.1                     <pip>
django-hermes             1.4.2                     <pip>
django-htmlmin            0.10.0                    <pip>
django-import-export      0.5.1                     <pip>
django-ipware             1.1.6                     <pip>
django-queryset-csv       0.2.10                    <pip>
django-storages           1.6.5                     <pip>
django-toolbelt           0.0.1                     <pip>
django-user-accounts      2.0.3                     <pip>
django-uuidfield          0.5.0                     <pip>
docutils                  0.14                     py27_0  
enum-compat               0.0.2                     <pip>
enum34                    1.1.6                    py27_0  
enum34                    1.1.6                     <pip>
et-xmlfile                1.0.1                     <pip>
eventlet                  0.20.0                    <pip>
Fabric                    1.14.0                    <pip>
flask                     0.12.2                   py27_0  
freetype                  2.5.5                         2  
funcsigs                  1.0.2                    py27_0  
functools32               3.2.3.post2               <pip>
functools32               3.2.3.2                  py27_0  
futures                   3.1.1                    py27_0  
gevent                    1.2.2                    py27_0  
gnureadline               6.3.8                     <pip>
greenlet                  0.4.12                   py27_0  
gunicorn                  19.7.1                    <pip>
html5lib                  0.999999999               <pip>
icu                       54.1                          0  
idna                      2.6                       <pip>
imagesize                 0.7.1                    py27_0  
ipaddress                 1.0.18                    <pip>
ipdb                      0.8                       <pip>
ipython                   2.2.0                     <pip>
ipython-genutils          0.2.0                     <pip>
itsdangerous              0.24                     py27_0  
jbig                      2.1                           0  
jdcal                     1.3                       <pip>
jinja2                    2.9.6                    py27_0  
jpeg                      9b                            0  
jsonfield                 2.0.2                     <pip>
jsonschema                2.6.0                     <pip>
jupyter-core              4.3.0                     <pip>
kombu                     4.1.0                     <pip>
libpng                    1.6.30                        1  
libtiff                   4.0.6                         3  
livereload                2.5.1                     <pip>
llvmlite                  0.18.0                   py27_0  
markupsafe                1.0                      py27_0  
matplotlib                2.0.2               np112py27_0  
meld3                     1.0.2                     <pip>
mkl                       2017.0.3                      0  
mock                      2.0.0                     <pip>
nbformat                  4.4.0                     <pip>
newrelic                  2.58.2.45                 <pip>
nomkl                     1.0                           0  
numba                     0.33.0              np112py27_0  
numpy                     1.12.1             py27_nomkl_0  [nomkl]
odfpy                     1.3.5                     <pip>
ogusa                     0.5.9                    py27_0    ospc/label/dev
olefile                   0.44                     py27_0  
openpyxl                  2.4.9                     <pip>
openssl                   1.0.2l                        0  
pandas                    0.20.3                   py27_0  
paramiko                  2.3.1                     <pip>
pathtools                 0.1.2                     <pip>
pbr                       3.1.1                     <pip>
pdfkit                    0.4.1                     <pip>
pillow                    4.2.1                    py27_0  
pip                       9.0.1                    py27_1  
plotly                    2.2.0                     <pip>
port-for                  0.3.1                     <pip>
psycopg2                  2.7.3.2                   <pip>
py                        1.4.34                   py27_0  
pyasn1                    0.3.7                     <pip>
pycparser                 2.18                      <pip>
pygments                  2.2.0                    py27_0  
PyNaCl                    1.1.2                     <pip>
pyparsing                 2.2.0                    py27_0  
pyqt                      5.6.0                    py27_2  
pytest                    3.2.1                    py27_0  
pytest-django             3.1.2                     <pip>
python                    2.7.12                        1  
python-dateutil           2.6.1                    py27_0  
python-http-client        3.0.0                     <pip>
pytz                      2017.2                   py27_0  
PyYAML                    3.12                      <pip>
pyyaml                    3.12                     py27_0  
qt                        5.6.2                         2  
readline                  6.2                           2  
redis                     3.2.0                         0  
redis                     2.10.6                    <pip>
requests                  2.14.2                   py27_0  
requests-mock             1.3.0                     <pip>
retrying                  1.3.3                     <pip>
scipy                     0.19.1          np112py27_nomkl_0  [nomkl]
sendgrid                  3.6.5                     <pip>
sendgrid-django           4.1.0                     <pip>
setuptools                36.4.0                   py27_1  
singledispatch            3.4.0.3                  py27_0  
singledispatch            3.4.0.3                   <pip>
sip                       4.18                     py27_0  
six                       1.10.0                   py27_0  
snowballstemmer           1.2.1                    py27_0  
sphinx                    1.6.3                    py27_0  
sphinx-autobuild          0.7.1                     <pip>
sphinx-rtd-theme          0.2.4                     <pip>
sphinxcontrib             1.0                      py27_0  
sphinxcontrib-websupport  1.0.1                    py27_0  
sqlite                    3.13.0                        0  
sqlparse                  0.2.4                     <pip>
ssl_match_hostname        3.5.0.1                  py27_0  
static3                   0.5.1                     <pip>
subprocess32              3.2.7                    py27_0  
supervisor                3.3.3                     <pip>
tablib                    0.12.1                    <pip>
taxcalc                   0.10.2                   py27_0    ospc
tk                        8.5.18                        0  
toolz                     0.8.2                    py27_0  
tornado                   4.5.2                    py27_0  
tornado                   4.5.2                     <pip>
traitlets                 4.3.2                     <pip>
typing                    3.6.2                    py27_0  
unicodecsv                0.14.1                    <pip>
vine                      1.1.4                     <pip>
waitress                  1.1.0                     <pip>
watchdog                  0.8.3                     <pip>
webencodings              0.5.1                     <pip>
werkzeug                  0.12.2                   py27_0  
wheel                     0.29.0                   py27_0  
xlrd                      1.1.0                    py27_0  
xlwt                      1.3.0                     <pip>
xz                        5.2.3                         0  
yaml                      0.1.6                         0  
yolk                      0.4.3                     <pip>
zlib                      1.2.11                        0  

You can see that pandas is 0.20.3 instead of 0.18.0, numpy is 1.12.1 instead of 1.11.2, taxcalc is 0.10.2 instead of 0.8.3, scipy is 0.19.1 instead of 0.18.1

[jdebacker]

@hdoupe I'm not 100% sure, but I think the int/float issue is due to the difference in versions from NumPy.

If you can pin 1.11.2 for just OG-USA, great. If not, the fix will take longer (but needs to be done anyway) to go into OG-USA and make sure we have the correct data types specified. I did try to do this before, but couldn't figure it out before giving up and pinning to the older NumPy... now that's coming back to bite.

[hdoupe]

I'm running the same run_ogusa.py file again, but I'm running it in the ospcdyn environment. I did something along the lines:

rm -rf OG-USA/
git clone https://www.github.com/hdoupe/OG-USA
cd OG-USA/
git remote add upstream https://github.com/open-source-economics/OG-USA/
git fetch upstream pull/316/head:regression_testing
git checkout regression_testing
conda env remove --name ospcdyn
conda env create
source activate ospcdyn
python setup.py develop
cd ~/Documents/PolicyBrain/deploy/taxbrain_server
python run_ogusa.py

If this runs OK, then we will know it's the environment.

[UPDATE] The model appears to be running OK.

Here's the result of the conda export for ospcdyn:

name: ospcdyn
channels:
- ospc
- defaults
dependencies:
- appnope=0.1.0=py27_0
- backports=1.0=py27_0
- backports_abc=0.5=py27_0
- bleach=1.5.0=py27_0
- certifi=2016.2.28=py27_0
- configparser=3.5.0=py27_0
- cycler=0.10.0=py27_0
- decorator=4.1.2=py27_0
- entrypoints=0.2.3=py27_0
- enum34=1.1.6=py27_0
- freetype=2.5.5=2
- funcsigs=1.0.2=py27_0
- functools32=3.2.3.2=py27_0
- get_terminal_size=1.0.0=py27_0
- html5lib=0.9999999=py27_0
- icu=54.1=0
- ipykernel=4.6.1=py27_0
- ipython=5.3.0=py27_0
- ipython-notebook=4.0.4=py27_0
- ipython_genutils=0.2.0=py27_0
- jinja2=2.9.6=py27_0
- jsonschema=2.6.0=py27_0
- jupyter_client=5.1.0=py27_0
- jupyter_core=4.3.0=py27_0
- libpng=1.6.30=1
- llvmlite=0.20.0=py27_0
- markupsafe=1.0=py27_0
- matplotlib=2.0.2=np111py27_0
- mistune=0.7.4=py27_0
- mock=2.0.0=py27_0
- nbconvert=5.2.1=py27_0
- nbformat=4.4.0=py27_0
- nomkl=1.0=0
- notebook=5.0.0=py27_0
- numba=0.35.0=np111py27_0
- numpy=1.11.2=py27_nomkl_0
- openssl=1.0.2l=0
- pandas=0.18.0=np111py27_0
- pandocfilters=1.4.2=py27_0
- path.py=10.3.1=py27_0
- pathlib2=2.3.0=py27_0
- pbr=1.10.0=py27_0
- pexpect=4.2.1=py27_0
- pickleshare=0.7.4=py27_0
- pip=9.0.1=py27_1
- prompt_toolkit=1.0.15=py27_0
- ptyprocess=0.5.2=py27_0
- py=1.4.34=py27_0
- pygments=2.2.0=py27_0
- pyparsing=2.2.0=py27_0
- pyqt=5.6.0=py27_2
- pytest=3.2.1=py27_0
- python=2.7.13=0
- python-dateutil=2.6.1=py27_0
- pytz=2017.2=py27_0
- pyzmq=16.0.2=py27_0
- qt=5.6.2=2
- readline=6.2=2
- scandir=1.5=py27_0
- scipy=0.18.1=np111py27_nomkl_1
- setuptools=36.4.0=py27_1
- simplegeneric=0.8.1=py27_1
- singledispatch=3.4.0.3=py27_0
- sip=4.18=py27_0
- six=1.10.0=py27_0
- sqlite=3.13.0=0
- ssl_match_hostname=3.5.0.1=py27_0
- subprocess32=3.2.7=py27_0
- terminado=0.6=py27_0
- testpath=0.3.1=py27_0
- tk=8.5.18=0
- toolz=0.8.2=py27_0
- tornado=4.5.2=py27_0
- traitlets=4.3.2=py27_0
- wcwidth=0.1.7=py27_0
- wheel=0.29.0=py27_0
- zlib=1.2.11=0
- taxcalc=0.8.3=np111py27_0
- pip:
  - backports-abc==0.5
  - backports.shutil-get-terminal-size==1.0.0
  - backports.ssl-match-hostname==3.5.0.1
  - ipython-genutils==0.2.0
  - jupyter-client==5.1.0
  - jupyter-core==4.3.0
  - ogusa (/Users/henrydoupe/Documents/OG-USA)==0.5.8+55.gd7a77ff
  - prompt-toolkit==1.0.15
prefix: /anaconda/envs/ospcdyn

[hdoupe]

@jdebacker I'm not sure if we want to go back to numpy 0.11.2. @martinholmer how would moving back to numpy 0.11.1 affect Tax-Calculator?

[martinholmer]

@jdebacker said:

If you can pin 1.11.2 for just OG-USA, great. If not, the fix will take longer (but needs to be done anyway) to go into OG-USA and make sure we have the correct data types specified. I did try to do this before, but couldn't figure it out before giving up and pinning to the older NumPy... now that's coming back to bite.

Yes, yet another example of the evils of exact pinning (see this discussion for a general discussion of why exact pinning should be avoided).

@hdoupe asked:

[H]ow would moving back to numpy 0.11.1 affect Tax-Calculator?

Tax-Calculator uses at-least pinning for numpy and pandas:

    - numpy >=1.12.1
    - pandas >=0.20.1

And the current version of those two packages are: numpy 1.13.1 and pandas 0.20.3 (at least this is what's installed on my computer now).

The solution, as @jdebacker realizes, is to fix the code in OG-USA so that it runs using modern versions of numpy and pandas. Tax-Calculator developers have had to do this in the past as described in item 2 in this discussion. Doing this made the code less buggy and Tax-Calculator ran much faster because the the efficiency improvements in the newer versions of numpy and pandas. (Remember exact pinning to an old version of numpy will logically imply the use of an older version of pandas, which is likely to be much slower than modern versions.)

I suppose you could run OG-USA on its own AWS instance, which would allow it to have a different environment than used by Tax-Calculator, but that approach is likely to take more time than simply fixing the OG-USA bugs that arise when using modern versions of numpy and pandas.

[jdebacker]

@hdoupe You probably don't. I'm awaiting my thrashing from @martinholmer.

I might be able to try to figure out some packaging stuff in the next 24 hours. Can you identify other conflicts between ospcdyn and what is needed for PolicyBrain?

[hdoupe]

@martinholmer Thanks for your input and I agree.

@jdebacker after comparing the environment.yml files for OG-USA and Tax-Calculator, I think we'll have to make changes so that the OG-USA environment file will look like:

name: ospcdyn
channels:
- ospc
dependencies:
- ipython
- ipython-notebook
- setuptools
- nomkl
- scipy=0.18.1 # --> >=1.0.0?
- matplotlib
- taxcalc=0.8.3 # --> >=0.12.0
- numba
- numpy=1.11.2 # --> >=1.12.1
- pandas==0.18.0 # --> >=0.20.1
- pytest
- python=2.7

I'll add some tolerance checking on the regression testing PR (PSLmodels/OG-Core#316) that I opened up ages ago. That way once we fix the version problems we can make sure that we didn't break anything in the process.

[martinholmer]

@hdoupe said:

- scipy=0.18.1 # --> >=1.0.0?

conda search scipy shows that the most recent version of scipy is 0.19.1
Why use any kind of pinning for scipy?

[hdoupe]

@martinholmer We could stick with >=0.18.1 version but the 1.0.0 version was released two days ago. I'm not sure why it's not available on conda yet. Maybe we should just stick with >=0.18.1.

@jdebacker I don't think Tax-Calculator or B-Tax uses scipy. So whichever version you can get to work with the proper versions of Numpy will work.

[jdebacker]

Ok, thanks for the input @hdoupe and @martinholmer. Other than the integer/float issue, I think some changes to the SciPy's optimization routines are relevant to the model and so it's good to know what the other PolicyBrain dependencies are that and related packages.

[hdoupe]

With the merge of PSLmodels/OG-Core#322, the version issue should be resolved.

I'm still not sure how to handle the issue where OG-USA expects there to be a puf.csv file but we instead have a puf.csv.gz file. How does CCC handle this so that it does not throw an error?

[martinholmer]

Either have both puff.csv and puf.csv.gz available or bring OG-USA into line with taxcalc.

…

Sent from my iPhone

On Oct 27, 2017, at 11:46 AM, Henry Doupe ***@***.***> wrote: With the merge of PSLmodels/OG-Core#322, the version issue should be resolved. I'm still not sure how to handle the issue where OG-USA expects there to be a puf.csv file but we instead have a puf.csv.gz file. How does CCC handle this so that it does not throw an error? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

[hdoupe]

I just checked and the worker nodes have both puf.csv and puf.csv.gz. So, I think #710 will be ready to be merged once the next version of OG-USA is released and PB is pinned to taxcalc >=0.12.0.

The next version of OG-USA is pinned to taxcalc >=0.12.0. Thus, we will need to submit another PR that updates PB to taxcalc >=0.12.0 before we can merge this one.

@brittainhard @PeterDSteinberg