acqs.make_report() throws IndexError

[jeanconn]

In [2]: proseco.test(get_version=True)
Out[2]: '4.4-r506-3ea42f4'

n [8]: kw
Out[8]: 
{'att': [344.985493, -52.392549, 244.667835],
 'date': '2019:018:17:34:50.824',
 'detector': 'ACIS-S',
 'dither': (8.0, 8.0),
 'focus_offset': 0,
 'man_angle': 85.0,
 'n_fid': 0,
 'n_guide': 8,
 'obsid': 48492,
 'sim_offset': 0,
 't_ccd': -11.1}

In [9]: cat = proseco.catalog.get_aca_catalog(**kw)

In [10]: cat.acqs.make_report()
/home/jeanconn/git/proseco/proseco/report_acq.py:367: VisibleDeprecationWarning: boolean index did not match indexed array along dimension 0; dimension is 260 but corresponding boolean dimension is 486
  bad_stars = acqs.bad_stars_mask[ok]
---------------------------------------------------------------------------
IndexError                                Traceback (most recent call last)
<ipython-input-10-8ce44e148bf5> in <module>()
----> 1 cat.acqs.make_report()

~/git/proseco/proseco/acq.py in make_report(self, rootdir)
    226         """
    227         from .report_acq import make_report
--> 228         make_report(self, rootdir=rootdir)
    229 
    230     def update_p_acq_column(self):

~/git/proseco/proseco/report_acq.py in make_report(obsid, rootdir)
    334     make_cand_acqs_report(acqs, cand_acqs, events, context, outdir)
    335     make_initial_cat_report(events, context)
--> 336     make_acq_star_details_report(acqs, cand_acqs, events, context, outdir)
    337     make_optimize_catalog_report(events, context)
    338 

~/git/proseco/proseco/report_acq.py in make_acq_star_details_report(acqs, cand_acqs, events, context, obsdir)
    209         cca['spoilers_plot'] = basename
    210         if not filename.exists():
--> 211             plot_spoilers(acq, acqs, filename=filename)
    212 
    213         # Make the acq detail plot with spoilers and imposters

~/git/proseco/proseco/report_acq.py in plot_spoilers(acq, acqs, filename)
    365           (np.abs(stars['zang'] - acq['zang']) < plot_hw))
    366     stars = stars[ok]
--> 367     bad_stars = acqs.bad_stars_mask[ok]
    368 
    369     fig = plt.figure(figsize=(5, 5))

IndexError: index 298 is out of bounds for axis 1 with size 260

[taldcroft]

That's pretty weird, I cannot reproduce on my mac at exactly the same version.

In [1]: from proseco import get_aca_catalog

In [2]: kw = {'att': [344.985493, -52.392549, 244.667835],
   ...:  'date': '2019:018:17:34:50.824',
   ...:  'detector': 'ACIS-S',
   ...:  'dither': (8.0, 8.0),
   ...:  'focus_offset': 0,
   ...:  'man_angle': 85.0,
   ...:  'n_fid': 0,
   ...:  'n_guide': 8,
   ...:  'obsid': 48492,
   ...:  'sim_offset': 0,
   ...:  't_ccd': -11.1}
   ...:  

In [3]: aca = get_aca_catalog(**kw)

In [4]: aca.acqs.make_report()

In [6]: import proseco

In [7]: proseco.test(get_version=True)
Out[7]: '4.4-r506-3ea42f4'

[taldcroft]

And I get a pass from that example on HEAD linux ska3-flight.

[jeanconn]

I'll look at this more later

fido: source /proj/sot/ska3/flight/bin/ska_envs.csh
fido: cd ~/git/proseco
fido: ipython
Python 3.6.2 |Continuum Analytics, Inc.| (default, Jul 20 2017, 13:51:32) 
Type 'copyright', 'credits' or 'license' for more information
IPython 6.1.0 -- An enhanced Interactive Python. Type '?' for help.

In [1]: kw = {'att': [344.985493, -52.392549, 244.667835],
   ...:  'date': '2019:018:17:34:50.824',
   ...:  'detector': 'ACIS-S',
   ...:  'dither': (8.0, 8.0),
   ...:  'focus_offset': 0,
   ...:  'man_angle': 85.0,
   ...:  'n_fid': 0,
   ...:  'n_guide': 8,
   ...:  'obsid': 48492,
   ...:  'sim_offset': 0,
   ...:  't_ccd': -11.1}
   ...:  

In [2]: import proseco.catalog

In [3]: cat = proseco.catalog.get_aca_catalog(**kw)

In [4]: cat.acqs.make_report()
/proj/sot/ska/jeanproj/git/proseco/proseco/report_acq.py:367: VisibleDeprecationWarning: boolean index did not match indexed array along dimension 0; dimension is 260 but corresponding boolean dimension is 486
  bad_stars = acqs.bad_stars_mask[ok]
---------------------------------------------------------------------------
IndexError                                Traceback (most recent call last)
<ipython-input-4-8ce44e148bf5> in <module>()
----> 1 cat.acqs.make_report()

/proj/sot/ska/jeanproj/git/proseco/proseco/acq.py in make_report(self, rootdir)
    226         """
    227         from .report_acq import make_report
--> 228         make_report(self, rootdir=rootdir)
    229 
    230     def update_p_acq_column(self):

/proj/sot/ska/jeanproj/git/proseco/proseco/report_acq.py in make_report(obsid, rootdir)
    334     make_cand_acqs_report(acqs, cand_acqs, events, context, outdir)
    335     make_initial_cat_report(events, context)
--> 336     make_acq_star_details_report(acqs, cand_acqs, events, context, outdir)
    337     make_optimize_catalog_report(events, context)
    338 

/proj/sot/ska/jeanproj/git/proseco/proseco/report_acq.py in make_acq_star_details_report(acqs, cand_acqs, events, context, obsdir)
    209         cca['spoilers_plot'] = basename
    210         if not filename.exists():
--> 211             plot_spoilers(acq, acqs, filename=filename)
    212 
    213         # Make the acq detail plot with spoilers and imposters

/proj/sot/ska/jeanproj/git/proseco/proseco/report_acq.py in plot_spoilers(acq, acqs, filename)
    365           (np.abs(stars['zang'] - acq['zang']) < plot_hw))
    366     stars = stars[ok]
--> 367     bad_stars = acqs.bad_stars_mask[ok]
    368 
    369     fig = plt.figure(figsize=(5, 5))

IndexError: index 298 is out of bounds for axis 1 with size 260

In [5]: import proseco

In [6]: proseco.test(get_version=True)
Out[6]: '4.4-r506-3ea42f4'

In [7]:                                                                         
Do you really want to exit ([y]/n)? y
efido: echo $SKA
/proj/sot/ska3/flight
fido: echo $PYTHONPATH
PYTHONPATH: Undefined variable.
fido: pwd
/proj/sot/ska/jeanproj/git/proseco
fido: git status
On branch master
Your branch is up-to-date with 'origin/master'.
nothing to commit, working tree clean

[taldcroft]

FWIW I ran the same on fido and did not get an error. I even went into your git repo and ran from that (with rootdir='/home/aldcroft/tmp'). So something funky in your setup. Agreed it can wait, I was concerned there might be a real problem.

[jeanconn]

"real problem"?

[taldcroft]

"real problem" => something that makes proseco work differently on a different platform for a correctly configured user environment. (E.g. when SOT MP runs proseco).

[jeanconn]

Well, so far I've gotten it to fail twice as the ACA user in what I think is a correctly configured environment, and what I think is a correctly configured environment for me is now failing intermittently. So it is certainly interesting.

I took minor offense at "real problem" because this was a real problem for me when I couldn't make an acq report for 48492 in JAN1419A.

[taldcroft]

OK well maybe it is a real problem (in the code), not user configuration error. hmm.

[jeanconn]

Heck, it probably is a user configuration error of some kind, but if the user can't figure it out the code may have to provide more breadcrumbs.

I was doing most of my broken tests with proseco.catalog.get_aca_catalog and with a pylab ipython session, but didn't narrow it down more to figure out why the acqs._bad_star_mask sometimes seems to get set to a value that is inconsistent with the acqs. But already tried too many combinations today trying to get it to break to get useful debug.

[taldcroft]

I still can't reproduce, even as aca on kadi with current master '4.4-r506-3ea42f4'. You should try yourself in ~aca/git/proseco as aca.

[taldcroft]

Dang, I thought the pylab might have been the problem. In general using pylab is a really bad idea because it does things like overload the Python sum with numpy sum, etc. So just don't do that ever, and instead use --matplotlib.

[jeanconn]

OK. It was intermittent even with pylab but if you narrowed it down to something in there that's fine. I wasn't sure if the problem was somewhere in matplotlib that was just "showing its face" more with pylab.

[taldcroft]

I forgot the second half of the first sentence, namely, "but no, it still worked for me".

[jeanconn]

Hah. OK, I'll make some time this week to see if I can reproduce it as the ACA user with --matplotlib .

[taldcroft]

Well I ran 3 times in a row with --pylab as aca on kadi and had no problem. Are you on NX? I'm not (and running in pylab mode on mac client is quite annoying for this because it pops up windows constantly and steals focus so no chance of multitasking while it runs.).

[jeanconn]

Yes, I'm working in NX (even when I'm on console I tend to work in the NX session). But if related to nx this would be the first really intermittent issue.