Larger-than-normal flux fluctuations in simulations give offset transit depths

[nespinoza]

Hi @hover2pi,

For my JDAT notebooks, I had to take a deep look at end-to-end awesimsoss simulations, and found that if the data is passed through the CalWebb pipeline, the transit depths one gets back are offset from the input ones to awesimsoss. Here's one of the final results:

Note the transit depths match the input ones only in Order 2. In order 1, they seem to go astray as you move to longer wavelengths --- also, between ~1-1.3 microns they are totally off.

The end-to-end simulation can be checked at: https://github.com/nespinoza/dat_pyinthesky/tree/test/jdat_notebooks/soss-transit-spectroscopy (PART1 are the actual simulations --- on which I implemented my latest pull request here: #88, PART2 are the pass of the data through the CalWebb pipeline and PART3 is the data analysis to get to those transit depths).

In particular, in the third notebook of the series (https://github.com/nespinoza/dat_pyinthesky/blob/test/jdat_notebooks/soss-transit-spectroscopy/HAT-P-1b-PART3-data-analysis.ipynb) I left an analysis at the very end of the notebook in which I checked that the problem is really at the pixel-level (and not, e.g., a problem with the transit fitting routine, or the simple aperture extraction algorithm I wrote). My hunch is that there is something funky going on with the non-linearity correction --- most likely because the "forward" coefficients you are using are not the same as the ones the pipeline grabs from CRDS (in fact, the forward coefficients have a different order (4) to the ones CRDS has (6)). Given the 1-1.3 um region is the one with the largest counts (and that's where the non-linearity correction is worse), the problem blows up over there. Again, this is just a hunch --- would have to run a simulation without non-linearity to check.

If you could have a look at this, it would be great. Sadly, this implies that I will not be turning in the PART1 and PART2 notebooks, and will have to just create a fake observation for the JDAT notebooks on my own :(.