Preprocess: Speedup run time

[msilvafe]

This is a general effort for profiling, and then speeding up underlying code by:

1. Evaluating if there are inefficiencies in the implementation
2. Evaluating if there's repeated or unneeded steps
3. Replacing the slowest steps with python wrapped C-code where appropriate
4. Evaluating our top-bottom parallelization strategy

[skhrg]

I had brought this up on slack before. But I think numba is a pretty safe and stable way to speed up a lot of this stuff without needing to write as much C. Or at least to provide intertim speedups.

At one point @mhasself seemed to be anti-JIT but I would be curious to know if that stance has softened.

[mmccrackan]

Just to follow up on today's FLP telecon discussion on profiling, here is the profiling file used to make the profiling figure and table from the recent satp3 site-pipeline run. You can generate the plot and table by installing snakeviz and running it on this file.

In terms of speeding things up, I have so3g C++ code in the works for the main preprocessing time-sinks (interpolation, PSD calculation, and noise fitting).

combined_profile.txt

[msilvafe]

We should decide when we think this issue is resolved i.e. the speed up is good-enough. @mhasself you had a target in mind of the time it should take for processing which I thought was order 10 sec per obs-wafer-band does hitting 10 sec sound like a safe target? We have a ways to go for that...maybe we'll have @mmccrackan rewrite everything in C in so3g