TOF resolution run-dependence in 2020 data

[jrstevenjlab]

I'm not sure exactly where to put this issue, but I found a surprising change in the TOF resolution in the 2019-11 data. The plots below show the pi+ and pi- TOF DeltaT resolution from the standard p3pi_hists plugin run during the REST reconstruction (histograms available at /work/halld/data_monitoring/RunPeriod-YYYY-MM/recon_verNN/rootfiles/).

For the 2018-08 data, there is some variation with run number, with the most significant difference in the low coherent peak energy region 51384-51457 deviating from the average by only a few ps.

RunPeriod-2018-08 data:

For the 2019-11 data however, there are a couple discrete changes at the 25 ps level in "Batch 10" where the data were produced at different sites according to the links on our Dataset Summary page https://halldweb.jlab.org/wiki-private/index.php/Spring_2020_Dataset_Summary

The first 9 batches (71350-72760) have a resolution consistently larger than previous run periods. Note: the COVID break between Spring and Summer runs starts at 72435 where no discrete change occurs.

The magenta vertical lines indicate runs 72761-72771 which were produced on PSC https://halldweb.jlab.org/data_monitoring/recon/summary_swif2_output_recon_2019-11_ver01_batch10_PSC2/index.html

The blue vertical lines indicate runs 72804-72830 which were produced on PSC(?) https://halldweb.jlab.org/data_monitoring/recon/summary_swif2_output_recon_2019-11_ver01_batch10_PSC2_ver2/index.html

After 72831 the resolution is consistent with the values from RunPeriods. These runs were primarily produced on the farm at JLab.

RunPeriod-2019-11 data:

Zoomed version of RunPeriod-2019-11 data

It's not to me clear what the cause of this discrete change is, but my guess would be a something different in the environment used for reconstruction at these sites, such as a different version of the CCDB calibration sqlite file or timestamp being used somewhere.

I would suggest that for now we use different TOF2/paddle_resolutions values in mcsmear for these discrete regions, so we roughly match the effect in simulation. If the cause of this difference is due to stale CCDB tables, there could be other subtle issues to consider. Comments and suggestions are welcome.