In user manual/notebook list out the expected outputs of each step

[falkben]

Currently it is not stated what the file outputs are for each step of the pipeline. It would be helpful during debugging to know what are the expected outputs of each step.

[falkben]

See below. Will possibly change as we move the steps around. Has relevance to #13 and #8

Since the underlying heasoft commands create most of the outputs, it can be a little difficult to identify where the outputs come from for each step.

---

Inputs/Outputs Cheatsheet

step	script	input	output
1	uvotimage	*rw.img or *.evt and attfile (*pat.fits)	output_uvotimage_{...}_rw.txt,output_uvotimage_{...}w1po_uf.txt, {...}_{evt,img}_{FFF}_{rw,sk}.img
2	uvotskycorr	*sk.img, attfile (*pat.fits), catfile (usnob1.spec)	{...}_evt_{FFF}_aspcorr.ALL,output_uvotskycorrID_{...}_{evt,img}_{FFF}_sk.txt
3	uvotattcorr	*pat.fits	output_uvotattcorr_{...}pat.txt,{...}_aspcorr_{NNN}filters.ALL,{...}uat.fits
4	uvotimage2	(*rw.img without _img_ and _evt_ in fname) or *.evt	output_uvotimage_{...}{FFF}_{uat,w1po_uf}.txt,{...}_uat_evt_{FFF}_{rw,sk}.img
5	uvotbadpix	*sk.img with uat in fname	output_uvotbadpix_{...}_uat_{evt,img}_{FFF}_sk.txt,quality_{...}_uat_{evt,img}_{FFF}_badpix.img
6	uvotexpmap	*sk.img with uat in fname	output_uvotexpmap_{...}_uat_{evt,img}_{FFF}_sk.txt,quality_{...}_uat_{evt,img}_{FFF}_badpix_new.img,{...}_uat_{evt,img}_{FFF}_{ex,mk}.img
7	uvotskycorr2	*sk.img with uat in fname	{output_uvotskycorrID,output_uvotskycorrSKY}_{...}_uat_{evt,img}_{FFF}_{sk,ex,mk}.txt,{...}_uat_{evt,img}_{FFF}_aspcorr.ALL,{...}_uat_{evt,img}_{FFF}_{ex,mk,sk}_corr.img
8	uvotskylss	*sk_corr.img	output_uvotskylss_{...}_uat_{evt,img}_{FFF}_sk_corr.txt,{...}_uat_{evt,img}_{FFF}_lss_corr.img
9	sort_by_year		files moved into folders
10	uvotimsum		all_{FFF}_{ex,lss,mk,sk}.img,output_uvotimsum_{FFF}_{ex,lss,sk}.txt,sum_{FFF}_{ex,lss,nm,sk}.img,{...}_uat_{evt,img}_{FFF}_mk_corr_new.img
11	corrections		sum_{FFF}_nm_{coi,coilss,coilsszp}.img
12	combine		total_sum_{FFF}_{ex,nm,sk}.fits, sum_{FFF}_ex_r.img, sum_{FFF}_nm_coilsszp_{c,c_r}.img
13	calibration		NGC0628_final_{FFF}_Jy.fits

Legend

• {...} = filename (sw00032891004, sw00032891012)
• {FFF} = filter (um2, uw1, uw2)
• {NNN} = number of filters (2,3)
• anything lowercase separated by commas indicates both options may be present in the filename, e.g. {evt,img}

Notes

This was generated by adding the following to each step in pipeline.bash and then running the pipeline from start to finish with the small set of data we use in GH actions. This creates a snapshot of the files at each step

ls -lR /astro/dust_kg3/bfalk/test_pipeline/NGC0628/working_dir > 13_file_listings.txt

The pulled out just the filenames

find . -name '*_file_listings.txt' -exec awk 'FNR > 2 {print $9 > (FILENAME "-new")}' {} +

Then did a comparison from one step to the next to determine the outputs of each step