Some RadMon extracted data for regional sources is missing and/or incorrect

[EdwardSafford-NOAA]

Recent work with the RRFS data has revealed these pieces are missing/wrong from the extracted data files:

• summary data files are missing analysis data
• time series data files have incorrect sdv values for scan angle correction -- should be all 0

Additionally the RRFS folks would like to see different time intervals for plots. Single cycle plots have so much variance from cycle to cycle that they don't mean much. The exact ranges are yet to be worked out but it's likely something like 6 cycles, 24 cycles, and 120 cycles. Change the extraction executables accordingly.

[EdwardSafford-NOAA]

Returning to this issue, looks like things stalled out at re-grouping the summary plots. Can't remember what happened but that was about the time RRFS underwent some changes in direction.

The branch I'd done this work on got deleted but I was able to reconstruct it using this command:

git branch feature/rrfs-128 4edf138

[EdwardSafford-NOAA]

@xyzemc I was reviewing open issues in GSI-monitor and found this one. I'm sorry it got dropped. Please take a look at it and tell me if these changes are still needed. If so I'll make it happen. If not I'll close this issue.

[xyzemc]

Yes, this change is still very useful to regional. Please go ahead make it happen. I am sorry I did not comments this issue earlier. Thanks!

[EdwardSafford-NOAA]

@xyzemc not your fault. I tagged the wrong person so you didn't know where this stood. We'd talked about the problems via email but I don't think you'd seen this issue because of my error. I'll update things here as I progress.

[EdwardSafford-NOAA]

@xyzemc I've studied the regional summary plots.

With regard to the first plot (currently last 4 cycles), what would you like to see there? The global plots have 4 curves for the last 4 cycles. Would 4 curves for the previous 6 hr intervals make sense? Meaning:

• curve 1 = PDATE to PDATE-6hr
• curve 2 = PDATE-7h to PDATE -12 hr
• curve 3 = PDATE - 13hr to PDATE - 18hr
• curve 4 = PDATE - 19hr to PDATE -24hr

For the Penalty and Total Bias plots instead of curves for 1 cycle, 1 day (4 cycles) and 30 days (120 cycles) I propose making those 6hr (6 cycles), 1 day (24 cycles) and 5 days (120 cycles).

For the omgbc plots instead of curves for avg/sdv plots for 1 cycle and 30 days I propose making those periods of 6 hr (6 cycles) and 5 days (120 cycles).

Thoughts?

[xyzemc]

@xyzemc I've studied the regional summary plots.

With regard to the first plot (currently last 4 cycles), what would you like to see there? The global plots have 4 curves for the last 4 cycles. Would 4 curves for the previous 6 hr intervals make sense? Meaning:

• curve 1 = PDATE to PDATE-6hr
• curve 2 = PDATE-7h to PDATE -12 hr
• curve 3 = PDATE - 13hr to PDATE - 18hr
• curve 4 = PDATE - 19hr to PDATE -24hr

For the Penalty and Total Bias plots instead of curves for 1 cycle, 1 day (4 cycles) and 30 days (120 cycles) I propose making those 6hr (6 cycles), 1 day (24 cycles) and 5 days (120 cycles).

For the omgbc plots instead of curves for avg/sdv plots for 1 cycle and 30 days I propose making those periods of 6 hr (6 cycles) and 5 days (120 cycles).

Thoughts?

4 curves for previous 1 hour will make more sense for regional summary plots, which is showing on current rrfs radmon. This is good enough, and do not need to change.
For the Penalty and Total Bias plots, 1 day (24 cycles) and 5 days (120 cycles) is good. And please do same thing for omgbc plots.

[EdwardSafford-NOAA]

@xyzemc OK, here's what I'll do:

• No change to Counts plot.
• Use increments of 1 day (24 cycles) and 5 days (120 cycles) in the other 3 plots and drop the 1 cycle from Penalty and Total Bias.

[EdwardSafford-NOAA]

@xyzemc Please take a look at the nam summary plots. I think I've made all the changes we've discussed. Please let me know if you see anything you'd like changed. https://www.emc.ncep.noaa.gov/gmb/gdas/radiance/esafford/regional/nam/index.html

[xyzemc]

@xyzemc Please take a look at the nam summary plots. I think I've made all the changes we've discussed. Please let me know if you see anything you'd like changed. https://www.emc.ncep.noaa.gov/gmb/gdas/radiance/esafford/regional/nam/index.html

It seems the total bias and penalty plots are still 30 days instead of 5 days.

[EdwardSafford-NOAA]

D'oh! I forgot to change those labels. The data is 5 days though. I'll get that fixed.

[EdwardSafford-NOAA]

How's it look now?

[xyzemc]

How's it look now?

Looks good!

[EdwardSafford-NOAA]

The angle plots currently show curves for 1 day, 7day and 30 days, but those are probably using 4 cycles/day instead of 24, so I'll fix that. Does it make sense to reduce these to two curves, 1 day (24 cycles) and 5 days (120 cycles), like the summary plots?

[xyzemc]

The angle plots currently show curves for 1 day, 7day and 30 days, but those are probably using 4 cycles/day instead of 24, so I'll fix that. Does it make sense to reduce these to two curves, 1 day (24 cycles) and 5 days (120 cycles), like the summary plots?

Yes, two curves are good to go. Thanks!

[EdwardSafford-NOAA]

@xyzemc Please take a look at the nam angle plots when you have a moment. I've made the changes we've discussed. Note that some sources have incomplete data sets and the data shows up as unconnected points rather than curves. Sources like iasi_metop-b and mhs_n19 always have full data sets. Please let me know if you see anything wrong/suspect.

[xyzemc]

@xyzemc Please take a look at the nam angle plots when you have a moment. I've made the changes we've discussed. Note that some sources have incomplete data sets and the data shows up as unconnected points rather than curves. Sources like iasi_metop-b and mhs_n19 always have full data sets. Please let me know if you see anything wrong/suspect.

The unconnected point for atms_npp looks same as gdas radiance monitor. I think it is ok. But for all amsua instrument, all the data from certain scan position are missing. This is probably something wrong. But I don't know why. I am not sure how much effort we should put since NAM is probably expecting fully retire before move to wcoss3.

[EdwardSafford-NOAA]

If you're ok with this approach here's what I'd like to do:

1. Put these changes together in a PR.
2. Open a new issue to investigate the missing scan positions in regional amsua angle plots. I'll treat this as low a priority.
3. Continue with the final step of the regional website update (in a new issue) which will provide a scripted way to create regional web sites like the global sources have now. This also includes putting the regional html (and other web resource) files into the GSI-monitor repo.

Does that sound OK?

[xyzemc]

If you're ok with this approach here's what I'd like to do:

1. Put these changes together in a PR.
2. Open a new issue to investigate the missing scan positions in regional amsua angle plots. I'll treat this as low a priority.
3. Continue with the final step of the regional website update (in a new issue) which will provide a scripted way to create regional web sites like the global sources have now. This also includes putting the regional html (and other web resource) files into the GSI-monitor repo.

Does that sound OK?

Sounds good. Thanks!