-
Notifications
You must be signed in to change notification settings - Fork 27
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
REFS CAPE setting update and T/Td unit conversion #614
base: feature/rrfs_refs_v1
Are you sure you want to change the base?
REFS CAPE setting update and T/Td unit conversion #614
Conversation
…PREPBUFR_SPCoutlook.conf
Thanks, Binbin, for putting this together! @malloryprow - Should this REFS PR from @BinbinZhou-NOAA actually be submitted to the RRFS/REFS branch and not to the develop branch? CC @AndrewBenjamin-NOAA |
It should! @BinbinZhou-NOAA The branch should have been made off of feature/rrfs_refs_v1 as well. |
@malloryprow I remember that I built this branch off the https://github.com/NOAA-EMC/EVS/tree/feature/rrfs_refs_v1 Thanks! Binbin |
At the top of this PR, there is an "Edit" button, which should allow you to change the branch it will be merged into. Select I have never done this but we can see how it goes. |
Mallory,
Yes, it works. The base has been changed to feature/rrfs_refs_v1
Thanks!
Binbin
…On Tue, Dec 3, 2024 at 3:28 PM Mallory Row ***@***.***> wrote:
At the top of this PR, there is an "Edit" button, which should allow you
to change the branch it will be merged into. Select
base:feature/rrfs_refs_v1 and then save.
I have never done this but we can see how it goes.
—
Reply to this email directly, view it on GitHub
<#614 (comment)>, or
unsubscribe
<https://github.com/notifications/unsubscribe-auth/AQBMPFP7MCB3M57FD2NBYQ32DYH7VAVCNFSM6AAAAABSXQHEH6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDKMJVGQ4DOMJQGI>
.
You are receiving this because you were mentioned.Message ID:
***@***.***>
--
Binbin Zhou
Physical Scientist
Lynker at NOAA/NWS/NCEP/EMC
5830 University Research Ct.
College Park, MD 20740
***@***.***
301-683-3683
|
Great news! Thank you I'll get to reviewing it today. |
Mallory,
2 types of the snowfall plots:
evs.refs.ctc.weasd_06h(or
24h).last31days.perfdiag_valid00z12z.buk_conus.png
evs.refs.ets.weasd_06h(or
24h).last31days.threshmean_valid00z12z.buk_conus.png
"valid00z12z" means the scores are for both 00z and 12z. Not sure this is
the correct way.
Binbin
…On Wed, Dec 4, 2024 at 8:07 AM Mallory Row ***@***.***> wrote:
***@***.**** commented on this pull request.
------------------------------
In scripts/plots/cam/exevs_refs_snowfall_plots.sh
<#614 (comment)>:
> @@ -275,14 +275,15 @@ for var in weasd ; do
for domain in conus conus_east conus_west conus_south conus_central ; do
if [ -s ${score_type}_regional_${domain}_${valid}_${level}_${var}_${lead}.png ] ; then
- mv ${score_type}_regional_${domain}_${valid}_${level}_${var}_${lead}.png evs.refs.ctc.${var}_${level}.last${last_days}days.${scoretype}_valid_all_times.buk_${domain}.png
+ mv ${score_type}_regional_${domain}_${valid}_${level}_${var}_${lead}.png evs.refs.ctc.${var}_${level}.last${last_days}days.${scoretype}_valid00z12z.buk_${domain}.png
What type of plot is this renaming for?
—
Reply to this email directly, view it on GitHub
<#614 (review)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AQBMPFIPXPKMEQVE5AQSLJ32D35B5AVCNFSM6AAAAABSXQHEH6VHI2DSMVQWIX3LMV43YUDVNRWFEZLROVSXG5CSMV3GSZLXHMZDINZYGU2DANZUG4>
.
You are receiving this because you were mentioned.Message ID:
***@***.***>
--
Binbin Zhou
Physical Scientist
Lynker at NOAA/NWS/NCEP/EMC
5830 University Research Ct.
College Park, MD 20740
***@***.***
301-683-3683
|
Mallory,
I am not the original programmer of the plotting scripts but just copied
the scripts
from Marcel's (from the global_ens plotting suit). So not sure if the above
are needed
or not.
Binbin
…On Wed, Dec 4, 2024 at 8:34 AM Mallory Row ***@***.***> wrote:
***@***.**** commented on this pull request.
------------------------------
In ush/cam/ush_refs_plot_py/plot_util.py
<#614 (comment)>:
> @@ -1586,7 +1814,13 @@ def calculate_stat(logger, model_data, stat):
rmse = model_data.loc[:]['RMSE']
spread = model_data.loc[:]['SPREAD']
me = model_data.loc[:]['ME']
- mae = model_data.loc[:]['MAE']
+ mae = model_data.loc[:]['MAE']
+ if bool_convert:
Is there no need for const to be added here?
—
Reply to this email directly, view it on GitHub
<#614 (review)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AQBMPFKH5P4YZMO4W2HAPX32D4AGZAVCNFSM6AAAAABSXQHEH6VHI2DSMVQWIX3LMV43YUDVNRWFEZLROVSXG5CSMV3GSZLXHMZDINZYGYZTEMRUHA>
.
You are receiving this because you were mentioned.Message ID:
***@***.***>
--
Binbin Zhou
Physical Scientist
Lynker at NOAA/NWS/NCEP/EMC
5830 University Research Ct.
College Park, MD 20740
***@***.***
301-683-3683
|
Sure, this section can be deleted
Thanks!
Binbin
…On Wed, Dec 4, 2024 at 8:38 AM Mallory Row ***@***.***> wrote:
***@***.**** commented on this pull request.
------------------------------
In ush/cam/ush_refs_plot_py/refs_atmos_plots.py
<#614 (comment)>:
> @@ -109,7 +109,7 @@
logger.error("TOO MANY MODELS LISTED ("+str(len(model_list))
+", ["+', '.join(model_list)+"]), maximum is 10")
sys.exit(1)
-
+'''
If commenting out this section, it can be deleted.
—
Reply to this email directly, view it on GitHub
<#614 (review)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AQBMPFIS53OKJPSYLI6D3IT2D4AUZAVCNFSM6AAAAABSXQHEH6VHI2DSMVQWIX3LMV43YUDVNRWFEZLROVSXG5CSMV3GSZLXHMZDINZYGY2DINRTHA>
.
You are receiving this because you were mentioned.Message ID:
***@***.***>
--
Binbin Zhou
Physical Scientist
Lynker at NOAA/NWS/NCEP/EMC
5830 University Research Ct.
College Park, MD 20740
***@***.***
301-683-3683
|
So snowfall stats are being combined for 00z and 12z? I would think we would want separate plots for the two valid hours. @AliciaBentley-NOAA @AndrewBenjamin-NOAA @PerryShafran-NOAA, thoughts? |
Mallory,
Now the grid2obs, precip and spcoutlook REFS stats jobs have completed.
The output stat files for all 3 jobs are normal without any ERROR messages
in their log files.
There are several WARNING messages in the precip and grid2obs log files.
The WARNING messages say " Found fewer files than expected. Found 13
expected 14
Adding 1 fake files to ensure ens_thresh check is accurate". This WARNING
should be OK
and won't impact the METplus execution since 13/14 > the ens_thresh (set as
0.5).
This type of WARNING will be removed in the METplus beta6 version.
Binbin
On Thu, Dec 5, 2024 at 9:57 AM Binbin Zhou - NOAA Affiliate <
***@***.***> wrote:
… The spcoutlook job has completed, without ERRORs and WARNINGs in the log
file.
Its output stat file looks good.
The grid2obs and precip jobs are still running.
Binbin
On Thu, Dec 5, 2024 at 9:42 AM Mallory Row ***@***.***>
wrote:
> stats Testing
>
> stmp significantly cleared out!! COMOUT is
> /lfs/h2/emc/vpppg/noscrub/mallory.row/verification/EVS_PRs/pr614/evs/v2.0/stats/cam
> 1. jevs_cam_refs_grid2obs_stats.sh
>
> *Log File:*
> /lfs/h2/emc/vpppg/noscrub/mallory.row/verification/EVS_PRs/pr614/EVS/dev/drivers/scripts/stats/cam/jevs_cam_refs_grid2obs_stats.o208382218
> *DATA:*
> /lfs/h2/emc/stmp/mallory.row/evs_test/prod/tmp/jevs_cam_refs_grid2obs_stats.208382218.dbqs01
> 2. jevs_cam_refs_precip_stats.sh
>
> *Log File:*
> /lfs/h2/emc/vpppg/noscrub/mallory.row/verification/EVS_PRs/pr614/EVS/dev/drivers/scripts/stats/cam/jevs_cam_refs_precip_stats.o208382220
> *DATA:*
> /lfs/h2/emc/stmp/mallory.row/evs_test/prod/tmp/jevs_cam_refs_precip_stats.208382220.dbqs01
> 3. jevs_cam_refs_spcoutlook_stats.sh
>
> *Log File:*
> /lfs/h2/emc/vpppg/noscrub/mallory.row/verification/EVS_PRs/pr614/EVS/dev/drivers/scripts/stats/cam/jevs_cam_refs_spcoutlook_stats.o208382222
> *DATA:*
> /lfs/h2/emc/stmp/mallory.row/evs_test/prod/tmp/jevs_cam_refs_spcoutlook_stats.208382222.dbqs01
>
> —
> Reply to this email directly, view it on GitHub
> <#614 (comment)>, or
> unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AQBMPFNSPBGPDYLVQ2L3YML2EBQ3NAVCNFSM6AAAAABSXQHEH6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDKMRQGUYTAMRQGE>
> .
> You are receiving this because you were mentioned.Message ID:
> ***@***.***>
>
--
Binbin Zhou
Physical Scientist
Lynker at NOAA/NWS/NCEP/EMC
5830 University Research Ct.
College Park, MD 20740
***@***.***
301-683-3683
--
Binbin Zhou
Physical Scientist
Lynker at NOAA/NWS/NCEP/EMC
5830 University Research Ct.
College Park, MD 20740
***@***.***
301-683-3683
|
We have no control of such a WARNING message, but it will be removed in
METplus beta6.
Binbin
…On Thu, Dec 5, 2024 at 11:29 AM Mallory Row ***@***.***> wrote:
I noticed in the grid2obs and precip logs there is statements like
WARNING: Found fewer files than expected. Found 13 expected 14
Is this okay? Is this something that can be avoid? Does it have to do with
refs data being scrubbed off in ptmp?
—
Reply to this email directly, view it on GitHub
<#614 (comment)>, or
unsubscribe
<https://github.com/notifications/unsubscribe-auth/AQBMPFNSTANJ37XYPXMLP2T2EB5PPAVCNFSM6AAAAABSXQHEH6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDKMRQHA2DAMRRHA>
.
You are receiving this because you were mentioned.Message ID:
***@***.***>
--
Binbin Zhou
Physical Scientist
Lynker at NOAA/NWS/NCEP/EMC
5830 University Research Ct.
College Park, MD 20740
***@***.***
301-683-3683
|
Noted. Moving on to plots. |
plotsCOMIN is pointing to COMIN=/lfs/h2/emc/vpppg/noscrub/binbin.zhou/evs/v2.0. COMOUT is /lfs/h2/emc/vpppg/noscrub/mallory.row/verification/EVS_PRs/pr614/evs/v2.0/plots/cam/atmos.20241203. 1. jevs_cam_refs_grid2obs_cape_last31days_plots.shLog File: /lfs/h2/emc/vpppg/noscrub/mallory.row/verification/EVS_PRs/pr614/EVS/dev/drivers/scripts/plots/cam/jevs_cam_refs_grid2obs_cape_last31days_plots.o208389931 2. jevs_cam_refs_grid2obs_cape_last90days_plots.shLog File: /lfs/h2/emc/vpppg/noscrub/mallory.row/verification/EVS_PRs/pr614/EVS/dev/drivers/scripts/plots/cam/jevs_cam_refs_grid2obs_cape_last90days_plots.o208389933 3. jevs_cam_refs_grid2obs_ctc_last31days_plots.shLog File: /lfs/h2/emc/vpppg/noscrub/mallory.row/verification/EVS_PRs/pr614/EVS/dev/drivers/scripts/plots/cam/jevs_cam_refs_grid2obs_ctc_last31days_plots.o208390097 4. jevs_cam_refs_grid2obs_ctc_last90days_plots.shLog File: /lfs/h2/emc/vpppg/noscrub/mallory.row/verification/EVS_PRs/pr614/EVS/dev/drivers/scripts/plots/cam/jevs_cam_refs_grid2obs_ctc_last90days_plots.o208390101 5. jevs_cam_refs_grid2obs_ecnt_last31days_plots.shLog File: /lfs/h2/emc/vpppg/noscrub/mallory.row/verification/EVS_PRs/pr614/EVS/dev/drivers/scripts/plots/cam/jevs_cam_refs_grid2obs_ecnt_last31days_plots.o208390129 6. jevs_cam_refs_grid2obs_ecnt_last90days_plots.shLog File: /lfs/h2/emc/vpppg/noscrub/mallory.row/verification/EVS_PRs/pr614/EVS/dev/drivers/scripts/plots/cam/jevs_cam_refs_grid2obs_ecnt_last90days_plots.o208390131 7. jevs_cam_refs_precip_last31days_plots.shLog File: /lfs/h2/emc/vpppg/noscrub/mallory.row/verification/EVS_PRs/pr614/EVS/dev/drivers/scripts/plots/cam/jevs_cam_refs_precip_last31days_plots.o208390164 8. jevs_cam_refs_precip_last90days_plots.shLog File: /lfs/h2/emc/vpppg/noscrub/mallory.row/verification/EVS_PRs/pr614/EVS/dev/drivers/scripts/plots/cam/jevs_cam_refs_precip_last90days_plots.o208390165 9. jevs_cam_refs_precip_spatial_plots.shLog File: /lfs/h2/emc/vpppg/noscrub/mallory.row/verification/EVS_PRs/pr614/EVS/dev/drivers/scripts/plots/cam/jevs_cam_refs_precip_spatial_plots.o208390177 10. jevs_cam_refs_profile_last31days_plots.shLog File: /lfs/h2/emc/vpppg/noscrub/mallory.row/verification/EVS_PRs/pr614/EVS/dev/drivers/scripts/plots/cam/jevs_cam_refs_profile_last31days_plots.o208390379 11. jevs_cam_refs_profile_last90days_plots.shLog File: /lfs/h2/emc/vpppg/noscrub/mallory.row/verification/EVS_PRs/pr614/EVS/dev/drivers/scripts/plots/cam/jevs_cam_refs_profile_last90days_plots.o208390402 12. jevs_cam_refs_snowfall_last31days_plots.shLog File: /lfs/h2/emc/vpppg/noscrub/mallory.row/verification/EVS_PRs/pr614/EVS/dev/drivers/scripts/plots/cam/jevs_cam_refs_snowfall_last31days_plots.o208390544 13. jevs_cam_refs_snowfall_last90days_plots.shLog File: /lfs/h2/emc/vpppg/noscrub/mallory.row/verification/EVS_PRs/pr614/EVS/dev/drivers/scripts/plots/cam/jevs_cam_refs_snowfall_last90days_plots.o208390545 14. jevs_cam_refs_spcoutlook_last31days_plots.shLog File: /lfs/h2/emc/vpppg/noscrub/mallory.row/verification/EVS_PRs/pr614/EVS/dev/drivers/scripts/plots/cam/jevs_cam_refs_spcoutlook_last31days_plots.o208390567 15. jevs_cam_refs_spcoutlook_last90days_plots.shLog File: /lfs/h2/emc/vpppg/noscrub/mallory.row/verification/EVS_PRs/pr614/EVS/dev/drivers/scripts/plots/cam/jevs_cam_refs_spcoutlook_last90days_plots.o208390568 |
Mallory,
All of the REFS plotting jobs have completed. The output graphic files
are all good.
No ERROR messages in their log files. There are several WARNING messages
for CAPE
in the grid2obs, and high thresholds for precip and snowfall are expected
in the log files
but these WARNINgs have no impact on the plotting jobs.
This stat data missing issue for CAPE, precip and snow is always existing
for some regions and
seasons since no events happened in those regions and seasons.
Binbin
…On Thu, Dec 5, 2024 at 12:41 PM Mallory Row ***@***.***> wrote:
plots
COMIN is pointing to COMIN=/lfs/h2/emc/vpppg/noscrub/binbin.zhou/evs/v2.0.
COMOUT is
1. jevs_cam_refs_grid2obs_cape_last31days_plots.sh
*Log File:*
/lfs/h2/emc/vpppg/noscrub/mallory.row/verification/EVS_PRs/pr614/EVS/dev/drivers/scripts/plots/cam/jevs_cam_refs_grid2obs_cape_last31days_plots.o208389931
*DATA:*
/lfs/h2/emc/stmp/mallory.row/evs_test/prod/tmp/jevs_cam_refs_grid2obs_cape_last31days_plots.208389931.dbqs01
2. jevs_cam_refs_grid2obs_cape_last90days_plots.sh
*Log File:*
/lfs/h2/emc/vpppg/noscrub/mallory.row/verification/EVS_PRs/pr614/EVS/dev/drivers/scripts/plots/cam/jevs_cam_refs_grid2obs_cape_last90days_plots.o208389933
*DATA:*
/lfs/h2/emc/stmp/mallory.row/evs_test/prod/tmp/jevs_cam_refs_grid2obs_cape_last90days_plots.208389933.dbqs01
3. jevs_cam_refs_grid2obs_ctc_last31days_plots.sh
*Log File:*
/lfs/h2/emc/vpppg/noscrub/mallory.row/verification/EVS_PRs/pr614/EVS/dev/drivers/scripts/plots/cam/jevs_cam_refs_grid2obs_ctc_last31days_plots.o208390097
*DATA:*
/lfs/h2/emc/stmp/mallory.row/evs_test/prod/tmp/jevs_cam_refs_grid2obs_ctc_last31days_plots.208390097.dbqs01
4. jevs_cam_refs_grid2obs_ctc_last90days_plots.sh
*Log File:*
/lfs/h2/emc/vpppg/noscrub/mallory.row/verification/EVS_PRs/pr614/EVS/dev/drivers/scripts/plots/cam/jevs_cam_refs_grid2obs_ctc_last90days_plots.o208390101
*DATA:*
/lfs/h2/emc/stmp/mallory.row/evs_test/prod/tmp/jevs_cam_refs_grid2obs_ctc_last90days_plots.208390101.dbqs01
5. jevs_cam_refs_grid2obs_ecnt_last31days_plots.sh
*Log File:*
/lfs/h2/emc/vpppg/noscrub/mallory.row/verification/EVS_PRs/pr614/EVS/dev/drivers/scripts/plots/cam/jevs_cam_refs_grid2obs_ecnt_last31days_plots.o208390129
*DATA:*
/lfs/h2/emc/stmp/mallory.row/evs_test/prod/tmp/jevs_cam_refs_grid2obs_ecnt_last31days_plots.208390129.dbqs01
6. jevs_cam_refs_grid2obs_ecnt_last90days_plots.sh
*Log File:*
/lfs/h2/emc/vpppg/noscrub/mallory.row/verification/EVS_PRs/pr614/EVS/dev/drivers/scripts/plots/cam/jevs_cam_refs_grid2obs_ecnt_last90days_plots.o208390131
*DATA:*
/lfs/h2/emc/stmp/mallory.row/evs_test/prod/tmp/jevs_cam_refs_grid2obs_ecnt_last90days_plots.208390131.dbqs01
7. jevs_cam_refs_precip_last31days_plots.sh
*Log File:*
/lfs/h2/emc/vpppg/noscrub/mallory.row/verification/EVS_PRs/pr614/EVS/dev/drivers/scripts/plots/cam/jevs_cam_refs_precip_last31days_plots.o208390164
*DATA:*
/lfs/h2/emc/stmp/mallory.row/evs_test/prod/tmp/jevs_cam_refs_precip_last31days_plots.208390164.dbqs01
8. jevs_cam_refs_precip_last90days_plots.sh
*Log File:*
/lfs/h2/emc/vpppg/noscrub/mallory.row/verification/EVS_PRs/pr614/EVS/dev/drivers/scripts/plots/cam/jevs_cam_refs_precip_last90days_plots.o208390165
*DATA:*
/lfs/h2/emc/stmp/mallory.row/evs_test/prod/tmp/jevs_cam_refs_precip_last90days_plots.208390165.dbqs01
9. jevs_cam_refs_precip_spatial_plots.sh
*Log File:*
/lfs/h2/emc/vpppg/noscrub/mallory.row/verification/EVS_PRs/pr614/EVS/dev/drivers/scripts/plots/cam/jevs_cam_refs_precip_spatial_plots.o208390177
*DATA:*
/lfs/h2/emc/stmp/mallory.row/evs_test/prod/tmp/jevs_cam_refs_precip_spatial_plots.208390177.dbqs01
10. jevs_cam_refs_profile_last31days_plots.sh
*Log File:*
/lfs/h2/emc/vpppg/noscrub/mallory.row/verification/EVS_PRs/pr614/EVS/dev/drivers/scripts/plots/cam/jevs_cam_refs_profile_last31days_plots.o208390379
*DATA:*
/lfs/h2/emc/stmp/mallory.row/evs_test/prod/tmp/jevs_cam_refs_profile_last31days_plots.208390379.dbqs01
11. jevs_cam_refs_profile_last90days_plots.sh
*Log File:*
/lfs/h2/emc/vpppg/noscrub/mallory.row/verification/EVS_PRs/pr614/EVS/dev/drivers/scripts/plots/cam/jevs_cam_refs_profile_last90days_plots.o208390402
*DATA:*
/lfs/h2/emc/stmp/mallory.row/evs_test/prod/tmp/jevs_cam_refs_profile_last90days_plots.208390402.dbqs01
12. jevs_cam_refs_snowfall_last31days_plots.sh
*Log File:*
/lfs/h2/emc/vpppg/noscrub/mallory.row/verification/EVS_PRs/pr614/EVS/dev/drivers/scripts/plots/cam/jevs_cam_refs_snowfall_last31days_plots.o208390544
*DATA:*
/lfs/h2/emc/stmp/mallory.row/evs_test/prod/tmp/jevs_cam_refs_snowfall_last31days_plots.208390544.dbqs01
13. jevs_cam_refs_snowfall_last90days_plots.sh
*Log File:*
/lfs/h2/emc/vpppg/noscrub/mallory.row/verification/EVS_PRs/pr614/EVS/dev/drivers/scripts/plots/cam/jevs_cam_refs_snowfall_last90days_plots.o208390545
*DATA:*
/lfs/h2/emc/stmp/mallory.row/evs_test/prod/tmp/jevs_cam_refs_snowfall_last90days_plots.208390545.dbqs01
14. jevs_cam_refs_spcoutlook_last31days_plots.sh
*Log File:*
/lfs/h2/emc/vpppg/noscrub/mallory.row/verification/EVS_PRs/pr614/EVS/dev/drivers/scripts/plots/cam/jevs_cam_refs_spcoutlook_last31days_plots.o208390567
*DATA:*
/lfs/h2/emc/stmp/mallory.row/evs_test/prod/tmp/jevs_cam_refs_spcoutlook_last31days_plots.208390567.dbqs01
15. jevs_cam_refs_spcoutlook_last90days_plots.sh
*Log File:*
/lfs/h2/emc/vpppg/noscrub/mallory.row/verification/EVS_PRs/pr614/EVS/dev/drivers/scripts/plots/cam/jevs_cam_refs_spcoutlook_last90days_plots.o208390568
*DATA:*
/lfs/h2/emc/stmp/mallory.row/evs_test/prod/tmp/jevs_cam_refs_spcoutlook_last90days_plots.208390568.dbqs01
—
Reply to this email directly, view it on GitHub
<#614 (comment)>, or
unsubscribe
<https://github.com/notifications/unsubscribe-auth/AQBMPFPQKWKT2Y2XXO56YM32ECF4NAVCNFSM6AAAAABSXQHEH6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDKMRRGAZTANJQGU>
.
You are receiving this because you were mentioned.Message ID:
***@***.***>
--
Binbin Zhou
Physical Scientist
Lynker at NOAA/NWS/NCEP/EMC
5830 University Research Ct.
College Park, MD 20740
***@***.***
301-683-3683
|
Great! One thing I noticed is it seems a few of the jobs are asking for more resources than they are using. These are jevs_cam_refs_grid2obs_cape_last[31][90]days_plots.sh, jevs_cam_refs_grid2obs_ctc_last[31][90]days_plots.sh, jevs_cam_refs_grid2obs_ecnt_last[31][90]days_plots.sh, jevs_cam_refs_precip_last[31][90]days_plots.sh. jevs_cam_refs_grid2obs_cape_last31days_plots.sh 820 cpus ( Can we get the resources correctly so the jobs are asking for what they need or less (both in dev drivers and ecf scripts)? I also see in /lfs/h2/emc/vpppg/noscrub/mallory.row/verification/EVS_PRs/pr614/EVS/dev/drivers/scripts/plots/cam/jevs_cam_refs_snowfall_last31days_plots.o208390544, 'cp: cannot create regular file '/lfs/h2/emc/vpppg/noscrub/mallory.row/verification/EVS_PRs/pr614/evs/v2.0/plots/cam/atmos.20241203/restart/31/refs_snowfall_plots/threshold_average_regional_conus_east_valid_12z_24h_weasd_ets_f24-30-36-42-48.png': File exists' |
Mallory,
Thanks for pointing these issues out. I'll look at them and make some
adjustments.
Binbin
…On Thu, Dec 5, 2024 at 2:57 PM Mallory Row ***@***.***> wrote:
Great! One thing I noticed is it seems a few of the jobs are asking for
more resources than they are using. These are
jevs_cam_refs_grid2obs_cape_last[31][90]days_plots.sh,
jevs_cam_refs_grid2obs_ctc_last[31][90]days_plots.sh,
jevs_cam_refs_grid2obs_ecnt_last[31][90]days_plots.sh,
jevs_cam_refs_precip_last[31][90]days_plots.sh.
jevs_cam_refs_grid2obs_cape_last31days_plots.sh 820 cpus (
select=10:ncpus=82). It runs the commands for all the parallel jobs as mpiexec
-np 820 -ppn 82 --cpu-bind verbose,depth cfp
/lfs/h2/emc/stmp/mallory.row/evs_test/prod/tmp/jevs_cam_refs_grid2obs_ctc_last31days_plots.20839009
7.dbqs01/run_all_poe.sh. In run_all_poe.sh, only 512 jobs are list, so
that is 308 cpus not being used. You can see this looking in the log for TOTAL
RANK RUN TIME: 0.0 sec.
Can we get the resources correctly so the jobs are asking for what they
need or less (both in dev drivers and ecf scripts)?
I also see in
/lfs/h2/emc/vpppg/noscrub/mallory.row/verification/EVS_PRs/pr614/EVS/dev/drivers/scripts/plots/cam/jevs_cam_refs_snowfall_last31days_plots.o208390544,
'cp: cannot create regular file
'/lfs/h2/emc/vpppg/noscrub/mallory.row/verification/EVS_PRs/pr614/evs/v2.0/plots/cam/atmos.20241203/restart/31/refs_snowfall_plots/threshold_average_regional_conus_east_valid_12z_24h_weasd_ets_f24-30-36-42-48.png':
File exists'
—
Reply to this email directly, view it on GitHub
<#614 (comment)>, or
unsubscribe
<https://github.com/notifications/unsubscribe-auth/AQBMPFO2VR6C24YKH4QOS532ECV2NAVCNFSM6AAAAABSXQHEH6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDKMRRGI3TKOBSGA>
.
You are receiving this because you were mentioned.Message ID:
***@***.***>
--
Binbin Zhou
Physical Scientist
Lynker at NOAA/NWS/NCEP/EMC
5830 University Research Ct.
College Park, MD 20740
***@***.***
301-683-3683
|
Mallory,
The cpu resources have been adjusted to match the number of sub-jobs in
cape, ctc and ecnt jobs. The "File exists" WARNINGs have been fixed in the
snowfall plotting jobs.
Please re-run the plotting jobs to see if there are other issues.
Thanks!
Binbin
…On Thu, Dec 5, 2024 at 2:57 PM Mallory Row ***@***.***> wrote:
Great! One thing I noticed is it seems a few of the jobs are asking for
more resources than they are using. These are
jevs_cam_refs_grid2obs_cape_last[31][90]days_plots.sh,
jevs_cam_refs_grid2obs_ctc_last[31][90]days_plots.sh,
jevs_cam_refs_grid2obs_ecnt_last[31][90]days_plots.sh,
jevs_cam_refs_precip_last[31][90]days_plots.sh.
jevs_cam_refs_grid2obs_cape_last31days_plots.sh 820 cpus (
select=10:ncpus=82). It runs the commands for all the parallel jobs as mpiexec
-np 820 -ppn 82 --cpu-bind verbose,depth cfp
/lfs/h2/emc/stmp/mallory.row/evs_test/prod/tmp/jevs_cam_refs_grid2obs_ctc_last31days_plots.20839009
7.dbqs01/run_all_poe.sh. In run_all_poe.sh, only 512 jobs are list, so
that is 308 cpus not being used. You can see this looking in the log for TOTAL
RANK RUN TIME: 0.0 sec.
Can we get the resources correctly so the jobs are asking for what they
need or less (both in dev drivers and ecf scripts)?
I also see in
/lfs/h2/emc/vpppg/noscrub/mallory.row/verification/EVS_PRs/pr614/EVS/dev/drivers/scripts/plots/cam/jevs_cam_refs_snowfall_last31days_plots.o208390544,
'cp: cannot create regular file
'/lfs/h2/emc/vpppg/noscrub/mallory.row/verification/EVS_PRs/pr614/evs/v2.0/plots/cam/atmos.20241203/restart/31/refs_snowfall_plots/threshold_average_regional_conus_east_valid_12z_24h_weasd_ets_f24-30-36-42-48.png':
File exists'
—
Reply to this email directly, view it on GitHub
<#614 (comment)>, or
unsubscribe
<https://github.com/notifications/unsubscribe-auth/AQBMPFO2VR6C24YKH4QOS532ECV2NAVCNFSM6AAAAABSXQHEH6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDKMRRGI3TKOBSGA>
.
You are receiving this because you were mentioned.Message ID:
***@***.***>
--
Binbin Zhou
Physical Scientist
Lynker at NOAA/NWS/NCEP/EMC
5830 University Research Ct.
College Park, MD 20740
***@***.***
301-683-3683
|
@BinbinZhou-NOAA I missed your comment that the cape, ctc ecnt, and snowfall jobs are ready. Will there be update ncpus for the precip jobs as well? |
Mallory,
Yes, the ncpus and select have been adjusted.
Binbin
…On Tue, Dec 10, 2024 at 12:41 PM Mallory Row ***@***.***> wrote:
@BinbinZhou-NOAA <https://github.com/BinbinZhou-NOAA> I missed your
comment that the cape, ctc ecnt, and snowfall jobs are ready. Will there be
update ncpus for the precip jobs as well?
—
Reply to this email directly, view it on GitHub
<#614 (comment)>, or
unsubscribe
<https://github.com/notifications/unsubscribe-auth/AQBMPFIIFD5XRCRMVC264GD2E4RTFAVCNFSM6AAAAABSXQHEH6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDKMZSGM4TAOJVGQ>
.
You are receiving this because you were mentioned.Message ID:
***@***.***>
--
Binbin Zhou
Physical Scientist
Lynker at NOAA/NWS/NCEP/EMC
5830 University Research Ct.
College Park, MD 20740
***@***.***
301-683-3683
|
Mallory,
The number of jobs in the cape (31 or 90days) are 768, if 768/86 ~ 8.93,
and 768/85~9.04.
Running with select=9 and ncpus=85 will get ERROR.
So reset select=9 and ncpus=85, and rum mpiexec with -np 765 npp 85
(765/85=9)
Same reason for ctc, set select=6 and ncpus=85. and run mpiexec with -np
6 ppn 510
(510/85=6)
Binbin
…On Tue, Dec 10, 2024 at 12:51 PM Mallory Row ***@***.***> wrote:
***@***.**** commented on this pull request.
------------------------------
In scripts/plots/cam/exevs_refs_grid2obs_ctc_plots.sh
<#614 (comment)>:
> @@ -302,7 +302,7 @@ chmod +x run_all_poe.sh
# Run the POE script in parallel or in sequence order to generate png files
#**************************************************************************
if [ $run_mpi = yes ] ; then
- mpiexec -np 820 -ppn 82 --cpu-bind verbose,depth cfp ${DATA}/run_all_poe.sh
+ mpiexec -np 512 -ppn 86 --cpu-bind verbose,depth cfp ${DATA}/run_all_poe.sh
The refs_grid2obs_ctc jobs are requesting select=6:ncpus=86, so 6 * 86 is
516 so need to adjust resources so the come together to 512
—
Reply to this email directly, view it on GitHub
<#614 (review)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AQBMPFPFMYQAVPHDWIDF46D2E4SZNAVCNFSM6AAAAABSXQHEH6VHI2DSMVQWIX3LMV43YUDVNRWFEZLROVSXG5CSMV3GSZLXHMZDIOJTGEZDINJSGE>
.
You are receiving this because you were mentioned.Message ID:
***@***.***>
--
Binbin Zhou
Physical Scientist
Lynker at NOAA/NWS/NCEP/EMC
5830 University Research Ct.
College Park, MD 20740
***@***.***
301-683-3683
|
Will there be update ncpus for the precip jobs as well? I have that written down as requesting too many resources but I don't think I see those changes. |
Mallory,
I might miss the precip job. Now the cpu request for the precip has been
updated.
Thanks!
Binbin
…On Tue, Dec 10, 2024 at 2:38 PM Mallory Row ***@***.***> wrote:
Will there be update ncpus for the precip jobs as well? I have that
written down as requesting too many resources but I don't think I see those
changes.
—
Reply to this email directly, view it on GitHub
<#614 (comment)>, or
unsubscribe
<https://github.com/notifications/unsubscribe-auth/AQBMPFKC6FUA4754VJ3JACD2E47MVAVCNFSM6AAAAABSXQHEH6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDKMZSG4YDCNRWHA>
.
You are receiving this because you were mentioned.Message ID:
***@***.***>
--
Binbin Zhou
Physical Scientist
Lynker at NOAA/NWS/NCEP/EMC
5830 University Research Ct.
College Park, MD 20740
***@***.***
301-683-3683
|
plotsCOMIN is pointing to COMIN=/lfs/h2/emc/vpppg/noscrub/binbin.zhou/evs/v2.0. COMOUT is /lfs/h2/emc/vpppg/noscrub/mallory.row/verification/EVS_PRs/pr614/evs/v2.0/plots/cam/atmos.20241208. 1. jevs_cam_refs_grid2obs_cape_last31days_plots.shLog File: /lfs/h2/emc/vpppg/noscrub/mallory.row/verification/EVS_PRs/pr614/EVS/dev/drivers/scripts/plots/cam/jevs_cam_refs_grid2obs_cape_last31days_plots.o208832862 2. jevs_cam_refs_grid2obs_cape_last90days_plots.shLog File: /lfs/h2/emc/vpppg/noscrub/mallory.row/verification/EVS_PRs/pr614/EVS/dev/drivers/scripts/plots/cam/jevs_cam_refs_grid2obs_cape_last90days_plots.o208832876 3. jevs_cam_refs_grid2obs_ctc_last31days_plots.shLog File: /lfs/h2/emc/vpppg/noscrub/mallory.row/verification/EVS_PRs/pr614/EVS/dev/drivers/scripts/plots/cam/jevs_cam_refs_grid2obs_ctc_last31days_plots.o208832883 4. jevs_cam_refs_grid2obs_ctc_last90days_plots.shLog File: /lfs/h2/emc/vpppg/noscrub/mallory.row/verification/EVS_PRs/pr614/EVS/dev/drivers/scripts/plots/cam/jevs_cam_refs_grid2obs_ctc_last90days_plots.o208832896 5. jevs_cam_refs_grid2obs_ecnt_last31days_plots.shLog File: /lfs/h2/emc/vpppg/noscrub/mallory.row/verification/EVS_PRs/pr614/EVS/dev/drivers/scripts/plots/cam/jevs_cam_refs_grid2obs_ecnt_last31days_plots.o208832902 6. jevs_cam_refs_grid2obs_ecnt_last90days_plots.shLog File: /lfs/h2/emc/vpppg/noscrub/mallory.row/verification/EVS_PRs/pr614/EVS/dev/drivers/scripts/plots/cam/jevs_cam_refs_grid2obs_ecnt_last90days_plots.o208832903 7. jevs_cam_refs_precip_last31days_plots.shLog File: /lfs/h2/emc/vpppg/noscrub/mallory.row/verification/EVS_PRs/pr614/EVS/dev/drivers/scripts/plots/cam/jevs_cam_refs_precip_last31days_plots.o208833716 8. jevs_cam_refs_precip_last90days_plots.shLog File: /lfs/h2/emc/vpppg/noscrub/mallory.row/verification/EVS_PRs/pr614/EVS/dev/drivers/scripts/plots/cam/jevs_cam_refs_precip_last90days_plots.o208833718 9. jevs_cam_refs_snowfall_last31days_plots.shLog File: /lfs/h2/emc/vpppg/noscrub/mallory.row/verification/EVS_PRs/pr614/EVS/dev/drivers/scripts/plots/cam/jevs_cam_refs_snowfall_last31days_plots.o208832907 10. jevs_cam_refs_snowfall_last90days_plots.shLog File: /lfs/h2/emc/vpppg/noscrub/mallory.row/verification/EVS_PRs/pr614/EVS/dev/drivers/scripts/plots/cam/jevs_cam_refs_snowfall_last90days_plots.o208832908 |
Mallory,
All of the jobs are good with correct tar files and no ERROR in the log
files.
It seems the memory in many plot jobs are requested too much. I think they
can be
reduced.
Binbin
…On Tue, Dec 10, 2024 at 3:06 PM Mallory Row ***@***.***> wrote:
plots
COMIN is pointing to COMIN=/lfs/h2/emc/vpppg/noscrub/binbin.zhou/evs/v2.0.
COMOUT is
/lfs/h2/emc/vpppg/noscrub/mallory.row/verification/EVS_PRs/pr614/evs/v2.0/plots/cam/atmos.20241208.
1. jevs_cam_refs_grid2obs_cape_last31days_plots.sh
*Log File:*
/lfs/h2/emc/vpppg/noscrub/mallory.row/verification/EVS_PRs/pr614/EVS/dev/drivers/scripts/plots/cam/jevs_cam_refs_grid2obs_cape_last31days_plots.o208832862
*DATA:*
/lfs/h2/emc/stmp/mallory.row/evs_test/prod/tmp/jevs_cam_refs_grid2obs_cape_last31days_plots.208832862.dbqs01
2. jevs_cam_refs_grid2obs_cape_last90days_plots.sh
*Log File:*
/lfs/h2/emc/vpppg/noscrub/mallory.row/verification/EVS_PRs/pr614/EVS/dev/drivers/scripts/plots/cam/jevs_cam_refs_grid2obs_cape_last90days_plots.o208832876
*DATA:*
/lfs/h2/emc/stmp/mallory.row/evs_test/prod/tmp/jevs_cam_refs_grid2obs_cape_last90days_plots.208832876.dbqs01
3. jevs_cam_refs_grid2obs_ctc_last31days_plots.sh
*Log File:*
/lfs/h2/emc/vpppg/noscrub/mallory.row/verification/EVS_PRs/pr614/EVS/dev/drivers/scripts/plots/cam/jevs_cam_refs_grid2obs_ctc_last31days_plots.o208832883
*DATA:*
/lfs/h2/emc/stmp/mallory.row/evs_test/prod/tmp/jevs_cam_refs_grid2obs_ctc_last31days_plots.208832883.dbqs01
4. jevs_cam_refs_grid2obs_ctc_last90days_plots.sh
*Log File:*
/lfs/h2/emc/vpppg/noscrub/mallory.row/verification/EVS_PRs/pr614/EVS/dev/drivers/scripts/plots/cam/jevs_cam_refs_grid2obs_ctc_last90days_plots.o208832896
*DATA:*
/lfs/h2/emc/stmp/mallory.row/evs_test/prod/tmp/jevs_cam_refs_grid2obs_ctc_last90days_plots.208832896.dbqs01
5. jevs_cam_refs_grid2obs_ecnt_last31days_plots.sh
*Log File:*
/lfs/h2/emc/vpppg/noscrub/mallory.row/verification/EVS_PRs/pr614/EVS/dev/drivers/scripts/plots/cam/jevs_cam_refs_grid2obs_ecnt_last31days_plots.o208832902
*DATA:*
/lfs/h2/emc/stmp/mallory.row/evs_test/prod/tmp/jevs_cam_refs_grid2obs_ecnt_last31days_plots.208832902.dbqs01
6. jevs_cam_refs_grid2obs_ecnt_last90days_plots.sh
*Log File:*
/lfs/h2/emc/vpppg/noscrub/mallory.row/verification/EVS_PRs/pr614/EVS/dev/drivers/scripts/plots/cam/jevs_cam_refs_grid2obs_ecnt_last90days_plots.o208832903
*DATA:*
/lfs/h2/emc/stmp/mallory.row/evs_test/prod/tmp/jevs_cam_refs_grid2obs_ecnt_last90days_plots.208832903.dbqs01
7. jevs_cam_refs_precip_last31days_plots.sh
*Log File:*
/lfs/h2/emc/vpppg/noscrub/mallory.row/verification/EVS_PRs/pr614/EVS/dev/drivers/scripts/plots/cam/jevs_cam_refs_precip_last31days_plots.o208833716
*DATA:*
/lfs/h2/emc/stmp/mallory.row/evs_test/prod/tmp/jevs_cam_refs_precip_last31days_plots.208833716.dbqs01
8. jevs_cam_refs_precip_last90days_plots.sh
*Log File:*
/lfs/h2/emc/vpppg/noscrub/mallory.row/verification/EVS_PRs/pr614/EVS/dev/drivers/scripts/plots/cam/jevs_cam_refs_precip_last90days_plots.o208833718
*DATA:*
/lfs/h2/emc/stmp/mallory.row/evs_test/prod/tmp/jevs_cam_refs_precip_last90days_plots.208833718.dbqs01
9. jevs_cam_refs_snowfall_last31days_plots.sh
*Log File:*
/lfs/h2/emc/vpppg/noscrub/mallory.row/verification/EVS_PRs/pr614/EVS/dev/drivers/scripts/plots/cam/jevs_cam_refs_snowfall_last31days_plots.o208832907
*DATA:*
/lfs/h2/emc/stmp/mallory.row/evs_test/prod/tmp/jevs_cam_refs_snowfall_last31days_plots.208832907.dbqs01
10. jevs_cam_refs_snowfall_last90days_plots.sh
*Log File:*
/lfs/h2/emc/vpppg/noscrub/mallory.row/verification/EVS_PRs/pr614/EVS/dev/drivers/scripts/plots/cam/jevs_cam_refs_snowfall_last90days_plots.o208832908
*DATA:*
/lfs/h2/emc/stmp/mallory.row/evs_test/prod/tmp/jevs_cam_refs_snowfall_last90days_plots.208832908.dbqs01
—
Reply to this email directly, view it on GitHub
<#614 (comment)>, or
unsubscribe
<https://github.com/notifications/unsubscribe-auth/AQBMPFJ77WUBEETVLTBYBJT2E5CVXAVCNFSM6AAAAABSXQHEH6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDKMZSG42TQMZQGU>
.
You are receiving this because you were mentioned.Message ID:
***@***.***>
--
Binbin Zhou
Physical Scientist
Lynker at NOAA/NWS/NCEP/EMC
5830 University Research Ct.
College Park, MD 20740
***@***.***
301-683-3683
|
Running with the memory changes! COMIN is pointing to COMIN=/lfs/h2/emc/vpppg/noscrub/binbin.zhou/evs/v2.0. COMOUT is /lfs/h2/emc/vpppg/noscrub/mallory.row/verification/EVS_PRs/pr614/evs/v2.0/plots/cam/atmos.20241209. 1. jevs_cam_refs_grid2obs_cape_last31days_plots.shLog File: /lfs/h2/emc/vpppg/noscrub/mallory.row/verification/EVS_PRs/pr614/EVS/dev/drivers/scripts/plots/cam/jevs_cam_refs_grid2obs_cape_last31days_plots.o208900124 2. jevs_cam_refs_grid2obs_cape_last90days_plots.shLog File: /lfs/h2/emc/vpppg/noscrub/mallory.row/verification/EVS_PRs/pr614/EVS/dev/drivers/scripts/plots/cam/jevs_cam_refs_grid2obs_cape_last90days_plots.o208900133 3. jevs_cam_refs_grid2obs_ctc_last31days_plots.shLog File: /lfs/h2/emc/vpppg/noscrub/mallory.row/verification/EVS_PRs/pr614/EVS/dev/drivers/scripts/plots/cam/jevs_cam_refs_grid2obs_ctc_last31days_plots.o208900137 4 jevs_cam_refs_grid2obs_ctc_last90days_plots.shLog File: /lfs/h2/emc/vpppg/noscrub/mallory.row/verification/EVS_PRs/pr614/EVS/dev/drivers/scripts/plots/cam/jevs_cam_refs_grid2obs_ctc_last90days_plots.o208900138 5. jevs_cam_refs_grid2obs_ecnt_last31days_plots.shLog File: /lfs/h2/emc/vpppg/noscrub/mallory.row/verification/EVS_PRs/pr614/EVS/dev/drivers/scripts/plots/cam/jevs_cam_refs_grid2obs_ecnt_last31days_plots.o208900139 6. jevs_cam_refs_grid2obs_ecnt_last90days_plots.shLog File: /lfs/h2/emc/vpppg/noscrub/mallory.row/verification/EVS_PRs/pr614/EVS/dev/drivers/scripts/plots/cam/jevs_cam_refs_grid2obs_ecnt_last90days_plots.o208900140 7. jevs_cam_refs_profile_last31days_plots.shLog File: /lfs/h2/emc/vpppg/noscrub/mallory.row/verification/EVS_PRs/pr614/EVS/dev/drivers/scripts/plots/cam/jevs_cam_refs_profile_last31days_plots.o208900143 8. jevs_cam_refs_profile_last90days_plots.shLog File: /lfs/h2/emc/vpppg/noscrub/mallory.row/verification/EVS_PRs/pr614/EVS/dev/drivers/scripts/plots/cam/jevs_cam_refs_profile_last90days_plots.o208900145 9. jevs_cam_refs_snowfall_last31days_plots.shLog File: /lfs/h2/emc/vpppg/noscrub/mallory.row/verification/EVS_PRs/pr614/EVS/dev/drivers/scripts/plots/cam/jevs_cam_refs_snowfall_last31days_plots.o208900161 10. jevs_cam_refs_snowfall_last90days_plots.shLog File: /lfs/h2/emc/vpppg/noscrub/mallory.row/verification/EVS_PRs/pr614/EVS/dev/drivers/scripts/plots/cam/jevs_cam_refs_snowfall_last90days_plots.o208900165 11. jevs_cam_refs_spcoutlook_last31days_plots.shLog File: /lfs/h2/emc/vpppg/noscrub/mallory.row/verification/EVS_PRs/pr614/EVS/dev/drivers/scripts/plots/cam/jevs_cam_refs_spcoutlook_last31days_plots.o208900167 12. jevs_cam_refs_spcoutlook_last90days_plots.shLog File: /lfs/h2/emc/vpppg/noscrub/mallory.row/verification/EVS_PRs/pr614/EVS/dev/drivers/scripts/plots/cam/jevs_cam_refs_spcoutlook_last90days_plots.o208900169 |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Changes are good and testing successful!
Thanks for your work @BinbinZhou-NOAA!
All jobs are good with correct output, no ERROR messages in the log files.
Thanks!
Binbin
…On Wed, Dec 11, 2024 at 9:08 AM Mallory Row ***@***.***> wrote:
Running with the memory changes!
COMIN is pointing to COMIN=/lfs/h2/emc/vpppg/noscrub/binbin.zhou/evs/v2.0.
COMOUT is
/lfs/h2/emc/vpppg/noscrub/mallory.row/verification/EVS_PRs/pr614/evs/v2.0/plots/cam/atmos.20241209.
1. jevs_cam_refs_grid2obs_cape_last31days_plots.sh
*Log File:*
/lfs/h2/emc/vpppg/noscrub/mallory.row/verification/EVS_PRs/pr614/EVS/dev/drivers/scripts/plots/cam/jevs_cam_refs_grid2obs_cape_last31days_plots.o208900124
*DATA:*
/lfs/h2/emc/stmp/mallory.row/evs_test/prod/tmp/jevs_cam_refs_grid2obs_cape_last31days_plots.208900124.dbqs01
2. jevs_cam_refs_grid2obs_cape_last90days_plots.sh
*Log File:*
/lfs/h2/emc/vpppg/noscrub/mallory.row/verification/EVS_PRs/pr614/EVS/dev/drivers/scripts/plots/cam/jevs_cam_refs_grid2obs_cape_last90days_plots.o208900133
*DATA:*
/lfs/h2/emc/stmp/mallory.row/evs_test/prod/tmp/jevs_cam_refs_grid2obs_cape_last90days_plots.208900133.dbqs01
3. jevs_cam_refs_grid2obs_ctc_last31days_plots.sh
*Log File:*
/lfs/h2/emc/vpppg/noscrub/mallory.row/verification/EVS_PRs/pr614/EVS/dev/drivers/scripts/plots/cam/jevs_cam_refs_grid2obs_ctc_last31days_plots.o208900137
*DATA:*
/lfs/h2/emc/stmp/mallory.row/evs_test/prod/tmp/jevs_cam_refs_grid2obs_ctc_last31days_plots.208900137.dbqs01
4 jevs_cam_refs_grid2obs_ctc_last90days_plots.sh
*Log File:*
/lfs/h2/emc/vpppg/noscrub/mallory.row/verification/EVS_PRs/pr614/EVS/dev/drivers/scripts/plots/cam/jevs_cam_refs_grid2obs_ctc_last90days_plots.o208900138
*DATA:*
/lfs/h2/emc/stmp/mallory.row/evs_test/prod/tmp/jevs_cam_refs_grid2obs_ctc_last90days_plots.208900138.dbqs01
5. jevs_cam_refs_grid2obs_ecnt_last31days_plots.sh
*Log File:*
/lfs/h2/emc/vpppg/noscrub/mallory.row/verification/EVS_PRs/pr614/EVS/dev/drivers/scripts/plots/cam/jevs_cam_refs_grid2obs_ecnt_last31days_plots.o208900139
*DATA:*
/lfs/h2/emc/stmp/mallory.row/evs_test/prod/tmp/jevs_cam_refs_grid2obs_ecnt_last31days_plots.208900139.dbqs01
6. jevs_cam_refs_grid2obs_ecnt_last90days_plots.sh
*Log File:*
/lfs/h2/emc/vpppg/noscrub/mallory.row/verification/EVS_PRs/pr614/EVS/dev/drivers/scripts/plots/cam/jevs_cam_refs_grid2obs_ecnt_last90days_plots.o208900140
*DATA:*
/lfs/h2/emc/stmp/mallory.row/evs_test/prod/tmp/jevs_cam_refs_grid2obs_ecnt_last90days_plots.208900140.dbqs01
7. jevs_cam_refs_profile_last31days_plots.sh
*Log File:*
/lfs/h2/emc/vpppg/noscrub/mallory.row/verification/EVS_PRs/pr614/EVS/dev/drivers/scripts/plots/cam/jevs_cam_refs_profile_last31days_plots.o208900143
*DATA:*
/lfs/h2/emc/stmp/mallory.row/evs_test/prod/tmp/jevs_cam_refs_profile_last31days_plots.208900143.dbqs01
8. jevs_cam_refs_profile_last90days_plots.sh
*Log File:*
/lfs/h2/emc/vpppg/noscrub/mallory.row/verification/EVS_PRs/pr614/EVS/dev/drivers/scripts/plots/cam/jevs_cam_refs_profile_last90days_plots.o208900145
*DATA:*
/lfs/h2/emc/stmp/mallory.row/evs_test/prod/tmp/jevs_cam_refs_profile_last90days_plots.208900145.dbqs01
9. jevs_cam_refs_snowfall_last31days_plots.sh
*Log File:*
/lfs/h2/emc/vpppg/noscrub/mallory.row/verification/EVS_PRs/pr614/EVS/dev/drivers/scripts/plots/cam/jevs_cam_refs_snowfall_last31days_plots.o208900161
*DATA:*
/lfs/h2/emc/stmp/mallory.row/evs_test/prod/tmp/jevs_cam_refs_snowfall_last31days_plots.208900161.dbqs01
10. jevs_cam_refs_snowfall_last90days_plots.sh
*Log File:*
/lfs/h2/emc/vpppg/noscrub/mallory.row/verification/EVS_PRs/pr614/EVS/dev/drivers/scripts/plots/cam/jevs_cam_refs_snowfall_last90days_plots.o208900165
*DATA:*
/lfs/h2/emc/stmp/mallory.row/evs_test/prod/tmp/jevs_cam_refs_snowfall_last90days_plots.208900165.dbqs01
11. jevs_cam_refs_spcoutlook_last31days_plots.sh
*Log File:*
/lfs/h2/emc/vpppg/noscrub/mallory.row/verification/EVS_PRs/pr614/EVS/dev/drivers/scripts/plots/cam/jevs_cam_refs_spcoutlook_last31days_plots.o208900167
*DATA:*
=/lfs/h2/emc/stmp/mallory.row/evs_test/prod/tmp/jevs_cam_refs_spcoutlook_last31days_plots.208900167.dbqs01
12. jevs_cam_refs_spcoutlook_last90days_plots.sh
*Log File:*
/lfs/h2/emc/vpppg/noscrub/mallory.row/verification/EVS_PRs/pr614/EVS/dev/drivers/scripts/plots/cam/jevs_cam_refs_spcoutlook_last90days_plots.o208900169
*DATA:*
/lfs/h2/emc/stmp/mallory.row/evs_test/prod/tmp/jevs_cam_refs_spcoutlook_last90days_plots.208900169.dbqs01
—
Reply to this email directly, view it on GitHub
<#614 (comment)>, or
unsubscribe
<https://github.com/notifications/unsubscribe-auth/AQBMPFNODIURZAMOXQAJM6D2FBBO5AVCNFSM6AAAAABSXQHEH6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDKMZWGA4DQMZSGM>
.
You are receiving this because you were mentioned.Message ID:
***@***.***>
--
Binbin Zhou
Physical Scientist
Lynker at NOAA/NWS/NCEP/EMC
5830 University Research Ct.
College Park, MD 20740
***@***.***
301-683-3683
|
Mallory,
I tried to sync the refs_unit_conversion fork, but there are 38
conflicts to be
discarded before it can be synced. What is the action should I do?
Thanks!
Binbin
On Wed, Dec 11, 2024 at 10:30 AM Binbin Zhou - NOAA Affiliate <
***@***.***> wrote:
… All jobs are good with correct output, no ERROR messages in the log files.
Thanks!
Binbin
On Wed, Dec 11, 2024 at 9:08 AM Mallory Row ***@***.***>
wrote:
> Running with the memory changes!
>
> COMIN is pointing to
> COMIN=/lfs/h2/emc/vpppg/noscrub/binbin.zhou/evs/v2.0. COMOUT is
> /lfs/h2/emc/vpppg/noscrub/mallory.row/verification/EVS_PRs/pr614/evs/v2.0/plots/cam/atmos.20241209.
> 1. jevs_cam_refs_grid2obs_cape_last31days_plots.sh
>
> *Log File:*
> /lfs/h2/emc/vpppg/noscrub/mallory.row/verification/EVS_PRs/pr614/EVS/dev/drivers/scripts/plots/cam/jevs_cam_refs_grid2obs_cape_last31days_plots.o208900124
> *DATA:*
> /lfs/h2/emc/stmp/mallory.row/evs_test/prod/tmp/jevs_cam_refs_grid2obs_cape_last31days_plots.208900124.dbqs01
> 2. jevs_cam_refs_grid2obs_cape_last90days_plots.sh
>
> *Log File:*
> /lfs/h2/emc/vpppg/noscrub/mallory.row/verification/EVS_PRs/pr614/EVS/dev/drivers/scripts/plots/cam/jevs_cam_refs_grid2obs_cape_last90days_plots.o208900133
> *DATA:*
> /lfs/h2/emc/stmp/mallory.row/evs_test/prod/tmp/jevs_cam_refs_grid2obs_cape_last90days_plots.208900133.dbqs01
> 3. jevs_cam_refs_grid2obs_ctc_last31days_plots.sh
>
> *Log File:*
> /lfs/h2/emc/vpppg/noscrub/mallory.row/verification/EVS_PRs/pr614/EVS/dev/drivers/scripts/plots/cam/jevs_cam_refs_grid2obs_ctc_last31days_plots.o208900137
> *DATA:*
> /lfs/h2/emc/stmp/mallory.row/evs_test/prod/tmp/jevs_cam_refs_grid2obs_ctc_last31days_plots.208900137.dbqs01
> 4 jevs_cam_refs_grid2obs_ctc_last90days_plots.sh
>
> *Log File:*
> /lfs/h2/emc/vpppg/noscrub/mallory.row/verification/EVS_PRs/pr614/EVS/dev/drivers/scripts/plots/cam/jevs_cam_refs_grid2obs_ctc_last90days_plots.o208900138
> *DATA:*
> /lfs/h2/emc/stmp/mallory.row/evs_test/prod/tmp/jevs_cam_refs_grid2obs_ctc_last90days_plots.208900138.dbqs01
> 5. jevs_cam_refs_grid2obs_ecnt_last31days_plots.sh
>
> *Log File:*
> /lfs/h2/emc/vpppg/noscrub/mallory.row/verification/EVS_PRs/pr614/EVS/dev/drivers/scripts/plots/cam/jevs_cam_refs_grid2obs_ecnt_last31days_plots.o208900139
> *DATA:*
> /lfs/h2/emc/stmp/mallory.row/evs_test/prod/tmp/jevs_cam_refs_grid2obs_ecnt_last31days_plots.208900139.dbqs01
> 6. jevs_cam_refs_grid2obs_ecnt_last90days_plots.sh
>
> *Log File:*
> /lfs/h2/emc/vpppg/noscrub/mallory.row/verification/EVS_PRs/pr614/EVS/dev/drivers/scripts/plots/cam/jevs_cam_refs_grid2obs_ecnt_last90days_plots.o208900140
> *DATA:*
> /lfs/h2/emc/stmp/mallory.row/evs_test/prod/tmp/jevs_cam_refs_grid2obs_ecnt_last90days_plots.208900140.dbqs01
> 7. jevs_cam_refs_profile_last31days_plots.sh
>
> *Log File:*
> /lfs/h2/emc/vpppg/noscrub/mallory.row/verification/EVS_PRs/pr614/EVS/dev/drivers/scripts/plots/cam/jevs_cam_refs_profile_last31days_plots.o208900143
> *DATA:*
> /lfs/h2/emc/stmp/mallory.row/evs_test/prod/tmp/jevs_cam_refs_profile_last31days_plots.208900143.dbqs01
> 8. jevs_cam_refs_profile_last90days_plots.sh
>
> *Log File:*
> /lfs/h2/emc/vpppg/noscrub/mallory.row/verification/EVS_PRs/pr614/EVS/dev/drivers/scripts/plots/cam/jevs_cam_refs_profile_last90days_plots.o208900145
> *DATA:*
> /lfs/h2/emc/stmp/mallory.row/evs_test/prod/tmp/jevs_cam_refs_profile_last90days_plots.208900145.dbqs01
> 9. jevs_cam_refs_snowfall_last31days_plots.sh
>
> *Log File:*
> /lfs/h2/emc/vpppg/noscrub/mallory.row/verification/EVS_PRs/pr614/EVS/dev/drivers/scripts/plots/cam/jevs_cam_refs_snowfall_last31days_plots.o208900161
> *DATA:*
> /lfs/h2/emc/stmp/mallory.row/evs_test/prod/tmp/jevs_cam_refs_snowfall_last31days_plots.208900161.dbqs01
> 10. jevs_cam_refs_snowfall_last90days_plots.sh
>
> *Log File:*
> /lfs/h2/emc/vpppg/noscrub/mallory.row/verification/EVS_PRs/pr614/EVS/dev/drivers/scripts/plots/cam/jevs_cam_refs_snowfall_last90days_plots.o208900165
> *DATA:*
> /lfs/h2/emc/stmp/mallory.row/evs_test/prod/tmp/jevs_cam_refs_snowfall_last90days_plots.208900165.dbqs01
> 11. jevs_cam_refs_spcoutlook_last31days_plots.sh
>
> *Log File:*
> /lfs/h2/emc/vpppg/noscrub/mallory.row/verification/EVS_PRs/pr614/EVS/dev/drivers/scripts/plots/cam/jevs_cam_refs_spcoutlook_last31days_plots.o208900167
> *DATA:*
> =/lfs/h2/emc/stmp/mallory.row/evs_test/prod/tmp/jevs_cam_refs_spcoutlook_last31days_plots.208900167.dbqs01
> 12. jevs_cam_refs_spcoutlook_last90days_plots.sh
>
> *Log File:*
> /lfs/h2/emc/vpppg/noscrub/mallory.row/verification/EVS_PRs/pr614/EVS/dev/drivers/scripts/plots/cam/jevs_cam_refs_spcoutlook_last90days_plots.o208900169
> *DATA:*
> /lfs/h2/emc/stmp/mallory.row/evs_test/prod/tmp/jevs_cam_refs_spcoutlook_last90days_plots.208900169.dbqs01
>
> —
> Reply to this email directly, view it on GitHub
> <#614 (comment)>, or
> unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AQBMPFNODIURZAMOXQAJM6D2FBBO5AVCNFSM6AAAAABSXQHEH6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDKMZWGA4DQMZSGM>
> .
> You are receiving this because you were mentioned.Message ID:
> ***@***.***>
>
--
Binbin Zhou
Physical Scientist
Lynker at NOAA/NWS/NCEP/EMC
5830 University Research Ct.
College Park, MD 20740
***@***.***
301-683-3683
--
Binbin Zhou
Physical Scientist
Lynker at NOAA/NWS/NCEP/EMC
5830 University Research Ct.
College Park, MD 20740
***@***.***
301-683-3683
|
@BinbinZhou-NOAA Please do not sync anything to your branch. |
OK, Thanks
…On Wed, Dec 11, 2024 at 10:41 AM Mallory Row ***@***.***> wrote:
@BinbinZhou-NOAA <https://github.com/BinbinZhou-NOAA> Please do not sync
anything to your branch.
—
Reply to this email directly, view it on GitHub
<#614 (comment)>, or
unsubscribe
<https://github.com/notifications/unsubscribe-auth/AQBMPFKAWUYHU7YZ5AGZFTL2FBMIRAVCNFSM6AAAAABSXQHEH6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDKMZWGM2DOMJUGM>
.
You are receiving this because you were mentioned.Message ID:
***@***.***>
--
Binbin Zhou
Physical Scientist
Lynker at NOAA/NWS/NCEP/EMC
5830 University Research Ct.
College Park, MD 20740
***@***.***
301-683-3683
|
Note to developers: You must use this PR template!
Description of Changes
Following updates are included in this PR:
1.. Update CAPE settings in the conf files
2. Update plotting scripts for T and Td unit conversion
3. Change subprocess.Popen to subprocess.run in EVS v2.0 (if they are used at all).
This change also requires ps.communicated()[0] to be changed to ps.stdout to work correctly
4. Define $NET prior to $HOMEevs in dev scripts
5. Match settings (walltime, place, cpu, etc) in dev and ecf scripts
6. Update the graphic file name to follow required format
7. Tested in Metplus-beta5
Developer Questions and Checklist
Yes, Maat's RRFS and REFS development runs will be temporally stopped around Dec. 2 or 3 until at least next spring
in order to save the resources for the RRFS/REFS retrospective runs for the RRFS/REFS field evaluation. So it is better to test the stat jobs before Dec. 3.
No
No
${USER}
where necessary throughout the code.HOMEevs
are removed from the code.dev/drivers/scripts
ordev/modulefiles
have been made in the correspondingecf/scripts
andecf/defs/evs-nco.def
?Note: there are some WARINGs that could not be removed until the METplus-beta6 is implemented
Testing Instructions
Note: If the testing is on a personal account, the following line should be added to the spcoutlook stat driver script:
export EVSINspcotlk=/lfs/h2/emc/vpppg/noscrub/emc.vpppg/evs/v2.0/prep/cam
or
export EVSINspcotlk=/lfs/h1/ops/prod/com/evs/v1.0/prep/cam
Test procedures:
Part 1. For the stats generation jobs
There are 3 independent stats jobs:
jevs_cam_refs_grid2obs_stats.sh
jevs_cam_refs_precip_stats.sh
jevs_cam_refs_spcoutlook_stats.sh
Just submit them at same time or at different times
Part 2 For the plot generation jobs
Note: since REFS is a new component for EVS, there are no REFS stat files in vpppg diectory
/lfs/h2/emc/vpppg/noscrub/emc.vpppg/evs/v2.0/stats/cam
So please set COMIN in the plot driver scripts to be /lfs/h2/emc/vpppg/noscrub/binbin.zhou/evs/v2.0/stats/cam
There are 15 independent graphics generation jobs:
7 jobs for 31-day score plots, 7 jobs for 90-day score plots and 1 job for precip spatial map:
All of those 15 jobs are pretty faster, most of them are less than 15i min, but still have restart capability except for spatial map job.
jevs_cam_refs_grid2obs_cape_past31days_plots.sh
jevs_cam_refs_grid2obs_cape_past90days_plots.sh
jevs_cam_refs_grid2obs_ctc_past31days_plots.sh
jevs_cam_refs_grid2obs_ctc_past90days_plots.sh
jevs_cam_refs_precip_past31days_plots.sh
jevs_cam_refs_precip_past90days_plots.sh
jevs_cam_refs_profile_past31days_plots.sh
jevs_cam_refs_profile_past90days_plots.sh
jevs_cam_refs_grid2obs_ecnt_past31days_plots.sh
jevs_cam_refs_grid2obs_ecnt_past90days_plots.sh
jevs_cam_refs_snowfall_past31days_plots.sh
jevs_cam_refs_snowfall_past90days_plots.sh
jevs_cam_refs_spcoutlook_past31days_plots.sh
jevs_cam_refs_spcoutlook_past90days_plots.sh
jevs_cam_refs_precip_spatial_plots.sh
Submit them at same time or at different times.