Plotting issues/errors with 'plot_allvars' task

[swomantree]

Hello,
I am an intern working on testing and verification of the UFS-SRW app. Whenever I try to use the COM-OUT_REF on the plot_all_vars task, I get a weird error.
This is what I'm running in my config.yaml file
task_plot_allvars:
COMOUT_REF: '{EXPT_BASEDIR}/jun_23_23_12z_rap/${PDY}${cyc}/postprd'

This command should just plot the differences between two experiments. When I use the FV3-HRRR physics, the plot_allvars task fails on hour 36 of my forecast, when it's trying to plot CAPE.
The log file on my plot_all_vars task gives this error.

ERROR: TopologyException: side location conflict at -101.92278893881431 40.306652768131286
INFO: Self-intersection at or near point -104.51722419282993 42.156323502581415

When the plot_allvars tasks complete on my FV3_RAP run, the plots are not very useful. The plots should have some transparency, be transparent at 0, or have the map drawn on top of the data for spatial reference. I'm attaching examples of some of the problematic plots.
Thank you for your help.

Output (optional)

[gspetro-NOAA]

Hi @swomantree,

I am going to see if someone on my team can look into this, but in the meantime, it would be helpful to have a bit more information. Since you're plotting the difference, would it be possible to send us the config files for both experiments?

Thanks,
Gillian Petro | EPIC Advanced User Support Team

[gspetro-NOAA]

Hi @swomantree,

@danielabdi-noaa suggested that the error message could be related to the use of corrupted input data for plotting. The "TopologyException" is a Cartopy error not directly related to the plotting script. If you are able to provide the config.yaml script and any other info on the test setup, we can try to reproduce the error, but we do need that information to be able to help you troubleshoot.

Daniel also thought that the diff plot looked ok (i.e., similar to plots he's gotten in the past), so it seems like perhaps the transparency/overlap over reference area are wishlist items. @BenjaminBlake-NOAA worked on the plotting scripts, so he may be able to provide more information as to whether these look normal and/or whether there is a way to modify the plots for the result you want.

Best,
Gillian

[swomantree]

Here are the config.yaml files I used to try to create this difference plot.
config_hrrr.yaml.txt
config_rap.yaml.txt

[gspetro-NOAA]

Thanks for the config files! In general, they look fine, so it might be helpful for me to take a look at the logs. Where are your experiment directories on Hera? Sometimes the final error that causes a failure is actually a result of issues further up in the log, so that might give us an idea of what's going wrong, especially given what Daniel was saying about TopologyException being a Cartopy-specific error.

[swomantree]

Yes, my experiment directories are on Hera. I can send you the path for all of the log files if that's easiest.

…

On Thu, Aug 17, 2023, 7:03 PM Gillian Petro ***@***.***> wrote: Thanks for the config files! In general, they look fine, so it might be helpful for me to take a look at the logs. Where are your experiment directories on Hera? Sometimes the final error that causes a failure is actually a result of issues further up in the log, so that might give us an idea of what's going wrong, especially given what Daniel was saying about TopologyException being a Cartopy-specific error. — Reply to this email directly, view it on GitHub <#865 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AZLMKX6BCQRIGRUF77UCPTDXV25MLANCNFSM6AAAAAA2QGGAOY> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[gspetro-NOAA]

Yes, that would be helpful! I don't have access to HPSS, so while I'm rectifying that situation, I can take a look at the logs and perhaps get someone with HPSS access to run the experiment/replicate the bug if I'm not able to do so myself.

[danielabdi-noaa]

@gspetro-NOAA @swomantree I do have access to HPSS but it looks like tasks are failing at the get_extrn_ics step because it could not find the necessary files. The hrrr config file fails with this error message:

Username: Daniel.Abdi  UID: 20429  Acct: 20429(20429) Copies: 1 COS: 0 Firewall: off [hsi.9.3.0.p0 Tue Oct 4 14:08:46 UTC 2022] 
*** ls: No such file or directory [-2: HPSS_ENOENT] 
    /NCEPPROD/hpssprod/runhistory/rh2023/202306/20230624/gpfs_hps_nco_ops_com_hrrr_prod_hrrr.20230624_conus12-17.wrf.tar
###WARNING  hsi returned non-zero exit status.
            64 = /apps/hpss/bin/hsi ls /NCEPPROD/hpssprod/runhistory/rh2023/202306/20230624/gpfs_hps_nco_ops_com_hrrr_prod_hrrr.20230624_conus12-17.wrf.tar
WARNING: /NCEPPROD/hpssprod/runhistory/rh2023/202306/20230624/gpfs_hps_nco_ops_com_hrrr_prod_hrrr.20230624_conus12-17.wrf.tar is not available! 
 
INFO: Running command 
 hsi ls /NCEPPROD/hpssprod/runhistory/rh2023/202306/20230624/com_hrrr_prod_hrrr.20230624_conus12-17.wrf.tar 

The RAP config file has a typo in the specification of COMOUT_REF but the same error occurs after fixing that. Also it is trying to get a "HRRR" ICs file while I expected it to fetch a RAP ICs file. Looking into the HPSS directory, I see files like com_rap/hrrr_v5.. etc but not com_rap/hrrr_prod. Maybe there are changes in HPSS that I am not aware of, or I don't have full access to all files.

[gsketefian]

@danielabdi-noaa @gspetro-NOAA @swomantree During Sarah's internship this summer, we encountered (at least) a couple of issues with the get_extr_[ics|lbcs] tasks. I documented them at #854 and #856.

@danielabdi-noaa If you haven't resolved the above problem yet, I'm happy to help if you send me your experiment and clone paths (hopefully you're working on Hera or Jet). Thanks.

[danielabdi-noaa]

Thank you @gsketefian @swomantree I copied the data_locations.yaml file from Sarah's directory that had the changes you made and it worked for me after that. Yes, this seems to be a critical bug that needs to be fixed soon.
Anway, I am running the HRRR test case and comparing against Sarah's RAP run and will let you know how it goes later.
I don't think the problem lies with the plotting scripts but with the output data used plotting, mainly because the non-diff plots worked for Sarah for all 48 hrs for the RAP run, and upto 36 hrs for the HRRR run.

[danielabdi-noaa]

@swomantree The HRRR run seems to work for me. Here is my experiment directory:

/scratch2/BMC/gsd-hpcs/Daniel.Abdi/expt_dirs/june_24_23_12z_hrrr

The plot task run out of wallclock time but it was going well until then. I may try to re-run it with a larger wallclock time limit for the plot task to see if it can produce all diff plots. Despite what I said earlier, your HRRR run did not actually start making diff plots, it failed at the 36th hour of the regular plot task. Also your COMOUT_REF path should not use EXPTDIR since the experiment directory we want to compare to is that of RAP's, so set it explicitly like this:

COMOUT_REF: '/scratch2/BMC/fv3lam/Sarah.Womantree/expt_dirs/june_24_23_12z_rap/${PDY}${cyc}/postprd'

[danielabdi-noaa]

After increasing walltime for plot_allvars task to 08:00:00 hrs, both the regular and diff plots files finished in about 2 hrs.
I increased the walltime directly in the worklflow xml file for this run, but you would need to add something like this in the config file under rocoto: tasks: section.

task_plot_allvars:
   walltime: 08:00:00

The plots are here:

/scratch2/BMC/gsd-hpcs/Daniel.Abdi/expt_dirs/june_24_23_12z_hrrr/2023062412/postprd

[swomantree]

Nice, thank you Daniel! I'll try that out. I'm trying to run a new experiment, so I'll see if the SRW app can produce those regular and difference plots now.

…

On Tue, Aug 22, 2023 at 7:04 AM danielabdi-noaa ***@***.***> wrote: After increasing walltime for plot_allvars task to 08:00:00 hrs, both the regular and diff plots files finished in about 2 hrs. I increased the walltime directly in the worklflow xml file for this run, but you would need to add something like this in the config file under rocoto: tasks: section. task_plot_allvars: walltime: 08:00:00 The plots are here: /scratch2/BMC/gsd-hpcs/Daniel.Abdi/expt_dirs/june_24_23_12z_hrrr/2023062412/postprd — Reply to this email directly, view it on GitHub <#865 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AZLMKXYZ3XU666OP4ADKF7DXWSU4JANCNFSM6AAAAAA2QGGAOY> . You are receiving this because you were mentioned.Message ID: ***@***.***>