Processing LSRs

[spence98]

Hello! I am currently attempting to process local storm reports from the SPC site (so it is in the correct format for ASCII2NC) using METplus. I have edited the configuration file used for the 5.2.3.2. Point2Grid: Calculate Practically Perfect Probabilities Use Case so that it reflects the timing/data that I'm using. When I try running the python script to run everything, I get an error at the ASCII2NC step. The first time I tried, it didn't recognize the python flag as the ASCII format, so I changed that to 'met_point' and that seemed to fix that issue. However, it now can not find my OBS_INPUT file to be able to actually run the program. I'm attaching the log file and configuration file, but if you need any other files just let me know. Any assistance in figuring out why the program is having issues finding my input data would be greatly appreciated! I've tried reading through the documentation to find the root of where the problem may be but haven't had any luck with it yet.

metplus.log.20211022173531.txt
Point2Grid_obsLSR_ObsOnly_PracticallyPerfect.txt

Thank you,
Michelle

EDIT: I just tried running the 5.2.3.2. Point2Grid: Calculate Practically Perfect Probabilities Use Case again, but it is coming up with the same error as stated above.

[JohnHalleyGotway]

Hi Michelle, @georgemccabe is probably the best person to answer this question, but he's out of the office today. I'll take a crack at answering but would like George to review and offer up any additional advice.

First, thanks for sending the METplus log and configuration files. That really help diagnose the situation. You're adapting the Calculate Practically Perfect Probabilities use case. I see from the PROCESS_LIST that you're running ASCII2NC, Point2Grid, and RegridDataPlane.

Checking in the METplus log file, I see that it errored out before actually calling ascii2nc.

10/22 17:35:31.936 metplus (command_builder.py:255) ERROR: (command_builder.py:728) Could not find OBS_INPUT file: ...

For this use case, you're running ascii2nc using python embedding to serve up point observations. As such, the ascii2nc input format should be "python", not "met_point":

ASCII2NC_INPUT_FORMAT = python

Otherwise, METplus thinks that the python script you're running is an ACTUAL observation input file name, which it is not. And that's the immediate source of this error. But you state "The first time I tried, it didn't recognize the python flag as the ASCII format". I'm not totally sure what that means.

I'd recommend taking a step back. Let's first make sure that the python embedding command runs on the command line manually:

python /raid-08/LS/spence98/METplus-4.0.0/parm/use_cases/model_applications/convection_allowing_models/Point2Grid_obsLSR_ObsOnly_PracticallyPerfect/read_ascii_storm.py /home/spence98/spence98/METplus-4.0.0/sample_data/model_applications/convection_allowing_models/practically_perfect/080307_rpts_torn.csv

Assuming that runs without error, let's try running ascii2nc manually. I'll assume ascii2nc is in your path. If not, just pre-pend the path the path MET's /bin directory:

ascii2nc \
'/raid-08/LS/spence98/METplus-4.0.0/parm/use_cases/model_applications/convection_allowing_models/Point2Grid_obsLSR_ObsOnly_PracticallyPerfect/read_ascii_storm.py /home/spence98/spence98/METplus-4.0.0/sample_data/model_applications/convection_allowing_models/practically_perfect/080307_rpts_torn.csv' \
/home/spence98/spence98/METplus-4.0.0/cases/lsr_data/7mar08_StormReps.2008030712.nc \
-format python

Do those commands run without error?

If they do, please try rerunning using 'ASCII2NC_INPUT_FORMAT = python' and send the resulting METplus log file.

[spence98]

SUNDAY EVENING UPDATE:

Hello again! So, over the weekend and after further tries of getting things working my advisor and I have determined that the root of the problem may lie within our universities supercomputer and maybe something on the back end is trying to force the use of python, despite my commands to tell it where to point to. Maybe that's it, maybe it isn't, but regardless we have decided to forgo trying to use it on that system.

We have set up a separate system on a different computer and downloaded docker and are attempting to use METplus via that route instead. Upon doing this, I was able to successfully*(see below) run the example use case that takes LSRs and puts them on a grid. However, when I try to do it for my own data, it appears to run and I get the message at the end saying it successfully completed, but no observational data is retained. When I look into the log file that is created, the program is getting tripped up at the first ascii step and it says that modules can't be found. (Something like 'read_ascii_storm' module not found).

I am new to the docker world, so maybe I am incorrectly running the command or something, but I wanted to give you both an update. I'm happy to open a new ticket if you would like, just let me know! I can also provide the log file and such tomorrow when I have access to the computer I was using.

Michelle

• While the example use case appears to work, it does pull the sample data from a directory that I can not locate from my machine and puts it into a directory that I also can't seem to find, so I'm not actually 100% sure that it worked properly.

[spence98]

MONDAY UPDATE:

An update since our update from last night. I tried running the test case again and then checked the log file that it outputs. It turns out that upon checking the log file it actually didn't run successfully, despite the "METplus has successfully finished running" message that it output to the screen. The use case is also getting caught up at the first ascii2nc step when it tries to run the python script. Like previously mentioned, it is like it is trying to read in the read_ascii_storm.py file as a module. As a result it acts like it is trying to read the observation file, but it doesn't actually retain the observations. It continues through the processes, but doesn't actually read in any observations, so it just creates empty files. I don't know why that is the case, but I'm going to try to figure that out today. My theory is that it could be something related to the run_metplus.py file?

Michelle

[georgemccabe]

Hi Michelle, I tried to run the use case with a bad INPUT_BASE value and confirmed that it succeeds even though the input file is not found. The app ascii2nc creates a file even if it doesn't find any observations by design. However, the error is not properly caught by the script. This is happening for a few reasons.

The script that creates the tmp file (write_tmp_point.py) imports the python script to get the data and apparently an error in an imported module does not cause a failure. To test I added logic to the python embedding script (read_ascii_storm.py) to call sys.exit(1) if the file is not found and that caused an error in ascii2nc (as expected).

My guess is that the intended behavior of the read_ascii_storm.py script is to prevent an error from occurring if the input file is not found and log a message saying that it was not found. This is done because many run times can be processed and some times do not have an input file (I think @CPKalb wrote this script and could confirm this). Either way, in this example use case the file should exist and I think it should fail if it does not exist. This could be handled by adding an argument to the script to fail if any files are not found, but still allow the script to be used to write an empty file if no inputs are found. If the intended behavior is to actually fail if the input file is not found, then the new argument is not necessary and the script should exit with an error code if so.

Also, the read_ascii_storm.py script handles the NameError exception but the error that actually occurs is FileNotFoundError. I would have expected this to cause a failure, but apparently it does not. I think it would be worthwhile to see if the importlib functions that are used in the write_tmp_point.py script have a way to check if any uncaught exceptions occurred and error if so. We have also discussed changing the logic of the python embedding in MET so that the script is executed as if it was called from the command line instead of being imported, although I am not sure how to get the data into MET that way. @JohnHalleyGotway thoughts?

Long story short, I'm guessing that the input file does not exist in your Docker environment and that is why you are not getting any results. You can check this by checking if the file listed as the input file in the script log output exists. The sample input data for the use case is available on DockerHub so you can get the data by creating a Docker data volume and mounting that into your Docker container. To do that, run the following Docker command:

docker create --name convection_allowing_models dtcenter/metplus-data:4.0-convection_allowing_models

This creates a Docker data volume named convection_allowing_models from the dtcenter/metplus-data:4.0-convection_allowing_models image from DockerHub. Then add the volume to your docker run command by using the --volumes-from argument:

docker run --volumes-from convection_allowing_models ...

Let me know if that doesn't work.

[georgemccabe]

@spence98, I am just following up on this issue. Were you able to get the use case to run properly?

On a related note, pull request dtcenter/MET#1958 fixed the issue where failures in a python embedding script for point observations were not caught properly. Also pull request #1238 fixed the python embedding script from this use case so that it will report an error if the input file is not found. These two fixes should improve this use case to make it more clear when the input data is not available.

[spence98]

Hello George, I apologize for the delayed response.

Thank you for your different explanations and taking the time to try to figure out what was going on! I appreciate it a lot!

Over the weekend I worked to try to figure out these issues and what was happening. On Tuesday I came in, tried running things the exact same way as over the weekend, and for whatever reason everything worked properly. I honestly do not know what could have possibly changed because I (and two other people) were working throughout the weekend to get things going.

Regardless, I have been able to move on from that issue and have successfully processed all my LSR data using the docker!

Michelle