Fix multiprocessing and consolidate QC #68
Conversation
kyleredilla
changed the title
There was a problem hiding this comment.
I like the idea of folding all of the QC work into a single spot, and this looks great overall! I ran into a few small issues along the way though:
-
While running this Prefect flow using the
v1_configbranch of the prefect repo, it complained that the--qc_notebookoption was missing from the QC command and quit/stalled out. I replaced the system call here https://github.com/ua-snap/prefect/blob/58843c3f0070bd8ef9094326d826b58e92136bfa/regridding/regridding_functions.py#L394 with the following and it seemed to work after that:f"export PATH=$PATH:/opt/slurm-22.05.4/bin:/opt/slurm-22.05.4/sbin:$HOME/miniconda3/bin && python {run_qc_script} --qc_notebook '{visual_qc_notebook}' --conda_init_script '{conda_init_script}' --conda_env_name '{conda_env_name}' --cmip6_directory '{cmip6_directory}' --output_directory '{output_directory}' --repo_regridding_directory '{repo_regridding_directory}' --vars '{vars}' --freqs '{freqs}' --models '{models}' --scenarios '{scenarios}'" -
The QC notebook complained that
error_filewas not defined in a couple places. See PR code review comments. -
After removing the
error_filereferences to run the QC notebook to completion, the random src vs. regrid files it chose to inspect produced the following error:AssertionError: No files found for regridded file clt_Amon_MPI-ESM1-2-HR_historical_regrid_196201-196212.nc in /beegfs/CMIP6/arctic-cmip6/CMIP6/CMIP/DKRZ/MPI-ESM1-2-HR/historical with */Amon/clt/*/*/clt_Amon_MPI-ESM1-2-HR_historical_*.nc.My second run chose a different set of random files & succeeded, however. And it looks great!! Once those small issues are fixed, I think this is good to merge.
regridding/qc.ipynb
Outdated
| "print(f\"QC process complete: {error_count} errors found.\")\n", | ||
| "if len(ds_errors) > 0:\n", | ||
| " print(\n", | ||
| " f\"Errors in opening some datasets. {len(ds_errors)} files could not be opened. See {str(error_file)} for error log.\"\n", |
There was a problem hiding this comment.
The QC notebook complained that error_file was not defined here and execution stopped. I got around this temporarily just by removing the reference to error_file here.
regridding/qc.ipynb
Outdated
| " )\n", | ||
| "if len(value_errors) > 0:\n", | ||
| " print(\n", | ||
| " f\"Errors in dataset values. {len(value_errors)} files have regridded values outside of source file range. See {str(error_file)} for error log.\"\n", |
There was a problem hiding this comment.
Same as my above comment, error_file was not defined here either.
The initial goal with this PR was simply to improve the reliability of using multiprocessing to scrape the grid info from files in the batch file generation step of the regridding pipeline, and in the QC section of sanity checking all regridded files. The behavior we consistently see in using multiprocessing (or concurrent.futures paradigm) is that whatever method is used to dispatch some function that operates on multiple netcdf files can sometimes hang indefinitely. This PR has taken some steps to improve this, but it would seem that total reliability here is out of scope for now. It seems that simply processing fewer files by breaking things up into smaller groups, such as we have begun doing for the prefect flows (with v1_1 and v1_2 variables etc), is a sane way forward for now. The QC step that was checking every single new file has been changed as well to only check a random subset of the files, which should help with the hanging symptoms. You will notice that the quality control step step that was originally outside of the QC notebook has been moved into that notebook, so that we only have one QC product to evaluate following a flow run.
To test, simply run a regridding flow using prefect, probably for a subset of variables and frequencies, such as monthly v1_2 etc and check out the quality control notebook.