Script to generate validation material

[rchan26]

Working towards #20.

• Makes changes to run_forecast_plots.sh to allow passing of threshold, grid area, region
• Make changes to run_forecast_plots.sh to make all leadtime averaged plots and to save the metrics dataframe(s)
• Add better documentation of various options
• Create run_validation.sh to produce plots for validation
  • Can pass in metrics that you want to produce plots for
  • Can pass in thresholds of interest
  • Can pass in grid area size (for SIE error)
  • Can pass in region to focus on
  • It will produce all leadtime averaged plots and individual plots for each forecast (and stitch together to make a video)
  • It will also produce the same plots but compare with ECMWF foreasts

[rchan26]

For some examples:

./run_prediction.sh -m "binacc,sie,sic,mae" -l -v MAY_2018 dh23 north MAY_2018 current

• runs the predictions for the dates specified by MAY_2018_start and MAY_2018_end for model dh23 in hemisphere north which used current dataset
• generates plots for binary accuracy, SIE error, raw SIC error (videos), MAE
• (since we have -l) will generate leadtime averaged plots (averaged over all, averaged over monthly, and averaged over daily - will plot them for the initialisation day and target day)
  • standard deviation performance will be plotted too
• (since we have -v) these plots will be stitched together using ffmpeg
• will save the plots in plot/MAY_2018_north by default

./run_forecast_plots.sh -m "binacc,sie" -t 0.9 -g 20 -e -v -o plot/MAY_2018 MAY_2018 north

• creates the metric plots for binary accuracy and SIE error
• (since we have -t) uses threshold = 0.9
• (since we have -g) uses grid area size of 20km^2
• (since we have -e) will compare with ECMWF
• (since we have -v) will stitch together the plots using ffmpeg
• (since we have -o) will store the plots in plot/MAY_2018 (note, usually will save in MAY_2018_north by default)

./run_validation.sh -r "125,190,170,235" -t 0.2,0.9 MAY_2018 north

• since no metrics are passed, will produce metric plots for binary accuracy, SIE error, RMSE, MAE by default
• (since we have -r) will focus on the region that is passed in
• (since we have -t) will produce plots for thresholds = 0.2, 0.9
• since no grid area size is passed, will use grid area size of 25km^2 by default (for SIE error plots)
• will create all leadtime average plots
• will also compare with ECMWF
• will save plots in plot/validation/MAY_2018_north and in plot/validation/MAY_2018_north/ECMWF_comp for ECMWF comparisons

./run_validation.sh -m "binacc,mae" -o plot/validation/MAY_2018 MAY_2018 north

• will produce metric plots for binary accuracy and MAE
• since no threshold is passed, will produce plots for thresholds = 0.15, 0.8
• will create all leadtime average plots
• will also compare with ECMWF
• (since we have -o) will save plots in plot/validation/MAY_2018 and in plot/validation/MAY_2018/ECMWF_comp for ECMWF comparisons (usually will save in plot/validation/MAY_2018_north by default)

[JimCircadian]

Awesome, these are really easy changes to review too @rchan26. Let's merge this as soon as 0.2.4 is rolled out, if that's cool?