Future development plan for AI4S2S project

[geek-yang]

Value and vision of s2spy suite

Based on the exploration of recipes with our tools and some team discussions, here are some thoughts about the "role" of our packages:

• Pipeline structure may not be suitable for all users, it is specific for regression based forecasts.
• Our pipeline may not serve the deep learning workflow from the users (e.g. weather forecasts with transformer). Only a small part of the pipeline could be useful (e.g. lilio, preprocessing).
• However, the "easy-to-achieve" regression forecasts using our packages could be used as skillful baselines for deep learning forecasts, which also in a way that facilitates the deep learning forecasting research.
• Currently the benchmarking is not prioritized, transparent, and standardized (@semvijverberg can you put a link to the microsoft paper here?). Our package could help with this.

List of tasks for improvement

• Scaling to HPC and big datasets (e.g. CMIP6).
  • Conduct "pressure tests" with CMIP6/ERA5 data for lilio and s2spy.
  • Identify bottlenecks in terms of speed and memory usage
  • Implement dask (and other tools) to optimize the workflow
• Support more detrending operations
  • For detrending operations, we now assume a linear trend in the climate system. We could think of an ARIMA type of approach to predict the warming trend? The most neat way to model the global warming signal is to retrieve the signal by comparing the multi-model mean greenhouse gas (GHG) forced CMIP6 simulations versus Natural Only forced CMIP6 models. See paper by Andre Jueling (@semvijverberg Could you please add a link to the paper?).
  • Explore ESMValTool for more options for detrending
• From PR prospective, it might be more effective to promote lilio and preprocessing separately
  • Provide flexibility for use cases not suitable for our workflow.
  • This might be necessary when preprocessor becomes bigger.
• Support multiple lags for one RGDR instance
• Add more recipes to our cookbook
  • Basic workflow with our regression based forecasts
  • A deep learning workflow using part of our packages but a full baseline with regression based forecasts
  • A joint workflow of s2spy and ESMValTool
• Enable cross-validation for both model fitting and dimensionality reduction during hyper-parameter tuning

The priority of these tasks are to be determined.

[BSchilperoort]

@geek-yang and I had a discussion about supporting deep learning workflows:

• It's important to note that we only want to support deep learning where lilio is used. This also means that there is technically no timeseries component anymore (I.e. the NN forecasts A, using data points B, C, D... etc.).
• As there is a limited amount of weather data available, and compute resources are not infinite, there is also still space for dimensionality reduction, as this reduces the complexity and training time of the neural network.
• Additionally, we should have a good look at what researchers are doing now. We can support a hypothetical workflow that we think is best, but if no one is interested in doing it this way, then who are we developing the software for.

[semvijverberg]

Thanks for the overview @geek-yang!

The paper on S2S benchmarks, they may have done what we want.

• https://github.com/microsoft/subseasonal_toolkit (toolkit for learned benchmarks).
• https://github.com/microsoft/subseasonal_data (data downloader, don't know if it still works).

I've asked André for the paper on learning the climate change signal from cmip runs, will come back to it when I have the answer.

[semvijverberg]

I also found this paper (and code) on standardizing benchmarking, but for medium range forecasting!

WeatherBench: A Benchmark Data Set for Data‐Driven Weather Forecasting
https://onlinelibrary.wiley.com/doi/10.1029/2020MS002203

[semvijverberg]

Related to "stress testing" the package and related to accommodating an additional research community, we could think about how lilio can support the processing of ML-driven pipeline for post-processing of ensemble forecasts.

You can easily download EC46 data (within Python) using the climetlab package.

This is the data format:

I will raise an issue to give some additional context.