diff --git a/UCs-problems/Climate data processing with dask extremly slow.md b/UCs-problems/Climate data processing with dask extremly slow.md deleted file mode 100644 index 229417d..0000000 --- a/UCs-problems/Climate data processing with dask extremly slow.md +++ /dev/null @@ -1,10 +0,0 @@ -## Climate data processing with dask extremly slow - -| Section | Value | -| ------- | ---- | -| Use case | UC1 | -| Description | The scope is to process 1 year of hourly climate data (around 14GB in NetCDF) to produce daily statistics for selected EU cities. Using dask to (hopefully) speedup data loading and processing.
However, the script runtime is considerably slower when executed on FAIRiCube Hub than when executed locally. -| Impact on the project | This problem causes execution time and resource consumption to increase exponentially (up to 10x in some cases). | -| Component | Storage, CPU, RAM and Network | -| Potential solution | Traditional file formats (e.g. tiff, netCDF) cause a lot of network traffic and slow down the computation when the file resides on the cloud.
Cloud-optimized format like COG, zarr are designed to overcome this problem. | -| Solution benefits | The use of cloud-optimised formats results in exponentially better performance (in terms of execution time and resources consumed) than traditional formats such as NetCDF. |