From f2eff9f86bd7d09f3b75caadc43aed7f30af2793 Mon Sep 17 00:00:00 2001 From: Antonio Cozzolino Date: Thu, 6 Jun 2024 17:45:37 +0200 Subject: [PATCH] Update climate_data_processing_with_dask_extremly_slow.md --- .../climate_data_processing_with_dask_extremly_slow.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/UCs-lessons-learnt/climate_data_processing_with_dask_extremly_slow.md b/UCs-lessons-learnt/climate_data_processing_with_dask_extremly_slow.md index 5730425..6a4239b 100644 --- a/UCs-lessons-learnt/climate_data_processing_with_dask_extremly_slow.md +++ b/UCs-lessons-learnt/climate_data_processing_with_dask_extremly_slow.md @@ -10,7 +10,7 @@ The scope is to process 1 year of hourly climate data (around 14GB in NetCDF) to This problem causes execution time and resource consumption to increase exponentially (up to 10x in some cases). ## Affected component of FAIRiCUBE-Hub -Storage, CPU, RAM and Network +FAIRiCUBE Lab (Storage, CPU, RAM and Network) ## Proposed solution Traditional file formats (e.g. tiff, netCDF) cause a lot of network traffic and slow down the computation when the file resides on the cloud.
Cloud-optimized format like COG, zarr are designed to overcome this problem.