2021-12-13 PyAerocom meeting

[avaldebe]

Going through PRs

• Ove's station fix and test PR fix ungriddedata.UngriddedData.from_stationdata with single station #536 (merged)
• Daniel will follow up on tutorial w/Ove add tutorial notebook for adding station data pyaerocom-tutorials#9
• one logger per module #535 needs reviewer
• use plain dicts #530 needs reviewer

How have people used pyaerocom this week?

• Packaging and deployment: Charlie and Alvaro
• Pyaerocom workshop: Charlie, Alvaro, Anna, Daniel

Chunking

• One chunk: Need to fetch/unpack the entire chunk, expensive if chunks are too big
• Too small chunks: Too much overhead in assembling the fields
• Netcdf has decent defaults (chunk size 16 MB) when creating an unlimited dimension
• Compression requires chunks
• Compression is applied per chunk

netCDF rechunking commands

• NCO: ncks -O -4 --chunk_policy nco (needs twice the data size as RAM)
• CDO: cdo -O -f nc4 -k auto copy (pretty RAM efficient, but needs to recognise the grid and might not copy everything)

The outcome of the two commands above is not always the same!

CAMS2 83

Ecflow job handling, multiprocessing: env variable NSLOTS for number of cores/CPUs

CAMS2 83 meeting with Daniel, Krister, Charlie and Alvaro (Thursday or Friday)

Next meeting

Monday January 3rd, 2022