Skip to content

Latest commit

 

History

History
 
 

examples

Folders and files

NameName
Last commit message
Last commit date

parent directory

..
dataflow
dataflow
 
 
README.md
README.md
 
 
ee_to_zarr.py
ee_to_zarr.py
 
 
ee_to_zarr_reqs.txt
ee_to_zarr_reqs.txt
 
 

README.md

Xee Examples

Xee integrations & use cases.

Export Earth Engine ImageCollections to Zarr with Xarray-Beam

The following demonstrates how to export ~20 TiBs of NASA IMERG data to Zarr (in about ~25 LOC) with Xarray-Beam and GCP's Dataflow Runner. At the time of writing, we were able to export the data in about ~3 hours (using Google's internal Beam runner).

python3 ee_to_zarr.py \
  --input NASA/GPM_L3/IMERG_V06 \
  --output $BUCKET/imerg_006.zarr \
  --target_chunks "time=6" \
  --runner DataflowRunner \
  -- \
  --project $PROJECT \
  --region $REGION \
  --temp_location $BUCKET/tmp/ \
  --no_use_public_ips  \
  --network $NETWORK \
  --subnetwork regions/$REGION/subnetworks/$SUBNET \
  --requirements ee_to_zarr_reqs.txt \
  --job_name imerg-to-zarr