Load with cancel option

[johnkit]

Is it possible to add an Xarray.DataArray.start_load() method that uses a dask.distributed client and returns a Future object(s) that can be canceled by the application? Sometimes we want to kill a large download (too big, slow connection, operator error, etc.) and right now the only way we can stop Xarray/dask is to kill the application.

[keewis]

You might be able to use distributed.Client.compute for this. xarray objects are dask collections, so this should work immediately:

import xarray as xr
from distributed import Client

client = Client()

ds = xr.tutorial.open_dataset("air_temperature", chunks={})
f1 = client.compute(ds)
f1.cancel()
f2 = client.compute(ds.Tair)
f2.cancel()

[johnkit]

gosh thanks, I will give that a try

[johnkit]

Yes! initializing the distributed client indeed works. Thanks @keewis !

One observation: If I use dask processes, I seem to leave the system in some unhappy state with 1 worker process still alive and 1 "leaked semaphore object" at exit. Rather than wade into why that occurs, and because my code is local, I switched to threads

client = Client(processes=False)