dispatch.run is not resilient to worker loss

[hendrikmakait]

dispatch.run uses worker-restrictions to pin tasks to the workers they should get executed on. Should a worker get removed (or possibly restarted), this will cause the task to transition to the no-worker state and remain there indefinitely (see dask/distributed#7346). From what I see, there is no mechanism implemented to prevent this.

To circumvent this, dask-pytorch-ddp would probably also benefit from dask/distributed#8624.