DaskExecutor

[normanrz]

Description:

• Adds a DaskExecutor that can use an existing dask cluster + scheduler to distribute computation. We can use dask to easily schedule workloads across on-demand-created VMs
• Refactors Executor type to a protocol that should make the interface more explicit and therefore better enforceable by mypy
• Upgrades dependencies in cluster_tools and webknossos

How to test

• Start a dask scheduler python -m dask scheduler
• Start at least one dask worker python -m dask worker tcp://10.0.0.2:8786 (can be on the same machine)
• Open a Python shell and run:

import webknossos as wk
from cluster_tools import get_executor

m = wk.Dataset.open("path/to/l4_sample").get_layer("color").get_mag("1")
with get_executor("dask", address="10.0.0.2:8786") as exc:
  m.map_chunk(lambda view: view.read().mean(), executor=exc)

Todos:

Make sure to delete unnecessary points or to check all before merging:

• Updated Changelog
• Added / Updated Tests

[philippotto]

awesome stuff 👍 testing went also very smooth 🥇

there might be some potential to DRY things between the cluster executor and the dask executor, but I guess it's alright for now.