Add gpu_limit to make_pod_spec

[ddelange]

• Add support for scheduling GPUs (k8s docs) ref Pass worker resources to pod args #398 (comment)
• Correct memory_limit docstring: fractions (float) and 'GB' (str, internally converted to 'GiB') are supported by --memory-limit (CLI docs) but not by k8s' Pod resources spec (k8s docs). It took me some time to find this out also ref Distributed LocalCluster's memory_limit keyword argument needs documentation  dask#8224
• Sync docstring arg order with the function signature arg order
• People looking to schedule on AMD gpu's (vast minority) can still achieve this by passing amd.com/gpu along with memory and cpu requests/limits to extra_pod_config instead of using the separate request/limit args.

[ddelange]

Some CI errors maybe due to new major release of kubernetes_asyncio ref #419 ?

AttributeError: module 'kubernetes_asyncio.client' has no attribute 'V1beta1Eviction'

[jacobtomlinson]

This looks great! With GPUs on Kubernetes the limit and request must always be set to the same value, I wonder if we should not expose two different options.

As you say AMD GPUs and also be configured via extra pod config, along with NVIDIA MIG.

[ddelange]

Ah interesting, didn't know. Agreed :) Is it similar behaviour to memory, i.e. make a pod with memory limits and k8s will automatically set memory requests to the same value? In that case could leave only gpu_limit kwarg?

[ddelange]

Some CI errors maybe due to new major release of kubernetes_asyncio ref #419 ?

AttributeError: module 'kubernetes_asyncio.client' has no attribute 'V1beta1Eviction'

Pushed a fix, V1beta1Eviction was removed in v22

[jacobtomlinson]

Woo this looks great! Many thanks.