Define "GPU" as a worker resource #1401
Conversation
| resources = dict(pair.split("=") for pair in resources) | ||
| resources = valmap(float, resources) | ||
| gpu_resources = valmap(int, itemfilter(lambda x: x != "GPU", resources)) | ||
| resources = valmap(float, itemfilter(lambda x: x == "GPU", resources)) |
There was a problem hiding this comment.
@jacobtomlinson you have written this section originally, I presume mapping values to float was done to support a definition such as "MEMORY" but since I don't see any tests or any other explicit mention in Dask-CUDA, could you comment if that's right and whether you can think of more robust ways for us to handle types here other than what I wrote above as "GPU" or NOT "GPU"?
There was a problem hiding this comment.
Honestly I don't remember the reasoning. I think dask handles these values as floats, as you say it's to support values like MEMORY or other arbitrary quantities.
| if "GPU" not in worker_kwargs["resources"]: | ||
| worker_kwargs["GPU"] = 1 | ||
| else: | ||
| worker_kwargs["resources"] = {"GPU": 1} |
There was a problem hiding this comment.
I wonder if it makes sense to use {"cuda": 1} (i.e. use "cuda" as the resource name for a CUDA-capable GPU)?
This would align with the convention used in DL (e.g. pytorch)
There was a problem hiding this comment.
I wouldn't mind personally, but the keyword GPU is documented in Distributed worker resources docs, so that seems like a more universal option and better documented.
There was a problem hiding this comment.
Okay nice - That's good enough for me.
pentschev
added
3 - Ready for Review
Add "GPU" as a worker resource to each CUDA worker. This should support users in identifying whether the workers available contain a GPU resource.