kubernetes_asyncio major version release breaks: service account not used anymore ("system/anonymous" not allowed to create "pods")

[mkarbo]

First up: Apologies, I don't have a good example for reproducing the issue, as I use prefect and not dask directly.

TL;DR: New release of kubernetes-asyncio (22.6) broke something in dask-kubernetes (or in prefect, I don't know), pinning version fixed my workflows. Hope this helps someone.

---

Yesterday, my dask pipelines all worked perfectly in AWS EKS. Today, after re-building images, I consistently got the following:

...
  File "/pipelines/flow_test/.venv/lib/python3.9/site-packages/dask_kubernetes/core.py", line 474, in __init__
    super().__init__(**self.kwargs)
  File "/pipelines/flow_test/.venv/lib/python3.9/site-packages/distributed/deploy/spec.py", line 260, in __init__
    self.sync(self._start)
  File "/pipelines/flow_test/.venv/lib/python3.9/site-packages/distributed/utils.py", line 311, in sync
    return sync(
  File "/pipelines/flow_test/.venv/lib/python3.9/site-packages/distributed/utils.py", line 378, in sync
    raise exc.with_traceback(tb)
  File "/pipelines/flow_test/.venv/lib/python3.9/site-packages/distributed/utils.py", line 351, in f
    result = yield future
  File "/pipelines/flow_test/.venv/lib/python3.9/site-packages/tornado/gen.py", line 762, in run
    value = future.result()
  File "/pipelines/flow_test/.venv/lib/python3.9/site-packages/dask_kubernetes/core.py", line 603, in _start
    await super()._start()
  File "/pipelines/flow_test/.venv/lib/python3.9/site-packages/distributed/deploy/spec.py", line 293, in _start
    self.scheduler = await self.scheduler
  File "/pipelines/flow_test/.venv/lib/python3.9/site-packages/distributed/deploy/spec.py", line 59, in _
    await self.start()
  File "/pipelines/flow_test/.venv/lib/python3.9/site-packages/dask_kubernetes/core.py", line 182, in start
    await super().start(**kwargs)
  File "/pipelines/flow_test/.venv/lib/python3.9/site-packages/dask_kubernetes/core.py", line 87, in start
    raise e
  File "/pipelines/flow_test/.venv/lib/python3.9/site-packages/dask_kubernetes/core.py", line 77, in start
    self._pod = await self.core_api.create_namespaced_pod(
  File "/pipelines/flow_test/.venv/lib/python3.9/site-packages/kubernetes_asyncio/client/api_client.py", line 192, in __call_api
    raise e
  File "/pipelines/flow_test/.venv/lib/python3.9/site-packages/kubernetes_asyncio/client/api_client.py", line 185, in __call_api
    response_data = await self.request(
  File "/pipelines/flow_test/.venv/lib/python3.9/site-packages/kubernetes_asyncio/client/rest.py", line 223, in POST
    return (await self.request("POST", url,
  File "/pipelines/flow_test/.venv/lib/python3.9/site-packages/kubernetes_asyncio/client/rest.py", line 180, in request
    raise ApiException(http_resp=r)
kubernetes_asyncio.client.exceptions.ApiException: (403)
Reason: Forbidden
HTTP response headers: <...>
HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"pods is forbidden: User \"system:anonymous\" cannot create resource \"pods\" in API group \"\" in the namespace \"prefect\"","reason":"Forbidden","details":{"kind":"pods"},"code":403}

After some investigation, I found that a release of kubernetes-asyncio (version bump 21.7 -> 22.6) has broken dask-kubernetes in all my pipelines.

After adding constraint on version kubernetes-asyncio<22.6, all my pipelines work. Posting this in-case anyone else has similar issues.

[jacobtomlinson]

xref #420

Looks like this version bump has broken a few things. I'll look at pinning it.

[mkarbo]

Cheers :) And thank you for all the work

[ddelange]

I can confirm the new release 22.6.1 fixes the 401 Unauthorized from 22.6.0 for me

[jacobtomlinson]

That's great thanks for updating @ddelange. If this has been fixed upstream I'll close this out.