Permissions issues in the 22.6.0

[jonathon-love]

hi,

i've been happily using kubernetes_asyncio for a while now on a microk8s v1.21.10 cluster, but i recently had a container having permissions issues when executing a watch on persistent volumes:

kubernetes_asyncio.client.exceptions.ApiException: (401)
Reason: Unauthorized: Unauthorized

after tinkering for a while, i traced the issue back to the recent 22.6.0 release of kubernetes_asyncio ... if i roll back to an earlier version of kubernetes_asyncio, everything works again.

is this a problem in kubrnetes_asyncio, or has my microk8s fallen off the back of reasonable compatibility?

with thanks

[tomplus]

Do you watch PVs from the pod or from outside the cluster? Do you use this function: CoreV1Api.list_persistent_volume?

Could you provide steps to reproduce? I tried something on my minikube but with no luck.

[jonathon-love]

yeah, so this is from within a pod inside my cluster.

i basically go:

config.load_incluster_config()

client = ApiClient()
api = client.CoreV1Api(client)

watch = Watch()
async for update in watch.stream(self._api.list_persistent_volume):
    ...

is that enough to go on? i can whip you up a reproducible example if you need (but i'm obviously lazy and trying to avoid that :P).

with thanks

[tomplus]

I did some tests and it works on my site. I use minikube with k8s v1.21.10, kubernetes-asyncio 22.6.0. I'll check it on microk8s later.

[JacobHayes]

This issue may be related: dask/dask-kubernetes#419. I get:

HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"pods is forbidden: User \"system:anonymous\" cannot create resource \"pods\" in API group \"\" in the namespace \"default\"","reason":"Forbidden","details":{"kind":"pods"},"code":403}

when accessing k8s (version 1.21.9-gke.1002) with managed google-cloud-sdk auth (application default credentials):

apiVersion: v1
clusters:
- cluster:
    certificate-authority-data: ...
    server: https://...
  name: ...
contexts:
- context:
    cluster: ...
    user: ...
  name: ...
current-context: ...
kind: Config
preferences: {}
users:
- name: ...
  user:
    auth-provider:
      config:
        access-token: ...
        cmd-args: config config-helper --format=json
        cmd-path: /usr/local/Caskroom/google-cloud-sdk/latest/google-cloud-sdk/bin/gcloud
        expiry: ...
        expiry-key: '{.credential.token_expiry}'
        token-key: '{.credential.access_token}'
      name: gcp

Downgrading to kubernetes-asyncio==21.7.1 fixed things.

[tomplus]

Thanks for your input. It maybe related but there are 2 differences:

1. in-cluster config vs remote cluster (via kubeconfig witth gcloud command)
2. error code 401 vs. 403

I'm still investigating...

[tomplus]

It should be fixed now. Could you try with the latest version 22.6.1? Thanks.

[jonathon-love]

it works!