Controller loses API connection after token expiry on Azure Kubernetes Service (AKS) 1.30 due to kopf bug

[creste]

dask-operator fails to create Dask Jobs on Azure Kubernetes Service (AKS) 1.30:

See this kopf bug report for details.

Minimal Complete Verifiable Example:

1. Install dask-operator on AKS 1.30.
2. Wait an hour for the authentication token to expire.
3. Create a DaskJob resource.

dask-operator will not create the DaskJob because dask-operator's kubernetes authentication token has expired and kopf's watchers are no longer connected to kubeapi. A bug in kopf prevents kopf from refreshing the authentication token.

This only occurs on AKS 1.30+ because that is the first AKS version that sets --service-account-extend-token-expiration to false.

Environment:

• Dask operator version: 2024.9.0

[jacobtomlinson]

Thanks for flagging this here. I don't see any immediate solution we can implement in dask-kubernetes to work around this so I expect we will need to wait for a fix in nolar/kopf#980