Apps don't sync every are in stuck in refreshing

[haooliveira84]

Checklist:

• I've searched in the docs and FAQ for my answer: https://bit.ly/argocd-faq.
• I've included steps to reproduce the bug.
• I've pasted the output of argocd version.

Describe the bug
I've updated the ArgoCD from v2.9.5 to 2.11.0 with dynamicClusterDistribution: enabled and the ArgoCD Applications don't sync every are stuck in refreshing, I see the application controller does not have many logs.

To Reproduce
1 - Install Argocd with helm-chart with version: 6.9.3
2 - Edit the values.yaml and configure the controller to dynamicClusterDistribution: enabled

Expected behavior

The application controller balances the clusters with yours pods.
Sync stuck resolved

[haooliveira84]

tagging @ishitasequeira from #15036

[haooliveira84]

Anyone looking to this?

[jenna-foghorn]

same /similar: #18467

Temp band-aid is to kick the argocd-application-controller via kubectl rollout restart statefulset argocd-application-controller -n argocd

[oscrx]

I had this issue as well, I thought that it happened because the cluster ownership registration in the configmap was not being expired.
I didn't look into it further then because I ran it on prod and people started complaining :)

Maybe the application controllers can use a Kubernetes lease for every cluster instead of a configmap to keep track of the ownership between application controllers.

[oscrx]

Our configmap stays healthy after restarts, I'll keep it running for a while and see where it breaks. (3 instances currently)

❯ k get cm argocd-app-controller-shard-cm -o yaml | k neat
apiVersion: v1
data:
  shardControllerMapping: '[{"ShardNumber":0,"ControllerName":"argocd-application-controller-bb77ddc6-vlfhk","HeartbeatTime":"2024-09-18T07:06:37Z"},{"ShardNumber":1,"ControllerName":"argocd-application-controller-bb77ddc6-7fvrd","HeartbeatTime":"2024-09-18T07:06:36Z"},{"ShardNumber":2,"ControllerName":"argocd-application-controller-bb77ddc6-t8gpj","HeartbeatTime":"2024-09-18T07:06:35Z"}]'
kind: ConfigMap
metadata:
  name: argocd-app-controller-shard-cm
  namespace: argocd

[jmmclean]

Im having to kick the application controller every day at this point. I encounter cache: key is missing log events, then it seems ArgoCD goes into a deadlock.

Similar Potential issue #18503

The CPU usage is a clear indicator of the app controller being dead:

Also, the saw toothiness of the CPU usage correlates w/ this key is missing error. Also seems my workqueue depth peaks at 434:

ArgoCD version: 2.11.3

cc @crenshaw-dev

[jmmclean]

Following up on the above, after configuring shards with the application-controller and explicitly defining shard: n in the cluster secret, our caching issue went away as well as our workqueue deadlock (we did end up upgrade to 2.11.5 as well as there was a deadlock issue, but it did not resolve our specific deadlock issue).