ArgoCD Detects duplicate resource

[kgopi1]

Describe the bug
When we deploy Connaisseur helm package via argocd , it detect duplicate resource is added.

Resource admissionregistration.k8s.io/MutatingWebhookConfiguration//connaisseur-webhook appeared 2 times among application resources

Expected behavior
No warnings / duplicate resources should be.

Optional: To reproduce
Deploy the Helm package via Gitops.

Optional: Versions (please complete the following information as relevant):

• OS:
• Kubernetes Cluster:
• Notary Server:
• Container registry:
• Connaisseur:
• Other:

Optional: Additional context

[phbelitz]

Hello 👋 That is probably the case because we have two webhooks defined, that are meant to overwrite eachother. But I guess this only works for helm installation, since we use helm hooks.

I never tried deploying connaisseur via argoCD and I don't have an argoCD setup right now, BUT according to its documentation there are also hooks similar to helm. I created an extra branch experimental/argocd where the hooks were added to the mutatingWebHookConfigurations. So maybe you can tryout this branch and see if you argoCD deployment works fine with these changes.

If they do, I'd be happy to know, so I can deploy the changes into the master branch in one of our upcomming releases.

CHEERS.

[kgopi1]

I will clone repo and build helm package , keep you posted. Thank you quick support.

[kgopi1]

Hi @phbelitz , I tried with new helm files , still the same error. Just to let know that ArgoCD will not run helm install command, it runs helm template and get the *.yaml files and then do kubectl apply . Does this helps ?

[phbelitz]

Huh, ok. I introduced a new field inside the helm/values.yaml called argoCDDeployment (set to true for now).That should make sure only a single webhook gets deployed.

Please try again.

[kgopi1]

Thank you @phbelitz , it works !!!

[kgopi1]

Hi @phbelitz , When i stop and start the Cluster , im getting below error in the Connaisseur namespace ..
Internal error occurred: failed calling webhook "connaisseur-svc.connaisseur.svc":
failed to call webhook: Post "https://connaisseur-svc.connaisseur.svc:443/mutate?timeout=30s":
write unix @->/tunnel-uds/proxysocket: write: broken pipe

Is this due to changes made in the Helm package ??

[phbelitz]

Short answer: That's a standard issue of strict admission controllers (which connaisseur is). They are not suited for restarting of clusters and should be uninstalled/disabled before restarting.

Long answer: The issue here is a bit complex. Most admission controllers, like connaisseur, run inside the cluster they are working for. They are activated via a so called MutatingWebhookConfiguration, which essentially says: somewhere in this cluster a service is running, which I will contact to do validation/mutation of requests comming to the kubernetes API. The problem occurs should the service not be available, while the configuration is still active. Then no validation/mutation can be done and ANY request gets simply denied. This is essentially the case when you reboot a cluster. Before the service (connaisseur) can boot up and do the thing it's susposed to do, the MutatingWebhookConfiguration is already active. With the configuration active, the booting up of the service itself gets blocked, since the service isn't available yet. Quiet ironic.

Now I don't know what your usecase is and why you need to restart your cluster, but be advised to clean up connaisseur before rebooting.

[kgopi1]

Thanks @phbelitz for the response , we are testing Connaisseur in our non-prod environments , these clusters will stopped after business hours and started next day . Do you see any other options to make this work ?

[phbelitz]

That's a fairly specific usecase. Multiple options:

1. Write a CronJob that automatically uninstalls and installs Connaisseur at given times. Do this before stopping the cluster and shortly after restarting.
2. Use Connaisseurs namespace validation feature. With that, certain namespaces can be excluded from validation. Do this for the kube-system namespace and wherever you installed Connaisseur. This allows your cluster at least to start up. All other resources are still being denied, as long as Connaisseur isn't ready yet.

Unfortunately these are all just workarounds. The core of the problem won't be solved by this and I don't think this will change any time soon 🤷

[kgopi1]

Hi @phbelitz , Even the namespace validation exclusion didn't work after the Cluster restart. I deployed the 1.6.1 (public release helm package) and it works , but with same warning "Resource admissionregistration.k8s.io/MutatingWebhookConfiguration//connaisseur-webhook appeared 2 times among application resources."

Apart from this application functionality works as expected. So fix experimental/argocd creates different issue .

Can you advise on this ?