Add option to delete old Rancher webhooks

[richardcase]

What would you like to be added (User Story)?

As a operator
When i have an existing Rancher Manager Cluster
And i install Rancher Turtles
And disable the "embedded CAPI" feature in Rancher
I expect that i can apply CAPI cluster definitions to the cluster

Detailed Description

Currently doing the above results in validating/mutating webhooks being left behind after the feature is disabled. If you try and apply a CAPI manifest you will get an error:

time="2023-10-03T14:18:36Z" level=error msg="error syncing 'cluster-fleet-local-local-1a3d67d0a899/clusters-clusters': handler bundle-deploy: failed to create  │
│ resource: Internal error occurred: failed calling webhook \"default.cluster.cluster.x-k8s.io\": failed to call webhook: Post \"https://capi-webhook-service.cat/ │
│ tle-provisioning-capi-system.svc:443/mutate-cluster-x-k8s-io-v1beta1-cluster?timeout=10s\": service \"capi-webhook-service\" not found, requeuing"

Disabling the feature doesn't remove the webhooks. To fix this manually you can run the following commands after disabling the feature:

kubectl delete mutatingwebhookconfigurations.admissionregistration.k8s.io mutating-webhook-configuration
kubectl delete validatingwebhookconfigurations.admissionregistration.k8s.io validating-webhook-configuration

We need to:

• Update our docs so people can run those commands
• Look at incorporating feature flag enabled functionality to automatically delete the webhooks (i.e. engineer a fix). Like we are doing in feat: add v2prov kubeconfig label patching #178 for labels on the kubeconfig secrets
• Create a fix upstream in Rancher

Anything else you would like to add?

No response

Label(s) to be applied

/kind feature

[richardcase]

Another option is that we create a small utility as there may be other "fixes" we need to apply to Rancher.

[Danil-Grigorev]

As a pure cleanup option for rancher related leftovers, a pre-install hook with job cleaning these webhook configurations could be enough.

[salasberryfin]

Created a PR with a pre-install hook (thanks @Danil-Grigorev and @richardcase) proposal that disables the embedded feature and deletes webhooks. With this change helm install should manage all requirements in an existing Rancher installation. #192