fix: use pre-install helm hook to prepare rancher for turtles

[salasberryfin]

What this PR does / why we need it:

Currently, installing Rancher Turtles in an existing Rancher Manager after disabling embedded CAPI, results in validating/mutating webhooks being left behind after the feature is disabled. If you try and apply a CAPI manifest you will get an error:

time="2023-10-03T14:18:36Z" level=error msg="error syncing 'cluster-fleet-local-local-1a3d67d0a899/clusters-clusters': handler bundle-deploy: failed to create  │
│ resource: Internal error occurred: failed calling webhook \"default.cluster.cluster.x-k8s.io\": failed to call webhook: Post \"https://capi-webhook-service.cat/ │
│ tle-provisioning-capi-system.svc:443/mutate-cluster-x-k8s-io-v1beta1-cluster?timeout=10s\": service \"capi-webhook-service\" not found, requeuing"

To avoid having to manually delete these webhooks and disable the feature flag, this PR adds a pre-install Helm hook that:

1. Disables embedded CAPI.
2. Deletes mutating/validating webhooks (creating a Service Account, Cluster Role and Cluster Role Binding).

This is not the fanciest of solutions but it allows to install Rancher Turtles just using helm install in an existing Rancher Manager. We can later change Rancher and remove this hook if we find a more robust option.

Which issue(s) this PR fixes:
Fixes #185

Special notes for your reviewer:

When this gets merged, we'll need to update the docs accordingly.

Checklist:

• squashed commits into logical changes
• includes documentation
• adds unit tests
• adds or updates e2e tests

[salasberryfin]

We may need to install Rancher before Rancher Turtles in the E2E environment for the Feature kind to be available. Looks like this is triggering CI failures.

[Danil-Grigorev]

E2e tests are passing with this change, so failure here could be ignored.

[salasberryfin]

Thanks @Danil-Grigorev. Should we consider this PR superseded by yours #190 and close it?

[Danil-Grigorev]

I don’t know. Can go both ways, you can cherry pick the commit #190 and the other one could be closed. Or this PR could be merged without e2e tests passing, and followed by #190 separately.

[salasberryfin]

Let's merge this first and then #190 and keep them as independent PRs.

[richardcase]

In my head i was thinking of something actually in the Rancher Turtles binary but this is an interesting approach.

[salasberryfin]

I added flags for disabling embedded capi and deleting rancher webhooks:

rancherTurtles:
  ...
  features:
    disable-embedded-capi:
      enabled: true
    cleanup-rancher-webhook:
      enabled: true
      image: rancher/kubectl

Also, changed the deleting webhook Job to use rancher/kubectl image. This has to now be separated into two different jobs: one for each webhook deletion (mutating/validating).

[richardcase]

Looks good, thanks for adding the flags in the values.

Although there is a chart linrt failure.

[Danil-Grigorev]

Some minor suggestions, but LGTM as is.

[Danil-Grigorev]

Suggested change

	run: helm install rancher-turtles out/charts/rancher-turtles/ -n rancher-turtles-system --create-namespace --wait --set cluster-api-operator.cluster-api.enabled=false --set cluster-api-operator.enabled=false
	run: helm install rancher-turtles out/charts/rancher-turtles/ -n rancher-turtles-system --create-namespace --wait --set cluster-api-operator.cluster-api.enabled=false --set cluster-api-operator.enabled=false --set features. disable-embedded-capi.enabled=false --set features.cleanup-rancher-webhook.enabled=false

As this workflow represents the default flow of installing rancher-turtles in a fresh cluster, we might not need to bring rancher with it. Just making sure the chart is installable.

[salasberryfin]

I understand your point but I would say, considering these flags are enable by default, we should test them as well to validate the chart.

[Danil-Grigorev]

Suggested change

	- name: Add cert-manager chart repo
	run: helm repo add jetstack https://charts.jetstack.io
	- name: Add rancher chart repo
	run: helm repo add rancher-stable https://releases.rancher.com/server-charts/stable
	- name: Install cert-manager
	run: helm install cert-manager jetstack/cert-manager --namespace cert-manager --create-namespace --version ${{ env.CERT_MANAGER_VERSION }} --set installCRDs=true --wait
	- name: Install Rancher
	run: helm install rancher rancher-stable/rancher --namespace cattle-system --create-namespace --set bootstrapPassword=rancheradmin --set replicas=1 --set hostname="e2e.dev.rancher" --set 'extraEnv[0].name=CATTLE_FEATURES' --set global.cattle.psp.enabled=false --version ${{ env.RANCHER_VERSION }} --wait

In addition to https://github.com/rancher-sandbox/rancher-turtles/pull/192/files#r1348559835