Convert Fleet-Controller to StatefulSet

[manno]

Our controllers are running in a k8s deployment. We see that work is done outside the leader election if multiple replicas exist or during upgrades. E.g. fleet-controller runs "cleanup" multiple times, controllers start up and create CRDs in parallel, fleet-agent runs multiple registrations.

Leader election is an expensive operation, our default timing is 30s renew, 45s for the lease.
We also see problems with the lock files, client-go > 0.25 (?) will deprecate the current lock resource. The current lease mechanism causes problems with backup and restore. The lead election code lives in wrangler and keeps us from updating client-go.

From the controller-runtime docs:

Note that the Manager is run as a StatefulSet and not a Deployment. This is to ensure that only 1 instance of the Manager is run at a time (a Deployment may sometimes run multiple instances even with replicas set to 1).

See:

• [SURE-3805] Make leaderelection poll period configurable (was: Fleet lock api calls account for 33% of all api calls) #1491

[manno]

We discussed and will try to improve the existing leader election code first.

• move all code that interacts with the cluster into leader election, i.e. https://github.com/rancher/fleet/blob/master/internal/cmd/controller/controllers/content/controller.go#L45
• remove CRD creation, which is also outside leader election (see Replace Wrangler's CRD generator #1815)
• split fleet-controller, observe performance
• upgrade to wrangler/v2 and the lease resource lock (see Bump to wrangler/v2 #1777)
• look into making leaderelection timeouts configurable

[raulcabello]

move all code that interacts with the cluster into leader election, i.e. https://github.com/rancher/fleet/blob/master/internal/cmd/controller/controllers/content/controller.go#L45

This is going to be moved into leader election when we split the cleanup into a different container https://github.com/raulcabello/fleet/blob/split-cleanup/internal/cmd/controller/cleanup/start.go#L39

[manno]

We're going to fix leader election instead.

• bump to wrangler 2.1.2 (configurable leader election)
• move code into leader election, done? removed addcrd, cleanup