Configurable leader election via chart values

[aruiz14]

Part of #1491

It makes leader election parameters configurable via Helm chart values, and adapts the default values by applying the following reasoning:

• The purpose of leader election on Fleet is to ensure only instance is running at a time.
  • fleet-controller deployment is configured to use only 1 replica, but users could still try to scale it up.
  • Rolling updates could cause to temporarily have 2 living instances of fleet.
  • This could be mitigated by using StatefulSet instead of Deployment, although this would still not prevent users from manually scaling up the StatefulSet too
• The definition of LeaseDuration says that any candidate should wait at lease this duration before attempting to become leader, so long duration could slow down the rollout of new images.
  • For this reason, I've decreased LeaseDuration from 45 to 30 seconds.
    • core Kubernetes clients use 15 seconds, but they also keep a shorter retry period (2s), but 30s is enough for Fleet's use case.
• RetryPeriod defines the wait period between actions, including renewing the leader lease. This mean that every $retryPeriod, it will acquire the lease for up to $leaseDuration. It has a default period of 2 seconds, which causes too much pressure on the Kubernetes API
  • For this reason, and taking into account the value used for LeaseDuration, I'm changing RetryPeriod from 2 to 10 seconds.
• RenewalDeadline is the period during which an active master will keep trying to renew the lock before giving up, which in our implementation means exiting the program. Failures renewing the lease could happen due to network instability in the node running the controller.
  • Considering the LeaseDuration and RetryPeriod, in order to allow at least 2 attempts to renew the lease before it expires, I'm configuring RenewalDeadline to 25 seconds.

Reference: https://pkg.go.dev/k8s.io/client-go/tools/leaderelection#LeaderElectionConfig

[moio]

Reasoning sounds solid to me. The only part I believe not being accurate is:

Closes #1491

That issue is not only about making these parameters configurable, but crucially to make them configurable in Rancher for all downstream clusters at once.

This PR is a step in that direction, but crucially we will still need enablement in Rancher code to make use of these new Helm values, namely #1717.

Please do close #1491 when this PR is merged and backported and a fix for #1717 is also in place.

Thanks!