leaderelection: configure all timeouts via environment variables

[moio]

We have observed in fleet-agent user cases (SURE-6125/fleet#1465) that the leader election can be lost due to timeouts - not because pods are not working, but because the control plane is overwhelmed by load.

While of course having an overloaded control plane is a problem in and of itself, losing the election means that Wrangler-based pods will restart, which in turn means a new full listing of all interesting resources, creating additional load and exacerbating symptoms.

Ideally, the situation should improve when API Priority and Fairness goes out of beta and becomes commonplace - as it should ensure leader election API calls get served with higher priority, thus returning correctly within reasonable time frames even if the control plane is overloaded.

Meantime, pretty much the only other pragmatic option left is to give election more time before failure.

This PR allows to set all critical parameters via environment variables - with default values staying the same as before.

Note: this will also address (SURE-3805/#1491)

[rmweir]

One nit, otherwise LGTM.

[moio]

@rmweir fixed. Do changes in wrangler require two approvers? If so, who could be the right person?

[rmweir]

LGTM

[rmweir]

@moio yes, I assigned @KevinJoiner. Once approved it will be good to merge.

[KevinJoiner]

Also, I don't know what 🎯 policy is for Unit tests.

[moio]

Well, the Bullseye Team does not have a policy at this point.

There is a guiding principle, which is to adapt to each Team's patterns/habits/processes/policies to the extent reasonable and possible - the goal being to help as good as possible.

Question becomes: would you normally add (or ask to add) tests in a case like this? If so, how would you propose to set them up?

Currently code is basically a "main" method and the effect of setting environment variables is only time-detectable - not really amenable to unit tests. One possibility would be to refactor out the main added block:

https://github.com/rancher/wrangler/pull/305/files#diff-2b8e7c2af1963d0a67b49eb0af1b1107583a195b328f40db0fc28970d7ba4dc9R54-R77

to a separate function returning the three values or err, then test it the various error conditions. To my personal sensibility, that would not be worth the cost/benefit tradeoff, but you as the long-term maintainers are the ones to set the bar for contributions, so I will respect whatever decision you take here.

[moio]

@KevinJoiner do you have a comment on the last update?

[olblak]

@KevinJoiner , @rmweir What's the current state here?
By looking at the discussion history, it seems that this issue is stalled for a month now

[moio]

@KevinJoiner: rebased, refactored per your suggestion, added unit tests. Please review again.

[moio]

@rmweir do we already know when a new version of wrangler could be tagged? I am asking because of a fleet issue that would benefit from this change.