[release-1.29] Fix issue with loadbalancer failover to default server

[brandond]

Proposed Changes

The loadbalancer should only fail over to the default server if all other server have failed, and it should force fail-back to a preferred server as soon as one passes health checks.

The loadbalancer tests have been improved to ensure that this occurs.

Types of Changes

bugfix

Verification

See linked issue

Testing

yes

Linked Issues

• [Release-1.29] - Secondary etcd-only nodes do not reconnect to apiserver after outage if joined against an etcd-only node #11322

User-Facing Change

Further Comments

[codecov]

Codecov Report

Attention: Patch coverage is 73.91304% with 6 lines in your changes missing coverage. Please review.

Project coverage is 13.74%. Comparing base (39385c1) to head (f6f3060).
Report is 5 commits behind head on release-1.29.

Files with missing lines	Patch %	Lines
pkg/agent/loadbalancer/servers.go	78.94%	4 Missing ⚠️
pkg/util/apierrors.go	0.00%	2 Missing ⚠️

❗ There is a different number of reports uploaded between BASE (39385c1) and HEAD (f6f3060). Click for more details.

HEAD has 18 uploads less than BASE

Flag	BASE (39385c1)	HEAD (f6f3060)
inttests	11	0
e2etests	7	0

Additional details and impacted files

@@                Coverage Diff                @@
##           release-1.29   #11326       +/-   ##
=================================================
- Coverage         46.65%   13.74%   -32.92%     
=================================================
  Files               179      176        -3     
  Lines             18707    18646       -61     
=================================================
- Hits               8727     2562     -6165     
- Misses             8624    15787     +7163     
+ Partials           1356      297     -1059     

Flag	Coverage Δ
e2etests	?
inttests	?
unittests	13.74% <73.91%> (+0.17%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.