Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add a PodDisruptionBudget for CoreDNS #3585

Merged
merged 1 commit into from
Oct 26, 2023
Merged

Conversation

twz123
Copy link
Member

@twz123 twz123 commented Oct 13, 2023

Description

As CoreDNS is a rather critical component in Kubernetes clusters, k0s should ensure it remains available, even during node maintenance or other disruptions. A PodDisruptionBudget helps to maintain a minimum level of availability.

Some additional tweaks and their considerations

  • No pod anti-affinity for single replica deployments, so that CoreDNS can be rolled on a single node without downtime, as the single replica can remain operational until the new replica can take over.
  • No PodDisruptionBudget for single replica deployments, so that CoreDNS may be drained from the node running the single replica. This might leave CoreDNS eligible for eviction due to node pressure, but such clusters aren't HA in the first place and it seems more desirable to not block a drain.
  • Set maxUnavailable=1 only for deployments with two or three replicas. Use the Kubernetes defaults (maxUnavailable=25%, rounded down to get absolute values) for all other cases. For single replica deployments, maxUnavailable=1 meant that the deployment's available condition would be true even with zero ready replicas. For deployments with 4 to 7 replicas, this has been the same as the default, and for deployments with 8 or more replicas, this has been artificially constraining the rolling update speed.

Basic integration test

  • Use strings as identifier when detecting multiple CoreDNS pods on nodes. Makes for more readable log/error messages.
  • Ignore pods without pod anti-affinity, as they are remnants when scaling up single node clusters.

Type of change

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • Documentation update

How Has This Been Tested?

  • Manual test
  • Auto test added

Checklist:

  • My code follows the style guidelines of this project
  • My commit messages are signed-off
  • I have performed a self-review of my own code
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • My changes generate no new warnings
  • I have added tests that prove my fix is effective or that my feature works
  • New and existing unit tests pass locally with my changes
  • Any dependent changes have been merged and published in downstream modules
  • I have checked my code and corrected any misspellings

@twz123 twz123 requested a review from a team as a code owner October 13, 2023 16:00
jnummelin
jnummelin previously approved these changes Oct 13, 2023
@twz123 twz123 marked this pull request as draft October 18, 2023 11:16
@twz123
Copy link
Member Author

twz123 commented Oct 18, 2023

This more or less revealed CoreDNS failures in inttests: #3602

@twz123
Copy link
Member Author

twz123 commented Oct 19, 2023

Also revealed #3609.

@twz123 twz123 force-pushed the coredns-pdb branch 2 times, most recently from 4a50683 to 0b1b3e3 Compare October 24, 2023 15:21
@twz123 twz123 marked this pull request as ready for review October 24, 2023 16:07
As CoreDNS is a rather critical component in Kubernetes clusters, k0s
should ensure it remains available, even during node maintenance or
other disruptions. A PodDisruptionBudget helps to maintain a minimum
level of availability.

Some additional tweaks and their considerations:

* No pod anti-affinity for single replica deployments, so that CoreDNS
  can be rolled on a single node without downtime, as the single replica
  can remain operational until the new replica can take over.
* No PodDisruptionBudget for single replica deployments, so that CoreDNS
  may be drained from the node running the single replica. This might
  leave CoreDNS eligible for eviction due to node pressure, but such
  clusters aren't HA in the first place and it seems more desirable to
  not block a drain.
* Set maxUnavailable=1 only for deployments with two or three replicas.
  Use the Kubernetes defaults (maxUnavailable=25%, rounded down to get
  absolute values) for all other cases. For single replica deployments,
  maxUnavailable=1 meant that the deployment's available condition would
  be true even with zero ready replicas. For deployments with 4 to 7
  replicas, this has been the same as the default, and for deployments
  with 8 or more replicas, this has been artificially constraining the
  rolling update speed.

Basic integration test:

* Use strings as identifier when detecting multiple CoreDNS pods on
  nodes. Makes for more readable log/error messages.
* Ignore pods without pod anti-affinity, as they are remnants when
  scaling up single node clusters.

Signed-off-by: Tom Wieczorek <[email protected]>
@twz123 twz123 merged commit c0b94d8 into k0sproject:main Oct 26, 2023
71 checks passed
@twz123 twz123 deleted the coredns-pdb branch October 26, 2023 11:32
@twz123 twz123 mentioned this pull request Sep 6, 2024
16 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants