From b4be1ef1df95ff4e5eb8fc5c982caedcad2ad149 Mon Sep 17 00:00:00 2001 From: Clayton Gonsalves <101868649+clayton-gonsalves@users.noreply.github.com> Date: Fri, 15 Sep 2023 21:06:08 +0200 Subject: [PATCH] design: EndpointSlice support (#5703) Updates #2696. Signed-off-by: Clayton Gonsalves --- design/endpoint-slice-support.md | 119 +++++++++++++++++++++++++++++++ 1 file changed, 119 insertions(+) create mode 100644 design/endpoint-slice-support.md diff --git a/design/endpoint-slice-support.md b/design/endpoint-slice-support.md new file mode 100644 index 00000000000..2a15491b998 --- /dev/null +++ b/design/endpoint-slice-support.md @@ -0,0 +1,119 @@ +# Contour k8s EndpointSlice support + +Contour currently supports Kubernetes endpoints. +This document proposes a process to enable support for k8s EndpointSlices and, eventually, migrate to exclusively using it. + +## Goals + +- Contour will support k8s EndpointSlices. +- The EndpointSlices will be gated by a feature flag in the Contour config. +- Kubernetes endpoints will still be the default mode of operation. + +## Non-goals + +- Removal of k8s endpoints. +- Support advanced features such as topology aware routing/hints offered by k8s endpoints. + +## Background + +See [#2696](https://github.com/projectcontour/contour/issues/2696) + +Before the introduction of the EndpointSlices, the only method for a service to expose the IP addresses and ports of the pads it fronted was through Kubernetes endpoints. + +However, Kubernetes endpoints have a few challenges with scaling, such as +- transmission of the entire endpoint object even if one endpoint changes, leading to network contention and heavy recalculation on the consumers of this data. +- As of k8s v1.22, a hard limit of 1000 addresses per endpoint object. +- No topology aware information for more intelligent routing. + +To solve the above problems, EndpointSlices was introduced in 1.16 and, as of k8s v1.21 in [stable state](https://kubernetes.io/docs/concepts/services-networking/endpoint-slices/). + + +## High-Level Design + +A feature flag will gate the usage of EndpointSlices, and the implementation will use the same pattern as used in the `endpointtranslator.go` with a few differences listed in the detailed design. + +## Detailed Design + +### Contour Configuration changes +EndpointSlices will be configured by a boolean feature flag, which defaults to false and allows users to opt to use EndpointSlices. + +```yaml +... +server: + xds-server-type: contour +use-endpoint-slice: true +... +``` + +```go +type Parameters struct { + // useEndpointSlice configures contour to fetch endpoint data + // from k8s EndpointSlices. defaults to false and reading endpoint + // data from the k8s endpoints. + useEndpointSlice bool +} +``` +### Implementation + +#### Context +The implementation of k8s EndpointSlices is fundamentally different from how EndpointSlices work. + +Below is an example of a Kubernetes service represented in as endpoint and EndpointSlice objects + +endpoints +``` +NAME ENDPOINTS AGE +ingress-conformance-echo 10.244.0.10:3000,10.244.0.153:3000,10.244.0.154:3000 + 7 more... 3d +``` + +EndpointSlice +``` +NAME ADDRESSTYPE PORTS ENDPOINTS AGE +ingress-conformance-echo-8bcrv IPv4 3000 10.244.0.156,10.244.0.157 18h +ingress-conformance-echo-dr6d2 IPv4 3000 10.244.0.158,10.244.0.159 18h +ingress-conformance-echo-dt27v IPv4 3000 10.244.0.153,10.244.0.154 2d14h +ingress-conformance-echo-l2j48 IPv4 3000 10.244.0.10,10.244.0.155 3d +ingress-conformance-echo-z4bcw IPv4 3000 10.244.0.160,10.244.0.161 18h +``` + +#### EndpointSlice cache + +Because of the above mentioned differences, there is a need to adapt the cache logic from `endpointtranslator.go` such as; + +Instead of using `endpoints map[types.NamespacedName]*v1.Endpoints`, use the below format with a nested map `k,v` where `k` is the EndpointSlice name and the `v` is the EndpointSlice itself. + +```go +type EndpointSliceCache struct { + endpointSlice: map[types.NamespacedName]map[string]*discoveryv1.EndpointSlice{}, +} +``` + +The above structure lets us keep track of each EndpointSlice and change/update/delete an endpoint without shuffling all the existing endpoints. + +#### EndpointSlice conditions +The EndpointSlice has information about the condition of the endpoint, namely, `ready`, `serving` and `terminating`. The EndpointSlice implementation will only support the `ready` condition. +This means that only pods with `ready` status will be considered since `ready` on the EndpointSlice maps to `ready` status on the pods. This also keep contour inline with the existing implementation. + +refer [ready status](https://kubernetes.io/docs/concepts/services-networking/endpoint-slices/#ready) + +#### FQDN Address types +FQDN address types on EndpointSlices will not be supported. Contour will continue to support `externalName` service types. + +#### Disabled Mirroring +If a service has [endpoint mirroring](https://kubernetes.io/docs/concepts/services-networking/endpoint-slices/#endpointslice-mirroring) disabled, it will not be processed by Contour. + +## Tests + +To ensure the feature is well tested, this feature will have +- e2e tests +- unit tests +- daily build run with EndpointSlices enabled. + +## Migration Plan + +Once we have enough consensus on the correctness and performance, we can make the move to EndpointSlices permanent and deprecate use of endpoints. This can be a combination of daily builds, burn time and feedback from the community. + + +## Alternatives Considered + +N/A