[docs] CPLB virtualServers and keepalived

Signed-off-by: Juan-Luis de Sousa-Valadas Castaño <[email protected]>

cmd/controller/controller.go

@@ -241,7 +241,7 @@ func (c *command) start(ctx context.Context) error {
 			Config:          cplb.Keepalived,
 			DetailedLogging: c.Debug,
 			KubeConfigPath:  c.K0sVars.AdminKubeConfigPath,
-			APISpec:         nodeConfig.Spec.API,
+			APIPort:         nodeConfig.Spec.API.Port,
 		})
 	}
 

docs/configuration.md

@@ -304,14 +304,24 @@ node-local load balancing.
 
 Configuration options related to k0s's [control plane load balancing] feature
 
-| Element         | Description                                                                                                 |
-| --------------- | ----------------------------------------------------------------------------------------------------------- |
-| `enabled`       | Indicates if control plane load balancing should be enabled. Default: `false`.                              |
-| `vrrpInstances` | Configuration options related to the VRRP. This is an array which allows to configure multiple virtual IPs. |
+| Element      | Description                                                                                 |
+| ------------ | ------------------------------------------------------------------------------------------- |
+| `enabled`    | Indicates if control plane load balancing should be enabled. Default: `false`.              |
+| `type`       | Indicates the backend for CPLB. If this isn't defined to `Keepalived`, CPLB will not start. |
+| `keepalived` | Contains the keepalived configuration.                                                      |
 
 [control plane load balancing]: cplb.md
 
-##### `spec.network.controlPlaneLoadBalancing.VRRPInstances`
+##### `spec.network.controlPlaneLoadBalancing.Keepalived`
+
+Configuration options related to keepalived in [control plane load balancing]
+
+| Element          | Description                                                                                                 |
+| ---------------- | ----------------------------------------------------------------------------------------------------------- |
+| `vrrpInstances`  | Configuration options related to the VRRP. This is an array which allows to configure multiple virtual IPs. |
+| `virtualServers` | Configuration options related LoadBalancing. This is an array which allows to configure multiple LBs.       |
+
+##### `spec.network.controlPlaneLoadBalancing.keepalived.vrrpInstances`
 
 Configuration options required for using VRRP to configure VIPs in control plane load balancing.
 
@@ -324,6 +334,18 @@ Configuration options required for using VRRP to configure VIPs in control plane
 | `advertInterval`  | Advertisement interval in seconds. Default: `1`.                                                                  |
 | `authPass`        | The password used for accessing vrrpd. This field is mandatory and must be under 8 characters long                |
 
+##### `spec.network.controlPlaneLoadBalancing.keepalived.virtualServers`
+
+Configuration options required for using VRRP to configure VIPs in control plane load balancing.
+
+| Element           | Description                                                                                                                      |
+| ----------------- | -------------------------------------------------------------------------------------------------------------------------------- |
+| `ipAddress`          | The load balancer's listen address.                                                                                           |
+| `delayLoop`          | Delay timer for health check polling in seconds. Default: `0`.                                                                |
+| `lbAlgo`             | Algorithm used by keepalived. Supported algorithms: `rr`, `wrr`, `lc`, `wlc`, `lblc`, `dh`, `sh`, `sed`, `nq`. Default: `rr`. |
+| `lbKind`             | Kind of ipvs load balancer. Supported values: `NAT`, `DR`, `TUN`  Default: `DR`.                                              |
+| `persistenceTimeout` | Timeout value for persistent connections in seconds. Default: `360` (6 minutes).                                              |
+
 ### `spec.controllerManager`
 
 | Element     | Description                                                                                                             |

docs/cplb.md

@@ -3,17 +3,23 @@
 For clusters that don't have an [externally managed load balancer](high-availability.md#load-balancer) for the k0s
 control plane, there is another option to get a highly available control plane called control plane load balancing (CPLB).
 
-CPLB allows automatic assigned of predefined IP addresses using VRRP across masters.
+CPLB has two features that are independent, but normally will be used together: VRRP Instances, which allows
+automatic assignation of predefined IP addresses using VRRP across control plane nodes. VirtualServers allows to
+do Load Balancing to the other control plane nodes.
+
+This feature is intended to be used for external traffic. This feature is fully compatible with
+[node-local load balancing (NLLB)](nllb.md) which means CPLB can be used for external traffic and NLLB for
+internal traffic at the same time.
 
 ## Technical functionality
 
-The k0s control plane load balancer provides k0s with virtual IPs on each
-controller node. This allows the control plane to be highly available using
-VRRP (Virtual Router Redundancy Protocol) as long as the network
-infrastructure allows multicast and GARP.
+The k0s control plane load balancer provides k0s with virtual IPs and TCP
+load Balancing on each controller node. This allows the control plane to
+be highly available using VRRP (Virtual Router Redundancy Protocol) and
+IPVS long as the network infrastructure allows multicast and GARP.
 
 [Keepalived](https://www.keepalived.org/) is the only load balancer that is
-supported so far and currently there are no plans to support other alternatives.
+supported so far. Currently there are no plans to support other alternatives.
 
 ## VRRP Instances
 
@@ -46,19 +52,24 @@ following:
   These do not provide any sort of security against ill-intentioned attacks, they are
   safety features to prevent accidental conflicts between VRRP instances in the same
   network segment.
+* If `VirtualServers` are used, the cluster configuration mustn't specify a non-empty
+  [`spec.api.externalAddress`][specapi]. If only `VRRPInstances` are specified, a
+  non-empty [`spec.api.externalAddress`][specapi] may be specified.
 
 Add the following to the cluster configuration (`k0s.yaml`):
 
 ```yaml
 spec:
-  api:
-    externalAddress: <External address> # This isn't a requirement, but it's a common use case.
   network:
     controlPlaneLoadBalancing:
       enabled: true
-      vrrpInstances:
-      - virtualIPs: ["<External address IP>/<external address IP netmask"]
-        authPass: <password>
+      type: Keepalived
+      keepalived:
+        vrrpInstances:
+        - virtualIPs: ["<External address IP>/<external address IP netmask"]
+          authPass: <password>
+        virtualServers:
+        - ipAddress: "ipAddress"
 ```
 
 Or alternatively, if using [`k0sctl`](k0sctl-install.md), add the following to
@@ -69,24 +80,30 @@ spec:
   k0s:
     config:
       spec:
-        api:
-          externalAddress: <External address> # This isn't a requirement, but it's a common use case.
         network:
           controlPlaneLoadBalancing:
             enabled: true
-            vrrpInstances:
-            - virtualIPs: ["<External address IP>/<external address IP netmask>"]
-              authPass: <password>
+            type: Keepalived
+            keepalived:
+              vrrpInstances:
+              - virtualIPs: ["<External address IP>/<external address IP netmask>"]
+                authPass: <password>
+              virtualServers:
+              - ipAddress: "<External ip address>"
 ```
 
 Because this is a feature intended to configure the apiserver, CPLB noes not
 support dynamic configuration and in order to make changes you need to restart
 the k0s controllers to make changes.
 
+[specapi]: configuration.md#specapi
+
 ## Full example using `k0sctl`
 
 The following example shows a full `k0sctl` configuration file featuring three
-controllers and three workers with control plane load balancing enabled:
+controllers and three workers with control plane load balancing enabled.
+Additionally it defines [spec.api.sans](configuration.md#specapi) so that the
+kube-apiserver certificate is valid for the virtual IP:
 
 ```yaml
 apiVersion: k0sctl.k0sproject.io/v1beta1
@@ -142,13 +159,18 @@ spec:
     config:
       spec:
         api:
-          externalAddress: 192.168.122.200
+          sans:
+          - 192.168.122.200
         network:
           controlPlaneLoadBalancing:
             enabled: true
-            vrrpInstances:
-            - virtualIPs: ["192.168.122.200/24"]
-              authPass: Example
+            type: Keepalived:
+            keepalived:
+              vrrpInstances:
+              - virtualIPs: ["192.168.122.200/24"]
+                authPass: Example
+              virtualServers:
+              - ipAddress: "<External ip address>"
 ```
 
 Save the above configuration into a file called `k0sctl.yaml` and apply it in
@@ -319,6 +341,15 @@ controller-1
 controller-2
 2: eth0    inet 192.168.122.87/24 brd 192.168.122.255 scope global dynamic noprefixroute eth0\       valid_lft 2182sec preferred_lft 2182sec
 3: dummyvip0    inet 192.168.122.200/32 scope global dummyvip0\       valid_lft forever preferred_lft forever
+
+$ for i in controller-{0..2} ; do echo $i ; ipvsadm --save -n; done
+IP Virtual Server version 1.2.1 (size=4096)
+Prot LocalAddress:Port Scheduler Flags
+  -> RemoteAddress:Port           Forward Weight ActiveConn InActConn
+TCP  192.168.122.200:6443 rr persistent 360
+  -> 192.168.122.185:6443              Route   1      0          0
+  -> 192.168.122.87:6443               Route   1      0          0
+  -> 192.168.122.122:6443              Route   1      0          0
 ````
 
 And the cluster will be working normally:

pkg/apis/k0s/v1beta1/clusterconfig_types.go

@@ -335,6 +335,12 @@ func (s *ClusterSpec) Validate() (errs []error) {
 		errs = append(errs, err)
 	}
 
+	if s.Network != nil && s.Network.ControlPlaneLoadBalancing != nil {
+		for _, err := range s.Network.ControlPlaneLoadBalancing.Validate(s.API.ExternalAddress) {
+			errs = append(errs, fmt.Errorf("controlPlaneLoadBalancing: %w", err))
+		}
+	}
+
 	return
 }
 

pkg/apis/k0s/v1beta1/cplb.go

@@ -104,9 +104,10 @@ type VRRPInstance struct {
 
 type VirtualIPs []string
 
-// ValidateVRRPInstances validates existing configuration and sets the default
+// validateVRRPInstances validates existing configuration and sets the default
 // values of undefined fields.
-func (k *KeepalivedSpec) ValidateVRRPInstances(getDefaultNICFn func() (string, error)) error {
+func (k *KeepalivedSpec) validateVRRPInstances(getDefaultNICFn func() (string, error)) []error {
+	errs := []error{}
 	if getDefaultNICFn == nil {
 		getDefaultNICFn = getDefaultNIC
 	}
@@ -118,7 +119,7 @@ func (k *KeepalivedSpec) ValidateVRRPInstances(getDefaultNICFn func() (string, e
 		if k.VRRPInstances[i].Interface == "" {
 			nic, err := getDefaultNICFn()
 			if err != nil {
-				return fmt.Errorf("failed to get default NIC: %w", err)
+				errs = append(errs, fmt.Errorf("failed to get default NIC: %w", err))
 			}
 			k.VRRPInstances[i].Interface = nic
 		}
@@ -127,7 +128,7 @@ func (k *KeepalivedSpec) ValidateVRRPInstances(getDefaultNICFn func() (string, e
 			vrid := int32(defaultVirtualRouterID + i)
 			k.VRRPInstances[i].VirtualRouterID = &vrid
 		} else if *k.VRRPInstances[i].VirtualRouterID < 0 || *k.VRRPInstances[i].VirtualRouterID > 255 {
-			return errors.New("VirtualRouterID must be in the range of 1-255")
+			errs = append(errs, errors.New("VirtualRouterID must be in the range of 1-255"))
 		}
 
 		if k.VRRPInstances[i].AdvertInterval == nil {
@@ -136,22 +137,22 @@ func (k *KeepalivedSpec) ValidateVRRPInstances(getDefaultNICFn func() (string, e
 		}
 
 		if k.VRRPInstances[i].AuthPass == "" {
-			return errors.New("AuthPass must be defined")
+			errs = append(errs, errors.New("AuthPass must be defined"))
 		}
 		if len(k.VRRPInstances[i].AuthPass) > 8 {
-			return errors.New("AuthPass must be 8 characters or less")
+			errs = append(errs, errors.New("AuthPass must be 8 characters or less"))
 		}
 
 		if len(k.VRRPInstances[i].VirtualIPs) == 0 {
-			return errors.New("VirtualIPs must be defined")
+			errs = append(errs, errors.New("VirtualIPs must be defined"))
 		}
 		for _, vip := range k.VRRPInstances[i].VirtualIPs {
 			if _, _, err := net.ParseCIDR(vip); err != nil {
-				return fmt.Errorf("VirtualIPs must be a CIDR. Got: %s", vip)
+				errs = append(errs, fmt.Errorf("VirtualIPs must be a CIDR. Got: %s", vip))
 			}
 		}
 	}
-	return nil
+	return errs
 }
 
 // VirtualServers is a list of VirtualServer
@@ -218,13 +219,16 @@ type RealServer struct {
 	Weight int `json:"weight,omitempty"`
 }
 
-func (k *KeepalivedSpec) ValidateVirtualServers() error {
+// validateVRRPInstances validates existing configuration and sets the default
+// values of undefined fields.
+func (k *KeepalivedSpec) validateVirtualServers() []error {
+	errs := []error{}
 	for i := range k.VirtualServers {
 		if k.VirtualServers[i].IPAddress == "" {
-			return errors.New("IPAddress must be defined")
+			errs = append(errs, errors.New("IPAddress must be defined"))
 		}
 		if net.ParseIP(k.VirtualServers[i].IPAddress) == nil {
-			return fmt.Errorf("invalid IP address: %s", k.VirtualServers[i].IPAddress)
+			errs = append(errs, fmt.Errorf("invalid IP address: %s", k.VirtualServers[i].IPAddress))
 		}
 
 		if k.VirtualServers[i].LBAlgo == "" {
@@ -234,7 +238,7 @@ func (k *KeepalivedSpec) ValidateVirtualServers() error {
 			case RRAlgo, WRRAlgo, LCAlgo, WLCAlgo, LBLCAlgo, DHAlgo, SHAlgo, SEDAlgo, NQAlgo:
 				// valid LBAlgo
 			default:
-				return fmt.Errorf("invalid LBAlgo: %s ", k.VirtualServers[i].LBAlgo)
+				errs = append(errs, fmt.Errorf("invalid LBAlgo: %s ", k.VirtualServers[i].LBAlgo))
 			}
 		}
 
@@ -245,19 +249,44 @@ func (k *KeepalivedSpec) ValidateVirtualServers() error {
 			case NATLBKind, DRLBKind, TUNLBKind:
 				// valid LBKind
 			default:
-				return fmt.Errorf("invalid LBKind: %s ", k.VirtualServers[i].LBKind)
+				errs = append(errs, fmt.Errorf("invalid LBKind: %s ", k.VirtualServers[i].LBKind))
 			}
 		}
 
 		if k.VirtualServers[i].PersistenceTimeoutSeconds == 0 {
 			k.VirtualServers[i].PersistenceTimeoutSeconds = 360
 		} else if k.VirtualServers[i].PersistenceTimeoutSeconds < 0 {
-			return errors.New("PersistenceTimeout must be a positive integer")
+			errs = append(errs, errors.New("PersistenceTimeout must be a positive integer"))
 		}
 
 		if k.VirtualServers[i].DelayLoop < 0 {
-			return errors.New("DelayLoop must be a positive integer")
+			errs = append(errs, errors.New("DelayLoop must be a positive integer"))
 		}
 	}
-	return nil
+	return errs
+}
+
+// Validate validates the ControlPlaneLoadBalancingSpec
+func (c *ControlPlaneLoadBalancingSpec) Validate(externalAddress string) []error {
+	if c == nil || !c.Enabled {
+		return nil
+	}
+	errs := []error{}
+
+	switch c.Type {
+	case CPLBTypeKeepalived:
+	case "":
+		c.Type = CPLBTypeKeepalived
+	default:
+		errs = append(errs, fmt.Errorf("unsupported CPLB type: %s. Only allowed value: %s", c.Type, CPLBTypeKeepalived))
+	}
+
+	errs = append(errs, c.Keepalived.validateVRRPInstances(nil)...)
+	errs = append(errs, c.Keepalived.validateVirtualServers()...)
+	// CPLB reconciler relies in watching kubernetes.default.svc endpoints
+	if externalAddress != "" && len(c.Keepalived.VirtualServers) > 0 {
+		errs = append(errs, errors.New(".spec.api.externalAddress and VRRPInstances cannot be used together"))
+	}
+
+	return errs
 }

pkg/apis/k0s/v1beta1/cplb_test.go

@@ -127,11 +127,11 @@ func (s *CPLBSuite) TestValidateVRRPInstances() {
 			k := &KeepalivedSpec{
 				VRRPInstances: tt.vrrps,
 			}
-			err := k.ValidateVRRPInstances(returnNIC)
+			err := k.validateVRRPInstances(returnNIC)
 			if tt.wantErr {
-				s.Require().Errorf(err, "Test case %s expected error. Got none", tt.name)
+				s.Require().NotEmpty(err, "Test case %s expected error. Got none", tt.name)
 			} else {
-				s.Require().NoErrorf(err, "Test case %s expected no error. Got: %v", tt.name, err)
+				s.Require().Empty(err, "Test case %s expected no errors. Got: %v", tt.name, err)
 				s.T().Log(k.VRRPInstances)
 				s.Require().Equal(len(tt.expectedVRRPs), len(k.VRRPInstances), "Expected and actual VRRPInstances length mismatch")
 				for i := 0; i < len(tt.expectedVRRPs); i++ {
@@ -250,16 +250,16 @@ func (s *CPLBSuite) TestValidateVirtualServers() {
 	for _, tt := range tests {
 		s.Run(tt.name, func() {
 			k := &KeepalivedSpec{VirtualServers: tt.vss}
-			err := k.ValidateVirtualServers()
+			err := k.validateVirtualServers()
 			if tt.wantErr {
-				s.Require().Errorf(err, "Test case %s expected error. Got none", tt.name)
+				s.Require().NotEmpty(err, "Test case %s expected error. Got none", tt.name)
 			} else {
-				s.Require().NoErrorf(err, "Tedst case %s expected no error. Got: %v", tt.name, err)
+				s.Require().Empty(err, "Tedst case %s expected no error. Got: %v", tt.name, err)
 				for i := range tt.expectedVSS {
-					s.Require().Equal(tt.expectedVSS[i].DelayLoop, cplb.VirtualServers[i].DelayLoop, "DelayLoop mismatch")
-					s.Require().Equal(tt.expectedVSS[i].LBAlgo, cplb.VirtualServers[i].LBAlgo, "LBalgo mismatch")
-					s.Require().Equal(tt.expectedVSS[i].LBKind, cplb.VirtualServers[i].LBKind, "LBKind mismatch")
-					s.Require().Equal(tt.expectedVSS[i].PersistenceTimeoutSeconds, cplb.VirtualServers[i].PersistenceTimeoutSeconds, "PersistenceTimeout mismatch")
+					s.Require().Equal(tt.expectedVSS[i].DelayLoop, k.VirtualServers[i].DelayLoop, "DelayLoop mismatch")
+					s.Require().Equal(tt.expectedVSS[i].LBAlgo, k.VirtualServers[i].LBAlgo, "LBalgo mismatch")
+					s.Require().Equal(tt.expectedVSS[i].LBKind, k.VirtualServers[i].LBKind, "LBKind mismatch")
+					s.Require().Equal(tt.expectedVSS[i].PersistenceTimeoutSeconds, k.VirtualServers[i].PersistenceTimeoutSeconds, "PersistenceTimeout mismatch")
 				}
 			}
 		})