Full implementation for Cilium ENI-based IP allocation (#604)

giantswarm · May 2, 2024 · f50fa8d · f50fa8d
1 parent 9e4b742
commit f50fa8d
Show file tree

Hide file tree

Showing 9 changed files with 215 additions and 7 deletions.
diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -7,8 +7,11 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## [Unreleased]
 
-## [0.73.0] - 2024-04-30
+### Added
 
+- Make Cilium ENI-based IP allocation configurable with high-level `global.connectivity.cilium.ipamMode` value. This feature was previously introduced as prototype and is now fully working.
+
+## [0.73.0] - 2024-04-30
 
 ### Added
 
@@ -76,7 +79,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ### Added
 
-- Make Cilium ENI-based IP allocation configurable with new high-level `global.connectivity.cilium.ipamMode` value
+- Make Cilium ENI-based IP allocation configurable with new high-level `global.connectivity.cilium.ipamMode` value (prototype)
 - Add automatic support for deploying to AWS China.
 
 ### Changed

diff --git a/helm/cluster-aws/Chart.lock b/helm/cluster-aws/Chart.lock
@@ -1,9 +1,9 @@
 dependencies:
 - name: cluster
   repository: https://giantswarm.github.io/cluster-catalog
-  version: 0.18.0
+  version: 0.19.0
 - name: cluster-shared
   repository: https://giantswarm.github.io/cluster-catalog
   version: 0.7.0
-digest: sha256:1ce40f738ca0a191ac8400d41ee55b0d81698e5ba904ee563f758c88ae805b5a
-generated: "2024-04-11T15:48:38.252484+02:00"
+digest: sha256:46235919f30fc601869f7a6fa82c4eceb18f580ee3c1aa65e106af186ff9be60
+generated: "2024-04-30T08:48:36.369197+02:00"
diff --git a/helm/cluster-aws/Chart.yaml b/helm/cluster-aws/Chart.yaml
@@ -16,7 +16,7 @@ restrictions:
     - capa
 dependencies:
   - name: cluster
-    version: "0.18.0"
+    version: "0.19.0"
     repository: https://giantswarm.github.io/cluster-catalog
   - name: cluster-shared
     version: "0.7.0"

diff --git a/helm/cluster-aws/README.md b/helm/cluster-aws/README.md
@@ -104,11 +104,21 @@ Properties within the `.global.connectivity` object
 | `global.connectivity.cilium.ipamMode` | **IPAM mode (IP allocation strategy)** - Use `eni` for ENI (AWS Elastic Network Interfaces) allocation of IPs in Cilium (https://docs.cilium.io/en/latest/network/concepts/ipam/eni/). The default is `kubernetes` (https://docs.cilium.io/en/latest/network/concepts/ipam/kubernetes/). WARNING: The `eni` feature is currently in an early stage and there might be breaking changes in the future. The networking infrastructure will be made consistent with our vintage cluster implementations so that pod IPs are placed in a secondary VPC CIDR.|**Type:** `string`<br/>**Default:** `"kubernetes"`|
 | `global.connectivity.dns` | **DNS**|**Type:** `object`<br/>|
 | `global.connectivity.dns.resolverRulesOwnerAccount` | **Resolver rules owner** - ID of the AWS account that created the resolver rules to be associated with the workload cluster VPC.|**Type:** `string`<br/>|
+| `global.connectivity.eniModePodSubnets` | **Pod Subnets for Cilium ENI mode** - Pod subnets are created and tagged based on this definition. **Only used for `global.connectivity.cilium.ipamMode=eni`** which puts pods on a secondary CIDR block in the VPC, and therefore requires separate subnets. The subnets must be tagged with `sigs.k8s.io/cluster-api-provider-aws/association=secondary` to be correctly handled by CAPA (so those subnets aren't accidentally chosen for nodes). These subnets are always private.|**Type:** `array`<br/>**Default:** `[{"cidrBlocks":[{"availabilityZone":"a","cidr":"10.1.0.0/18","tags":{"sigs.k8s.io/cluster-api-provider-aws/association":"secondary"}},{"availabilityZone":"b","cidr":"10.1.64.0/18","tags":{"sigs.k8s.io/cluster-api-provider-aws/association":"secondary"}},{"availabilityZone":"c","cidr":"10.1.128.0/18","tags":{"sigs.k8s.io/cluster-api-provider-aws/association":"secondary"}}]}]`|
+| `global.connectivity.eniModePodSubnets[*]` | **Subnet**|**Type:** `object`<br/>|
+| `global.connectivity.eniModePodSubnets[*].cidrBlocks` | **Network**|**Type:** `array`<br/>|
+| `global.connectivity.eniModePodSubnets[*].cidrBlocks[*]` |**None**|**Type:** `object`<br/>|
+| `global.connectivity.eniModePodSubnets[*].cidrBlocks[*].availabilityZone` | **Availability zone**|**Type:** `string`<br/>**Example:** `"a"`<br/>|
+| `global.connectivity.eniModePodSubnets[*].cidrBlocks[*].cidr` | **Address range** - IPv4 address range, in CIDR notation.|**Type:** `string`<br/>|
+| `global.connectivity.eniModePodSubnets[*].cidrBlocks[*].tags` | **Tags** - AWS resource tags to assign to this subnet.|**Type:** `object`<br/>|
+| `global.connectivity.eniModePodSubnets[*].cidrBlocks[*].tags.*` | **Tag value**|**Type:** `string`<br/>**Value pattern:** `^[ a-zA-Z0-9\._:/=+-@]+$`<br/>|
+| `global.connectivity.eniModePodSubnets[*].tags` | **Tags** - AWS resource tags to assign to this CIDR block.|**Type:** `object`<br/>|
+| `global.connectivity.eniModePodSubnets[*].tags.*` | **Tag value**|**Type:** `string`<br/>**Value pattern:** `^[ a-zA-Z0-9\._:/=+-@]+$`<br/>|
 | `global.connectivity.network` | **Network**|**Type:** `object`<br/>|
 | `global.connectivity.network.allowAllEgress` | **Allow all egress**|**Type:** `boolean`<br/>**Default:** `false`|
 | `global.connectivity.network.internetGatewayId` | **Internet Gateway ID** - ID of the Internet gateway for the VPC.|**Type:** `string`<br/>|
 | `global.connectivity.network.pods` | **Pods**|**Type:** `object`<br/>|
-| `global.connectivity.network.pods.cidrBlocks` | **Pod subnets**|**Type:** `array`<br/>**Default:** `["100.64.0.0/12"]`|
+| `global.connectivity.network.pods.cidrBlocks` | **Pod subnets** - CIDR blocks used for pods. Right now, only one block is supported.<br/><br/>**Note if you use `global.connectivity.cilium.ipamMode=eni` (https://docs.cilium.io/en/latest/network/concepts/ipam/eni/#ipam-eni):** this will be associated as secondary VPC CIDR. Therefore, only sizes /16 to /28 sizes are possible (see https://docs.aws.amazon.com/vpc/latest/userguide/vpc-cidr-blocks.html). And `global.connectivity.eniModePodSubnets` must be a valid split of the CIDR you chose here – we recommend setting `10.1.0.0/16` here for ENI mode because the default values for `global.connectivity.eniModePodSubnets` match that CIDR.|**Type:** `array`<br/>**Default:** `["100.64.0.0/12"]`|
 | `global.connectivity.network.pods.cidrBlocks[*]` | **Pod subnet** - IPv4 address range for pods, in CIDR notation.|**Type:** `string`<br/>**Example:** `"10.244.0.0/16"`<br/>|
 | `global.connectivity.network.services` | **Services**|**Type:** `object`<br/>|
 | `global.connectivity.network.services.cidrBlocks` | **K8s Service subnets**|**Type:** `array`<br/>**Default:** `["172.31.0.0/16"]`|

diff --git a/helm/cluster-aws/ci/test-eni-mode.yaml b/helm/cluster-aws/ci/test-eni-mode.yaml
@@ -0,0 +1,14 @@
+global:
+  metadata:
+    name: test-wc
+    organization: "test"
+  connectivity:
+    baseDomain: example.com
+    cilium:
+      ipamMode: eni
+    network:
+      pods:
+        cidrBlocks:
+          - 10.1.0.0/16
+  providerSpecific:
+    region: "eu-west-1"
diff --git a/helm/cluster-aws/templates/_aws_cluster.tpl b/helm/cluster-aws/templates/_aws_cluster.tpl
@@ -56,6 +56,16 @@ spec:
       cidrBlocks: {{- toYaml ((concat .Values.global.controlPlane.loadBalancerIngressAllowCidrBlocks (list "95.179.153.65/32" "185.102.95.187/32")) | uniq) | nindent 6 }}
     {{- end }}
   network:
+    {{- if eq (required "global.connectivity.cilium.ipamMode is required"  .Values.global.connectivity.cilium.ipamMode) "eni" }}
+    additionalControlPlaneIngressRules:
+      - description: "Allow traffic from pods to control plane nodes for access of applications to Kubernetes API"
+        protocol: "-1" # all
+        fromPort: -1
+        toPort: -1
+
+        # We could also use `sourceSecurityGroupIds` here, but the ID of the "<cluster>-pods" security group isn't known yet
+        cidrBlocks: {{ required "global.connectivity.network.pods.cidrBlocks is required" .Values.global.connectivity.network.pods.cidrBlocks | toYaml | nindent 10 }}
+    {{- end }}
     cni:
       cniIngressRules:
       - description: allow AWS CNI traffic across nodes and control plane
@@ -73,6 +83,14 @@ spec:
       {{- if .Values.global.connectivity.network.internetGatewayId }}
       internetGatewayId: {{ .Values.global.connectivity.network.internetGatewayId }}
       {{- end }}
+      {{- if eq (required "global.connectivity.cilium.ipamMode is required"  .Values.global.connectivity.cilium.ipamMode) "eni" }}
+      secondaryCidrBlocks:
+        # Managed by Cilium in ENI mode
+        {{- if not (required "global.connectivity.network.pods.cidrBlocks is required" .Values.global.connectivity.network.pods.cidrBlocks | first | regexMatch "/(1[6-9]|2[0-8])$") }}
+          {{ fail (printf "You have set `global.connectivity.cilium.ipamMode=eni`, but the pod CIDR %s is not supported as AWS VPC CIDR (see https://docs.aws.amazon.com/vpc/latest/userguide/vpc-cidr-blocks.html: /16 to /28 sizes are supported). Please change `global.connectivity.network.pods.cidrBlocks` to a valid value (see https://github.com/giantswarm/cluster-aws/tree/main/helm/cluster-aws#connectivity)." (.Values.global.connectivity.network.pods.cidrBlocks | first | quote)) }}
+        {{- end }}
+        - ipv4CidrBlock: {{ .Values.global.connectivity.network.pods.cidrBlocks | first | quote }}
+      {{- end }}
     subnets:
     {{- range $j, $subnet := .Values.global.connectivity.subnets }}
     {{- if $subnet.id }}
@@ -107,6 +125,30 @@ spec:
     {{- end }}
     {{- end }}
     {{- end }}
+
+    {{- if eq (required "global.connectivity.cilium.ipamMode is required"  .Values.global.connectivity.cilium.ipamMode) "eni" }}
+    {{- range $j, $subnet := .Values.global.connectivity.eniModePodSubnets }}
+    {{- range $i, $cidr := $subnet.cidrBlocks }}
+    - id: "{{ include "resource.default.name" $ }}-subnet-secondary-{{ if eq (len $cidr.availabilityZone) 1 }}{{ include "aws-region" $ }}{{ end }}{{ $cidr.availabilityZone }}"
+      cidrBlock: "{{ $cidr.cidr }}"
+      {{- if eq (len $cidr.availabilityZone) 1 }}
+      availabilityZone: "{{ include "aws-region" $ }}{{ $cidr.availabilityZone }}"
+      {{- else }}
+      availabilityZone: "{{ $cidr.availabilityZone }}"
+      {{- end }}
+      isPublic: false
+      {{- if ne (index $cidr.tags "sigs.k8s.io/cluster-api-provider-aws/association") "secondary" }}
+        {{ fail (printf "You have set `global.connectivity.cilium.ipamMode=eni`, but the pod subnet %q in `global.connectivity.eniModePodSubnets` is not tagged with `sigs.k8s.io/cluster-api-provider-aws/association=secondary`, as required so that CAPA does not accidentally choose the subnet for nodes (see https://github.com/giantswarm/cluster-aws/tree/main/helm/cluster-aws#connectivity)." $cidr.cidr) }}
+      {{- end }}
+      {{- if or $subnet.tags $cidr.tags }}
+      tags:
+        {{- if $cidr.tags }}
+        {{- toYaml $cidr.tags | nindent 8 }}
+        {{- end }}
+      {{- end }}
+    {{- end }}
+    {{- end }}
+    {{- end }}
   sshKeyName: ssh-key
   s3Bucket:
     controlPlaneIAMInstanceProfile: control-plane-{{ include "resource.default.name" $ }}

diff --git a/helm/cluster-aws/templates/cilium-crossplane-resources-app.yaml b/helm/cluster-aws/templates/cilium-crossplane-resources-app.yaml
@@ -0,0 +1,45 @@
+{{- if eq (required "global.connectivity.cilium.ipamMode is required"  .Values.global.connectivity.cilium.ipamMode) "eni" }}
+apiVersion: v1
+data:
+  values: |
+    aws_eni_mode:
+      enabled: true
+kind: ConfigMap
+metadata:
+  labels:
+    app-operator.giantswarm.io/version: 0.0.0
+    {{- include "labels.common" $ | nindent 4 }}
+  name: {{ printf "%s-cilium-crossplane-resources" (include "resource.default.name" $) | quote }}
+  namespace: {{ $.Release.Namespace | quote }}
+---
+apiVersion: application.giantswarm.io/v1alpha1
+kind: App
+metadata:
+  labels:
+    app-operator.giantswarm.io/version: 0.0.0
+    {{- include "labels.common" $ | nindent 4 }}
+  name: {{ printf "%s-cilium-crossplane-resources" (include "resource.default.name" $) | quote }}
+  namespace: {{ $.Release.Namespace | quote }}
+spec:
+  catalog: cluster
+  install:
+    timeout: "10m"
+  upgrade:
+    timeout: "10m"
+  kubeConfig:
+    inCluster: true # in management cluster context
+  name: cilium-crossplane-resources
+  namespace: {{ $.Release.Namespace | quote }}
+  # used by renovate
+  # repo: giantswarm/cilium-crossplane-resources
+  version: 0.1.0
+  extraConfigs:
+    # See above
+    - kind: configMap
+      name: {{ printf "%s-cilium-crossplane-resources" (include "resource.default.name" $) | quote }}
+      namespace: {{ $.Release.Namespace | quote }}
+    # Created by aws-crossplane-cluster-config-operator once the cluster gets provisioned
+    - kind: configMap
+      name: {{ printf "%s-crossplane-config" (include "resource.default.name" $) | quote }}
+      namespace: {{ $.Release.Namespace | quote }}
+{{- end }}
diff --git a/helm/cluster-aws/values.schema.json b/helm/cluster-aws/values.schema.json
@@ -652,6 +652,85 @@
                                 }
                             }
                         },
+                        "eniModePodSubnets": {
+                            "type": "array",
+                            "title": "Pod Subnets for Cilium ENI mode",
+                            "description": "Pod subnets are created and tagged based on this definition. **Only used for `global.connectivity.cilium.ipamMode=eni`** which puts pods on a secondary CIDR block in the VPC, and therefore requires separate subnets. The subnets must be tagged with `sigs.k8s.io/cluster-api-provider-aws/association=secondary` to be correctly handled by CAPA (so those subnets aren't accidentally chosen for nodes). These subnets are always private.",
+                            "items": {
+                                "type": "object",
+                                "title": "Subnet",
+                                "properties": {
+                                    "cidrBlocks": {
+                                        "type": "array",
+                                        "title": "Network",
+                                        "items": {
+                                            "type": "object",
+                                            "required": [
+                                                "availabilityZone",
+                                                "cidr"
+                                            ],
+                                            "properties": {
+                                                "availabilityZone": {
+                                                    "type": "string",
+                                                    "title": "Availability zone",
+                                                    "examples": [
+                                                        "a"
+                                                    ]
+                                                },
+                                                "cidr": {
+                                                    "type": "string",
+                                                    "title": "Address range",
+                                                    "description": "IPv4 address range, in CIDR notation."
+                                                },
+                                                "tags": {
+                                                    "type": "object",
+                                                    "title": "Tags",
+                                                    "description": "AWS resource tags to assign to this subnet.",
+                                                    "additionalProperties": {
+                                                        "$ref": "#/$defs/awsResourceTagValue"
+                                                    }
+                                                }
+                                            }
+                                        }
+                                    },
+                                    "tags": {
+                                        "type": "object",
+                                        "title": "Tags",
+                                        "description": "AWS resource tags to assign to this CIDR block.",
+                                        "additionalProperties": {
+                                            "$ref": "#/$defs/awsResourceTagValue"
+                                        }
+                                    }
+                                }
+                            },
+                            "default": [
+                                {
+                                    "cidrBlocks": [
+                                        {
+                                            "availabilityZone": "a",
+                                            "cidr": "10.1.0.0/18",
+                                            "tags": {
+                                                "sigs.k8s.io/cluster-api-provider-aws/association": "secondary"
+                                            }
+                                        },
+                                        {
+                                            "availabilityZone": "b",
+                                            "cidr": "10.1.64.0/18",
+                                            "tags": {
+                                                "sigs.k8s.io/cluster-api-provider-aws/association": "secondary"
+                                            }
+                                        },
+                                        {
+                                            "availabilityZone": "c",
+                                            "cidr": "10.1.128.0/18",
+                                            "tags": {
+                                                "sigs.k8s.io/cluster-api-provider-aws/association": "secondary"
+                                            }
+                                        }
+                                    ]
+                                }
+                            ]
+                        },
                         "network": {
                             "type": "object",
                             "title": "Network",
@@ -673,6 +752,7 @@
                                         "cidrBlocks": {
                                             "type": "array",
                                             "title": "Pod subnets",
+                                            "description": "CIDR blocks used for pods. Right now, only one block is supported.<br/><br/>**Note if you use `global.connectivity.cilium.ipamMode=eni` (https://docs.cilium.io/en/latest/network/concepts/ipam/eni/#ipam-eni):** this will be associated as secondary VPC CIDR. Therefore, only sizes /16 to /28 sizes are possible (see https://docs.aws.amazon.com/vpc/latest/userguide/vpc-cidr-blocks.html). And `global.connectivity.eniModePodSubnets` must be a valid split of the CIDR you chose here – we recommend setting `10.1.0.0/16` here for ENI mode because the default values for `global.connectivity.eniModePodSubnets` match that CIDR.",
                                             "items": {
                                                 "type": "string",
                                                 "title": "Pod subnet",

diff --git a/helm/cluster-aws/values.yaml b/helm/cluster-aws/values.yaml
@@ -194,6 +194,20 @@ global:
     cilium:
       ipamMode: kubernetes
     dns: {}
+    eniModePodSubnets:
+      - cidrBlocks:
+          - availabilityZone: a
+            cidr: 10.1.0.0/18
+            tags:
+              sigs.k8s.io/cluster-api-provider-aws/association: secondary
+          - availabilityZone: b
+            cidr: 10.1.64.0/18
+            tags:
+              sigs.k8s.io/cluster-api-provider-aws/association: secondary
+          - availabilityZone: c
+            cidr: 10.1.128.0/18
+            tags:
+              sigs.k8s.io/cluster-api-provider-aws/association: secondary
     network:
       allowAllEgress: false
       pods: