All notable changes to this project will be documented in this file.
The format is based on Keep a Changelog, and this project adheres to Semantic Versioning.
2.5.0 - 2024-12-13
- Add aws-node-termination-handler bundle
- Values: Add
global.providerSpecific.controlPlaneAmi
&global.providerSpecific.nodePoolAmi
. - Make ASG lifecycle hook heartbeat timeout configurable
- Chart: Update
cluster
to v1.7.0.- Add
teleport-init
systemd unit to handle initial token setup beforeteleport
service starts - Improve
teleport
service reliability by adding proper file and service dependencies and pre-start checks
- Add
-
Fix aws-nth-bundle to use the MC's kubeconfig context if it's in a different organization namespace.
Workload clusters outside the MC's
org-giantswarm
namespace failed to deploy the bundle becauseHelmRelease
does not allow specifying the MC's kubeconfig secret namespace. The bundle was therefore switched to anApp
.
2.4.0 - 2024-11-12
- Add
global.providerSpecific.additionalNodeTags
. Field used to specify tags applied to nodes only.
- Only try to render subnet tags if they are defined by the user.
2.3.0 - 2024-10-17
- Expose the
maxHealthyPercentage
property to allow setting the maximum percentage of healthy machines in the Auto Scaling Group during upgrades.
2.2.0 - 2024-09-23
- Allow to enable
auditd
throughglobal.components.auditd.enabled
helm value. - Chart: Support multiple service account issuers.
This is used for example in the migration from Vintage AWS clusters to CAPA. Multiple issuers were previously supported only through internal chart values (this change removesinternal.migration.irsaAdditionalDomain
). The internal annotationaws.giantswarm.io/irsa-additional-domain
on AWSMachineTemplate objects is changed to pluralaws.giantswarm.io/irsa-trust-domains
on the AWSCluster object.
- Chart: Update
cluster
to v1.4.1. - Set provider specific configuration for cilium CNI ENI values.
- Remove Cilium app deprecated values.
2.1.0 - 2024-08-29
- Do not allow additional properties in the following fields in order to avoid unnoticed typos:
- global.connectivity.network
- global.connectivity.network.pods
- global.connectivity.network.services
- global.connectivity.subnets[]
- global.connectivity.topology
- global.controlPlane
- global.controlPlane.additionalSecurityGroups[]
- global.controlPlane.machineHealthCheck
- global.controlPlane.oidc
- global.providerSpecific
- global.providerSpecific.instanceMetadataOptions
- Validate that machine pool availability zones belong to the selected region.
- CI: Bump release version.
- Apps: Use
catalog
from Release CR.
- Remove unused kubectl image Helm value.
2.0.0 - 2024-08-07
Important
Releases that include this cluster-aws version must have the os-tooling
component in the Release resource .spec.components
,
as well as observability-policies app in the Release resource .spec.apps
.
See ami
changes below for more details about the change and see AWS (CAPA) release v29.0.0 for a Release resource example.
- Add
global.metadata.labels
to values schema. This field is used to add labels to the cluster resources. - Enable
observability-policies
app.
- Update cluster chart to v1.1.0.
- This sets cilium
kubeProxyReplacement
config to"true"
instead to"strict"
("strict"
has been deprecated since cilium v1.14, see this upstream cilium issue for more details).
- This sets cilium
- Update
ami
named template to correctly render OS image name with the new formatflatcar-stable-<flatcar version>-kube-<kubernetes version>-tooling-<capi-image-builder version>-gs
.
1.3.0 - 2024-07-25
- Update cluster chart version to v1.0.0. This update adds MC Zot deployment as a registry mirror for
gsoci.azurecr.io
registry. This is the new default behavior.
1.2.1 - 2024-07-22
- Apps: Fix service monitor dependencies.
1.2.0 - 2024-07-22
This release removes the CronJobTimeZone
feature gate as it becomes stable and is included in Kubernetes v1.29.
For Kubernetes <v1.29, you will need to re-enable it using the respective values.
- Chart: Update
cluster
chart to v0.36.0. (#703)
- Feature Gates: Remove
CronJobTimeZone
. (#699)
1.1.0 - 2024-07-11
- Update cluster chart to 0.35.0
- Fixed China IRSA suffix
1.0.1 - 2024-07-09
- Add the Management Cluster name as a tag to the AWS resources created by CAPA.
- Add the node pool name as a tag to the AWS resources associated with the node pool.
- Update cluster chart to 0.33.1
1.0.0 - 2024-06-20
- First major release, breaking changes not allowed in minor releases anymore.
0.79.0 - 2024-06-19
- Set maximum number of pods (kubelet
--max-pods
) for Cilium ENI mode due to restrictions by number of ENIs and IPs per ENI. This change will roll nodes even if not using Cilium ENI mode, since a new script is introduced.
0.78.2 - 2024-06-12
- Update cluster chart to 0.31.4
0.78.1 - 2024-06-12
- Update cluster chart to 0.31.2
0.78.0 - 2024-06-10
- Add
irsa-servicemonitors
andaws-ebs-csi-driver-servicemonitors
apps.
- Set
prometheus-blackbox-exporter
andk8s-audit-metrics
as enabled.
0.77.0 - 2024-06-07
- Add configuration for
aws-pod-web-identity-webook
app to include region into the IRSA enabled pods.
0.76.2 - 2024-06-05
- Set environment variable
COREOS_EC2_IPV4_LOCAL
to inject value to kubeadm configuration. - Set environmane variable
COREOS_EC2_HOSTNAME
to inject value to kubeadm configuration. - Update aws-cloud-controller-manager-app to v1.25.14-gs3.
- Update cluster chart to v0.27.0. More details in cluster chart v0.27.0 release notes.
- Always render
userConfig
values reference to configmap foraws-pod-identity-webhook-app
. - Store EC2 user data (Ignition config) for machine pools in S3 bucket to overcome the size limit (requires this new CAPA feature and
AWSMachinePool.spec.ignition
field) - Update CAPA CR references to API version
v1beta2
0.76.1 - 2024-05-16
- Set token audience for
aws-pod-identity-webhook
based on AWS region.
0.76.0 - 2024-05-14
- Update cluster chart to v0.25.0 and enable all default apps. More details in cluster chart v0.25.0 release notes.
Cluster upgrade steps are the following:
- Upgrade default-apps-aws App to v0.52.0 or newer.
- Update default-apps-aws Helm value
.Values.deleteOptions.moveAppsHelmOwnershipToClusterAws
to true.- All apps, except observability-bundle and security-bundle will get
app-operator.giantswarm.io/paused: true
annotation, so wait few minutes for the change to get applied by the Helm post-upgrade hook.
- All apps, except observability-bundle and security-bundle will get
- Delete default-apps-aws.
- App resources for all default apps will get deleted. Wait few minutes for this to happen.
- Chart resources on the workload cluster will stay, so all apps will continue running.
- Upgrade cluster-aws App to v0.76.0.
- cluster-aws will deploy all default apps, wait a few minutes for all Apps to be successfully deployed.
- Chart resources on the workload cluster will get updated, as newly deployed App resources will take over the reconciliation of the existing Chart resources.
- Chart: Add
aws-pod-identity-webhook
app. (#581).
0.75.0 - 2024-05-09
- Worker nodes - Add
nonRootVolumes
fields to mount/var/lib
and/var/log
on separate disk volumes. - BREAKING CHANGE: values
global.controlplane.containerdVolumeSizeGB
andglobal.controlplane.kubeletVolumeSizeGB
merged into single value.global.controlPlane.libVolumeSizeGB
which define size of disk volume used for/var/lib
mount point.
- Control-plane nodes - combine kubelet disk
/var/lib/kubelet
and containerd disk/var/lib/containerd
into single disk/var/lib
to share the volume space and save cost.
0.74.0 - 2024-05-08
- Added an annotation to Kubernetes resources to resolve an issue where deletion was stuck due to hanging load balancers.
- Make Cilium ENI-based IP allocation configurable with high-level
global.connectivity.cilium.ipamMode
value. This feature was previously introduced as prototype and is now fully working. - Allow to configure SELinux mode through
global.components.selinux.mode
helm value.
- Update cluster chart to v0.22.0.
0.73.0 - 2024-04-30
- Add
log
volume to control-plane nodes.
0.72.0 - 2024-04-24
- Add option to configure instance metadata http tokens for EC2 instances to enable or disable IMDSv2 enforcement.
0.71.0 - 2024-04-16
- Update cluster chart to v0.18.0. This updates teleport node labels and will roll nodes.
- Update instanceWarmup to 10' to be on pair with Vintage
- Enable extraPolicies from network-policies-app.
- Disable and remove extraPolicies from cilium-app.
- Values: Separate
app
andhelmRelease
definition. (#581)
0.70.0 - 2024-04-15
- Update cluster chart to v0.17.0. This updates cilium app from v0.21.0 to v0.22.0.
- Update aws-ebs-csi-driver-app from v2.30.0 to v2.30.1. This fixes accidental installation of PSPs which could break the upgrade to previous
cluster-aws
versions which didn't have this fix yet.
0.69.0 - 2024-04-11
- Allow customizing instance refresh parameters.
- Fix selecting right AWS region for Machine Pools when cluster is in different AWS region than MC.
- Update Availability Zones in helm/cluster-aws/files/azs-in-region.yaml
- AMI: Use new AMI which includes latest teleport binary v15.1.7
0.68.0 - 2024-03-27
- Chart: Bump
cluster
to v0.16.0.
0.67.0 - 2024-03-26
- Smart defaulting for AWS availability zones using actual AZs in the region of choice rather than hardcoded values.
0.66.1 - 2024-03-21
- Update Chart.lock with current version of dependencies.
0.66.0 - 2024-03-21
- Make Cilium ENI-based IP allocation configurable with new high-level
global.connectivity.cilium.ipamMode
value (prototype) - Add automatic support for deploying to AWS China.
- Use cleanup hook job HelmRelease from cluster chart.
- Chart: Bump
cluster
to v0.13.0.
0.65.0 - 2024-03-07
- Pass
clusterID
toaws-ebs-csi-driver
app's values for volume tagging purposes.
- Change image lookup format for base OS image. osImageVariant is set to "2" for this kubernetes version. This is a breaking change that requires manual steps. For the next kubernetes versions, osImageVariant should not be set.
0.64.2 - 2024-03-06
- Fix allow list API port 443.
0.64.1 - 2024-02-29
- Chart: Bump
cluster
to v0.11.1.
0.64.0 - 2024-02-28
- Chart: Bump
cluster
to v0.11.0. - Use cilium and network-policies from cluster chart, and remove them from cluster-aws.
0.63.0 - 2024-02-22
- Use default HelmRepositories from cluster chart.
- Use vertical-pod-autoscaler-crd HelmRelease from cluster chart.
- Use coredns HelmRelease from cluster chart.
- Remove default HelmRepositories from cluster-aws.
- Remove vertical-pod-autoscaler-crd HelmRelease from cluster-aws.
- Remove coredns HelmRelease from cluster-aws.
0.62.1 - 2024-02-19
- Update network-policies to avoid installing deny-all policy.
0.62.0 - 2024-02-14
- Add network-policies helm release.
0.61.0 - 2024-02-12
- Render MachineHealthCheck resource from the cluster chart.
- Remove MachineHealthCheck resource.
- Render MachinePool and KubeadmConfig resources from the cluster chart.
- Remove MachinePool and KubeadmConfig resources.
- Remove duplicate containerd config as it's already deployed by the cluster chart.
0.60.1 - 2024-02-05
- Allow customers to specify optional extraConfigs in HelmRelease apps.
- Include cluster-test-catalog in the CI, so we can more easily test dev builds of subcharts.
- Update cluster chart version to the latest v0.7.1 release.
- Render control plane resources from the cluster chart.
- Remove KubeadmControlPlane resource.
- Use
cluster.connectivity.proxy.noProxy
Helm template from cluster chart to render NO_PROXY in cluster-aws. - Rename CI files, so they are used in GitHub action that checks Helm rendering.
- Remove ingress and egress rules from the security group that AWS creates by default when creating a new VPC.
- Remove unnecessary architect brackets cleanup.
- Use CI values to render templates locally.
0.60.0 - 2024-01-29
- Bumped kubernetes version to 1.25.16. This change also enforces PSS.
0.59.1 - 2024-01-24
- Do not hardcode cilium k8s service port. Use
global.controlPlane.apiServerPort
.
0.59.0 - 2024-01-23
- Use
gsoci.azurecr.io
forkubeadm
container images. - Use
gsoci.azurecr.io
for sandbox container image (pause container). - Update
coredns
to1.21.0
to usegsoci.azurecr.io
. - Update
aws-cloud-controller-manager
to1.25.14-gs2
to usegsoci.azurecr.io
.
0.58.0 - 2024-01-22
- Bump cilium-app to v0.19.2 (upgrades Cilium to v1.14.5 and fixes a
CiliumNetworkPolicy
definition for reaching coredns)
- Fix removing allow-all Cilium network policies
0.57.0 - 2024-01-10
- Add propagating tags from
cluster-aws
to resources managed myebs-csi-driver
. - CI: trigger automated e2e tests on Renovate PRs.
- Add new annotation for vintage irsa domain which is only used for migrating vintage clusters.
- Use 443 as the default api-server Load Balancer port.
- Remove allow-all Cilium network policies.
0.56.0 - 2024-01-08
- Add topology annotations to AWSCluster
- Add
cluster
chart as subchart. - Render Cluster resource from the
cluster
chart. - Delete Cluster resource template.
- Add missing kubelet configuration to align it with vintage config.
0.55.0 - 2023-12-21
- Change
KubeadmConfig
bootstrap config reference to ensure nodes get rolled when making changes to node specification (requires newer versions of CAPI/CAPA as shown in the original issue). Add machine pool instance warmup setting (5 minutes) to ensure nodes do not get replaced too quickly. - Run kubeadm after containerd to avoid node startup problems
0.54.0 - 2023-12-21
- Added option to customize app via configmap or secret with
global.apps.{app_name}.extraConfigs
. - In-line custom values for app moved from
global.apps.{app_name}
toglobal.apps.{app_name}.values
.
0.53.0 - 2023-12-13
- Remove bastion and ssh configuration on nodes.
0.52.0 - 2023-12-11
- Set node pool subnet filters to include avaiability zone.
- Fix error messages if
global.connectivity.baseDomain
is missing
0.51.0 - 2023-12-07
- Fill
AWSCluster.spec.network.subnets[*].id
field for managed subnets for compatibility with CAPA v2.3.0
0.50.0 - 2023-12-04
How to migrate to v0.50.0
Please ensure you did install yq first.
To migrate values from cluster-aws v0.50.0
, we provide below a bash script which writes an app.yaml
file which you need to apply.
This will move the existing user config values into global
and it also increases the version
field of cluster-aws
app to v0.50.0
.
- Login to the management cluster and run the script (e.g:
./migrate.sh organization my-cluster
) - Verify the
app.yaml
file and apply it to the management cluster (e.g:kubectl apply -f app.yaml
)
#!/bin/bash
# Check if two arguments are provided
if [ $# -ne 2 ]
then
echo "Incorrect number of arguments supplied. Please provide the organization name and the cluster name."
exit 1
fi
# Use the first argument as the organization name and the second as the cluster name
org=$1
cluster=$2
# Fetch the ConfigMap YAML
kubectl get cm -n org-$org ${cluster}-userconfig -o yaml > ${cluster}_cm.yaml
# Extract the ConfigMap values into a temporary file
yq eval '.data.values' ${cluster}_cm.yaml > tmp_cm_values.yaml
##### OPTIONAL START
# Fetch AppCatalog YAML
kubectl get helmreleases.helm.toolkit.fluxcd.io -n flux-giantswarm appcatalog-cluster -o yaml > catalog.yaml
# Extract the AppCatalog values into a temporary file
yq eval '.spec.values.appCatalog.config.configMap.values' catalog.yaml >> tmp_cm_values.yaml
###### OPTIONAL END
# Modify the values in tmp_cm_values.yaml as needed
yq eval --inplace 'with(select(.metadata != null); .global.metadata = .metadata) |
with(select(.connectivity != null); .global.connectivity = .connectivity) |
with(select(.controlPlane != null); .global.controlPlane = .controlPlane) |
with(select(.nodePools != null); .global.nodePools = .nodePools) |
with(select(.managementCluster != null); .global.managementCluster = .managementCluster ) |
with(select(.providerSpecific != null); .global.providerSpecific = .providerSpecific) |
with(select(.baseDomain != null); .global.connectivity.baseDomain = .baseDomain) |
with(select(.managementCluster != null); .global.managementCluster = .managementCluster) |
del(.metadata) |
del(.connectivity) |
del(.controlPlane) |
del(.nodePools) |
del(.managementCluster) |
del(.baseDomain) |
del(.provider) |
del(.providerSpecific)' tmp_cm_values.yaml
# Merge the modified values back into the ConfigMap YAML
yq eval-all 'select(fileIndex==0).data.values = select(fileIndex==1) | select(fileIndex==0)' ${cluster}_cm.yaml tmp_cm_values.yaml > app.yaml
## Multi-line
sed -i '' 's/values:/values: \|/g' app.yaml
# Fetch the App YAML
kubectl get app -n org-$org $cluster -o yaml > ${cluster}_app.yaml
## Update the version of the App YAML
yq eval --inplace 'with(select(.spec.version != null); .spec.version = "0.50.0")' ${cluster}_app.yaml
# Merge the App YAML and ConfigMap YAML
echo "---" >> app.yaml
cat ${cluster}_app.yaml >> app.yaml
# Clean up
rm ${cluster}_cm.yaml
rm tmp_cm_values.yaml
rm ${cluster}_app.yaml
rm catalog.yaml
- Move Helm values property
.Values.metadata
to.Values.global.metadata
. - Move Helm values property
.Values.connectivity
to.Values.global.connectivity
. - Move Helm values property
.Values.controlPlane
to.Values.global.controlPlane
. - Move Helm values property
.Values.nodePools
to.Values.global.nodePools
. - Move Helm values property
.Values.managementCluster
to.Values.global.managementCluster
. - Move Helm values property
.Values.baseDomain
to.Values.global.connectivity.baseDomain
. - Move Helm values property
.Values.providerSpecific
to.Values.global.providerSpecific
. - Move Helm values property
.Values.global.connectivity.containerRegistries
to.Values.global.components.containerd.containerRegistries
.
- Bump the Kubernetes version to
v1.24.16
. - Bump Teleport version to
v14.1.3
. - Enable Teleport by default.
- Make Helm values configurable for aws-cloud-controller-manager, aws-ebs-csi-driver, cilium, coredns and vertical-pod-autoscaler-crd
- Expose value to configure launch template overrides, used to override the instance type specified by the launch template with multiple instance types that can be used to launch instances.
- Fixed issue when deleting node pools that would prevent the deletion, caused by the fact that
MachinePool
andAWSMachinePool
CRs were annotated with"helm.sh/resource-policy": keep
.
0.49.0 - 2023-11-23
- Change schema validation allowing to add additional properties in
global
. - Support longer node pool names and allow dashes.
- Bump cilium-app to v0.18.0 (upgrades Cilium to v1.14.3)
- Fix containerd config that was breaking in newer flatcar versions.
0.48.1 - 2023-11-13
- The value to configure the control plane load balancer ingress rules is filtered to avoid duplicates and to always contain GiantSwarm VPN IPs.
0.48.0 - 2023-11-13
- Add
global.metadata.preventDeletion
to add the deletion prevention label to cluster resources
0.47.0 - 2023-11-07
- Allow cluster-autoscaler handling MachinePools.
- Add additional tag for cluster autoscaler to MachinePool ASGs.
- Add option to force CGroups v1.
0.46.1 - 2023-11-01
- Allow configuration of
AWSCluster.spec.AdditionalTags
value and add a default giantswarm tag.
0.46.0 - 2023-10-24
- Move labels to AWSMachineTemplate manifest to avoid unnecessary rolling/no rolling.
- Add teleport.service: Secure SSH access via Teleport.
- Add
controlPlane.loadBalancerIngressAllowCidrBlocks
to configure control plane load balancer ingress rules.
- Bump
coredns
version to1.19.0
and fix values.
0.45.0 - 2023-10-04
- Add values neccessery for migration from vintage.
0.44.0 - 2023-09-28
- Make AWS instances names independent of helm label to prevent unnecessary rolling.
- Align job that cleans
HelmReleases
andHelmCharts
with other providers.
0.43.1 - 2023-09-27
- Revert to install default Cilium policies again. Some operators' "allow access to API nodes"
NetworkPolicy
s are not effective and Cilium first needs to be upgraded, including a recent upstream fix to the known issue.
0.43.0 - 2023-09-27
- Remove installation of Cilium policies that allow certain cluster traffic unconditionally (
defaultPolicies.enabled
incilium-app
). This is no longer necessary as all operators have been adapted with own network policies.
0.42.0 - 2023-09-21
- Remove
connectivity.dns.mode
andconnectivity.dns.additionalVpc
properties due dropping support for private DNS.
0.41.0 - 2023-09-19
- Accept old service account issuer URI without
https://
prefix as well. This fixes the breaking change introduced in v0.38.4. Existing service account tokens, and the operators/applications using them, will keep working even before the tokens get rotated with the new service account issuer URI. When upgrading, it is recommended to skip earlier releases and immediately jump from v0.38.3 (or older) to this one.
0.40.0 - 2023-09-18
- Add support for Spot instances.
0.39.0 - 2023-09-12
- Support creating
CiliumNetworkPolicy
manifests that allow egress requests to DNS and conditionally the proxy host (viacilium-app
)
0.38.5 - 2023-09-12
- Remove dependency between
cilium
and CPI so thatcilium
is installed as soon as possible.
0.38.4 - 2023-08-30
https://
prefix, which is a breaking change), avoiding that operators lose access to the Kubernetes API which could render the cluster unhealthy.
- Add
https://
scheme prefix to service-account-issuer URI
0.38.3 - 2023-08-29
- Fix job that removes
HelmReleases
andHelmCharts
. - Delete
HelmReleases
andHelmCharts
clean-up jobs when they are successful.
0.38.2 - 2023-08-29
- Delete all
HelmCharts
on the organization namespace that contain the cluster name on its name.
0.38.1 - 2023-08-25
- Update kubernetes version to
1.24.14
.
0.38.0 - 2023-08-24
- Add always-required values to
noProxy
list for aws-cloud-controller-manager-app and aws-ebs-csi-driver-app (only relevant for private clusters with proxy) - Forbid additional properties under
connectivity.proxy
to avoid typos
- Use fixed alias CloudFront domain for IRSA
- Tolerate CAPI taints on uninitialized nodes when scheduling cilium relay and ui.
0.37.0 - 2023-07-19
- Decrease
interval
onHelmReleases
to make things more reactive.
- Fix RBAC for
HelmReleases
clean up job.
0.36.2 - 2023-07-12
- Specify
HelmChart
type when patchingHelmCharts
in job that removes finalizers.
0.36.1 - 2023-07-11
- Fix job that removes finalizers by dropping namespace from the
HelmChart
name when using it for patching.
0.36.0 - 2023-07-10
- Remove finalizers from
HelmCharts
when removing this app to avoid leaving leftovers in the management cluster.
- Set value for
controller-manager
terminated-pod-gc-threshold
to125
( consistent with vintage )
0.35.1 - 2023-06-29
- Fix defaulting of node pool for AWSCluster CR.
0.35.0 - 2023-06-28
- Add CNI/CSI/coredns apps as HelmReleases.
0.34.0 - 2023-06-21
- Migrating from Ubuntu AMI to Flatcar AMI is a breaking change that requires manual steps.
- Use CAPBK to provision bastion node with Flatcar AMI.
- Use CAPBK to provision control plane nodes with Flatcar AMI.
- Use CAPBK to provision worker nodes with Flatcar AMI.
- Migrating from Ubuntu AMI to Flatcar AMI is a breaking change that requires manual steps.
- Apply default OS setting for flatcar and os hardening.
- Update CAPA CRs API version from
v1beta1
tov1beta2
. - Values schema: disallow additional properties on the
.nodePools
object. This is a breaking change where node pool names are in use that do not match the pattern^[a-z0-9]{5,10}$
.
0.33.0 - 2023-06-07
Note: this release includes values schema changes which break compatibility with previous versions.
- Removed
connectivity.network.podCidr
andconnectivity.network.serviceCidr
. Replaced byconnectivity.network.pods.cidrBlocks
andconnectivity.network.services.cidrBlocks
. - Remove
app.kubernetes.io/version
from common labels. They are part of hashes, but we don't want to always roll nodes just because we are deploying a new version. - Remove
architect
templating fromChart.yaml
file. - Remove control plane replicas value
controlPlane.replicas
. Now it's hardcoded to 3 nodes. - Set
r6i.xlarge
as the new default AWS instance type for the control plane and node pools. - Added value
.metadata.servicePriority
to the schema to set the cluster's relative priority. - Updated
cluster-shared
chart dependency to0.6.5
- Add JSON schema related makefile.
- generate
values.yaml
fromvalues.schema.json
withmake generate-values
- normalize
values.schema.json
withmake normalize-schema
- validate that
values.schema.json
is according to requirements withmake validate-schema
- generate
- Add full configuration values documentation.
- Add
"helm.sh/resource-policy": keep
annotation to AWSCluster, (AWS)MachineDeployments, (AWS)MachinePools and KubeadmControlPlane. The deletion of these resources has to be in order and must be handled by the CAPI and CAPA controllers.
0.32.1 - 2023-04-27
- Moved the core components feature flags to their configuration, as the
featureGates
field is forkubeadm
feature flags.
- Remove
TTLAfterFinished
because it defaults to true.
0.32.0 - 2023-04-26
- Enable
CronJobTimeZone
feature gate in the kubelet. - Set kubernetes
1.24.10
as the default version. - Switch from the in-tree cloud-controller-manager to the external one. This requires version
v0.26.0
ofdefault-apps-aws
.
- Remove old JSON schema workflow.
0.31.0 - 2023-04-24
- Rename
defaultMachinePools
tointernal.nodePools
to fit new schema requirements and make clear that it should not be changed by customers. - Default to using
giantswarm.azurecr.io
as Docker Hub mirror.
- Remove duplicate label
cluster.x-k8s.io/cluster-name
in bastion MachineDeployment.
- Remove
image-pull-progress-deadline
kubelet flag, as it's Docker only, and it's removed in k8s v1.24+.
0.30.0 - 2023-04-06
- Configure kubelet
ShutdownGracePeriod
to 5m andShutdownGracePeriodCriticalPods
to 1m. These options letkubelet
prevent a node from shutting down until it has evicted all the pods from the node. The critical pods will be removed in the last 1m of the total 5m grace period and include pods with their priorityClassName set to system-cluster-critical or system-node-critical. - Set default Node systemd logind
InhibitDelayMaxSec
to 5m.
0.29.1 - 2023-04-03
- Fix rendering
oidc.pem
by mistake when not specified
0.29.0 - 2023-03-27
- Run machine pools and control plane nodes on private subnets.
0.28.0 - 2023-03-23
Note: this release includes values schema changes which break compatibility with previous versions.
How to migrate from v0.27.0
To migrate values from cluster-aws v0.27.0, we provide below yq script, which assumes your values (not a ConfigMap!) are available in the file values.yaml
. Note that the file will be overwritten.
Also be aware that if you were using .aws.awsClusterRole
to specify a role in v0.27.0, this cannot be migrated automatically. Instead you have to make sure to have a AWSClusterRoleIdentity resource in the management cluster which specifies the identity to use. The name of that resource then has to be specified as .providerSpecific.awsClusterRoleIdentityName
in the new values for v.28.0.
yq eval --inplace '
with(select(.ami != null); .providerSpecific.ami = .ami) |
with(select(.aws.awsClusterRoleIdentityName != null); .providerSpecific.awsClusterRoleIdentityName = .aws.awsClusterRoleIdentityName) |
with(select(.aws.region != null); .providerSpecific.region = .aws.region) |
with(select(.bastion != null); .connectivity.bastion = .bastion) |
with(select(.clusterDescription != null); .metadata.description = .clusterDescription) |
with(select(.clusterName != null); .metadata.name = .clusterName) |
with(select(.flatcarAWSAccount != null); .providerSpecific.flatcarAwsAccount = .flatcarAWSAccount) |
with(select(.hashSalt != null); .internal.hashSalt = .hashSalt) |
with(select(.kubernetesVersion != null); .internal.kubernetesVersion = .kubernetesVersion) |
with(select(.machinePools != null); .nodePools = .machinePools) |
with(select(.network.apiMode != null); .controlPlane.apiMode = .network.apiMode) |
with(select(.network.availabilityZoneUsageLimit != null); .connectivity.availabilityZoneUsageLimit = .network.availabilityZoneUsageLimit) |
with(select(.network.dnsAssignAdditionalVPCs != null); .connectivity.dns.additionalVpc = (.network.dnsAssignAdditionalVPCs | split(","))) |
with(select(.network.dnsMode != null); .connectivity.dns.mode = .network.dnsMode) |
with(select(.network.podCIDR != null); .connectivity.network.podCidr = .network.podCIDR) |
with(select(.network.prefixListID != null); .connectivity.topology.prefixListId = .network.prefixListID) |
with(select(.network.resolverRulesOwnerAccount != null); .connectivity.dns.resolverRulesOwnerAccount = .network.resolverRulesOwnerAccount) |
with(select(.network.serviceCIDR != null); .connectivity.network.serviceCidr = .network.serviceCIDR) |
with(select(.network.subnets != null); .connectivity.subnets = .network.subnets) |
with(select(.network.topologyMode != null); .connectivity.topology.mode = .network.topologyMode) |
with(select(.network.transitGatewayID != null); .connectivity.topology.transitGatewayId = .network.transitGatewayID) |
with(select(.network.vpcCIDR != null); .connectivity.network.vpcCidr = .network.vpcCIDR) |
with(select(.network.vpcEndpointMode != null); .connectivity.vpcEndpointMode = .network.vpcEndpointMode) |
with(select(.network.vpcMode != null); .connectivity.vpcMode = .network.vpcMode) |
with(select(.oidc != null); .controlPlane.oidc = .oidc) |
with(select(.organization != null); .metadata.organization = .organization) |
with(select(.proxy.enabled != null); .connectivity.proxy.enabled = .proxy.enabled) |
with(select(.proxy.http_proxy != null); .connectivity.proxy.httpProxy = .proxy.http_proxy) |
with(select(.proxy.https_proxy != null); .connectivity.proxy.httpsProxy = .proxy.https_proxy) |
with(select(.proxy.no_proxy != null); .connectivity.proxy.noProxy = .proxy.no_proxy) |
with(select(.sshSSOPublicKey != null); .connectivity.sshSsoPublicKey = .sshSSOPublicKey) |
del(.ami) |
del(.aws) |
del(.bastion) |
del(.clusterDescription) |
del(.clusterName) |
del(.flatcarAWSAccount) |
del(.hashSalt) |
del(.includeClusterResourceSet) |
del(.kubernetesVersion) |
del(.machinePools) |
del(.network) |
del(.oidc) |
del(.organization) |
del(.proxy) |
del(.releaseVersion) |
del(.sshSSOPublicKey)
' ./values.yaml
- Values schema:
- Added annotations
- Applied normalization using
schemalint normalize
- Added property schema for /connectivity/containerRegistries
- Added property schema for subnetTags objects
- Added default values
- Move /ami to /providerSpecific/ami
- Move /awsClusterRoleIdentityName to /providerSpecific/awsClusterRoleIdentityName
- Move /region to /providerSpecific/region
- Move /flatcarAWSAccount to /providerSpecific/flatcarAwsAccount
- Move /clusterName to /metadata/name
- Move /clusterDescription to /medatada/description
- Move /organization to /metadata/organization
- Move /oidc to /controlPlane/oidc
- Move /bastion to /connectivity/bastion
- Move /network/serviceCIDR to /connectivity/network/serviceCidr
- Move /network/podCIDR to /connectivity/network/podCidr
- Move /proxy to /connectivity/proxy
- Rename /proxy/no_proxy to /connectivity/proxy/noProxy
- Rename /proxy/http_proxy to /connectivity/proxy/httpProxy
- Rename /proxy/https_proxy to /connectivity/proxy/httpsProxy
- Move /sshSSOPublicKey to /connectivity/sshSsoPublicKey
- Remove unused /includeClusterResourceSet
- Remove /aws/awsClusterRole (previously deprecated)
- Move /hashSalt to /internal/hashSalt
- Move /kubernetesVersion to /internal/kubernetesVersion
- Move /network/dnsMode to /connectivity/dns/mode
- Move /network/dnsAssignAdditionalVPCs to /connectivity/dns/additionalVpc and change to type array
- Move /network/vpcCIDR to /connectivity/network/vpcCidr
- Move /network/apiMode to /controlPlane/apiMode
- Move /network/resolverRulesOwnerAccount to /connectivity/dns/resolverRulesOwnerAccount
- Move /network/prefixListID to /connectivity/topology/prefixListId
- Move /network/topologyMode to /connectivity/topology/mode
- Move /network/transitGatewayID to /connectivity/topology/transitGatewayId
- Move /network/vpcEndpointMode to /connectivity/vpcEndpointMode
- Move /network/vpcMode to /connectivity/vpcMode
- Move /network/availabilityZoneUsageLimit to /connectivity/availabilityZoneUsageLimit
- Move /network/subnets to /connectivity/subnets
- Rename /machinePools to /nodePools
- Disallow additional properties on the root level
- Values schema:
- Add /managementCluster and /provider to account for values injected by controllers.
- Use region defaulting wherever possible, removing
region
from schema.
0.27.0 - 2023-03-01
- Remove unused
releaseVersion
setting fromvalues.yaml
.
0.26.0 - 2023-03-01
- Add
MachineHealthCheck
for control plane nodes.
- Fail in Helm template if
dnsMode=public
is combined with abaseDomain
ending with.internal
.
0.25.1 - 2023-02-16
- Quote bastion subnet tag filters in order to avoid type conversion errors.
0.25.0 - 2023-02-16
- Replaced
registry
parameter toconnectivity.containerRegistries
in the values schema.
- Quote subnet tag filters in order to avoid type conversion errors.
- Made registry configurations
connectivity.containerRegistries
dynamic to accept as many container registries and mirrors as needed. - Expose helm value for customers to decide whether VPC endpoint should be created by Giantswarm.
- Set
/var/lib/kubelet
permissions to0750
to fixnode-exporter
issue.
0.24.1 - 2023-02-07
- Customize tags per individual subnet.
0.24.0 - 2023-02-02
- Use object for
.machinePools
schema instead of array. This is to make it easier to overwrite values when using GitOps. For migration steps see the "Upgrading tov0.24.0
" section in the readme.
0.23.0 - 2023-02-01
- Add value to specify which AWS account ID to use when associating Route53 Resolver Rules with workload cluster VPC.
0.22.0 - 2023-01-24
- Bump kubernetes version to
1.23.16
0.21.0 - 2023-01-19
- For private clusters, where
network.vpcMode
is set toprivate
, the subnets property has changed. Instead of previously being a list of CIDR strings the property now include a more complex object providing more configuration options. For migration steps see the "Upgrading tov0.21.0
" section in the readme.
- More configuration options when defining subnets to be created
controlPlane.subnetTags
,bastion.subnetTags
andmachinePools[].subnetTags
to target specific subnets- Add icon to Chart.yaml
- Subnets are now specified on the
AWSCluster
resource by default rather than relying on CAPA code to default them. The same sizing as the CAPA default have been used.
0.20.7 - 2023-01-12
- Use Giant Swarm image repository for official Kubernetes images
0.20.6 - 2023-01-11
- Add and propagate
no_proxy
value to the underlying components.
0.20.5 - 2023-01-11
- Override image repository to
registry.k8s.io
because kubeadm of Kubernetes v1.23.15 tries to pull the official image incorrectly, resulting in failing cluster upgrades, andk8s.gcr.io
is outdated
0.20.4 - 2023-01-05
- Change default NTP server as AWS NTP server.
- Deprecate confusingly named
aws.awsClusterRole
in favor ofaws.awsClusterRoleIdentityName
. The value refers to anAWSClusterRoleIdentity
object, not directly to an IAM role name/ARN. - Bump Kubernetes to 1.23.15
0.20.3 - 2022-12-22
- Add cluster base domain to no proxy config.
0.20.2 - 2022-12-09
- Dowgrade to using Ubuntu 20.04 as base OS.
- Run bastion on private IP if vpc mode is set to private.
- Remove registry authetication workaround.
0.20.1 - 2022-12-07
0.20.0 - 2022-12-06
- Add schema for items of the arrays
.machinePools[*].availabilityZones
and.machinePools[*].customNodeTaints
. - Add IRSA domain placeholder replacer as postKubeadm script.
- Add
containerd
registry auth workaround to bug giantswarm/roadmap#1737.
0.19.0 - 2022-11-29
- Add option to specify oidc CA PEM in order to autheticate againts OIDC with custom CA.
- Add option to configure containerd registry authentication for
docker.io
.
0.18.0 - 2022-11-24
- Add external resource gc annotation.
- Change sed to fix replacement for Cloudfront placeholder.
- Added missing prefixListID for UserManaged network topology
- Make
baseDomain
a required value.
0.17.1 - 2022-11-22
- Add
https://
for IRSA service account issuer.
0.17.0 - 2022-11-18
- Add full proxy configuration for private clusters.
0.16.1 - 2022-11-15
- Allow scraping of k8s core components.
- Bump external-dns to latest release
0.16.0 - 2022-11-10
- Make
kubeadm
skip the phase where it installscoredns
as it will be installed by as a default app.
0.15.2 - 2022-11-08
- Bumped cluster-shared to latest with coredns-adopter apiserver polling
0.15.1 - 2022-11-07
- Handle default values in worker machine pool values
0.15.0 - 2022-11-07
- Support setting node taints using
customNodeTaints
0.14.0 - 2022-11-03
- Bumped Kubernetes to v1.23
- Immutable AWSMachineTemplate
0.13.2 - 2022-11-03
- Ensure the
KubeadmControlPlane
.spec.version
value is always prefixed withv
0.13.1 - 2022-10-27
- Add the missing
api-audiences
attribute to theKubeadmControlPlane
template, to fix the use of IRSA service account tokens.
- Update cluster-shared from v0.3.0 to v0.6.3.
0.13.0 - 2022-10-19
- Make
kubeadm
skip the phase where it installskube-proxy
as we will usecilium
as a replacement.
0.12.0 - 2022-10-14
- IRSA for CAPA.
- Make subnets configurable.
- Re-added Ubuntu 22.04 with correct lookup
- Enable tcp forwarding for sshd on bastion.
0.11.1 - 2022-10-14
- Rolled back to Ubuntu 20.04
0.11.0 - 2022-10-14
- Set
aws.giantswarm.io/vpc-mode
annotation on AWSCluster. - Set cluster to paused when vpcMode is set to private.
- Updated to Kubernetes 1.22.15
- Updated to using Ubuntu 22.04 as base OS
0.10.0 - 2022-10-04
.Values.controlPlane.apiLoadbalancerScheme
has been removed in favour of.Values.network.apiMode
- Support for specifying private VPC configuration (not yet used)
- Support for specifying private DNS zone configuration.
- Validation of vpcMode and apiMode combination being valid
0.9.2 - 2022-09-16
- Default network topology mode changed to 'None'
0.9.1 - 2022-09-06
- Fix helm context for proxy helper function.
0.9.0 - 2022-09-06
- Add support for configuring outgoing proxy for the cluster.
- Allow configuration of loadbalancer for Control Plane API (
internet-facing
will be default).
0.8.7 - 2022-08-26
- Improved hash function to hash based on whole
.Spec
rather than just provided values
0.8.6 - 2022-08-23
- AZ list rendering
0.8.5 - 2022-08-17
- Network topology mode annotations
- Add role label to bastion machine.
0.8.4 - 2022-08-17
- Ensure availability zone restrictions are added to the subnet filters
0.8.3 - 2022-08-15
- Fix subnet filter to relevant with
tag:
prefix.
0.8.1 - 2022-08-15
- Limit subnet filter to relevant, cluster owned, subnets
0.8.0 - 2022-08-15
hash
function to ensure immutable resources change be changed via recreate/replacement
0.7.4 - 2022-08-11
0.7.3 - 2022-08-11
- Ensure worker nodes are only launched in private subnets
0.7.2 - 2022-08-11
- Add OIDC support for k8s api.
0.7.1 - 2022-08-09
- Added the OS version to the imageLookupBaseOS
0.7.0 - 2022-08-09
- Use our Giant Swarm built AMIs
- Bump default Kubernetes version to 1.22.12
0.6.2 - 2022-08-06
- Fixed app version label.
0.6.1 - 2022-08-03
- Add
localhost
andapi
domain to the certSANs of apiserver certificates.
0.6.0 - 2022-07-28
replicas
value fromcontrolPlane
no longer configurable - always set to 3 for HA
0.5.2 - 2022-07-26
- Quoted boolean to a string
0.5.1 - 2022-07-26
- Pod CIDR as array rather than string
0.5.0 - 2022-07-26
- Set pod CIDR to 100.64.0.0/12 to match what we set in Cilium (and to not clash with AWS CIDR)
0.4.2 - 2022-07-25
- Fix values schema.
- Make bastion optional.
0.4.1 - 2022-07-15
- Add team label to helm resources.
- Add
values.schema.json
file. - Remove helm lookup function for SSH CA cert and use value fro central vault instead.
0.4.0 - 2022-04-14
- Updated to latest
cluster-shared
library chart
- Support for specifying the
clusterName
(defaults to chart release name)
0.3.0 - 2022-04-12
- Switched to using
cluster-shared
for PSPs and coredns-adopter
0.2.1 - 2022-03-31
- Lookup AWS region if not set in values
- Lookup AWS Availability Zones if not set in values
0.2.0 - 2022-03-29
- Allow app platform to take over managing coredns
0.1.14 - 2022-03-22
0.1.13 - 2022-03-21
- Rename
networkSpec
tonetwork
in AWSCluster CR due renaming inv1beta1
.
0.1.12 - 2022-03-18
- Prefix machine pool with cluster id.
- Set etcd max db size to 8 GB.
- Add encryption provider config for k8s secrets.
0.1.11 - 2022-03-15
- Add
audit-policy
to kubernetes api. - Fix AWSMachinePool min and max values.
0.1.10 - 2022-03-09
0.1.9 - 2022-03-07
- Upgrade to
vbeta1
version for all CRs.
0.1.8 - 2022-03-07
0.1.7 - 2022-03-07
0.1.6 - 2022-03-07
0.1.5 - 2022-03-04
- Remove
AWSClusterRole
CR from the repository to prevent deletion of the role before the cluster is deleted.
0.1.4 - 2022-03-03
- Add labels to machine metadata to
AWSMachineTemplate
CRs.
0.1.3 - 2022-03-02
- Add
sourceIdenityRef
to AWSClusterRoleIdentity CR.
0.1.2 - 2022-02-25
- Fix aws cluster role identity value reference.
0.1.1 - 2022-02-25
- Fix bastion secret.