Merge branch 'main' into feat/opentofu

doc/Upgrade-Guide.md

@@ -10,8 +10,9 @@ state: Draft (v0.7)
 This document explains the steps to upgrade the SCS Kubernetes cluster-API
 based cluster management solution as follows:
 - from the R2 (2022-03) to the R3 (2022-09) state
-- from the R3 (2022-09) to the R4 state
-- from the R4 (2023-09) to the R5 state
+- from the R3 (2022-09) to the R4 (2023-03) state
+- from the R4 (2023-03) to the R5 (2022-09) state
+
 The document explains how the management cluster and the workload clusters can be
 upgraded without disruption. It is highly recommended to do a step-by-step upgrade
 across major releases i.e. upgrade from R2 to R3 and then to R4 in the case of
@@ -23,22 +24,22 @@ take, and it is advisable that cluster operators get some experience with
 this kind of cluster management before applying this to customer clusters
 that carry important workloads.
 
-Note that while the detailed steps are tested and targeted to a R2 -> R3 move,
+Note that while the detailed steps are tested and targeted to an R2 -> R3 move,
 R3 -> R4 move or R4 -> R5 move, many of the steps are a generic approach that will apply also for other
 upgrades, so expect a lot of similar steps when moving beyond R5.
 
-Upgrades from cluster management prior to R2 is difficult; many changes before
+Upgrades from cluster management prior to R2 are difficult; many changes before
 R2 assumed that you would redeploy the management cluster. Redeploying the
 management cluster can of course always be done, but it's typically disruptive
 to your workload clusters, unless you move your cluster management state into
 a new management cluster with `clusterctl move`.
 
 ## Management host (cluster) vs. Workload clusters
 
-When you initially deployed the SCS k8s-cluster-api-provider, you create a
+When you initially deployed the SCS k8s-cluster-api-provider, you created a
 VM with a [kind](https://kind.sigs.k8s.io/) cluster inside and a number of
 templates, scripts and binaries that are then used to do the cluster management.
-This is your management host (or more precisely you single-host management
+This is your management host (or more precisely your single-host management
 cluster). Currently, all cluster management including upgrading etc. is done
 by connecting to this host via ssh and performing commands there. (You don't
 need root privileges to do cluster management there, the normal ubuntu user
@@ -134,8 +135,11 @@ You can now apply the upgrade by executing the following command:
 clusterctl upgrade apply --contract v1beta1
 ```
 
-You can then upgrade the components. You can do them one-by-one or simply do
-`clusterctl upgrade apply --contract v1beta1`
+You can then upgrade the components. You can do them one-by-one, e.g.:
+```bash
+clusterctl upgrade apply --infrastructure capo-system/openstack:v0.7.3 --core capi-system/cluster-api:v1.5.1 -b capi-kubeadm-bootstrap-system/kubeadm:v1.5.1 -c capi-kubeadm-control-plane-system/kubeadm:v1.5.1
+```
+Or simply do `clusterctl upgrade apply --contract v1beta1`.
 
 #### New templates
 
@@ -337,11 +341,11 @@ If you decide to migrate your existing Kubernetes cluster from R4 to R5 be aware
 
 Follow the below steps if you want to migrate an existing cluster from R4 to R5:
 1. Access your management node
-2. Checkout R5 tag
+2. Checkout R5 branch
    ```bash
    cd k8s-cluster-api-provider
    git pull
-   git checkout tags/v6.0.0
+   git checkout maintained/v6.x
    ```
 3. Backup an existing cluster configuration files (recommended)
    ```bash
@@ -357,7 +361,7 @@ Follow the below steps if you want to migrate an existing cluster from R4 to R5:
    and are not directly mentioned in the cluster configuration files, but they are hardcoded
    in R5 scripts (e.g. ingress nginx controller, metrics server). Hence, read carefully the
    R5 release notes too. Also see that Kubernetes version was not updated, and it is still v1.25.6.
-6. Update an existing cluster (expect Kubernetes version)
+6. Update an existing cluster (except Kubernetes version)
    ```bash
    create_cluster.sh <CLUSTER_NAME>
    ```
@@ -380,7 +384,7 @@ Follow the below steps if you want to migrate an existing cluster from R4 to R5:
 10. Bump Kubernetes version to R5 v1.27.5 and increase the generation counter for node and control plane nodes
    ```bash
    sed -i 's/^KUBERNETES_VERSION: v1.26.8/KUBERNETES_VERSION: v1.27.5/' <CLUSTER_NAME>/clusterctl.yaml
-   sed -i 's/^OPENSTACK_IMAGE_NAME: ubuntu-capi-image-v1.26.8 /OPENSTACK_IMAGE_NAME: ubuntu-capi-image-v1.27.5/' <CLUSTER_NAME>/clusterctl.yaml
+   sed -i 's/^OPENSTACK_IMAGE_NAME: ubuntu-capi-image-v1.26.8/OPENSTACK_IMAGE_NAME: ubuntu-capi-image-v1.27.5/' <CLUSTER_NAME>/clusterctl.yaml
    sed -r 's/(^CONTROL_PLANE_MACHINE_GEN: genc)([0-9][0-9])/printf "\1%02d" $((\2+1))/ge' -i <CLUSTER_NAME>/clusterctl.yaml
    sed -r 's/(^WORKER_MACHINE_GEN: genw)([0-9][0-9])/printf "\1%02d" $((\2+1))/ge' -i <CLUSTER_NAME>/clusterctl.yaml
    ```
@@ -395,7 +399,7 @@ OCCM, CNI (calico/cilium), CSI
 
 ### New versions for optional components
 
-nginx, metrics (nothing to do), cert-manager, flux
+nginx, metrics server, cert-manager, flux
 
 ### etcd leader changes
 

doc/configuration.md

@@ -21,8 +21,8 @@ Parameters controlling the Cluster-API management server (capi management server
 | `kind_flavor`            |                 | SCS        | `SCS-2V-4`        | Flavor to be used for the k8s capi mgmt server                                                             |
 | `image`                  |                 | SCS        | `Ubuntu 22.04`    | Image for the capi mgmt server. Use `Ubuntu 22.04` or `Debian 12`. Check also the `ssh_username` parameter |
 | `ssh_username`           |                 | SCS        | `ubuntu`          | Name of the default user for the `image`                                                                   |
-| `clusterapi_version`     |                 | SCS        | `1.5.3` <!-- renovate: datasource=github-releases depName=kubernetes-sigs/cluster-api -->          | Version of the cluster-API incl. `clusterctl`                                                              |
-| `capi_openstack_version` |                 | SCS        | `0.8.0` <!-- renovate: datasource=github-releases depName=kubernetes-sigs/cluster-api-provider-openstack -->         | Version of the cluster-api-provider-openstack (needs to fit the CAPI version)                              |
+| `clusterapi_version`     |                 | SCS        | `1.6.1` <!-- renovate: datasource=github-releases depName=kubernetes-sigs/cluster-api -->          | Version of the cluster-API incl. `clusterctl`                                                              |
+| `capi_openstack_version` |                 | SCS        | `0.9.0` <!-- renovate: datasource=github-releases depName=kubernetes-sigs/cluster-api-provider-openstack -->         | Version of the cluster-api-provider-openstack (needs to fit the CAPI version)                              |
 | `cilium_binaries`        |                 | SCS        | `v0.15.7;v0.12.0` | Versions of the cilium and hubble CLI in the vA.B.C;vX.Y.Z format                                          |
 | `restrict_mgmt_server` |                                 | SCS        | `["0.0.0.0/0"]`                      | Allows restricting access to the management server by the given list of CIDRs. Empty value (default) means public.           |
 | `mgmt_cidr`              |                 | SCS        | `10.0.0.0/24` | IPv4 address range (CIDR notation) for management cluster                                          |

terraform/cleanup/cleanup.sh

@@ -209,7 +209,7 @@ while read LB FIP; do
 		if test -z "$DEBUG"; then $OPENSTACK floating ip delete $FID; fi
 	fi
 done < <(echo "$LBS")
-SRV=$(resourcelist server "$CAPIPRE-$CLUSTER\|$CLUSTER-control-plane")
+SRV=$(resourcelist server "$CAPIPRE-$CLUSTER\|$CLUSTER-")
 SRVVOL=$(server_vols $SRV)
 if test -n "$DEBUG"; then echo "### Attached volumes to "${SRV}": $SRVVOL"; fi
 #cleanup server $CAPIPRE-$CLUSTER
@@ -219,7 +219,7 @@ if test -n "$NOCASCADE"; then
 else
 	cleanup_list loadbalancer 1 "--cascade" "$LBS"
 fi
-cleanup port "$CAPIPRE-$CLUSTER\|$CLUSTER-control-plane"
+cleanup port "$CAPIPRE-$CLUSTER\|$CLUSTER-"
 RTR=$(resourcelist router "$CAPIPRE2ALL")
 SUBNETS=$(resourcelist subnet "$CAPIPRE2ALL")
 if test -n "$RTR" -a -n "$SUBNETS"; then
@@ -245,7 +245,7 @@ cleanup_list volume "" "" "$SRVVOL"
 #cleanup "image" ubuntu-capi-image
 cleanup "server group" "$CAPIPRE-$CLUSTER"
 # Normally, the volumes should be all gone, but if there's one left, take care of it
-cleanup volume "$CAPIPRE-$CLUSTER\|$CLUSTER-control-plane"
+cleanup volume "$CAPIPRE-$CLUSTER\|$CLUSTER-"
 cleanup "application credential" "$CAPIPRE-$CLUSTER-appcred"
 cleanup container "$CAPIPRE-$CLUSTER-harbor-registry"
 

terraform/files/bin/apply_cindercsi.sh

@@ -34,32 +34,29 @@ else
 fi
 echo "# Install Cinder CSI persistent storage support $CCSI_VERSION to $CLUSTER_NAME"
 
-if test -n "$SNAP_VERSION"; then
-  # deploy snapshot CRDs
-  for name in snapshot.storage.k8s.io_volumesnapshotcontents.yaml snapshot.storage.k8s.io_volumesnapshotclasses.yaml snapshot.storage.k8s.io_volumesnapshots.yaml; do
-    NAME=${name%.yaml}-$SNAP_VERSION.yaml
-    if ! test -s $NAME; then
-	curl -L https://raw.githubusercontent.com/kubernetes-csi/external-snapshotter/$SNAP_VERSION/client/config/crd/$name -o $NAME
-	echo -e "\n---" >> $NAME
-    fi
-  done
-  # FIXME: Should we ignore non-working snapshots?
-  cat snapshot.storage.k8s.io_volumesnapshot* > cindercsi-snapshot-$SNAP_VERSION.yaml
+SNAP_VERSION="master"
+# deploy snapshot CRDs
+for name in snapshot.storage.k8s.io_volumesnapshotcontents.yaml snapshot.storage.k8s.io_volumesnapshotclasses.yaml snapshot.storage.k8s.io_volumesnapshots.yaml; do
+  NAME=${name%.yaml}-$SNAP_VERSION.yaml
+  if ! test -s $NAME; then
+    curl -L https://raw.githubusercontent.com/kubernetes-csi/external-snapshotter/$SNAP_VERSION/client/config/crd/$name -o $NAME
+    echo -e "\n---" >> $NAME
+  fi
+done
+# FIXME: Should we ignore non-working snapshots?
+cat snapshot.storage.k8s.io_volumesnapshot* > cindercsi-snapshot-$SNAP_VERSION.yaml
 
-  # deploy snapshot controller
-  for name in rbac-snapshot-controller.yaml setup-snapshot-controller.yaml; do
-    NAME=${name%.yaml}-$SNAP_VERSION.yaml
-    if ! test -s $NAME; then
-      curl -L https://raw.githubusercontent.com/kubernetes-csi/external-snapshotter/$SNAP_VERSION/deploy/kubernetes/snapshot-controller/$name -o $NAME
-      echo -e "\n---" >> $NAME
-    fi
-    cat $NAME >> cindercsi-snapshot-$SNAP_VERSION.yaml
-  done
+# deploy snapshot controller
+for name in rbac-snapshot-controller.yaml setup-snapshot-controller.yaml; do
+  NAME=${name%.yaml}-$SNAP_VERSION.yaml
+  if ! test -s $NAME; then
+    curl -L https://raw.githubusercontent.com/kubernetes-csi/external-snapshotter/$SNAP_VERSION/deploy/kubernetes/snapshot-controller/$name -o $NAME
+    echo -e "\n---" >> $NAME
+  fi
+  cat $NAME >> cindercsi-snapshot-$SNAP_VERSION.yaml
+done
 
-  cp -p cindercsi-snapshot-$SNAP_VERSION.yaml ~/${CLUSTER_NAME}/deployed-manifests.d/cindercsi-snapshot.yaml
-else
-  cp -p external-snapshot-crds.yaml ~/$CLUSTER_NAME/deployed-manifests.d/cindercsi-snapshot.yaml
-fi
+cp -p cindercsi-snapshot-$SNAP_VERSION.yaml ~/${CLUSTER_NAME}/deployed-manifests.d/cindercsi-snapshot.yaml
 
 if test -n "$CCSI_VERSION"; then
   # Now get cinder

terraform/files/bin/install_k9s.sh

@@ -5,7 +5,7 @@
 . /etc/profile.d/proxy.sh
 
 # install k9s
-K9S_VERSION=v0.30.6 # renovate: datasource=github-releases depName=derailed/k9s
+K9S_VERSION=v0.31.7 # renovate: datasource=github-releases depName=derailed/k9s
 echo "# install k9s $K9S_VERSION"
 ARCH=$(uname -m | sed 's/x86_64/amd64/')
 # TODO: Check signature

terraform/files/bin/openstack-kube-versions.inc

@@ -3,13 +3,13 @@
 # (c) Kurt Garloff <[email protected]>, 3/2022
 # SPDX-License-Identifier: Apache-2.0
 # Images from https://swift.services.a.regiocloud.tech/swift/v1/AUTH_b182637428444b9aa302bb8d5a5a418c/openstack-k8s-capi-images
-k8s_versions=("v1.18.20" "v1.19.16" "v1.20.15" "v1.21.14" "v1.22.17" "v1.23.16" "v1.24.15" "v1.25.15" "v1.26.11" "v1.27.8" "v1.28.4")
+k8s_versions=("v1.21.14" "v1.22.17" "v1.23.16" "v1.24.15" "v1.25.15" "v1.26.11" "v1.27.8" "v1.28.4")
 # OCCM, CCM-RBAC, Cinder CSI, Cinder-Snapshot (TODO: Manila CSI)
-occm_versions=(""         ""       "v1.21.1"   "v1.21.1"  "v1.22.2"  "v1.23.4"  "v1.24.6" "v1.25.6" "v1.26.4" "v1.27.3" "v1.28.1")
-#ccmr_versions=(""         ""        ""         ""         "v1.22.2"  "v1.23.4"  "v1.24.6" "v1.25.6" "v1.26.4" "v1.27.3" "v1.28.1")
-ccmr_versions=(""         ""        "v1.22.2"  "v1.22.2"  "v1.22.2"  "v1.23.4"  "v1.24.6" "v1.25.6" "v1.26.4" "v1.27.3" "v1.28.1")
-ccsi_versions=(""         ""        "v1.20.5"  "v1.21.1"  "v1.22.2"  "v1.23.4"  "v1.24.6" "v1.25.6" "v1.26.4" "v1.27.3" "v1.28.1")
-min_snapshot_master="v1.21.0"
+occm_versions=("v1.21.1"  "v1.22.2"  "v1.23.4"  "v1.24.6" "v1.25.6" "v1.26.4" "v1.27.3" "v1.28.1")
+#ccmr_versions=(""         "v1.22.2"  "v1.23.4"  "v1.24.6" "v1.25.6" "v1.26.4" "v1.27.3" "v1.28.1")
+ccmr_versions=("v1.22.2"  "v1.22.2"  "v1.23.4"  "v1.24.6" "v1.25.6" "v1.26.4" "v1.27.3" "v1.28.1")
+ccsi_versions=("v1.21.1"  "v1.22.2"  "v1.23.4"  "v1.24.6" "v1.25.6" "v1.26.4" "v1.27.3" "v1.28.1")
+
 # Versions that require a --allow-preview-versions flag
 techprev_versions=("v1.28" "v2")
 
@@ -83,8 +83,6 @@ find_openstack_versions()
 	NUMV=${#k8s_versions[*]}
 	k8min=$(dotversion ${k8s_versions[0]%.*})
 	k8max=$(dotversion ${k8s_versions[$((NUMV-1))]%.*})
-	snapmaster=$(dotversion $min_snapshot_master)
-	if test $k8vers -lt $snapmaster; then SNAP_VERSION=""; else SNAP_VERSION="master"; fi
 	#echo "$k8vers $k8min $k8max"
 	if test $k8vers -lt $k8min; then setversions 0; return 0; fi
 	if test $k8vers -gt $((k8max+99)); then setversions $((NUMV-1)); return 0; fi

terraform/files/bin/update-R4-to-R5.sh

@@ -46,8 +46,7 @@ cp -p cluster-template.yaml cluster-template.yaml.backup
 cp -p clusterctl.yaml clusterctl.yaml.backup
 
 # Update cluster-template.yaml
-# TODO: Update `update-cluster-template.diff` file when R5 release will be stabilized
-if grep -q "SERVICE_CIDR\|POD_CIDR\|# Defragment & backup & trim script for SCS k8s-cluster-api-provider etcd cluster.\|# Allow to configure registry hosts in containerd" cluster-template.yaml; then
+if grep -q "SERVICE_CIDR\|POD_CIDR\|# Defragment & backup & trim script for SCS k8s-cluster-api-provider etcd cluster.\|# Allow to configure registry hosts in containerd\|tweak-kubeapi-memlimit" cluster-template.yaml; then
   echo "cluster-template.yaml already updated"
 else
   # The default template file `cluster-defaults/cluster-template.yaml` of version R4 still references the old `k8s.gcr.io` container registry.