From b384331c36d136d284d62bca82178b949f951f50 Mon Sep 17 00:00:00 2001 From: Noah Merlis-Stephens Date: Sat, 23 Nov 2024 13:58:11 -0800 Subject: [PATCH 01/10] Add etcd configuration and custom flag migration --- .../docs/configuration/kubernetes/_index.md | 1 + content/docs/configuration/kubernetes/etcd.md | 17 ++++++++ content/docs/migrate-from-MKE-3.md | 42 +++++++++++++++++++ 3 files changed, 60 insertions(+) create mode 100644 content/docs/configuration/kubernetes/etcd.md diff --git a/content/docs/configuration/kubernetes/_index.md b/content/docs/configuration/kubernetes/_index.md index 830ca568..c658e7d7 100644 --- a/content/docs/configuration/kubernetes/_index.md +++ b/content/docs/configuration/kubernetes/_index.md @@ -9,6 +9,7 @@ MKE 4 uses K0s to deploy core Kubernetes components, including: - [kube-apiserver](../../configuration/kubernetes/kube-apiserver) - [kube-controller-manager](../../configuration/kubernetes/kube-controller-manager) - [kube-scheduler](../../configuration/kubernetes/kube-scheduler) +- [etcd](../../configuration/kubernetes/etcd) You can configure all of the Kubernetes components through the MKE configuration file. \ No newline at end of file diff --git a/content/docs/configuration/kubernetes/etcd.md b/content/docs/configuration/kubernetes/etcd.md new file mode 100644 index 00000000..f54e8bca --- /dev/null +++ b/content/docs/configuration/kubernetes/etcd.md @@ -0,0 +1,17 @@ +--- +title: etcd +weight: 2 +--- + +etcd is a consistent and highly-available key value store used as Kubernetes' backing store for all cluster data. + +You can find in-depth information about etcd in the official [documentation](https://etcd.io/docs/). + +You can configure etcd through the `etcd` section of the MKE configuration file, an example of which follows: + +```yaml +spec: + etcd: + storageQuota: 2GB + tlsCipherSuites: TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 +``` diff --git a/content/docs/migrate-from-MKE-3.md b/content/docs/migrate-from-MKE-3.md index 1b12eb0e..b85be51e 100644 --- a/content/docs/migrate-from-MKE-3.md +++ b/content/docs/migrate-from-MKE-3.md @@ -85,6 +85,48 @@ mkectl init --mke3-config ensure you complete the [GPU prerequisites](/docs/configuration/gpu/#prerequisites) before starting the upgrade process. {{< /callout >}} +### Kubernetes Custom Flags + +MKE 3 supports applying additional flags to Kubernetes components with the following fields in the configuration file, +each specified as a list of strings. +``` +custom_kube_api_server_flags +custom_kube_controller_manager_flags +custom_kubelet_flags +custom_kube_scheduler_flags +custom_kube_proxy_flags +``` + +MKE 4 supports an 'extraArgs' field for each of these components, accepting a map of key-value pairs. +MKE 4 will convert these custom flags to the corresponding `extraArgs` field during the upgrade. +Any flags that cannot be automatically converted will be listed in the upgrade summary. + +#### Example of custom flags conversion + +MKE 3 configuration file: +``` +[cluster_config.custom_kube_api_server_flags] = ["--enable-garbage-collector=false"] +``` + +MKE 4 configuration file: +``` +spec: + apiServer: + extraArgs: + enable-garbage-collector: false +``` + +### Kubelet Custom Flag Profiles + +MKE 3 supports a map of kubelet flag profiles to specific nodes using the `custom_kubelet_flags_profiles` setting in the toml configuration file. + +MKE 4 does not support kubelet flag profiles, but you can map `KubeletConfiguration` values to specific +nodes using [Kubelet custom profiles](../configuration/kubernetes/kubelet.md#kubelet-custom-profiles). +MKE 4 supports migration of MKE 3 kubelet flag profiles to kubelet custom profiles. +Conversion of flags to `KubeletConfiguration` values is best-effort and any flags that cannot be +converted are listed in the upgrade summary. Hosts with a custom flag profile label are marked for the +corresponding kubelet custom profile. + ## Perform the migration An upgrade from MKE 3 to MKE 4 consists of the following steps, all of which From 001acd9c0ce0a0d4252aa4d2315ca3ad6c4b330d Mon Sep 17 00:00:00 2001 From: Noah Merlis-Stephens Date: Wed, 11 Dec 2024 10:15:52 -0800 Subject: [PATCH 02/10] Update content/docs/migrate-from-MKE-3.md Co-authored-by: Kory <57411706+KoryKessel-Mirantis@users.noreply.github.com> --- content/docs/migrate-from-MKE-3.md | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/content/docs/migrate-from-MKE-3.md b/content/docs/migrate-from-MKE-3.md index b85be51e..1ec5bb62 100644 --- a/content/docs/migrate-from-MKE-3.md +++ b/content/docs/migrate-from-MKE-3.md @@ -97,9 +97,7 @@ custom_kube_scheduler_flags custom_kube_proxy_flags ``` -MKE 4 supports an 'extraArgs' field for each of these components, accepting a map of key-value pairs. -MKE 4 will convert these custom flags to the corresponding `extraArgs` field during the upgrade. -Any flags that cannot be automatically converted will be listed in the upgrade summary. +MKE 4 supports an `extraArgs` field for each of these components, though, which accepts a map of key-value pairs. During upgrade from MKE 3, MKE 4 converts these custom flags to the corresponding `extraArgs` field. Any flags that cannot be automatically converted are listed in the upgrade summary. #### Example of custom flags conversion From a7995e99ee343c12f90be322cb9f0f33df8c3e39 Mon Sep 17 00:00:00 2001 From: Noah Merlis-Stephens Date: Wed, 11 Dec 2024 10:16:03 -0800 Subject: [PATCH 03/10] Update content/docs/migrate-from-MKE-3.md Co-authored-by: Kory <57411706+KoryKessel-Mirantis@users.noreply.github.com> --- content/docs/migrate-from-MKE-3.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/docs/migrate-from-MKE-3.md b/content/docs/migrate-from-MKE-3.md index 1ec5bb62..cfaa3dea 100644 --- a/content/docs/migrate-from-MKE-3.md +++ b/content/docs/migrate-from-MKE-3.md @@ -99,7 +99,7 @@ custom_kube_proxy_flags MKE 4 supports an `extraArgs` field for each of these components, though, which accepts a map of key-value pairs. During upgrade from MKE 3, MKE 4 converts these custom flags to the corresponding `extraArgs` field. Any flags that cannot be automatically converted are listed in the upgrade summary. -#### Example of custom flags conversion +Example of custom flags conversion: MKE 3 configuration file: ``` From 7a09f5d89054660da716a441d4bf2c85e5473d3a Mon Sep 17 00:00:00 2001 From: Noah Merlis-Stephens Date: Wed, 11 Dec 2024 10:17:26 -0800 Subject: [PATCH 04/10] Apply suggestions from code review Co-authored-by: Kory <57411706+KoryKessel-Mirantis@users.noreply.github.com> --- content/docs/migrate-from-MKE-3.md | 10 ++++------ 1 file changed, 4 insertions(+), 6 deletions(-) diff --git a/content/docs/migrate-from-MKE-3.md b/content/docs/migrate-from-MKE-3.md index cfaa3dea..7fe4c982 100644 --- a/content/docs/migrate-from-MKE-3.md +++ b/content/docs/migrate-from-MKE-3.md @@ -87,8 +87,7 @@ starting the upgrade process. {{< /callout >}} ### Kubernetes Custom Flags -MKE 3 supports applying additional flags to Kubernetes components with the following fields in the configuration file, -each specified as a list of strings. +MKE 3 and MKE 4 both support the application of additional flags to Kubernetes components that have the following fields in the MKE configuration file, each specified as a list of strings: ``` custom_kube_api_server_flags custom_kube_controller_manager_flags @@ -118,10 +117,9 @@ spec: MKE 3 supports a map of kubelet flag profiles to specific nodes using the `custom_kubelet_flags_profiles` setting in the toml configuration file. -MKE 4 does not support kubelet flag profiles, but you can map `KubeletConfiguration` values to specific -nodes using [Kubelet custom profiles](../configuration/kubernetes/kubelet.md#kubelet-custom-profiles). -MKE 4 supports migration of MKE 3 kubelet flag profiles to kubelet custom profiles. -Conversion of flags to `KubeletConfiguration` values is best-effort and any flags that cannot be +MKE 4 does not support kubelet flag profiles, but you can use [Kubelet custom profiles](../configuration/kubernetes/kubelet.md#kubelet-custom-profiles) to map `KubeletConfiguration` values to specific nodes. MKE 4 does supports the migration of MKE 3 kubelet flag profiles to kubelet custom profiles. + +The conversion of flags to `KubeletConfiguration` values is best-effort ,and any flags that cannot be converted are listed in the upgrade summary. Hosts with a custom flag profile label are marked for the corresponding kubelet custom profile. From 913253467f84302c31c192cbba34335d4f6c7cb3 Mon Sep 17 00:00:00 2001 From: Noah Merlis-Stephens Date: Wed, 11 Dec 2024 10:38:27 -0800 Subject: [PATCH 05/10] Bullet MKE 3 and MKE 4 config examples --- content/docs/migrate-from-MKE-3.md | 24 +++++++++++++----------- 1 file changed, 13 insertions(+), 11 deletions(-) diff --git a/content/docs/migrate-from-MKE-3.md b/content/docs/migrate-from-MKE-3.md index 7fe4c982..bb2b9d50 100644 --- a/content/docs/migrate-from-MKE-3.md +++ b/content/docs/migrate-from-MKE-3.md @@ -100,18 +100,20 @@ MKE 4 supports an `extraArgs` field for each of these components, though, which Example of custom flags conversion: -MKE 3 configuration file: -``` -[cluster_config.custom_kube_api_server_flags] = ["--enable-garbage-collector=false"] -``` +- MKE 3 configuration file: -MKE 4 configuration file: -``` -spec: - apiServer: - extraArgs: - enable-garbage-collector: false -``` + ``` + [cluster_config.custom_kube_api_server_flags] = ["--enable-garbage-collector=false"] + ``` + +- MKE 4 configuration file: + + ``` + spec: + apiServer: + extraArgs: + enable-garbage-collector: false + ``` ### Kubelet Custom Flag Profiles From 9101503554c7d55f1df15d04250e6ae048fbc9e6 Mon Sep 17 00:00:00 2001 From: Noah Merlis-Stephens Date: Wed, 11 Dec 2024 10:50:19 -0800 Subject: [PATCH 06/10] Base etcd documentation on existing MKE 3 docs --- content/docs/configuration/kubernetes/etcd.md | 14 +++++++++++++- 1 file changed, 13 insertions(+), 1 deletion(-) diff --git a/content/docs/configuration/kubernetes/etcd.md b/content/docs/configuration/kubernetes/etcd.md index f54e8bca..ab82b006 100644 --- a/content/docs/configuration/kubernetes/etcd.md +++ b/content/docs/configuration/kubernetes/etcd.md @@ -3,7 +3,9 @@ title: etcd weight: 2 --- -etcd is a consistent and highly-available key value store used as Kubernetes' backing store for all cluster data. +etcd is a consistent, distributed key-value store that provides a reliable way to store data that needs to be accessed by a distributed system or cluster of machines. It handles leader elections during network partitions and can tolerate machine failure, even in the leader node. + +For MKE, etcd serves as the Kubernetes backing store for all cluster data, with an etcd replica deployed on each MKE manager node. This is a primary reason why Mirantis recommends that you deploy an odd number of MKE manager nodes, as etcd uses the Raft consensus algorithm and thus requires that a quorum of nodes agree on any updates to the cluster state. You can find in-depth information about etcd in the official [documentation](https://etcd.io/docs/). @@ -15,3 +17,13 @@ spec: storageQuota: 2GB tlsCipherSuites: TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 ``` + +## Configure etcd storage quota + +You can control the etcd distributed key-value storage quota using the `etcd.storageQuota` parameter in the MKE configuration file. By default, the value of the parameter is 2GB. + +If you choose to increase the etcd quota, be aware that this quota has a limit and such action should be used in conjunction with other strategies, such as decreasing events TTL to ensure that the etcd database does not run out of space. + +{{< callout type="warning" >}} If a manager node virtual machine runs out of disk space, or if all of its system memory is depleted, etcd can cause the MKE cluster to move into an irrecoverable state. To prevent this from happening, configure the disk space and the memory of the manager node VMs to levels that are well in excess of the set etcd storage quota. {{< /callout >}} + +{{< callout type="info" >}} etcd official documentation, [How to debug large db size issue?](https://etcd.io/blog/2023/how_to_debug_large_db_size_issue/) {{< /callout >}} From 8c028fab963663d4e71de54c0e9fea4c5bf22d15 Mon Sep 17 00:00:00 2001 From: Noah Merlis-Stephens Date: Thu, 9 Jan 2025 12:40:50 -0800 Subject: [PATCH 07/10] Apply suggestions from code review Co-authored-by: Kory <57411706+KoryKessel-Mirantis@users.noreply.github.com> --- content/docs/configuration/kubernetes/etcd.md | 12 +++++++++--- 1 file changed, 9 insertions(+), 3 deletions(-) diff --git a/content/docs/configuration/kubernetes/etcd.md b/content/docs/configuration/kubernetes/etcd.md index ab82b006..def3ab3e 100644 --- a/content/docs/configuration/kubernetes/etcd.md +++ b/content/docs/configuration/kubernetes/etcd.md @@ -1,13 +1,15 @@ --- title: etcd -weight: 2 +weight: 6 --- etcd is a consistent, distributed key-value store that provides a reliable way to store data that needs to be accessed by a distributed system or cluster of machines. It handles leader elections during network partitions and can tolerate machine failure, even in the leader node. For MKE, etcd serves as the Kubernetes backing store for all cluster data, with an etcd replica deployed on each MKE manager node. This is a primary reason why Mirantis recommends that you deploy an odd number of MKE manager nodes, as etcd uses the Raft consensus algorithm and thus requires that a quorum of nodes agree on any updates to the cluster state. -You can find in-depth information about etcd in the official [documentation](https://etcd.io/docs/). +For detailed information, refer to the official [etcd documentation](https://etcd.io/docs/). + +## Configure etcd You can configure etcd through the `etcd` section of the MKE configuration file, an example of which follows: @@ -26,4 +28,8 @@ If you choose to increase the etcd quota, be aware that this quota has a limit a {{< callout type="warning" >}} If a manager node virtual machine runs out of disk space, or if all of its system memory is depleted, etcd can cause the MKE cluster to move into an irrecoverable state. To prevent this from happening, configure the disk space and the memory of the manager node VMs to levels that are well in excess of the set etcd storage quota. {{< /callout >}} -{{< callout type="info" >}} etcd official documentation, [How to debug large db size issue?](https://etcd.io/blog/2023/how_to_debug_large_db_size_issue/) {{< /callout >}} +{{< callout type="info" >}} + +For additional information, refer to the etcd official documentation, [How to debug large db size issue?](https://etcd.io/blog/2023/how_to_debug_large_db_size_issue/) + +{{< /callout >}} From 4bf88d4fb84f4119fe1b4bf7b5a1dd02129a4dfe Mon Sep 17 00:00:00 2001 From: Kory <57411706+KoryKessel-Mirantis@users.noreply.github.com> Date: Mon, 13 Jan 2025 14:32:31 +0100 Subject: [PATCH 08/10] Spacing fix. Co-authored-by: Magdalena Dziadosz <160592158+MagdaDziadosz@users.noreply.github.com> --- content/docs/migrate-from-MKE-3.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/docs/migrate-from-MKE-3.md b/content/docs/migrate-from-MKE-3.md index df63a87f..5cffabd2 100644 --- a/content/docs/migrate-from-MKE-3.md +++ b/content/docs/migrate-from-MKE-3.md @@ -162,7 +162,7 @@ MKE 3 supports a map of kubelet flag profiles to specific nodes using the `custo MKE 4 does not support kubelet flag profiles, but you can use [Kubelet custom profiles](../configuration/kubernetes/kubelet.md#kubelet-custom-profiles) to map `KubeletConfiguration` values to specific nodes. MKE 4 does supports the migration of MKE 3 kubelet flag profiles to kubelet custom profiles. -The conversion of flags to `KubeletConfiguration` values is best-effort ,and any flags that cannot be +The conversion of flags to `KubeletConfiguration` values is best-effort, and any flags that cannot be converted are listed in the upgrade summary. Hosts with a custom flag profile label are marked for the corresponding kubelet custom profile. From 43a48938dff84ecd6089e9c7914422e2be0696a6 Mon Sep 17 00:00:00 2001 From: Kory <57411706+KoryKessel-Mirantis@users.noreply.github.com> Date: Mon, 13 Jan 2025 14:36:17 +0100 Subject: [PATCH 09/10] Add 's' to 'agree'. Co-authored-by: Magdalena Dziadosz <160592158+MagdaDziadosz@users.noreply.github.com> --- content/docs/configuration/kubernetes/etcd.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/docs/configuration/kubernetes/etcd.md b/content/docs/configuration/kubernetes/etcd.md index def3ab3e..17a9e24d 100644 --- a/content/docs/configuration/kubernetes/etcd.md +++ b/content/docs/configuration/kubernetes/etcd.md @@ -5,7 +5,7 @@ weight: 6 etcd is a consistent, distributed key-value store that provides a reliable way to store data that needs to be accessed by a distributed system or cluster of machines. It handles leader elections during network partitions and can tolerate machine failure, even in the leader node. -For MKE, etcd serves as the Kubernetes backing store for all cluster data, with an etcd replica deployed on each MKE manager node. This is a primary reason why Mirantis recommends that you deploy an odd number of MKE manager nodes, as etcd uses the Raft consensus algorithm and thus requires that a quorum of nodes agree on any updates to the cluster state. +For MKE, etcd serves as the Kubernetes backing store for all cluster data, with an etcd replica deployed on each MKE manager node. This is a primary reason why Mirantis recommends that you deploy an odd number of MKE manager nodes, as etcd uses the Raft consensus algorithm and thus requires that a quorum of nodes agrees on any updates to the cluster state. For detailed information, refer to the official [etcd documentation](https://etcd.io/docs/). From b889a3632df30dc0c4264d1d465108b4ab7b8f9c Mon Sep 17 00:00:00 2001 From: Kory <57411706+KoryKessel-Mirantis@users.noreply.github.com> Date: Mon, 13 Jan 2025 14:45:41 +0100 Subject: [PATCH 10/10] Remove errant 's'. --- content/docs/migrate-from-MKE-3.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/docs/migrate-from-MKE-3.md b/content/docs/migrate-from-MKE-3.md index 5cffabd2..3ef5746e 100644 --- a/content/docs/migrate-from-MKE-3.md +++ b/content/docs/migrate-from-MKE-3.md @@ -160,7 +160,7 @@ Example of custom flags conversion: MKE 3 supports a map of kubelet flag profiles to specific nodes using the `custom_kubelet_flags_profiles` setting in the toml configuration file. -MKE 4 does not support kubelet flag profiles, but you can use [Kubelet custom profiles](../configuration/kubernetes/kubelet.md#kubelet-custom-profiles) to map `KubeletConfiguration` values to specific nodes. MKE 4 does supports the migration of MKE 3 kubelet flag profiles to kubelet custom profiles. +MKE 4 does not support kubelet flag profiles, but you can use [Kubelet custom profiles](../configuration/kubernetes/kubelet.md#kubelet-custom-profiles) to map `KubeletConfiguration` values to specific nodes. MKE 4 does support the migration of MKE 3 kubelet flag profiles to kubelet custom profiles. The conversion of flags to `KubeletConfiguration` values is best-effort, and any flags that cannot be converted are listed in the upgrade summary. Hosts with a custom flag profile label are marked for the