Merge pull request #53696 from mburke5678/okd-cgroup-v2-by-default

GH53536: [OKD] Documentation is unclear about status of cgroups v2
openshift · Dec 16, 2022 · af257f4 · af257f4
2 parents 157b72a + f5ce270
commit af257f4
Show file tree

Hide file tree

Showing 6 changed files with 214 additions and 2 deletions.
diff --git a/_topic_maps/_topic_map.yml b/_topic_maps/_topic_map.yml
@@ -489,6 +489,7 @@ Topics:
     File: configuring-firewall
   - Name: Enabling Linux control group version 2 (cgroup v2)
     File: enabling-cgroup-v2
+    Distros: openshift-enterprise
 - Name: Validating an installation
   File: validating-an-installation
   Distros: openshift-origin,openshift-enterprise
@@ -2160,7 +2161,10 @@ Topics:
     Distros: openshift-enterprise,openshift-origin
   - Name: Enabling Linux control group version 2 (cgroup v2)
     File: nodes-cluster-cgroups-2
-    Distros: openshift-enterprise,openshift-origin
+    Distros: openshift-enterprise
+  - Name: Configuring the Linux cgroup on your nodes
+    File: nodes-cluster-cgroups-okd
+    Distros: openshift-origin
   - Name: Enabling features using FeatureGates
     File: nodes-cluster-enabling-features
     Distros: openshift-enterprise,openshift-origin

diff --git a/modules/nodes-clusters-cgroups-okd-configure.adoc b/modules/nodes-clusters-cgroups-okd-configure.adoc
@@ -0,0 +1,126 @@
+// Module included in the following assemblies:
+//
+// * nodes/clusters/nodes-cluster-cgroups-okd.adoc
+// * post_installation_configuration/cluster-tasks.adoc
+
+ifeval::["{context}" == "nodes-cluster-cgroups-2"]
+:node:
+endif::[]
+ifeval::["{context}" == "post-install-cluster-tasks"]
+:post:
+endif::[]
+
+:_content-type: PROCEDURE
+[id="nodes-clusters-cgroups-okd-configure_{context}"]
+= Configuring Linux cgroup
+
+ifdef::post[]
+By default, {product-title} uses link:https://www.kernel.org/doc/html/latest/admin-guide/cgroup-v2.html[Linux control group version 2] (cgroup v2) in your cluster. You can switch to link:https://www.kernel.org/doc/html/latest/admin-guide/cgroup-v1.html[Linux control group version 1] (cgroup v1), if needed.
+
+cgroup v2 is the next version of the kernel link:https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/6/html/resource_management_guide/ch01[control group] and offers multiple improvements. However, it can have some unwanted effects on your nodes.
+endif::post[]
+
+You can configure whether your cluster uses cgroup v1 or cgroup v2 by editing the `node.config` object. Enabling the other version of cgroup in {product-title} disables the current cgroup controllers and hierarchies in your cluster.
+
+.Prerequisites
+* Have administrative privilege to a working {product-title} cluster.
+
+.Procedure
+
+. Create a `MachineConfig` object file that identifies the kernel argument (for example, `worker-cgroup-v1.yaml`)
++
+[source,yaml]
+----
+apiVersion: machineconfiguration.openshift.io/v1
+kind: MachineConfig
+metadata:
+  labels:
+    machineconfiguration.openshift.io/role: worker <1>
+  name: worker-cgroup-v1 <2>
+spec:
+  config:
+    ignition:
+      version: 3.2.0
+  kernelArguments:
+    - systemd.unified_cgroup_hierarchy=0 <3>
+----
++
+<1> Applies the new kernel argument only to worker nodes.
+<2> Applies a name to the machine config.
+<3> Configures cgroup v1 on the associated nodes.
+
+. Create the new machine config:
++
+[source,terminal]
+----
+$ oc create -f 05-worker-cgroup-v1.yaml
+----
+
+. Check to see that the new machine config was added:
++
+[source,terminal]
+----
+$ oc get MachineConfig
+----
++
+.Example output
+[source,terminal]
+----
+NAME                                               GENERATEDBYCONTROLLER                      IGNITIONVERSION   AGE
+00-master                                          52dd3ba6a9a527fc3ab42afac8d12b693534c8c9   3.2.0             33m
+00-worker                                          52dd3ba6a9a527fc3ab42afac8d12b693534c8c9   3.2.0             33m
+01-master-container-runtime                        52dd3ba6a9a527fc3ab42afac8d12b693534c8c9   3.2.0             33m
+01-master-kubelet                                  52dd3ba6a9a527fc3ab42afac8d12b693534c8c9   3.2.0             33m
+01-worker-container-runtime                        52dd3ba6a9a527fc3ab42afac8d12b693534c8c9   3.2.0             33m
+01-worker-kubelet                                  52dd3ba6a9a527fc3ab42afac8d12b693534c8c9   3.2.0             33m
+99-worker-cgroup-v1                                                                           3.2.0             105s
+99-master-generated-registries                     52dd3ba6a9a527fc3ab42afac8d12b693534c8c9   3.2.0             33m
+99-master-ssh                                                                                 3.2.0             40m
+99-worker-generated-registries                     52dd3ba6a9a527fc3ab42afac8d12b693534c8c9   3.2.0             33m
+99-worker-ssh                                                                                 3.2.0             40m
+rendered-master-23e785de7587df95a4b517e0647e5ab7   52dd3ba6a9a527fc3ab42afac8d12b693534c8c9   3.2.0             33m
+rendered-master-c5e92d98103061c4818cfcefcf462770   60746a843e7ef8855ae00f2ffcb655c53e0e8296   3.2.0             115s
+rendered-worker-5d596d9293ca3ea80c896a1191735bb1   52dd3ba6a9a527fc3ab42afac8d12b693534c8c9   3.2.0             33m
+----
+
+. Check the nodes:
++
+[source,terminal]
+----
+$ oc get nodes
+----
++
+.Example output
+[source,terminal]
+----
+NAME                           STATUS                     ROLES    AGE   VERSION
+ip-10-0-136-161.ec2.internal   Ready                      worker   28m   v1.25.0
+ip-10-0-136-243.ec2.internal   Ready                      master   34m   v1.25.0
+ip-10-0-141-105.ec2.internal   Ready,SchedulingDisabled   worker   28m   v1.25.0
+ip-10-0-142-249.ec2.internal   Ready                      master   34m   v1.25.0
+ip-10-0-153-11.ec2.internal    Ready                      worker   28m   v1.25.0
+ip-10-0-153-150.ec2.internal   Ready                      master   34m   v1.25.0
+----
++
+You can see that The command disables scheduling on each worker node.
+
+. Check that the sys/fs/cgroup/cgroup2fs file has been moved to the `tmpfs` file system:
++
+[source,terminal]
+----
+$ stat -c %T -f /sys/fs/cgroup
+----
++
+.Example output
++
+[source,terminal]
+----
+tmpfs
+----
+
+ifeval::["{context}" == "nodes-cluster-cgroups-2"]
+:!node:
+endif::[]
+ifeval::["{context}" == "post-install-cluster-tasks"]
+:!post:
+endif::[]
diff --git a/modules/nodes-nodes-kernel-arguments.adoc b/modules/nodes-nodes-kernel-arguments.adoc
@@ -20,7 +20,30 @@ Examples of kernel arguments you could set include:
 
 * **nosmt**: Disables symmetric multithreading (SMT) in the kernel. Multithreading allows multiple logical threads for each CPU. You could consider `nosmt` in multi-tenant environments to reduce risks from potential cross-thread attacks. By disabling SMT, you essentially choose security over performance.
 
+ifndef::openshift-origin[]
 * **systemd.unified_cgroup_hierarchy**: Enables link:https://www.kernel.org/doc/html/latest/admin-guide/cgroup-v2.html[Linux control group version 2] (cgroup v2). cgroup v2 is the next version of the kernel link:https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/6/html/resource_management_guide/ch01[control group] and offers multiple improvements.
++
+[IMPORTANT]
+====
+[subs="attributes+"]
+{product-title} cgroups version 2 support is a Technology Preview feature only. Technology Preview features
+are not supported with Red Hat production service level agreements (SLAs) and
+might not be functionally complete. Red Hat does not support Technology Preview features for use in production systems. These features provide early access to upcoming product
+features, enabling customers to test functionality and provide feedback during
+the development process.
+
+For more information about the support scope of Red Hat Technology Preview
+features, see https://access.redhat.com/support/offerings/techpreview/.
+====
+endif::openshift-origin[]
+ifdef::openshift-origin[]
+* **systemd.unified_cgroup_hierarchy**: Configures the version of Linux control group that is installed on your nodes: link:https://www.kernel.org/doc/html/latest/admin-guide/cgroup-v1.html[cgroup v1] or link:https://www.kernel.org/doc/html/latest/admin-guide/cgroup-v2.html[cgroup v2]. cgroup v2 is the next version of the kernel link:https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/6/html/resource_management_guide/ch01[control group] and offers multiple improvements. However, it can have some unwanted effects on your nodes. 
++
+[NOTE]
+====
+cgroup v2 is enabled by default. To disable cgroup v2, use the `systemd.unified_cgroup_hierarchy=0` kernel argument, as shown in the following procedure.
+====
+endif::openshift-origin[]
 
 See link:https://www.kernel.org/doc/Documentation/admin-guide/kernel-parameters.txt[Kernel.org kernel parameters] for a list and descriptions of kernel arguments.
 
@@ -61,6 +84,7 @@ rendered-master-23e785de7587df95a4b517e0647e5ab7   52dd3ba6a9a527fc3ab42afac8d12
 rendered-worker-5d596d9293ca3ea80c896a1191735bb1   52dd3ba6a9a527fc3ab42afac8d12b693534c8c9   3.2.0             33m
 ----
 
+ifndef::openshift-origin[]
 . Create a `MachineConfig` object file that identifies the kernel argument (for example, `05-worker-kernelarg-selinuxpermissive.yaml`)
 +
 [source,yaml]
@@ -83,13 +107,46 @@ spec:
 <2> Named to identify where it fits among the machine configs (05) and what it does (adds
 a kernel argument to configure SELinux permissive mode).
 <3> Identifies the exact kernel argument as `enforcing=0`.
+. Create the new machine config:
++
+[source,terminal]
+----
+$ oc create -f 05-worker-kernelarg-selinuxpermissive.yaml
+----
+endif::openshift-origin[]
+ifdef::openshift-origin[]
+. Create a `MachineConfig` object file that identifies the kernel argument (for example, `05-worker-kernelarg-selinuxpermissive.yaml`)
++
+[source,yaml]
+----
+apiVersion: machineconfiguration.openshift.io/v1
+kind: MachineConfig
+metadata:
+  labels:
+    machineconfiguration.openshift.io/role: worker <1>
+  name: 05-worker-kernelarg-selinuxpermissive <2>
+spec:
+  config:
+    ignition:
+      version: 3.2.0
+  kernelArguments:
+    - enforcing=0 <3>
+      systemd.unified_cgroup_hierarchy=0 <4> 
+----
++
+<1> Applies the new kernel argument only to worker nodes.
+<2> Named to identify where it fits among the machine configs (05) and what it does (adds
+a kernel argument to configure SELinux permissive mode).
+<3> Identifies the exact kernel argument as `enforcing=0`.
+<4> Configures cgroup v1 on the associated nodes. cgroup v2 is the default.
 
 . Create the new machine config:
 +
 [source,terminal]
 ----
 $ oc create -f 05-worker-kernelarg-selinuxpermissive.yaml
 ----
+endif::openshift-origin[]
 
 . Check the machine configs to see that the new one was added:
 +

diff --git a/nodes/clusters/nodes-cluster-cgroups-okd.adoc b/nodes/clusters/nodes-cluster-cgroups-okd.adoc
@@ -0,0 +1,19 @@
+:_content-type: ASSEMBLY
+:context: nodes-cluster-cgroups-2
+[id="nodes-cluster-cgroups-okd"]
+= Configuring the Linux cgroup on your nodes
+include::_attributes/common-attributes.adoc[]
+
+toc::[]
+
+By default, {product-title} uses link:https://www.kernel.org/doc/html/latest/admin-guide/cgroup-v2.html[Linux control group version 2] (cgroup v2) in your cluster. You can switch to link:https://www.kernel.org/doc/html/latest/admin-guide/cgroup-v1.html[Linux control group version 1] (cgroup v1), if needed.
+
+cgroup v2 is the next version of the kernel link:https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/6/html/resource_management_guide/ch01[control group] and offers multiple improvements. However, it can have some unwanted effects on your nodes.
+
+// The following include statements pull in the module files that comprise
+// the assembly. Include any combination of concept, procedure, or reference
+// modules required to cover the user story. You can also include other
+// assemblies.
+
+include::modules/nodes-clusters-cgroups-okd-configure.adoc[leveloffset=+1]
+
diff --git a/post_installation_configuration/cluster-tasks.adoc b/post_installation_configuration/cluster-tasks.adoc
@@ -587,8 +587,12 @@ include::modules/machine-autoscaler-cr.adoc[leveloffset=+2]
 :FeatureResourceName: MachineAutoscaler
 include::modules/deploying-resource.adoc[leveloffset=+2]
 
+ifndef::openshift-origin[]
 include::modules/nodes-clusters-cgroups-2.adoc[leveloffset=+1]
-
+endif::openshift-origin[]
+ifdef::openshift-origin[]
+include::modules/nodes-clusters-cgroups-okd-configure.adoc[leveloffset=+1]
+endif::openshift-origin[]
 [id="post-install-tp-tasks"]
 == Enabling Technology Preview features using FeatureGates
 

diff --git a/post_installation_configuration/machine-configuration-tasks.adoc b/post_installation_configuration/machine-configuration-tasks.adoc
@@ -74,3 +74,5 @@ Besides managing `MachineConfig` objects, the MCO manages two custom resources (
 include::modules/create-a-kubeletconfig-crd-to-edit-kubelet-parameters.adoc[leveloffset=+2]
 include::modules/create-a-containerruntimeconfig-crd.adoc[leveloffset=+2]
 include::modules/set-the-default-max-container-root-partition-size-for-overlay-with-crio.adoc[leveloffset=+2]
+
+
Original file line number	Diff line number	Diff line change
Expand Up		@@ -74,3 +74,5 @@ Besides managing `MachineConfig` objects, the MCO manages two custom resources (
		include::modules/create-a-kubeletconfig-crd-to-edit-kubelet-parameters.adoc[leveloffset=+2]
		include::modules/create-a-containerruntimeconfig-crd.adoc[leveloffset=+2]
		include::modules/set-the-default-max-container-root-partition-size-for-overlay-with-crio.adoc[leveloffset=+2]