From bf3f5777c6de9c72885405026392e6aef93fb820 Mon Sep 17 00:00:00 2001 From: Ariel Adam Date: Mon, 22 Jun 2020 10:54:07 +0300 Subject: [PATCH 01/10] Kata containers enhancement document Signed-off-by: Ariel Adam --- .../kata-containers-dev-preview.md | 259 ++++++++++++++++++ 1 file changed, 259 insertions(+) create mode 100644 enhancements/kata-containers/kata-containers-dev-preview.md diff --git a/enhancements/kata-containers/kata-containers-dev-preview.md b/enhancements/kata-containers/kata-containers-dev-preview.md new file mode 100644 index 0000000000..134bcdeb93 --- /dev/null +++ b/enhancements/kata-containers/kata-containers-dev-preview.md @@ -0,0 +1,259 @@ +--- +title: kata-containers-dev-preview +authors: + - "@aadam" +reviewers: + - "@mpatel" +approvers: + - "@mpatel" +creation-date: 2020-06-07 +last-updated: 2020-06-07 +status: implementable +see-also: + - TBD +replaces: + - TBD +superseded-by: + - TBD +--- + +# Kata Containers + +## Summary + +Kata containers is an open source project developing a container runtime using virtual machines and providing the same look and feel as with vanilla containers. +By leveraging hardware virtualization technologies kata containers provides powerful workload isolation compared to existing container solutions. + +We will be integrating kata containers into Openshift to provide the ability to run kernel isolated containers for any workload which requires custom kernel tuning (sysctl, scheduler changes, cache tuning, etc), custom kernel modules (out of tree, special arguments, etc), exclusive access to hardware, root privileges, or other administrative privileges above and beyond what is secure in a shared kernel environment (regular runc). + +It should be noted that kata containers differ from Kubevirt: + +- Kubevirt aims to run **VM images** on Openshift providing the VM with the look and feel of a virtual machine running as a legacy VM (CRD defining a Virtual Machine in Kubernetes) +- Kata containers aim to run **container images** on Openshift in an isolated manner using virtualization tools (uses Runtime Class resource to choose Kata only for specific workloads which require kernel level isolation) + +## Motivation +If we take telcos for example, there are a number of reasons they require isolated workloads: + +1. **Telco CNF deployments** - As part of 5G/NFV telco deployments there is a gradual migration from physical networking boxes to CNFs (container network functions). Some of these CNFs require root access while others could potentially create a threat on the operators cloud bypassing existing container isolation mechanisms. For addressing those issues a hardened isolated container solution is required +2. **Devops deployments** - Telcos are working hard to increase the rate of adding new capabilities to their clouds for competing with OTTs (over the top companies such as Netflix, Apple, Facebook etc…). This involves huge investments in devops tools and processes. In order for devops to deploy new features on production environments there is a need for high isolation of the deployed workloads in order to control and revert such changes. +3. **Hardware Vendors** - many hardware vendors require custom kernel parameters at boot, custom sysctl variables, etc. They want to use containers as a convenient packaging format for their entire application, but still need access to lower level tuning for portions of their applications + + +## Terminology +- **Kata artifacts** - includes the upstream kata project components and RHEL-virt artifacts including QEMU +- **RuntimeClass** - see https://kubernetes.io/docs/concepts/containers/runtime-class/ + + +## Goals +- Provide a way to enable running user workloads on kata containers on an OpenShift cluster. +- Clusters with kata container workloads should support upgrades. +- Add security policy to control which users can run what workloads on kata in an OpenShift cluster. + + +## Non-goals +- Although kata containers are capable of supporting running a different kernel then the one used on the Openshift node, such features will not be supported. This is in order to simplify the development cycles and testing efforts of this project. +- Running OpenShift control plane and core operators on kata. These will continue to run on runc. + + +## Kata containers architecture + +For details on the kata architecture see https://github.com/kata-containers/documentation/blob/master/design/architecture.md + +## Proposal - main development efforts +### Overview +Kubernetes provides support for RuntimeClasses. RuntimeClass is a feature for selecting the container runtime configuration. The container runtime configuration is used to run a Pod’s containers. + +CRI-O today comes out of the box with a runc as the default runtime. CRI-O also supports RUntimeClasses and using this configuration, it will support a KataContainers runtime as well. + +The 2 fundamental problems we need to address for getting kata to work with Openshift are: + +1. Getting the kata artifacts on to an RHCOS node +2. Deploying kata and configuring CRI-O to use it as the runtime + +For addressing the first issue of packaging and installing kata in openshift we will introduce 2 approaches, a short term and a long term. + +For addressing the second issue of deploying and configuring CRI-O we will use an operator. + +The next sections detailed the 2 solutions. + +### KataContainer Packaging +#### Short term (preview release): Use container images to deliver RPMs and install using rpm-ostree +In this option we build a container image containing RHEL AV RPMs (such as QEMU) and kata RPMs. The kata operator will create a pod based on this image and then mount the host filesystem and install the RPMs on the filesystem (kata operators will be described in a later section). + +**Pros:** +1. The RPMs can be installed only by the users who would be interested in using the kata runtime +2. There's no extra work related to properly loading qemu-kiwi dependencies, as all the packages would be in installed in the same PATH its build was intent for +3. QEMU RPMs have been tested / validated using an identical environment +4. No additional work with regarding to re-packaging those in another form other than the RPMs which is the way they are consumed by other Layered Products + +**Cons:** +1. ~200 mb extra, installed in the host, for those who'd be using kata runtime +2. Updates / Removal may be more complicated than having the RPMs as part of machine-os-content + +#### Long term: Use RHCOS extensions (qemu-kiwi and dependencies only) +In this option we build upon the RHCOS extension approach planned for OCP4.6. + +**What is the RHCOS extension?** + +This will add additional software onto the host - but this software will still be versioned with the host (included as part of the OpenShift release payload) and upgraded with the cluster. + +We have requests to ship things like `usbguard` that are needed to meet compliance in some scenarios (and `usbguard` is very useful on bare metal), but `usbguard` also makes no sense to ship in an OS for the public cloud. + +The `rpm-ostree` project has had as its goal from the start to re-cast RPMs as "operating system extensions" - much like how e.g. Firefox and its extensions work. By default it operates as a pure image system, but packages can be layered on (and overridden) client side. + +OpenShift is already making use of this today for the [realtime kernel](https://github.com/openshift/enhancements/blob/master/enhancements/support-for-realtime-kernel.md). + +We propose continuing the trail blazed with the RT kernel by adding additional RPMs to `machine-os-content` that aren't part of the base OS "image", but can be installed later. This is how `kernel-rt` works today; we added a `kernel-rt.rpm` that is committed to the container. In the future though, we may instead ship a separate `machine-os-extensions` container, or something else. The only API stable piece here is the `MachineConfig` field (same as for the `kernelType: realtime`). + +See additional details in: https://github.com/openshift/enhancements/pull/317 + +**So What do we want to do with this?** + +We start by creating a container image with only the kata upstream components as static binaries (no RHEL-AV components). The kata operator will pull down this image and mount it in the host mount namespace, i.e. without running it as a container. We then use the RHCOS extensions for deploying the qemu-kiwi RHEL-AV RPMs. + +**Pros:** +1. RHEL-AV depends on libraries and other tools which are all available as RPMs. Turning this into a static image instead of RPMs creates a huge single binary which we avoid with extensions +2. RHEL-AV has been tested / validated as RPMs and not a single binary which means we saves the extra overhead +3. RHEL-AV RPMs can be installed only by the customers who would be interested in using the kata runtime. QEMU (and its dependencies) get updated as part of the machine-os-content updates +4. There's no extra work related to properly loading qemu-kiwi dependencies, as all the packages would be installed in the same PATH its build in +5. Better control of the versions that are installed both for qemu (specific version for a given host) and for the dependencies (tracked by rpm) + +**Cons:** +1. ~20 mb extra, installed in the host, for those who'd be using kata runtime +2. machine-os-content would still carry this until there is a separate machine-os-content- extensions container + +This approach has a dependency on the extensions framework being delivered into Openshift. + + +### KataContainer Operator development +The goal is to develop a kata operator that can be used to manage the entire lifecycle of kata components in an OpenShift cluster. We will have a controller that watches for a kata **custom resource (CR)** and a daemon-set that acts upon changes to the (CR) and do the required work on the worker nodes (install, uninstall update,...). + +Via the CR it will be possible to select a subset of worker nodes. For deploying binaries on the host the current idea is to use a container image that can be mounted on the host level. The goal is to be able to install the operator via OperatorHub. The operator will also create a crio drop-in config file via machine config objects. The Operator will automatically select the payload image that contains correct version of the kata binaries for the given version of the OpenShift. The Operator will also configure a `RuntimeClass` called `kata` which can be used to deploy workload that uses kata runtime. The operator will create a machine-config object to enable the `qemu-kiwi` RHCOS extension. + +The `RuntimeClass` and payload image names will be visible in the CR status. + +#### Goals +- Create an API which supports installation of Kata Runtime on all or selected worker nodes +- Configure CRI-O to use Kata Runtime on those worker nodes +- Installation of the runtimeClass on the cluster +- Updates the Kata runtime +- Uninstall Kata Runtime and reconfigure CRI-O to not use it. + +#### Non-Goals +To keep the Kata Operator's goal to the lifecycle management of the Kata Runtime, it will only support installation configuration of the Kata Runtime. This operator will not interact with any runtime configuration, such as Pod Annoations supported by Kata. + +#### Proposal +The following new API is proposed `kataconfiguration.openshift.io/v1` + +```go +// Kataconfig is the Schema for the kataconfigs API +type Kataconfig struct { + metav1.TypeMeta `json:",inline"` + metav1.ObjectMeta `json:"metadata,omitempty"` + + Spec KataconfigSpec `json:"spec,omitempty"` + Status KataconfigStatus `json:"status,omitempty"` +} + +// KataconfigSpec defines the desired state of Kataconfig +type KataconfigSpec struct { + // KataConfigPoolSelector is used to filer the worker nodes + // if not specified, all worker nodes are selected + // +optional + KataConfigPoolSelector *metav1.LabelSelector + + Config KataInstallConfig `json:"config"` +} + +type KataInstallConfig struct { +} + +// KataconfigStatus defines the observed state of KataConfig +type KataconfigStatus struct { + // Name of the runtime class + runtimeClassName string `json:"runtimeClassName"` + + // +required + kataImage string `json:"kataImage"` + + // TotalNodesCounts is the total number of worker nodes targeted by this CR + TotalNodesCount int `json:"totalNodesCount"` + + // CompletedNodesCounts is the number of nodes that have successfully completed the given operation + CompletedNodesCount int `json:"CompletedNodesCount"` + + // InProgressNodesCounts is the number of nodes still in progress in completing the given operation + InProgressNodesCount int `json:"inProgressNodesCount"` + + // FailedNodes is the list of worker nodes failed to complete the given operation + FailedNodes []KataFailedNodes `json:"failedNodes"` + + // Conditions represents the latest available observations of current state. + Conditions []KataStatusCondition `json:"conditions"` +} + +type KataFailedNodes struct { + // Name is the worker node name + Name string `json:"name"` + + // Error returned by the daemon running on the worker node + Error string `json:"error"` +} + +// KataStatusCondition contains condition information for Status +type KataStatusCondition struct { + // type specifies the state of the operator's reconciliation functionality. + Type KataConfigStatusConditionType `json:"type"` + + // status of the condition, one of True, False, Unknown. + Status corev1.ConditionStatus `json:"status"` + + // lastTransitionTime is the time of the last update to the current status object. + // +nullable + LastTransitionTime metav1.Time `json:"lastTransitionTime"` +} + +type KataConfigStatusConditionType string + +const ( + // KataStatusInProgress means the kata controller is currently running. + KataStatusInProgress KataConfigStatusConditionType = "KataStatusInProgress" + + // KataStatusCompleted means the kata controller has completed reconciliation. + KataStatusCompleted KataConfigStatusConditionType = "KataStatusCompleted" + + // KataStatusFailed means the kata controller is failing. + KataStatusFailed KataConfigStatusConditionType = "KataStatusFailed" +) + +One of the ways Administrators can interact with the Kata Operator by providing a yaml file to the standard oc or kubectl command. +apiVersion: kataconfiguration.openshift.io/v1 +kind: KataConfig +metadata: + name: install-kata-1.0 +spec: + kataConfigPoolSelector: + matchLabels: + install-kata: kata-1.0 +``` + +### Openshift kata testing + +- Kubernetes E2E tests, matching the ones ran by CRI-O: +https://github.com/cri-o/cri-o/blob/master/contrib/test/integration/e2e.yml#L14 +- OpenShift "conformance/parallel" tests +- Get sample workloads from customers and add tests to CI if we are missing coverage for those workloads to ensure we support the workloads and don’t regress. + +### Openshift kata CI + +- Leverage existing openshift CI infrastructure for e2e tests on kata-containers upstream + +### Scale and performance + +These comparisons to be performed by targeted microbenchmarks. Instrumentation is WIP: +- Rate of Kata pod creation/deletion comparison with standard runtime +- Network traffic throughput/latency +- Storage throughput/latency +- CPU utilization +- Memory consumption/overhead + From ce4274d60939d7f22976d34733a20fd974662292 Mon Sep 17 00:00:00 2001 From: Jens Freimann Date: Tue, 16 Feb 2021 12:00:20 +0100 Subject: [PATCH 02/10] Update kata-containers-dev-preview.md fix review findings from Micah Abbot --- .../kata-containers-dev-preview.md | 25 ++++++++++--------- 1 file changed, 13 insertions(+), 12 deletions(-) diff --git a/enhancements/kata-containers/kata-containers-dev-preview.md b/enhancements/kata-containers/kata-containers-dev-preview.md index 134bcdeb93..23e4735fae 100644 --- a/enhancements/kata-containers/kata-containers-dev-preview.md +++ b/enhancements/kata-containers/kata-containers-dev-preview.md @@ -63,7 +63,7 @@ For details on the kata architecture see https://github.com/kata-containers/docu ### Overview Kubernetes provides support for RuntimeClasses. RuntimeClass is a feature for selecting the container runtime configuration. The container runtime configuration is used to run a Pod’s containers. -CRI-O today comes out of the box with a runc as the default runtime. CRI-O also supports RUntimeClasses and using this configuration, it will support a KataContainers runtime as well. +CRI-O today comes out of the box with a runc as the default runtime. CRI-O also supports RuntimeClasses and using this configuration, it will support a KataContainers runtime as well. The 2 fundamental problems we need to address for getting kata to work with Openshift are: @@ -87,25 +87,27 @@ In this option we build a container image containing RHEL AV RPMs (such as QEMU) 4. No additional work with regarding to re-packaging those in another form other than the RPMs which is the way they are consumed by other Layered Products **Cons:** -1. ~200 mb extra, installed in the host, for those who'd be using kata runtime +1. 20 MB extra, installed in the host, for those who'd be using kata runtime 2. Updates / Removal may be more complicated than having the RPMs as part of machine-os-content #### Long term: Use RHCOS extensions (qemu-kiwi and dependencies only) In this option we build upon the RHCOS extension approach planned for OCP4.6. -**What is the RHCOS extension?** +**What is the qemu-kiwi RHCOS extension?** This will add additional software onto the host - but this software will still be versioned with the host (included as part of the OpenShift release payload) and upgraded with the cluster. -We have requests to ship things like `usbguard` that are needed to meet compliance in some scenarios (and `usbguard` is very useful on bare metal), but `usbguard` also makes no sense to ship in an OS for the public cloud. +The additional RPMs required for qemu-kiwi and its dependencies are: +* qemu-kiwi +* qemu-kvm-common +* pxe-roms-qemu +* libpmem +* pixman +* seabios-bin +* seavgabions-bin +* sgabios-bin -The `rpm-ostree` project has had as its goal from the start to re-cast RPMs as "operating system extensions" - much like how e.g. Firefox and its extensions work. By default it operates as a pure image system, but packages can be layered on (and overridden) client side. - -OpenShift is already making use of this today for the [realtime kernel](https://github.com/openshift/enhancements/blob/master/enhancements/support-for-realtime-kernel.md). - -We propose continuing the trail blazed with the RT kernel by adding additional RPMs to `machine-os-content` that aren't part of the base OS "image", but can be installed later. This is how `kernel-rt` works today; we added a `kernel-rt.rpm` that is committed to the container. In the future though, we may instead ship a separate `machine-os-extensions` container, or something else. The only API stable piece here is the `MachineConfig` field (same as for the `kernelType: realtime`). - -See additional details in: https://github.com/openshift/enhancements/pull/317 +See additional details about RHCOS extensions in general in: https://github.com/openshift/enhancements/pull/317 **So What do we want to do with this?** @@ -122,7 +124,6 @@ We start by creating a container image with only the kata upstream components as 1. ~20 mb extra, installed in the host, for those who'd be using kata runtime 2. machine-os-content would still carry this until there is a separate machine-os-content- extensions container -This approach has a dependency on the extensions framework being delivered into Openshift. ### KataContainer Operator development From 3a9014d02b2829da15cd62ca8c1b8f16eb5d3ec6 Mon Sep 17 00:00:00 2001 From: Jens Freimann Date: Tue, 16 Feb 2021 12:23:19 +0100 Subject: [PATCH 03/10] Update kata-containers-dev-preview.md Fix c3d's review findings --- .../kata-containers/kata-containers-dev-preview.md | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/enhancements/kata-containers/kata-containers-dev-preview.md b/enhancements/kata-containers/kata-containers-dev-preview.md index 23e4735fae..cddfe9e521 100644 --- a/enhancements/kata-containers/kata-containers-dev-preview.md +++ b/enhancements/kata-containers/kata-containers-dev-preview.md @@ -21,15 +21,15 @@ superseded-by: ## Summary -Kata containers is an open source project developing a container runtime using virtual machines and providing the same look and feel as with vanilla containers. -By leveraging hardware virtualization technologies kata containers provides powerful workload isolation compared to existing container solutions. +[Kata Containers](https://katacontainers.io/) is an open source project developing a container runtime using virtual machines and providing the same look and feel as vanilla containers. +By leveraging hardware virtualization technologies, Kata Containers provides powerful workload isolation compared to existing container solutions. -We will be integrating kata containers into Openshift to provide the ability to run kernel isolated containers for any workload which requires custom kernel tuning (sysctl, scheduler changes, cache tuning, etc), custom kernel modules (out of tree, special arguments, etc), exclusive access to hardware, root privileges, or other administrative privileges above and beyond what is secure in a shared kernel environment (regular runc). +We will be integrating Kata Containers into Openshift to provide the ability to run kernel isolated containers for any workload which requires custom kernel tuning (sysctl, scheduler changes, cache tuning, etc), custom kernel modules (out of tree, special arguments, etc), exclusive access to hardware, root privileges, or other administrative privileges above and beyond what is secure in a shared kernel environment (regular runc). -It should be noted that kata containers differ from Kubevirt: +It should be noted that Kata Containers differ from Kubevirt: - Kubevirt aims to run **VM images** on Openshift providing the VM with the look and feel of a virtual machine running as a legacy VM (CRD defining a Virtual Machine in Kubernetes) -- Kata containers aim to run **container images** on Openshift in an isolated manner using virtualization tools (uses Runtime Class resource to choose Kata only for specific workloads which require kernel level isolation) +- Kata Containers aim to run **container images** on Openshift in an isolated manner using virtualization tools (uses Runtime Class resource to choose Kata only for specific workloads which require kernel level isolation) ## Motivation If we take telcos for example, there are a number of reasons they require isolated workloads: From f201a2333627c35d02ba5025e2f20dd583970070 Mon Sep 17 00:00:00 2001 From: Jens Freimann Date: Mon, 22 Feb 2021 11:48:25 +0100 Subject: [PATCH 04/10] Update goals of kata-operator Extend the goal "installation of the runtime" by also taking care of things like pod overhead etc --- enhancements/kata-containers/kata-containers-dev-preview.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/enhancements/kata-containers/kata-containers-dev-preview.md b/enhancements/kata-containers/kata-containers-dev-preview.md index cddfe9e521..b023e713c8 100644 --- a/enhancements/kata-containers/kata-containers-dev-preview.md +++ b/enhancements/kata-containers/kata-containers-dev-preview.md @@ -136,7 +136,7 @@ The `RuntimeClass` and payload image names will be visible in the CR status. #### Goals - Create an API which supports installation of Kata Runtime on all or selected worker nodes - Configure CRI-O to use Kata Runtime on those worker nodes -- Installation of the runtimeClass on the cluster +- Installation of the runtimeClass on the cluster, as well as of the required components for the runtime to be controlled by the orchestration layer. - Updates the Kata runtime - Uninstall Kata Runtime and reconfigure CRI-O to not use it. From e50fbdf6dcb587fb20cdb04ad958c930797accd1 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Fabiano=20Fid=C3=AAncio?= Date: Thu, 4 Mar 2021 18:02:32 +0100 Subject: [PATCH 05/10] kata-containers: Use OpenShift instead of Openshift MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Signed-off-by: Fabiano Fidêncio --- .../kata-containers/kata-containers-dev-preview.md | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/enhancements/kata-containers/kata-containers-dev-preview.md b/enhancements/kata-containers/kata-containers-dev-preview.md index b023e713c8..58d5392173 100644 --- a/enhancements/kata-containers/kata-containers-dev-preview.md +++ b/enhancements/kata-containers/kata-containers-dev-preview.md @@ -24,12 +24,12 @@ superseded-by: [Kata Containers](https://katacontainers.io/) is an open source project developing a container runtime using virtual machines and providing the same look and feel as vanilla containers. By leveraging hardware virtualization technologies, Kata Containers provides powerful workload isolation compared to existing container solutions. -We will be integrating Kata Containers into Openshift to provide the ability to run kernel isolated containers for any workload which requires custom kernel tuning (sysctl, scheduler changes, cache tuning, etc), custom kernel modules (out of tree, special arguments, etc), exclusive access to hardware, root privileges, or other administrative privileges above and beyond what is secure in a shared kernel environment (regular runc). +We will be integrating Kata Containers into OpenShift to provide the ability to run kernel isolated containers for any workload which requires custom kernel tuning (sysctl, scheduler changes, cache tuning, etc), custom kernel modules (out of tree, special arguments, etc), exclusive access to hardware, root privileges, or other administrative privileges above and beyond what is secure in a shared kernel environment (regular runc). It should be noted that Kata Containers differ from Kubevirt: -- Kubevirt aims to run **VM images** on Openshift providing the VM with the look and feel of a virtual machine running as a legacy VM (CRD defining a Virtual Machine in Kubernetes) -- Kata Containers aim to run **container images** on Openshift in an isolated manner using virtualization tools (uses Runtime Class resource to choose Kata only for specific workloads which require kernel level isolation) +- Kubevirt aims to run **VM images** on OpenShift providing the VM with the look and feel of a virtual machine running as a legacy VM (CRD defining a Virtual Machine in Kubernetes) +- Kata Containers aim to run **container images** on OpenShift in an isolated manner using virtualization tools (uses Runtime Class resource to choose Kata only for specific workloads which require kernel level isolation) ## Motivation If we take telcos for example, there are a number of reasons they require isolated workloads: @@ -51,7 +51,7 @@ If we take telcos for example, there are a number of reasons they require isolat ## Non-goals -- Although kata containers are capable of supporting running a different kernel then the one used on the Openshift node, such features will not be supported. This is in order to simplify the development cycles and testing efforts of this project. +- Although kata containers are capable of supporting running a different kernel then the one used on the OpenShift node, such features will not be supported. This is in order to simplify the development cycles and testing efforts of this project. - Running OpenShift control plane and core operators on kata. These will continue to run on runc. @@ -65,7 +65,7 @@ Kubernetes provides support for RuntimeClasses. RuntimeClass is a feature for se CRI-O today comes out of the box with a runc as the default runtime. CRI-O also supports RuntimeClasses and using this configuration, it will support a KataContainers runtime as well. -The 2 fundamental problems we need to address for getting kata to work with Openshift are: +The 2 fundamental problems we need to address for getting kata to work with OpenShift are: 1. Getting the kata artifacts on to an RHCOS node 2. Deploying kata and configuring CRI-O to use it as the runtime @@ -238,14 +238,14 @@ spec: install-kata: kata-1.0 ``` -### Openshift kata testing +### OpenShift kata testing - Kubernetes E2E tests, matching the ones ran by CRI-O: https://github.com/cri-o/cri-o/blob/master/contrib/test/integration/e2e.yml#L14 - OpenShift "conformance/parallel" tests - Get sample workloads from customers and add tests to CI if we are missing coverage for those workloads to ensure we support the workloads and don’t regress. -### Openshift kata CI +### OpenShift kata CI - Leverage existing openshift CI infrastructure for e2e tests on kata-containers upstream From de858c150c53465098c3d5570ac3d6066e236ec9 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Fabiano=20Fid=C3=AAncio?= Date: Thu, 4 Mar 2021 18:07:51 +0100 Subject: [PATCH 06/10] kata-containers: Remove trailing whitespaces MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit This commit silences "MD009/no-trailing-spaces Trailing space" markdown lint error. Signed-off-by: Fabiano Fidêncio --- enhancements/kata-containers/kata-containers-dev-preview.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/enhancements/kata-containers/kata-containers-dev-preview.md b/enhancements/kata-containers/kata-containers-dev-preview.md index 58d5392173..4b5fcf6a54 100644 --- a/enhancements/kata-containers/kata-containers-dev-preview.md +++ b/enhancements/kata-containers/kata-containers-dev-preview.md @@ -34,7 +34,7 @@ It should be noted that Kata Containers differ from Kubevirt: ## Motivation If we take telcos for example, there are a number of reasons they require isolated workloads: -1. **Telco CNF deployments** - As part of 5G/NFV telco deployments there is a gradual migration from physical networking boxes to CNFs (container network functions). Some of these CNFs require root access while others could potentially create a threat on the operators cloud bypassing existing container isolation mechanisms. For addressing those issues a hardened isolated container solution is required +1. **Telco CNF deployments** - As part of 5G/NFV telco deployments there is a gradual migration from physical networking boxes to CNFs (container network functions). Some of these CNFs require root access while others could potentially create a threat on the operators cloud bypassing existing container isolation mechanisms. For addressing those issues a hardened isolated container solution is required 2. **Devops deployments** - Telcos are working hard to increase the rate of adding new capabilities to their clouds for competing with OTTs (over the top companies such as Netflix, Apple, Facebook etc…). This involves huge investments in devops tools and processes. In order for devops to deploy new features on production environments there is a need for high isolation of the deployed workloads in order to control and revert such changes. 3. **Hardware Vendors** - many hardware vendors require custom kernel parameters at boot, custom sysctl variables, etc. They want to use containers as a convenient packaging format for their entire application, but still need access to lower level tuning for portions of their applications @@ -97,7 +97,7 @@ In this option we build upon the RHCOS extension approach planned for OCP4.6. This will add additional software onto the host - but this software will still be versioned with the host (included as part of the OpenShift release payload) and upgraded with the cluster. -The additional RPMs required for qemu-kiwi and its dependencies are: +The additional RPMs required for qemu-kiwi and its dependencies are: * qemu-kiwi * qemu-kvm-common * pxe-roms-qemu @@ -127,7 +127,7 @@ We start by creating a container image with only the kata upstream components as ### KataContainer Operator development -The goal is to develop a kata operator that can be used to manage the entire lifecycle of kata components in an OpenShift cluster. We will have a controller that watches for a kata **custom resource (CR)** and a daemon-set that acts upon changes to the (CR) and do the required work on the worker nodes (install, uninstall update,...). +The goal is to develop a kata operator that can be used to manage the entire lifecycle of kata components in an OpenShift cluster. We will have a controller that watches for a kata **custom resource (CR)** and a daemon-set that acts upon changes to the (CR) and do the required work on the worker nodes (install, uninstall update,...). Via the CR it will be possible to select a subset of worker nodes. For deploying binaries on the host the current idea is to use a container image that can be mounted on the host level. The goal is to be able to install the operator via OperatorHub. The operator will also create a crio drop-in config file via machine config objects. The Operator will automatically select the payload image that contains correct version of the kata binaries for the given version of the OpenShift. The Operator will also configure a `RuntimeClass` called `kata` which can be used to deploy workload that uses kata runtime. The operator will create a machine-config object to enable the `qemu-kiwi` RHCOS extension. From c0552910ce4037c08efaf68ffe009ab4420749dd Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Fabiano=20Fid=C3=AAncio?= Date: Thu, 4 Mar 2021 18:12:03 +0100 Subject: [PATCH 07/10] kata-containers: Make the headers more specific MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit This commit silences "MD024/no-duplicate-heading/no-duplicate-header Multiple headings with the same content" markdown lint error. Signed-off-by: Fabiano Fidêncio --- .../kata-containers/kata-containers-dev-preview.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/enhancements/kata-containers/kata-containers-dev-preview.md b/enhancements/kata-containers/kata-containers-dev-preview.md index 4b5fcf6a54..2d59dd362c 100644 --- a/enhancements/kata-containers/kata-containers-dev-preview.md +++ b/enhancements/kata-containers/kata-containers-dev-preview.md @@ -44,13 +44,13 @@ If we take telcos for example, there are a number of reasons they require isolat - **RuntimeClass** - see https://kubernetes.io/docs/concepts/containers/runtime-class/ -## Goals +## Enhancement Goals - Provide a way to enable running user workloads on kata containers on an OpenShift cluster. - Clusters with kata container workloads should support upgrades. - Add security policy to control which users can run what workloads on kata in an OpenShift cluster. -## Non-goals +## Enhancement Non-Goals - Although kata containers are capable of supporting running a different kernel then the one used on the OpenShift node, such features will not be supported. This is in order to simplify the development cycles and testing efforts of this project. - Running OpenShift control plane and core operators on kata. These will continue to run on runc. @@ -133,14 +133,14 @@ Via the CR it will be possible to select a subset of worker nodes. For deploying The `RuntimeClass` and payload image names will be visible in the CR status. -#### Goals +#### KataContainer Operator Goals - Create an API which supports installation of Kata Runtime on all or selected worker nodes - Configure CRI-O to use Kata Runtime on those worker nodes - Installation of the runtimeClass on the cluster, as well as of the required components for the runtime to be controlled by the orchestration layer. - Updates the Kata runtime - Uninstall Kata Runtime and reconfigure CRI-O to not use it. -#### Non-Goals +#### KataContainer Operator Non-Goals To keep the Kata Operator's goal to the lifecycle management of the Kata Runtime, it will only support installation configuration of the Kata Runtime. This operator will not interact with any runtime configuration, such as Pod Annoations supported by Kata. #### Proposal From 03c58469f2f99ba0007b3b97385f862a43c33d66 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Fabiano=20Fid=C3=AAncio?= Date: Thu, 4 Mar 2021 18:22:50 +0100 Subject: [PATCH 08/10] kata-containers: Use KataContainers instead of KataContainer MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Signed-off-by: Fabiano Fidêncio --- .../kata-containers/kata-containers-dev-preview.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/enhancements/kata-containers/kata-containers-dev-preview.md b/enhancements/kata-containers/kata-containers-dev-preview.md index 2d59dd362c..6960071f92 100644 --- a/enhancements/kata-containers/kata-containers-dev-preview.md +++ b/enhancements/kata-containers/kata-containers-dev-preview.md @@ -76,7 +76,7 @@ For addressing the second issue of deploying and configuring CRI-O we will use a The next sections detailed the 2 solutions. -### KataContainer Packaging +### KataContainers Packaging #### Short term (preview release): Use container images to deliver RPMs and install using rpm-ostree In this option we build a container image containing RHEL AV RPMs (such as QEMU) and kata RPMs. The kata operator will create a pod based on this image and then mount the host filesystem and install the RPMs on the filesystem (kata operators will be described in a later section). @@ -126,21 +126,21 @@ We start by creating a container image with only the kata upstream components as -### KataContainer Operator development +### KataContainers Operator development The goal is to develop a kata operator that can be used to manage the entire lifecycle of kata components in an OpenShift cluster. We will have a controller that watches for a kata **custom resource (CR)** and a daemon-set that acts upon changes to the (CR) and do the required work on the worker nodes (install, uninstall update,...). Via the CR it will be possible to select a subset of worker nodes. For deploying binaries on the host the current idea is to use a container image that can be mounted on the host level. The goal is to be able to install the operator via OperatorHub. The operator will also create a crio drop-in config file via machine config objects. The Operator will automatically select the payload image that contains correct version of the kata binaries for the given version of the OpenShift. The Operator will also configure a `RuntimeClass` called `kata` which can be used to deploy workload that uses kata runtime. The operator will create a machine-config object to enable the `qemu-kiwi` RHCOS extension. The `RuntimeClass` and payload image names will be visible in the CR status. -#### KataContainer Operator Goals +#### KataContainers Operator Goals - Create an API which supports installation of Kata Runtime on all or selected worker nodes - Configure CRI-O to use Kata Runtime on those worker nodes - Installation of the runtimeClass on the cluster, as well as of the required components for the runtime to be controlled by the orchestration layer. - Updates the Kata runtime - Uninstall Kata Runtime and reconfigure CRI-O to not use it. -#### KataContainer Operator Non-Goals +#### KataContainers Operator Non-Goals To keep the Kata Operator's goal to the lifecycle management of the Kata Runtime, it will only support installation configuration of the Kata Runtime. This operator will not interact with any runtime configuration, such as Pod Annoations supported by Kata. #### Proposal From d89d0660d659432240f8a46d0b8492120c4ac07b Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Fabiano=20Fid=C3=AAncio?= Date: Thu, 4 Mar 2021 18:19:57 +0100 Subject: [PATCH 09/10] kata-containers: Use Operator rather than operator MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Signed-off-by: Fabiano Fidêncio --- enhancements/kata-containers/kata-containers-dev-preview.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/enhancements/kata-containers/kata-containers-dev-preview.md b/enhancements/kata-containers/kata-containers-dev-preview.md index 6960071f92..70a70d9f00 100644 --- a/enhancements/kata-containers/kata-containers-dev-preview.md +++ b/enhancements/kata-containers/kata-containers-dev-preview.md @@ -129,7 +129,7 @@ We start by creating a container image with only the kata upstream components as ### KataContainers Operator development The goal is to develop a kata operator that can be used to manage the entire lifecycle of kata components in an OpenShift cluster. We will have a controller that watches for a kata **custom resource (CR)** and a daemon-set that acts upon changes to the (CR) and do the required work on the worker nodes (install, uninstall update,...). -Via the CR it will be possible to select a subset of worker nodes. For deploying binaries on the host the current idea is to use a container image that can be mounted on the host level. The goal is to be able to install the operator via OperatorHub. The operator will also create a crio drop-in config file via machine config objects. The Operator will automatically select the payload image that contains correct version of the kata binaries for the given version of the OpenShift. The Operator will also configure a `RuntimeClass` called `kata` which can be used to deploy workload that uses kata runtime. The operator will create a machine-config object to enable the `qemu-kiwi` RHCOS extension. +Via the CR it will be possible to select a subset of worker nodes. For deploying binaries on the host the current idea is to use a container image that can be mounted on the host level. The goal is to be able to install the operator via OperatorHub. The Operator will also create a crio drop-in config file via machine config objects. The Operator will automatically select the payload image that contains correct version of the kata binaries for the given version of the OpenShift. The Operator will also configure a `RuntimeClass` called `kata` which can be used to deploy workload that uses kata runtime. The Operator will create a machine-config object to enable the `qemu-kiwi` RHCOS extension. The `RuntimeClass` and payload image names will be visible in the CR status. From 043e2daeb519c28019e7295782ada0da7003cc06 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Fabiano=20Fid=C3=AAncio?= Date: Thu, 4 Mar 2021 18:15:48 +0100 Subject: [PATCH 10/10] kata-containers: Rewrite sentences in order to make them shorter MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Signed-off-by: Fabiano Fidêncio --- .../kata-containers-dev-preview.md | 22 ++++++++++++++++--- 1 file changed, 19 insertions(+), 3 deletions(-) diff --git a/enhancements/kata-containers/kata-containers-dev-preview.md b/enhancements/kata-containers/kata-containers-dev-preview.md index 70a70d9f00..9f79c515ce 100644 --- a/enhancements/kata-containers/kata-containers-dev-preview.md +++ b/enhancements/kata-containers/kata-containers-dev-preview.md @@ -24,7 +24,12 @@ superseded-by: [Kata Containers](https://katacontainers.io/) is an open source project developing a container runtime using virtual machines and providing the same look and feel as vanilla containers. By leveraging hardware virtualization technologies, Kata Containers provides powerful workload isolation compared to existing container solutions. -We will be integrating Kata Containers into OpenShift to provide the ability to run kernel isolated containers for any workload which requires custom kernel tuning (sysctl, scheduler changes, cache tuning, etc), custom kernel modules (out of tree, special arguments, etc), exclusive access to hardware, root privileges, or other administrative privileges above and beyond what is secure in a shared kernel environment (regular runc). +We will be integrating Kata Containers into OpenShift to provide the ability to run kernel isolated containers for any workload which requires: +- custom kernel tuning (sysctl, scheduler changes, cache tuning, etc); +- custom kernel modules (out of tree, special arguments, etc); +- exclusive access to hardware; +- root privileges; +- any other administrative privileges above and beyond what is secure in a shared kernel environment (regular runc). It should be noted that Kata Containers differ from Kubevirt: @@ -35,7 +40,8 @@ It should be noted that Kata Containers differ from Kubevirt: If we take telcos for example, there are a number of reasons they require isolated workloads: 1. **Telco CNF deployments** - As part of 5G/NFV telco deployments there is a gradual migration from physical networking boxes to CNFs (container network functions). Some of these CNFs require root access while others could potentially create a threat on the operators cloud bypassing existing container isolation mechanisms. For addressing those issues a hardened isolated container solution is required -2. **Devops deployments** - Telcos are working hard to increase the rate of adding new capabilities to their clouds for competing with OTTs (over the top companies such as Netflix, Apple, Facebook etc…). This involves huge investments in devops tools and processes. In order for devops to deploy new features on production environments there is a need for high isolation of the deployed workloads in order to control and revert such changes. +2. **Devops deployments** - Telcos are working hard to increase the rate of adding new capabilities to their clouds for competing with OTTs (over the top companies such as Netflix, Apple, Facebook etc…). + This involves huge investments in devops tools and processes. In order for devops to deploy new features on production environments there is a need for high isolation of the deployed workloads in order to control and revert such changes. 3. **Hardware Vendors** - many hardware vendors require custom kernel parameters at boot, custom sysctl variables, etc. They want to use containers as a convenient packaging format for their entire application, but still need access to lower level tuning for portions of their applications @@ -129,7 +135,17 @@ We start by creating a container image with only the kata upstream components as ### KataContainers Operator development The goal is to develop a kata operator that can be used to manage the entire lifecycle of kata components in an OpenShift cluster. We will have a controller that watches for a kata **custom resource (CR)** and a daemon-set that acts upon changes to the (CR) and do the required work on the worker nodes (install, uninstall update,...). -Via the CR it will be possible to select a subset of worker nodes. For deploying binaries on the host the current idea is to use a container image that can be mounted on the host level. The goal is to be able to install the operator via OperatorHub. The Operator will also create a crio drop-in config file via machine config objects. The Operator will automatically select the payload image that contains correct version of the kata binaries for the given version of the OpenShift. The Operator will also configure a `RuntimeClass` called `kata` which can be used to deploy workload that uses kata runtime. The Operator will create a machine-config object to enable the `qemu-kiwi` RHCOS extension. +Via the CR it will be possible to select a subset of worker nodes. + +For deploying binaries on the host the current idea is to use a container image that can be mounted on the host level. + +The goal is to be able to install the operator via OperatorHub. + +The Operator will: +- Create a crio drop-in config file via machine config objects; +- Automatically select the payload image that contains correct version of the kata binaries for the given version of the OpenShift; +- Configure a `RuntimeClass` called `kata` which can be used to deploy workload that uses kata runtime; +- Create a machine-config object to enable the `qemu-kiwi` RHCOS extension. The `RuntimeClass` and payload image names will be visible in the CR status.