rhcos/extensions: Support for CoreOS extensions #317
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,97 @@ | ||
| --- | ||
| title: support-for-coreos-extensions | ||
| authors: | ||
| - "@cgwalters" | ||
| reviewers: | ||
| - "@ashcrow" | ||
| approvers: | ||
| - "@ashcrow" | ||
| - "@crawford" | ||
| - "@imcleod" | ||
| - "@runcom" | ||
| creation-date: 2020-04-21 | ||
| last-updated: 2020-04-21 | ||
| status: provisional | ||
| see-also: | ||
| replaces: | ||
| superseded-by: | ||
| --- | ||
|
|
||
| # Support For CoreOS extensions | ||
|
|
||
| ## Release Signoff Checklist | ||
|
|
||
| - [ ] Enhancement is `implementable` | ||
| - [ ] Design details are appropriately documented from clear requirements | ||
| - [ ] Test plan is defined | ||
| - [ ] Graduation criteria for dev preview, tech preview, GA | ||
| - [ ] User-facing documentation is created in [openshift-docs](https://github.com/openshift/openshift-docs/) | ||
|
|
||
| ## Summary | ||
|
|
||
| This enhancement proposes a MachineConfig fragment like: | ||
| ``` | ||
| extensions: | ||
| - usbguard | ||
| ``` | ||
|
|
||
| This is the OpenShift version of the [Fedora CoreOS extension system tracker](https://github.com/coreos/fedora-coreos-tracker/issues/401). | ||
|
|
||
| That will add additional software onto the host - but this software will still be versioned with the host (included as part of the OpenShift release payload) and upgraded with the cluster. | ||
|
|
||
| ## Motivation | ||
|
|
||
| We have requests to ship things like `usbguard` that are needed to meet compliance in some scenarios (and `usbguard` is very useful on bare metal), but `usbguard` also makes no sense to ship in an OS for the public cloud. | ||
|
|
||
| The `rpm-ostree` project has had as its goal from the start to re-cast RPMs as "operating system extensions" - much like how e.g. Firefox and its extensions work. By default it operates as a pure image system, but packages can be layered on (and overridden) client side. | ||
|
|
||
| OpenShift is already making use of this today for the [realtime kernel](https://github.com/openshift/enhancements/blob/master/enhancements/support-for-realtime-kernel.md). | ||
|
|
||
| We propose continuing the trail blazed with the RT kernel by adding additional RPMs to `machine-os-content` that aren't part of the base OS "image", but can be installed later. This is how `kernel-rt` works today; we added a `kernel-rt.rpm` that is committed to the container. In the future though, we may instead ship a separate `machine-os-extensions` container, or something else. The only API stable piece here is the `MachineConfig` field (same as for the `kernelType: realtime`). | ||
|
There was a problem hiding this comment. If we're going this route, why not publish all OS content into a container as a repo, install the content from said repo. Then we can support RHEL utilizing the same source of packages published per version. There was a problem hiding this comment. First: that's a much much larger set of stuff, including a lot of stuff that has no business ever being installed on RHCOS (e.g. The scale of this proposal is probably (hopefully) at most 50-100MB of additional stuff, which is an order of magnitude or two smaller in size. And by package count it's probably 20 packages tops versus thousands. Second...people who have reason to access all of that should have entitlements set up anyways. There was a problem hiding this comment. In addition to usbguard and real-time kernel, there are use cases for this regarding kata (see https://github.com/harche/enhancements-1/blob/66675b4dd1fde442504ac331bc6343d90cfe1f1f/enhancements/kata/kata-operator.md) and, presumably, KubeVirt. In both cases, you need a There was a problem hiding this comment. KubeVirt runs qemu in a container, it's not needed on the host There was a problem hiding this comment. @cgwalters I didn't mean package all of RHEL. I mean all of OpenShift. So whatever we're packing into CoreOS today, make those the same type of installation method as the 'extensions'. There was a problem hiding this comment.
Ah. Well...the thing is I'd really like to keep the default RHCOS being in "image mode" by default where we're doing the depsolving, selinux labeling etc. all on the server and not per machine. This of course gets into coreos/rpm-ostree#1237 I see what you're thinking here but the counter is that if you want your OS content lifecycled with the cluster...that's what RHCOS is. There was a problem hiding this comment.
This is an undecipherable rabbit hole for me. I'm sure it makes a lot of sense to people heavily involved with these things, but I find it quite unapproachable.
I fail to see a distinction. We have some process that pulls in RPM content and delivers them to a host. The only thing that is different is the underlying implementation. If we add these 'extensions' we're opening ourselves up to the exact same problems RHEL has when it comes to shipping content. Some package the user installs introduces a conflict. Most packages also have some sort of configuration. How are users going to handle configuration conflicts across versions? Not to say these problems are blockers, but they seem to diminish the utility of RHCOS vs RHEL. There's another proposal about building RPM ostrees in-cluster, and then also uploading images to the cloud for each release. We might as well just build RHEL images at this point. We might as well run a repo pod and stream just the bits that are updated rather than a whole ostree to each host. I think the OStree model is good and has it's place. Our software delivery model doesn't seem to necessarily be the best match, and it seems we're slowly rebuilding RHEL. There was a problem hiding this comment. Sorry I meant to link coreos/rpm-ostree#1081
Conflict with what? We own the set of extensions and testing them.
Do you have a more specific exmaple? Something like usbguard changing its configuration file incompatibly? That seems like a "don't do that" situation. But...I can imagine eventually we will need some sort of special handling in the MCO for this.
I won't attach my name to an operating system that leaves the root filesystem in a corrupted state if the kernel freezes or you run ENOSPC etc. during an update. There are many other benefits to an "image based" system by default, among them clear versioning - typing By default, you are running an exact set of bits that has been tested in CI rather than dynamically reassembled on each machine. And notably with this extension proposal...we would probably start without any enabled extensions, so there's still no client-side depsolving. There was a problem hiding this comment. Generic lifecycle of RPMs on top of RHCoS should be completely out of scope. We aren't shipping RHEL with OpenShift - we're shipping the package set that is needed to run containers, and everything else should be in a container. There was a problem hiding this comment. Are we just using this to avoid the real issue, of defining, what goes on the host and what goes into a container? There was a problem hiding this comment.
How does this work with base OS versioning? I guess in the existing case you must be doing this in the Actually, I think the proposal as written is ambiguous about whether you're proposing "there will be a single container in the release image ( (Although, I guess, we could install the RPMs into "higher-level" images like Anyway, I feel like CNI / coreos/fedora-coreos-tracker#354 should be explicitly either a Goal or a Non-Goal... There was a problem hiding this comment. I think there's really two levels to this proposal, and I guess I need to cleanly separate them. The first part is really about the customer UX for adding things that are not conceptually on by default, such as But what this does quickly bleed into is using such a mechanism for default parts of OpenShift, such as But were we to ship ovs that way, I think it wouldn't be something an administrator could choose in the MCO And the same would be true for the CNI binaries. Does that makes sense? There was a problem hiding this comment. We don't want to expose CNI plugins / system-vs-containerized OVS as end-user options, but I don't think it makes sense to have the MCO try to figure out what networking RPMs to install by analyzing the network config itself. It seems like it would make more sense to have CNO tell MCO what RPMs it needs, somehow. That may or may not make sense as part of the same mechanism as for the end-user-selectable extensions. There was a problem hiding this comment. OK yep, we're in sync on that then. |
||
|
|
||
| An important way to think about this is that we can still ascribe a *single version number* to this set of content - all content is still versioned with and tested with the OS (and hence the main OpenShift 4 release image). | ||
|
|
||
| ### Goals | ||
|
|
||
| - Add support for cluster operators (notably the MCO) to deploy non-containerized software that is tested and versioned along with the OS, but not installed by default | ||
| - Avoid forcing the authors of software like `usbguard` to maintain two ways to ship their software (as RPM and as a container) | ||
|
|
||
cgwalters marked this conversation as resolved.
Show resolved
Hide resolved
|
||
| ### Non-Goals | ||
|
|
||
| - Direct support for installing RPMs *not* from the release image | ||
|
There was a problem hiding this comment. Isn't this what customers actually want? There was a problem hiding this comment. Some of them probably, for some things... But trying to support it would be a huge increase in scope; for things we ship we know we've tested it in combination with the particular OS version. Doing that for random 3rd party RPMs (e.g. PAM hooks or hardware raid tooling) is a totally different thing. The other big problem domain here is: who chooses when updates for those 3rd party extensions (RPMs) happen? The easiest would be to only trigger when the base OS updates, but... There was a problem hiding this comment. Agreed. Some customers will want to install RPMs, but that doesn't make it supportable, opinionated, or the right move. I'd argue most customers are more interested in a stable, upgradable OCP (which RHCOS is an important part of).
cgwalters marked this conversation as resolved.
Show resolved
Hide resolved
|
||
| - Support for traditional RHEL systems (see below) | ||
|
|
||
| ## Proposal | ||
|
There was a problem hiding this comment. I don’t see it called out, but we will need telemetry from the MCO telling us what extensions are being used by customers. |
||
|
|
||
| 1. RHCOS build system is updated to inject `usbguard` (and other software) into `machine-os-content` as an RPM | ||
|
There was a problem hiding this comment. How do we decide what to include in the set of extensions over time? Will we ever remove extensions? There was a problem hiding this comment. RFE's more than likely. It's not likely we'd remove any extensions. |
||
| 2. MachineConfig gains support for `extensions` that are layered on in the same way it applies OS updates and manages `kernel-rt` | ||
|
|
||
| ### Implementation Details/Notes/Constraints | ||
|
|
||
| Note this will also likely enable us to move `open-vm-tools` out of the host and have the MCO install it on VSphere environments for example. | ||
|
|
||
| ### Risks and Mitigations | ||
|
|
||
| #### 3rd party RPMs | ||
|
|
||
| This will blaze a trail that will make it easier to install 3rd party RPMs, which is much more of a risk in terms of compatibility and for upgrades. | ||
|
There was a problem hiding this comment. Does this perpetuate bad/lazy behaviors. I touch on this in https://github.com/openshift/enhancements/pull/317/files#r425535006 In short, ISV's, Integrator's, Developer's (in general) will take the path of least resistance to get work done, and complete tasks (under the guys that they work). It sound like this proposal - does not expose this to ISV's and third parties (which I am ok with), but if we do ever decide to open this up, do we run the risk of forcing components to be containerized. Do we begin to show / set a bad example, simply for the sake of easing friction? There was a problem hiding this comment. That's correct, this feature does not support 3rd parties.
The way I see things is we have a spectrum. We can't change the whole world at one time. |
||
|
|
||
| #### Non-Kubernetes native management | ||
|
|
||
| How one *ships* software has a huge impact on how one *manages* the software. Shipping software as an RPM implies nowadays it usually runs as a `systemd` unit, parses config files in `/etc`, logs to the journal etc. | ||
|
|
||
| When one ships software as a container for Kubernetes, one can use `oc/kubectl` to inspect its status, it might offer custom resource definitions, etc. And we have made investment in ensuring a "Kubernetes native" interface to important components. | ||
|
|
||
| Writing containers should be preferred - but, we cannot do that all at once, for all software that is relevant for host execution. | ||
|
|
||
| ### Upgrades | ||
|
|
||
| The MCO will update the OS and extensions as one unit. All shipped extensions will have been tested with the same version of the host. | ||
|
|
||
| ## Alternatives | ||
|
|
||
| Multiple OS builds: Becomes a combinatorial nightmare. | ||
|
|
||
| Do nothing: RHCOS may continue to grow with every new case that is expected to be supported. This would mean packages, such as `usbguard`, special cloud agents, debug or development tools, and other packages would increase the size of the base image while only providing features to a subset of customers | ||
|
|
||
| Force `usbguard` authors to containerize: Not a technical problem exactly but...it is *really* hard to have two ways to ship software; see above. | ||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
As far as I understand the list of extensions is going to increase in the future depending upon requirements.
One concern I have in terms of implementation is how we are going to manage additional dependencies on which these extension packages may depend? For example, installing usbguard also installs package foo as dependency but foo is not being shipped in BaseOS.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We include those dependencies in an extra
dependencies/directory currently.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
sweet, didn't know about that.