rhcos/extensions: Support for CoreOS extensions

This enhancement proposes a MachineConfig fragment like:
```
extensions:
  - usbguard
```

This is the OpenShift version of the [Fedora CoreOS extension system tracker](coreos/fedora-coreos-tracker#401).

That will add additional software onto the host,
but this software will still be versioned with the host
(included as part of the OpenShift release payload) and
upgraded with the cluster.

enhancements/rhcos/extensions.md

@@ -0,0 +1,96 @@
+---
+title: support-for-coreos-extensions
+authors:
+  - "@cgwalters"
+reviewers:
+  - "@ashcrow"
+approvers:
+  - "@ashcrow"
+  - "@crawford"
+  - "@imcleod"
+  - "@runcom"
+creation-date: 2020-04-21
+last-updated: 2020-04-21
+status: provisional
+see-also:
+replaces:
+superseded-by:
+---
+
+# Support For CoreOS extensions
+
+## Release Signoff Checklist
+
+- [ ] Enhancement is `implementable`
+- [ ] Design details are appropriately documented from clear requirements
+- [ ] Test plan is defined
+- [ ] Graduation criteria for dev preview, tech preview, GA
+- [ ] User-facing documentation is created in [openshift-docs](https://github.com/openshift/openshift-docs/)
+
+## Summary
+
+This enhancement proposes a MachineConfig fragment like:
+```
+extensions:
+  - usbguard
+```
+
+This is the OpenShift version of the [Fedora CoreOS extension system tracker](https://github.com/coreos/fedora-coreos-tracker/issues/401).
+
+That will add additional software onto the host - but this software will still be versioned with the host (included as part of the OpenShift release payload) and upgraded with the cluster.
+
+## Motivation
+
+We have requests to ship things like `usbguard` that are needed to meet compliance in some scenarios (and `usbguard` is very useful on bare metal), but `usbguard` also makes no sense to ship in an OS for the public cloud.
+
+The `rpm-ostree` project has had as its goal from the start to re-cast RPMs as "operating system extensions" - much like how e.g. Firefox and its extensions work.  By default it operates as a pure image system, but packages can be layered on (and overridden) client side.
+
+OpenShift is already making use of this today for the [realtime kernel](https://github.com/openshift/enhancements/blob/master/enhancements/support-for-realtime-kernel.md).
+
+We propose continuing the trail blazed with the RT kernel by adding additional RPMs to `machine-os-content` that aren't part of the base OS "image", but can be installed later.  This is how `kernel-rt` works today; we added a `kernel-rt.rpm` that is committed to the container.  In the future though, we may instead ship a separate `machine-os-extensions` container, or something else.  The only API stable piece here is the `MachineConfig` field (same as for the `kernelType: realtime`).
+
+An important way to think about this is that we can still ascribe a *single version number* to this set of content - all content is still versioned with and tested with the OS (and hence the main OpenShift 4 release image).
+
+### Goals
+
+- Add support for cluster operators (notably the MCO) to deploy non-containerized software that is maintained alongside OpenShift
+- Avoid forcing the authors of software like `usbguard` to maintain two ways to ship their software (as RPM and as a container)
+
+### Non-Goals
+
+- Direct support for installing RPMs *not* from the release image
+
+## Proposal
+
+1.  RHCOS build system is updated to inject `usbguard` (and other software) into `machine-os-content` as an RPM
+2.  MachineConfig gains support for `extensions` that are layered on in the same way it applies OS updates and manages `kernel-rt`
+
+### Implementation Details/Notes/Constraints
+
+Note this will also likely enable us to move `open-vm-tools` out of the host and have the MCO install it on VSphere environments for example.
+
+### Risks and Mitigations
+
+#### 3rd party RPMs
+
+This will blaze a trail that will make it easier to install 3rd party RPMs, which is much more of a risk in terms of compatibility and for upgrades.
+
+#### Non-Kubernetes native management
+
+How one *ships* software has a huge impact on how one *manages* the software.  Shipping software as an RPM implies nowadays it usually runs as a `systemd` unit, parses config files in `/etc`, logs to the journal etc.
+
+When one ships software as a container for Kubernetes, one can use `oc/kubectl` to inspect its status, it might offer custom resource definitions, etc.  And we have made investment in ensuring a "Kubernetes native" interface to important components.
+
+Writing containers should be preferred - but, we cannot do that all at once, for all software that is relevant for host execution.
+
+### Upgrades
+
+The MCO will update the OS and extensions as one unit.  All shipped extensions will have been tested with the same version of the host.
+
+## Alternatives
+
+Multiple OS builds: Becomes a combinatorial nightmare.
+
+Do nothing: Probably RHCOS will continue to grow and we will have to look embarrassed when someone asks us why `usbguard` is installed in their AWS cluster where the VMs don't have USB.
+
+Force `usbguard` authors to containerize: Not a technical problem exactly but...it is *really* hard to have two ways to ship software; see above.