How to implement approval workflows in Rukpak

[joelanford]

There's an existing issue for this topic in #113, but I figured it might make sense to start a discussion while we explore this space since we have nothing concrete to do yet.

Background

RukPak is designed to handle installation and some aspects of lifecycling arbitrary bundles that can be applied to Kubernetes clusters. These bundles often contain cluster extensions which means that cluster administrators must have awareness of their contents to ensure that they are secure and cannot interfere with or introduce vulnerabilities to their cluster.

To ensure that cluster admins have this awareness, a naive solution is to give only trusted administrators the RBAC that is necessary to create and update bundle instances.

While this solution would work, it limits the ways that RukPak can be used. It forces cluster administrators to directly touch individual bundle instance objects in order to pivot them between versions. Some cluster admins would find this behavior compelling because it guarantees that no unintended changes are applied to the system.

However another class of cluster administrators may want to build automation that enables bundle instances to pivot between versions as new versions of their bundles become available. For example, a cluster administrator might wish to upgrade their bundle instances as soon as new security patches are available for them. It would be tedious for a cluster admin to a) know when security upgrades are available and b) check often enough to ensure their systems are net left vulnerable and c) manually update each bundle instance.

With this latter use case in mind, the Operator Lifecycle Manager project introduced APIs that enable the cluster to automatically receive, discover, and upgrade applications without administrator interaction:

• A CatalogSource is an abstraction that allows OLM to find the set of available bundles to install. Catalogs are analogous to yum or apt repositories.
• A Subscription is (among other things) a declaration of a cluster users desire to install a package and keep it up-to-date.
• An InstallPlan is a realization of the set of packages to install.

OLM currently has a course-grained approval flow that enables cluster users to:

1. Specify whether packages should be automatically or manually upgraded
2. For manual approval mode, specify that a specific InstallPlan is approved (in automatic mode, OLM provides the approval)

There are several shortcomings in OLM's approval flow:

1. There's no concept of pinning the version of particular packages. So you either have to accept all upgrades or no upgrades at all
2. Cluster admins are not given easy insight into what they are approving. While a cluster admin may review the permissions requested by a bundle prior to the first install, they are not informed when a new bundle version requests additional permissions. Android's permission model prompts users when an application asks for permissions that have not yet been reviewed by the user.

Lastly, approval is inherently imperative, so there are UX concerns with implementing an approval workflow in a Kubernetes API because Kubernetes APIs are declarative. Our solution needs to be mindful of this fact. Some questions in this area are:

1. What is the mechanic that actually implements approval or denial?
2. What is the mechanic for revoking an approval?
3. If there is no revocation mechanic, what options does a cluster admin have to unwind or reverse their previous approval decision?
4. Does revocation or reversal of a decision affect only RBAC or does it also affect workloads?
   • Author note: it seems like both are applicable. If I detect that a bundle gave itself specific permissions I don't want it to have, can I revoke just that RBAC and leave the bundle's workloads running? If I detect that an operator bundle upgrade brought in a bitcoin mining daemonset alongside the controller workloads, can I downgrade or switch the implementation out without losing my operands?

Inspiration

I haven't looked much, but the closest thing I've found that seems applicable to our APIs is cert-manager's CertificateRequest API. It sits between Certificates and Issuers. A user creates a Certificate for which the cert-manager controller creates a CertificateRequest. An isssuer implementation does not act on a CertificateRequest until it has been approved.

Cert-manager has no opinion on what actually implements the approval of a CertificateRequest. It can be another controller that inspects the CertificateRequest and automatically approves it, or it can be a human.

Cert-manager defines a cluster role that approvers must have in order to be allowed to apply approvals or denials, and it implements the authorization using a validating webhook with a SubjectAccessReview (details can be found in the linked design doc)

Applications in RukPak

RukPak is the lowest layer of bundle lifecycling. It is not involved in sourcing repository metadata or determining when upgrades are available. It is simply the abstraction layer that allows for simple installation of bundle content and pivots from one bundle to another.

The obvious way to implement approval in RukPak similar to the cert-manager design is to introduce a new API that sits between BundleInstances and Provisioners. A provisioner would not act on installing or pivoting a BundleInstance until its BundleInstanceRequest was approved.

Is this the right layer for approval to work? Do we need multiple layers of approval (e.g. at higher levels of the API stack)? Deppy comes to mind as another API where some approval controls could be implemented. In the case of Deppy being used to generate a cluster-wide resolution of a set of compatible bundles, it probably does not make sense to allow some of those bundles to be upgraded but not others because that could cause incompatibilities between installed components. Some bundles may need to be upgraded in lock step in order to work properly.