From b4650860f904d0e3e020099e884f05cf26d97385 Mon Sep 17 00:00:00 2001 From: Dan Winship Date: Wed, 12 Jan 2022 13:51:28 -0500 Subject: [PATCH] initial wip synchronized-upgrades --- enhancements/update/synchronized-upgrades.md | 302 +++++++++++++++++++ 1 file changed, 302 insertions(+) create mode 100644 enhancements/update/synchronized-upgrades.md diff --git a/enhancements/update/synchronized-upgrades.md b/enhancements/update/synchronized-upgrades.md new file mode 100644 index 00000000000..5452e945cfa --- /dev/null +++ b/enhancements/update/synchronized-upgrades.md @@ -0,0 +1,302 @@ +--- +title: Synchronized Upgrades Between Clusters +authors: + - "@danwinship" +reviewers: + - "@dhellmann" + - "@sdodson" + - "@wking" + - "@romfreiman" + - "@zshi" +approvers: + - TBD +api-approvers: # in case of new or modified APIs or API extensions (CRDs, aggregated apiservers, webhooks, finalizers) + - TBD +creation-date: 2021-01-11 +last-updated: 2021-01-11 +tracking-link: + - https://issues.redhat.com/browse/SDN-2603 +see-also: + - "/enhancements/network/dpu/overview.md" +--- + +# Synchronized Upgrades Between Clusters + +## Summary + +Our architecture for [using DPUs (eg, BlueField-2 NICs) in +OCP](../network/dpu/overview.md) involves creating one ordinary +"tenant" OCP cluster consisting of x86 hosts, and a second "infra" OCP +cluster consisting of the ARM systems running on the NICs installed in +the x86 hosts. + +In the current Dev Preview of DPU support, there is no coordination +between the infra and tenant clusters during upgrades. Thus, when +upgrading the infra cluster, each NIC will end up rebooting at some +point, after it has drained _its own_ pods, but without having made +any attempt to drain the pods of its x86 tenant node. When this +happens, the x86 node (and its pods) will lose network connectivity +until the NIC finishes rebooting. + +In order to make upgrades in DPU clusters work smoothly, we need some +way to synchronize the reboots between the two clusters, so that the +NICs get rebooted during the period when their corresponding x86 hosts +are cordoned and drained. + +## Glossary + +(Copied from the [DPU Overview +Enhancement](../network/dpu/overview.md).) + +- **DPU** / **Data Processing Unit** - a Smart NIC with a full CPU, + RAM, storage, etc, running a full operating system, able to offload + network processing and potentially other tasks from its host system. + For example, the NVIDIA BlueField-2. + +- **DPU NIC** - refers to the DPU and its CPU, OS, etc (as opposed to + the CPU, OS, etc, of the x86 server that the DPU is installed in). A + **DPU NIC node** or **DPU NIC worker** is an OCP worker node + running on a DPU NIC. + +- **DPU Host** - refers to an x86 server (and its CPU, OS, etc) which + contains a DPU. A **DPU Host node** or **DPU Host worker** is an OCP + worker node running on a DPU Host. + +- **Two-Cluster Design** - any architecture for deploying an OCP + cluster with (some) nodes containing DPUs, where the DPUs are nodes + in a second OCP cluster. (For purposes of this term, the + architecture still counts as "two-cluster" even if HyperShift is + involved and there are actually three clusters.) + +- **Infra Cluster** - in a Two-Cluster Design, the OCP cluster which + contains the DPU NIC workers. (It may also include master nodes + and/or non-DPU workers, depending on the particular details of the + two-cluster design.) + +- **Tenant Cluster** - in a Two-Cluster Design, the OCP cluster + containing the DPU Host workers. (It may also include master nodes + and/or workers without DPUs.) + +## Motivation + +### Goals + +- Make upgrades work smoothly in clusters running with DPU support, by + synchronizing the reboots of nodes between the infra cluster and the + tenant cluster. + +### Non-Goals + +- Supporting synchronized upgrades of more than 2 clusters at once. + +## Proposal + +### User Stories + +As the administrator of a cluster using DPUs, I want to be able to do +y-stream and z-stream upgrades without causing unnecessary network outages. + +### API Extensions + +TBD + +### Implementation Details/Notes/Constraints + +#### Setup + +We can set some things up at install time (eg, creating credentials to +allow certain operators in the two clusters to talk to each other). + +#### Initiating Upgrades + +As part of the DPU security model, the tenant cluster cannot have any +power over the infra cluster. In particular, it can't be possible for +an administrator in the tenant cluster to force the infra cluster to +upgrade/downgrade to any particular version. + +Thus, the infra cluster upgrade must be initiated on the infra cluster +side. In theory, we could have the infra cluster then initiate the +tenant cluster upgrade, but that would require having a tenant cluster +cluster-admin admin credential lying around in the infra cluster +somewhat unnecessarily. It probably makes more sense to require the +administrator to initiate the upgrade on the tenant side as well. + +#### Inter-Cluster Version Skew + +There is very little explicit communication between the two clusters: + + 1. The dpu-operator in the infra cluster talks to the apiserver in + the tenant cluster. + + 2. The ovn-kubernetes components in the two clusters do not + communicate directly, but they do make some assumptions about + each other's behavior (which is necessary to coordinate the + plumbing of pod networking). + +The apiserver communication is very ordinary and boring and not likely +to be subject to any interesting version skew problems (even if we +ended up with, say, a 3-minor-version skew between the clusters). + +Thus, other than ovn-kubernetes, no OCP component needs to be +concerned about version skew between the two clusters, because the +components in the two clusters are completely unaware of each other. + +For ovn-kubernetes, if we ever change the details of the cross-cluster +communication, then we will need to add proper checks to enforce +tolerable cross-cluster skew at that time. + +## Design Details + +The upgrade plan takes advantage of two things: + +- Other than the reboots, no part of the upgrade needs to be + synchronized between the two clusters. + +- Each infra node only runs pods to support the pods on its + corresponding tenant node. Thus, if a tenant node reboots, it is + safe to immediately reboot its infra node as well without needing + to cordon and drain it, because it is guaranteed that none of its + pods are doing anything important at that point. + +So, both clusters' upgrades can proceed normally up until the MCO +upgrade. + +In the infra cluster, rather than cordoning, draining, and rebooting +nodes one-by-one, the MCO will instead queue up the new RHCOS image on +each node, and then mark the node (somehow) as "waiting for a reboot". +Then the MCO will watch the nodes and wait for each one to reboot (of +its own accord), and then finally mark itself fully-upgraded after +that happens. + +On the infra nodes, when they are "waiting for a reboot", they will +monitor the state of the ovn-kubernetes network; when they determine +that the tenant node is rebooting, the infra node will then also +reboot. + +In terms of synchronization, this means that we want to avoid the +tenant cluster reaching the MCO upgrade stage when the infra cluster +is not ready for the tenant cluster to start rebooting nodes: + +- One way to do this would be to not let a tenant cluster start an + upgrade until the infra cluster has already reached the "waiting + for reboots" stage. This could be done by having an operator in + the tenant cluster whose status is always `Upgradeable: False` + except when the infra cluster tells it that it is safe to + proceed. + +- Another possibility would be to have an operator that sits just + before the MCO in the upgrade order, which checks if the infra + cluster is in the middle of an upgrade, and if so, blocks until + the infra cluster is ready for the tenant cluster to proceed. (Or + rather, the infra cluster would notify it that it should/shouldn't + block the upgrade.) + + - This would allow for the possibility of doing tenant cluster + upgrades without corresponding infra cluster upgrades, which + might be useful? + +### Open Questions + +- How exactly will the coordination between the two MCO upgrades + occur? + +- How exactly will infra nodes detect their tenant node rebooting? + +- Do we want to allow (z-stream) upgrades of the tenant cluster + without also upgrading the infra cluster? + +### Risks and Mitigations + +The infra cluster could remain stuck at the pre-reboot stage of the +upgrade for arbitrarily long. In theory this should not cause any +problems, as all operators must be able to handle running with the old +RHCOS version anyway. But we should make sure that alerts eventually +get fired in this case. + +More TBD + +### Test Plan + +TBD + +### Graduation Criteria + +TBD + +#### Dev Preview -> Tech Preview + +#### Tech Preview -> GA + +#### Removing a deprecated feature + +### Upgrade / Downgrade Strategy + +This is a modification to the upgrade process, not something that can +be upgrade or downgraded on its own. + +TBD, as the details depend on the eventual design. + +### Version Skew Strategy + +TBD, as the details depend on the eventual design. + +### Operational Aspects of API Extensions + +TBD + +#### Failure Modes + +- The system might get confused and spuriously block upgrades that + should be allowed. + +- Communications failures might lead to upgrades failing without the + tenant cluster being able to figure out why they failed. + +- TBD + +#### Support Procedures + +TBD + +## Implementation History + +- Initial proposal: 2021-01-11 +- Updated for initial feedback: 2021-01-24 + +## Drawbacks + +This makes the upgrade process more complicated, which risks rendering +clusters un-upgradeable without manual intervention. + +However, without some form of synchronization, it is impossible to +have non-disruptive tenant cluster upgrades. + +## Alternatives + +### Never Reboot the DPUs + +This implies never upgrading OCP on the DPUs. I don't see how this +could work. + +### Don't Have an Infra Cluster + +If the DPUs were not all part of a single OCP cluster (for example, +they were just "bare" RHCOS hosts, or they were each running +Single-Node OpenShift), then it might be simpler to synchronize the +DPU upgrades with the tenant upgrades, because then each tenant could +coordinate the actions of its own DPU by itself. + +The big problem with this is that, for security reasons, we don't want +the tenants to have any control over their DPUs. (For some future use +cases, the DPUs will be used to enforce security policies on their +tenants.) + +### More closely coordinated reboots + +In the original proposal, the infra and tenant reboots were more +closely coordinated, with the infra and tenant MCOs communicating so +that the infra MCO could reboot each infra node at the same time as +the corresponding tenant node rebooted. But this (probably) turns out +to be unnecessary, as it should always be safe to just reboot the +infra nodes when their tenant nodes reboot, without needing to have +"planned" the reboot.