From 965d2c8fb1cf08aa84c196333707363544fa5d5a Mon Sep 17 00:00:00 2001 From: Eran Cohen Date: Sun, 13 Dec 2020 19:40:31 +0200 Subject: [PATCH 01/17] Add proposal for supporting single node production installation using Bootstrap-In-Place Signed-off-by: Eran Cohen --- ...le-node-installation-bootstrap-in-place.md | 335 ++++++++++++++++++ 1 file changed, 335 insertions(+) create mode 100644 enhancements/installer/single-node-installation-bootstrap-in-place.md diff --git a/enhancements/installer/single-node-installation-bootstrap-in-place.md b/enhancements/installer/single-node-installation-bootstrap-in-place.md new file mode 100644 index 0000000000..31ad21af8b --- /dev/null +++ b/enhancements/installer/single-node-installation-bootstrap-in-place.md @@ -0,0 +1,335 @@ +--- +title: single-node-deployment-with-bootstrap-in-place +authors: + - "@eranco" + - "@mrunalp" + - "@dhellmann" + - "@romfreiman" + - "@tsorya" +reviewers: + - TBD + - "@markmc" + - "@deads2k" + - "@wking" + - "@eparis" + - "@hexfusion" +approvers: + - TBD +creation-date: 2020-12-13 +last-updated: 2020-12-13 +status: implementable +see-also: + - https://github.com/openshift/enhancements/pull/560 + - https://github.com/openshift/enhancements/pull/302 +--- + +# Single Node deployment with bootstrap-in-place + +## Release Signoff Checklist + +- [x] Enhancement is `implementable` +- [ ] Design details are appropriately documented from clear requirements +- [ ] Test plan is defined +- [ ] Graduation criteria for dev preview, tech preview, GA +- [ ] User-facing documentation is created in [openshift-docs](https://github.com/openshift/openshift-docs/) + +## Summary + +As we add support for new features such as [single-node production deployment](https://github.com/openshift/enhancements/pull/560/files), +we need a way to install such clusters without an extra node dependency for bootstrap. + +This enhancement describes the flow for installing Single Node OpenShift using a liveCD that performs the bootstrap logic and reboots to become the single node. + +## Motivation + +Currently, all OpenShift installations use an auxiliary bootstrap node. +The bootstrap node creates a temporary control plane that is required for launching the actual cluster. + +Single Node OpenShift installations will often be performed in environments where there are no extra nodes, so it is highly desirable to remove the need for a separate bootstrap machine to reduce the resources required to install the cluster. + +The downsides of requiring a bootstrap node for Single Node OpenShift are: + +1. The obvious additional node. +2. Requires external dependencies: + a. Load balancer (only for bootstrap phase) + b. Requires DNS (configured per installation) +3. Cannot use Bare Metal IPI: + a. Adds irrelevant dependencies - vips, keepalived, mdns + b. Requires same L2 between bootstrap and the Single Node OpenShift + +### Goals + +* Describe an approach for installing Single Node OpenShift in a BareMetal environment for production use. +* Minimal changes to OpenShift installer and the implementation shouldn't affect existing deployment flows. +* Installation should result a clean Single Node OpenShift without any bootstrap leftovers. + + +### Non-Goals + +* Addressing a similar installation flow for multi-node clusters. +* High-availability for Single Node OpenShift. +* Single-node-developer (CRC) cluster-profile installation. +* Supporting cloud deployment for bootstrap in place (since livecd cannot be used). It will be addressed as part of future enhancement. +* Upgrading Single Node OpenShift will be addressed as part of a future enhancement. + +## Proposal + +The installer will be enhanced to provide a way to generate a single node ignition configuration. +The user will be able to boot a RHCOS live CD with that ignition to initiate the installation. +The live CD will perform the cluster bootstrap flow. +A master ignition including the control plane static pods will be created as part of the bootstrap. The master ignition will then be used on reboot to complete the installation and bring up Single Node OpenShift. +Use of the liveCD helps to ensure that we have a clean bootstrap +flow with just the master ignition as the handoff point. + + +### User Stories + +#### As an OpenShift user, I want to be able to deploy OpenShift on a supported single node configuration + +A user will be able to run the OpenShift installer to create a single-node +deployment, with some limitations (see non-goals above). The user +will not require special support exceptions to receive technical assistance +for the features supported by the configuration. + +### Implementation Details/Notes/Constraints + +The OpenShift installer `create ignition-configs` command will generate a `bootstrap-in-place-for-live-iso.ign` +file, when the number of replicas for the control plane (in the `install-config.yaml`) is `1`. + +The `bootstrap-in-place-for-live-iso.ign` will be embedded into an RHCOS liveCD by the user using the `coreos-install embed` command. + +The user will boot a machine with this liveCD and the liveCD will start executing a similar flow to a bootstrap node in a regular installation. + +`bootkube.sh` running on the live ISO will execute the rendering logic. +The live ISO environment provides a scratch place to write bootstrapping files so that they don't end up on the real node. This eliminates a potential source of errors and confusion when debugging problems. + +The bootstrap static pods will be generated in a way that the control plane operators will be able to identify them and either continue in a controlled way for the next revision, or just keep them as the correct revision and reuse them. + +`cluster-bootstrap` will apply all the required manifests (under ``/opt/openshift/manifests/``) + +Bootkube will get the master Ignition from `machine-config-server` and +generate an updated master Ignition combining the original ignition with the control plane static pods manifests and all required resources including etcd data. +`bootkube.sh` will write the new master Ignition along with RHCOS to disk. +At this point `bootkube` will reboot the node and let it complete the cluster creation. + +After the host reboots, the `kubelet` service will start the control plane static pods. +Kubelet will send a CSR (see below) and join the cluster. +CVO will deploy all cluster operators. +The control plane operators will rollout a new revision (if necessary). + +#### OpenShift-installer + +We will add logic to the installer to create `bootstrap-in-place-for-live-iso.ign` Ignition config. +This Ignition config will diverge from the default bootstrap Ignition: +bootkube.sh: +1. Start cluster-bootstrap without required pods (`--required-pods=''`) +2. Run cluster-bootstrap with `--bootstrap-in-place` entrypoint to enrich the master Ignition. +3. Write RHCOS image and the master Ignition to disk. +4. Reboot the node. + +#### Cluster-bootstrap + +By default, `cluster-bootstrap` starts the bootstrap control plane and creates all the manifests under ``/opt/openshift/manifests``. +`cluster-bootstrap` also waits for a list of required pods to be ready. These pods are expected to start running on the control plane nodes. +In case we are running the bootstrap in place, there is no control plane node that can run those pods. `cluster-bootstrap` should apply the manifest and tear down the control plane. If `cluster-bootstrap` fails to apply some of the manifests, it should return an error. + + +`cluster-bootstrap` will have a new entrypoint `--bootstrap-in-place` which will get the master Ignition as input and will enrich the master Ignition with control plane static pods manifests and all required resources including etcd data. + +#### Bootstrap / Control plane static pods + +The control plane components we will add to the master Ignition are (to be placed under `/etc/kubernetes/manifests`): + +1. etcd-pod +2. kube-apiserver-pod +3. kube-controller-manager-pod +4. kube-scheduler-pod + +Control plane required resources to be added to the Ignition: + +1. `/var/lib/etcd` +2. `/etc/kubernetes/bootstrap-configs` +3. /opt/openshift/tls/* (`/etc/kubernetes/bootstrap-secrets`) +4. /opt/openshift/auth/kubeconfig-loopback (`/etc/kubernetes/bootstrap-secrets/kubeconfig`) + +**Note**: `/etc/kubernetes/bootstrap-secrets` and `/etc/kubernetes/bootstrap-configs` will be deleted after the node reboots by the post-reboot service (see below), and the OCP control plane is ready. + +The control plane operators (that will run on the node post reboot) will manage the rollout of a new revision of the control plane pods. + +#### etcd data + +In order to add a viable, working etcd post reboot, we will take a snapshot of etcd and add it to the master Ignition. +Post reboot we will use the restored etcd-member from the snapshot. + +Another option is to stop the etcd pod (move the static pod manifest from `/etc/kubernetes/manifests`). +When stopped, etcd will save its state and exit. We can then add the `/var/lib/etcd` directory to the master Ignition config. +After the reboot, etcd should start with all the data it had prior to the reboot. + +#### Post reboot + +We will add a new `post-reboot` service for approving the kubelet and the node Certificate Signing Requests. +This service will also cleanup the bootstrap static pods resources once the OCP control plane is ready. +Since we start with a liveCD, the bootstrap services (`bootkube`, `approve-csr`, etc.), `/etc` and `/opt/openshift` "garbage" are written to the ephemeral filesystem of the liveCD, and not to the node's real filesystem. +The files that we need to delete are under: +`/etc/kubernetes/bootstrap-secrets` and `/etc/kubernetes/bootstrap-configs` +These files are required for the bootstrap control plane to start before it is replaced by the control plane operators. +Once the OCP control plane static pods are deployed we can delete the files as they are no longer required. + +### Initial Proof-of-Concept + +User flow +1. Generate bootstrap ignition using the OpenShift installer. +2. Embed this Ignition to an RHCOS liveCD. +3. Boot a machine with this liveCD. + +This POC uses the following services for mitigating some gaps: +- `patch.service` for allowing single node installation. it won't be required once [single-node production deployment](https://github.com/openshift/enhancements/pull/560/files) is implemented. +- `post_reboot.service` for approving the node CSR and bootstrap static pods resources cleanup post reboot. + +Steps to try it out: +- Clone the installer branch: `iBIP_4_6` from https://github.com/eranco74/installer.git +- Build the installer (`TAGS=libvirt hack/build.sh`) +- Add your ssh key and pull secret to the `./install-config.yaml` +- Generate ignition - `make generate` +- Set up networking - `make network` (provides DNS for `Cluster name: test-cluster, Base DNS: redhat.com`) +- Download rhcos image - `make embed` (download RHCOS liveCD and embed the bootstrap Ignition) +- Spin up a VM with the the liveCD - `make start-iso` +- Monitor the progress using `make ssh` and `journalctl -f -u bootkube.service` or `kubectl --kubeconfig ./mydir/auth/kubeconfig get clusterversion` + +### Risks and Mitigations + +*What are the risks of this proposal and how do we mitigate. Think broadly. For +example, consider both security and how this will impact the larger OKD +ecosystem.* + +*How will security be reviewed and by whom? How will UX be reviewed and by whom?* + +## Design Details + +### Open Questions + +1. How will the user specify custom configuration, such as installation disk, static IPs? +2. Number of revisions for the control plane - do we want to make changes to the bootstrap static pods to make them closer to the final ones? +3. What do we do with the CRDs that are generated by operators instead of installed from manifests managed by the CVO? + +### Bootable installation artifact (future work) + +In order to embed the bootstrap-in-place-for-live-iso Ignition config to the liveCD the user need to get the liveCD and the coreos-installer binary. +We consider adding `openshift-install create single-node-iso` command that that result a liveCD with the bootstrap-in-place-for-live-iso.ign embeded. +It can also take things like additional manifests for setting the RT kernel (and kernel args) via MachineConfig as well + as supporting injecting network configuration as files and choosing the target disk drive for coreos-installer. +Internally, create single-node-iso would compile a single-node-iso-target.yaml into Ignition (much like coreos/fcct) + and include it along with the Ignition it generates and embed it into the ISO. + +### Limitations + +While most CRDs get created by CVO some CRDs are created by the operators, since during the bootstrap phase there is no schedulable node, + operators can't run, these CRDs won't be created until the node pivot to become the master node. +This imposes a limitation on the user when specifying custom manifests prior to the installation. + +These are the CRDs that are not present during bootstrap: +* clusternetworks.network.openshift.io +* controllerconfigs.machineconfiguration.openshift.io +* egressnetworkpolicies.network.openshift.io +* hostsubnets.network.openshift.io +* ippools.whereabouts.cni.cncf.io +* netnamespaces.network.openshift.io +* network-attachment-definitions.k8s.cni.cncf.io +* overlappingrangeipreservations.whereabouts.cni.cncf.io +* volumesnapshotclasses.snapshot.storage.k8s.io +* volumesnapshotcontents.snapshot.storage.k8s.io +* volumesnapshots.snapshot.storage.k8s.io + +### Test Plan + +In order to claim full support for this configuration, we must have +CI coverage informing the release. +An end-to-end job using the bootstrap-in-place installation flow, +based on the [installer UPI CI](https://github.com/openshift/release/blob/master/ci-operator/templates/openshift/installer/cluster-launch-installer-metal-e2e.yaml#L507), + running an appropriate subset of the standard OpenShift tests +will be created and configured to block accepting release images unless it passes. +This job is a different CI from the Single node production edge CI that will run with a bootstrap vm on cloud environment. + +That end-to-end job should also be run against pull requests for +the control plane repos, installer and cluster-bootstrap. + +### Graduation Criteria + +**Note:** *Section not required until targeted at a release.* + +Define graduation milestones. + +These may be defined in terms of API maturity, or as something else. Initial proposal +should keep this high-level with a focus on what signals will be looked at to +determine graduation. + +Consider the following in developing the graduation criteria for this +enhancement: + +- Maturity levels + - [`alpha`, `beta`, `stable` in upstream Kubernetes][maturity-levels] + - `Dev Preview`, `Tech Preview`, `GA` in OpenShift +- [Deprecation policy][deprecation-policy] + +Clearly define what graduation means by either linking to the [API doc definition](https://kubernetes.io/docs/concepts/overview/kubernetes-api/#api-versioning), +or by redefining what graduation means. + +In general, we try to use the same stages (alpha, beta, GA), regardless how the functionality is accessed. + +[maturity-levels]: https://git.k8s.io/community/contributors/devel/sig-architecture/api_changes.md#alpha-beta-and-stable-versions +[deprecation-policy]: https://kubernetes.io/docs/reference/using-api/deprecation-policy/ + +#### Examples + +##### Dev Preview -> Tech Preview + +- Ability to utilize the enhancement end to end +- End user documentation, relative API stability +- Sufficient test coverage +- Gather feedback from users rather than just developers + +##### Tech Preview -> GA + +- More testing (upgrade, downgrade, scale) +- Sufficient time for feedback +- Available by default + +**For non-optional features moving to GA, the graduation criteria must include +end to end tests.** + +## Implementation History + +Major milestones in the life cycle of a proposal should be tracked in `Implementation +History`. + +## Drawbacks + +1. The API will be unavailable from time to time during the installation. +2. Coreos-installer cannot be used in the cloud environment. + +## Alternatives + +### Installing using remote bootstrap node + +Run the bootstrap node in a HUB cluster as VM. +This approach is appealing because it keeps the current installation flow. +Requires external dependencies. +However, there are drawbacks: +1. It will require Load balancer and DNS per installation. +2. Deployments run remotely via L3 connection (high latency (up to 150ms), low BW in some cases), this isn't optimal for etcd cluster (one member is running on the bootstrap during the installation) +3. Running the bootstrap on the HUB cluster present a (resources) scale issue (~50*(8GB+4cores)), limits ACM capacity + +### Installing without liveISO + +Run the bootstrap flow on the node disk and clean up all the bootstrap residues once the node fully configured. +This is very similar to the current enhancement installation approach but without the requirement to start from liveCD. +This approach advantage is that it will work on cloud environment. +The disadvantage is that it's more prune to result a single node deployment with bootstrap leftovers. + + +### Installing using a baked Ignition file. + +The installer will generate an ignition config. +This Ignition configuration includes all assets required for launching the single node cluster (including TLS certificates and keys). +When booting a machine with CoreOS and this Ignition configuration the Ignition config will lay down the control plane operator static pods. +The ignition config will also create a static pod that functions as cluster-bootstrap (this pod should delete itself once it’s done) and apply the OCP assets to the control plane. \ No newline at end of file From 7bd078a82e843abe66366aeea14c305c359e425a Mon Sep 17 00:00:00 2001 From: Doug Hellmann Date: Tue, 12 Jan 2021 15:04:14 -0500 Subject: [PATCH 02/17] rephrase goal about minimal changes to the installer Rephrase the non-goal for cloud support. clean up list of downsides for using a bootstrap node Remove the mistaken assertion that the bootstrap VM and production nodes need to be on the same L2 segment. This is no longer true with more recent versions of OpenShift that support virtual media. flesh out the proposal section add risk about custom resource definitions Remove the open question about using CRDs and add a more detailed risk and mitigation description, based on the non-standard "limitation" section. update the implementation section Expand on the reasons for some choices. move discarded option for etcd handling to alternatives section move future work information to alternatives section Signed-off-by: Doug Hellmann --- ...le-node-installation-bootstrap-in-place.md | 360 ++++++++++++------ 1 file changed, 234 insertions(+), 126 deletions(-) diff --git a/enhancements/installer/single-node-installation-bootstrap-in-place.md b/enhancements/installer/single-node-installation-bootstrap-in-place.md index 31ad21af8b..fa22216975 100644 --- a/enhancements/installer/single-node-installation-bootstrap-in-place.md +++ b/enhancements/installer/single-node-installation-bootstrap-in-place.md @@ -51,36 +51,73 @@ The downsides of requiring a bootstrap node for Single Node OpenShift are: 1. The obvious additional node. 2. Requires external dependencies: - a. Load balancer (only for bootstrap phase) - b. Requires DNS (configured per installation) + 1. Load balancer (only for bootstrap phase) + 2. Preconfigured DNS (for each deployment) 3. Cannot use Bare Metal IPI: - a. Adds irrelevant dependencies - vips, keepalived, mdns - b. Requires same L2 between bootstrap and the Single Node OpenShift + 1. Adds additional dependencies - VIPs, keepalived, mdns ### Goals * Describe an approach for installing Single Node OpenShift in a BareMetal environment for production use. -* Minimal changes to OpenShift installer and the implementation shouldn't affect existing deployment flows. +* The implementation should require minimal changes to the OpenShift installer and the should not affect existing deployment flows. * Installation should result a clean Single Node OpenShift without any bootstrap leftovers. - +* Describe an approach that can be carried out by a user manually or automated by an orchestration tool. ### Non-Goals * Addressing a similar installation flow for multi-node clusters. -* High-availability for Single Node OpenShift. * Single-node-developer (CRC) cluster-profile installation. -* Supporting cloud deployment for bootstrap in place (since livecd cannot be used). It will be addressed as part of future enhancement. -* Upgrading Single Node OpenShift will be addressed as part of a future enhancement. +* Supporting cloud deployment for bootstrap in place. Using a live CD image is challenging in cloud environments, so this work is postponed to a future enhancement. ## Proposal -The installer will be enhanced to provide a way to generate a single node ignition configuration. -The user will be able to boot a RHCOS live CD with that ignition to initiate the installation. -The live CD will perform the cluster bootstrap flow. -A master ignition including the control plane static pods will be created as part of the bootstrap. The master ignition will then be used on reboot to complete the installation and bring up Single Node OpenShift. -Use of the liveCD helps to ensure that we have a clean bootstrap -flow with just the master ignition as the handoff point. - +The OpenShift install process relies on an ephemeral bootstrap +environment so that none of the hosts in the running cluster end up +with unique configuration left over from computing how to create the +cluster. When the bootstrap virtual machine is removed from the +process, the temporary files, logs, etc. from that phase should still +be segregated from the "real" OpenShift files on the host. This means +it is useful to retain a "bootstrap environment", as long as we can +avoid requiring a separate host to run a virtual machine. + +The focus for single-node deployments right now is edge use cases, +either for telco RAN deployments or other situations where a user may +have several instances being managed centrally. That means it is +important to make it possible to automate the workflow for deploying, +even if we also want to retain the option for users to deploy by hand. +In the telco RAN case, single-node deployments will be managed from a +central "hub" cluster using tools like RHACM, Hive, and metal3. + +The baseboard management controller (BMC) in enterprise class hardware +can be given a URL to an ISO image and told to attach the image to the +host as though it was inserted into a CD-ROM or DVD drive. An image +booted from an ISO can use a ramdisk as a place to create temporary +files, without affecting the persistent storage in the host. This +capability makes the existing live ISO for RHCOS a good foundation on +which to build this feature. A live ISO can serve as the "bootstrap +environment", separate from the real OpenShift system on persistent +storage in the host. The BMC in the host can be used to automate +deployment via a multi-cluster orchestration tool. + +The RHCOS live ISO image uses Ignition to configure the host, just as +the RHCOS image used for running OpenShift does. This means Ignition +is the most effective way to turn an RHCOS live image into a +special-purpose image for performing the installation. + +We propose the following steps for deploying single-node instances of +OpenShift: + +1. Have the OpenShift installer generate a special Ignition config for + a single-node deployment. +2. Combine that Ignition config with an RHCOS live ISO image to build + an image for deploying OpenShift on a single node. +3. Boot the new image on the host. +4. Bootstrap the deployment, generating a new master Ignition config + and the static pod definitions for OpenShift. Write them, along + with an RHCOS image, to the disk in the host. +5. Reboot the host to the internal disk, discarding the ephemeral live + image environment and allowing the previously generated artifacts + to complete the installation and bring up OpenShift. ### User Stories @@ -93,108 +130,120 @@ for the features supported by the configuration. ### Implementation Details/Notes/Constraints -The OpenShift installer `create ignition-configs` command will generate a `bootstrap-in-place-for-live-iso.ign` -file, when the number of replicas for the control plane (in the `install-config.yaml`) is `1`. - -The `bootstrap-in-place-for-live-iso.ign` will be embedded into an RHCOS liveCD by the user using the `coreos-install embed` command. - -The user will boot a machine with this liveCD and the liveCD will start executing a similar flow to a bootstrap node in a regular installation. - -`bootkube.sh` running on the live ISO will execute the rendering logic. -The live ISO environment provides a scratch place to write bootstrapping files so that they don't end up on the real node. This eliminates a potential source of errors and confusion when debugging problems. - -The bootstrap static pods will be generated in a way that the control plane operators will be able to identify them and either continue in a controlled way for the next revision, or just keep them as the correct revision and reuse them. - -`cluster-bootstrap` will apply all the required manifests (under ``/opt/openshift/manifests/``) - -Bootkube will get the master Ignition from `machine-config-server` and -generate an updated master Ignition combining the original ignition with the control plane static pods manifests and all required resources including etcd data. -`bootkube.sh` will write the new master Ignition along with RHCOS to disk. -At this point `bootkube` will reboot the node and let it complete the cluster creation. - -After the host reboots, the `kubelet` service will start the control plane static pods. -Kubelet will send a CSR (see below) and join the cluster. -CVO will deploy all cluster operators. -The control plane operators will rollout a new revision (if necessary). +The installation images for single-node clusters will be unique for +each cluster. The user or orchestration tool will create an +installation image by combining the +`bootstrap-in-place-for-live-iso.ign` created by the installer with an +RHCOS live image using `coreos-install embed`. Making the image unique +allows us to build on the existing RHCOS live image, instead of +delivering a different base image, and means that the user does not +need any infrastructure to serve Ignition configs to hosts during +deployment. + +In order to add a viable, working etcd post reboot, we will take a +snapshot of etcd and add it to the Ignition config for the host. +After rebooting, we will use the restored `etcd-member` from the +snapshot to rebuild the database. This allows etcd and the API service +to come up on the host without having to re-apply all of the +kubernetes operations run during bootstrapping. #### OpenShift-installer -We will add logic to the installer to create `bootstrap-in-place-for-live-iso.ign` Ignition config. -This Ignition config will diverge from the default bootstrap Ignition: -bootkube.sh: -1. Start cluster-bootstrap without required pods (`--required-pods=''`) -2. Run cluster-bootstrap with `--bootstrap-in-place` entrypoint to enrich the master Ignition. -3. Write RHCOS image and the master Ignition to disk. +The OpenShift installer will be updated so that the `create +ignition-configs` command generates a new +`bootstrap-in-place-for-live-iso.ign` file when the number of replicas +for the control plane in the `install-config.yaml` is `1`. + +This Ignition config will have a different `bootkube.sh` from the +default bootstrap Ignition. In addition to the standard rendering +logic, the modified script will: + +1. Start `cluster-bootstrap` without required pods by setting `--required-pods=''` +2. Run `cluster-bootstrap` with the `--bootstrap-in-place` option. +3. Fetch the master Ignition and combine it with the original Ignition + config, the control plane static pod manifests, the required + kubernetes resources, and the bootstrap etcd database snapshot to + create a new Ignition config for the host. +3. Write the RHCOS image and the combined Ignition config to disk. 4. Reboot the node. #### Cluster-bootstrap -By default, `cluster-bootstrap` starts the bootstrap control plane and creates all the manifests under ``/opt/openshift/manifests``. -`cluster-bootstrap` also waits for a list of required pods to be ready. These pods are expected to start running on the control plane nodes. -In case we are running the bootstrap in place, there is no control plane node that can run those pods. `cluster-bootstrap` should apply the manifest and tear down the control plane. If `cluster-bootstrap` fails to apply some of the manifests, it should return an error. +`cluster-bootstrap` will have a new entrypoint `--bootstrap-in-place` +which will get the master Ignition as input and will enrich the master +Ignition with control plane static pods manifests and all required +resources, including the etcd database. +`cluster-bootstrap` normally waits for a list of required pods to be +ready. These pods are expected to start running on the control plane +nodes when the bootstrap and control plane run in parallel. That is +not possible when bootstrapping in place, so when `cluster-bootstrap` +runs with the `--bootstrap-in-place` option it should only apply the +manifests and then tear down the control plane. -`cluster-bootstrap` will have a new entrypoint `--bootstrap-in-place` which will get the master Ignition as input and will enrich the master Ignition with control plane static pods manifests and all required resources including etcd data. +If `cluster-bootstrap` fails to apply some of the manifests, it should +return an error. #### Bootstrap / Control plane static pods -The control plane components we will add to the master Ignition are (to be placed under `/etc/kubernetes/manifests`): +The control plane components we will copy from +`/etc/kubernetes/manifests` into the master Ignition are: 1. etcd-pod 2. kube-apiserver-pod 3. kube-controller-manager-pod 4. kube-scheduler-pod -Control plane required resources to be added to the Ignition: +These components also require other files generated during bootstrapping: 1. `/var/lib/etcd` 2. `/etc/kubernetes/bootstrap-configs` -3. /opt/openshift/tls/* (`/etc/kubernetes/bootstrap-secrets`) -4. /opt/openshift/auth/kubeconfig-loopback (`/etc/kubernetes/bootstrap-secrets/kubeconfig`) +3. `/opt/openshift/tls/*` (`/etc/kubernetes/bootstrap-secrets`) +4. `/opt/openshift/auth/kubeconfig-loopback` (`/etc/kubernetes/bootstrap-secrets/kubeconfig`) -**Note**: `/etc/kubernetes/bootstrap-secrets` and `/etc/kubernetes/bootstrap-configs` will be deleted after the node reboots by the post-reboot service (see below), and the OCP control plane is ready. +**Note**: `/etc/kubernetes/bootstrap-secrets` and `/etc/kubernetes/bootstrap-configs` will be deleted by the `post-reboot` service, after the node reboots (see below). -The control plane operators (that will run on the node post reboot) will manage the rollout of a new revision of the control plane pods. +The bootstrap static pods will be generated in a way that the control +plane operators will be able to identify them and either continue in a +controlled way for the next revision, or just keep them as the correct +revision and reuse them. -#### etcd data +#### Post-reboot service -In order to add a viable, working etcd post reboot, we will take a snapshot of etcd and add it to the master Ignition. -Post reboot we will use the restored etcd-member from the snapshot. +We will add a new `post-reboot` service for approving the kubelet and +the node Certificate Signing Requests. This service will also cleanup +the bootstrap static pods resources when the OpenShift control plane +is ready. -Another option is to stop the etcd pod (move the static pod manifest from `/etc/kubernetes/manifests`). -When stopped, etcd will save its state and exit. We can then add the `/var/lib/etcd` directory to the master Ignition config. -After the reboot, etcd should start with all the data it had prior to the reboot. +Since we start with a liveCD, the bootstrap services (`bootkube`, `approve-csr`, etc.), `/etc` and `/opt/openshift` temporary files are written to the ephemeral filesystem of the live image, and not to the node's real filesystem. -#### Post reboot - -We will add a new `post-reboot` service for approving the kubelet and the node Certificate Signing Requests. -This service will also cleanup the bootstrap static pods resources once the OCP control plane is ready. -Since we start with a liveCD, the bootstrap services (`bootkube`, `approve-csr`, etc.), `/etc` and `/opt/openshift` "garbage" are written to the ephemeral filesystem of the liveCD, and not to the node's real filesystem. The files that we need to delete are under: -`/etc/kubernetes/bootstrap-secrets` and `/etc/kubernetes/bootstrap-configs` + +* `/etc/kubernetes/bootstrap-secrets` +* `/etc/kubernetes/bootstrap-configs` + These files are required for the bootstrap control plane to start before it is replaced by the control plane operators. -Once the OCP control plane static pods are deployed we can delete the files as they are no longer required. +Once the OCP control plane static pods are deployed we can delete the files as they are no longer required. ### Initial Proof-of-Concept -User flow -1. Generate bootstrap ignition using the OpenShift installer. -2. Embed this Ignition to an RHCOS liveCD. -3. Boot a machine with this liveCD. +A proof-of-concept implementation is available for experimenting with +the design. This POC uses the following services for mitigating some gaps: -- `patch.service` for allowing single node installation. it won't be required once [single-node production deployment](https://github.com/openshift/enhancements/pull/560/files) is implemented. +- `patch.service` for allowing single node installation. This won't be required after [single-node production deployment](https://github.com/openshift/enhancements/pull/560) is implemented. - `post_reboot.service` for approving the node CSR and bootstrap static pods resources cleanup post reboot. -Steps to try it out: -- Clone the installer branch: `iBIP_4_6` from https://github.com/eranco74/installer.git -- Build the installer (`TAGS=libvirt hack/build.sh`) -- Add your ssh key and pull secret to the `./install-config.yaml` -- Generate ignition - `make generate` -- Set up networking - `make network` (provides DNS for `Cluster name: test-cluster, Base DNS: redhat.com`) -- Download rhcos image - `make embed` (download RHCOS liveCD and embed the bootstrap Ignition) -- Spin up a VM with the the liveCD - `make start-iso` -- Monitor the progress using `make ssh` and `journalctl -f -u bootkube.service` or `kubectl --kubeconfig ./mydir/auth/kubeconfig get clusterversion` +To try it out: + +1. Clone the installer branch: `iBIP_4_6` from https://github.com/eranco74/installer.git +2. Build the installer with the command: `TAGS=libvirt hack/build.sh` +3. Add your ssh key and pull secret to the `./install-config.yaml` +4. Generate the Ignition config with the command `make generate` +5. Set up DNS for `Cluster name: test-cluster, Base DNS: redhat.com` running: `make network` +6. Download an RHCOS live image and add the bootstrap Ignition config by running: `make embed` +7. Spin up a VM with the the liveCD with the command: `make start-iso` +8. Monitor the progress using `make ssh` and `journalctl -f -u bootkube.service` or `kubectl --kubeconfig ./mydir/auth/kubeconfig get clusterversion` ### Risks and Mitigations @@ -204,30 +253,25 @@ ecosystem.* *How will security be reviewed and by whom? How will UX be reviewed and by whom?* -## Design Details - -### Open Questions - -1. How will the user specify custom configuration, such as installation disk, static IPs? -2. Number of revisions for the control plane - do we want to make changes to the bootstrap static pods to make them closer to the final ones? -3. What do we do with the CRDs that are generated by operators instead of installed from manifests managed by the CVO? - -### Bootable installation artifact (future work) - -In order to embed the bootstrap-in-place-for-live-iso Ignition config to the liveCD the user need to get the liveCD and the coreos-installer binary. -We consider adding `openshift-install create single-node-iso` command that that result a liveCD with the bootstrap-in-place-for-live-iso.ign embeded. -It can also take things like additional manifests for setting the RT kernel (and kernel args) via MachineConfig as well - as supporting injecting network configuration as files and choosing the target disk drive for coreos-installer. -Internally, create single-node-iso would compile a single-node-iso-target.yaml into Ignition (much like coreos/fcct) - and include it along with the Ignition it generates and embed it into the ISO. - -### Limitations - -While most CRDs get created by CVO some CRDs are created by the operators, since during the bootstrap phase there is no schedulable node, - operators can't run, these CRDs won't be created until the node pivot to become the master node. -This imposes a limitation on the user when specifying custom manifests prior to the installation. +#### Custom Manifests for CRDs + +One limitation of single-node deployments not present in multi-node +clusters is handling some custom resource definitions (CRDs). During +bootstrapping of a multi-node cluster, the bootstrap host and real +cluster hosts run in parallel for a time. This means that the +bootstrap host can iterate publishing manifests to the API server +until the operators running on the other hosts are up and define their +CRDs. If it takes a little while for those operators to install their +CRDs, the bootstrap host can wait and retry the operation. In a +single-node deployment, the bootstrap environment and real node are +not active at the same time. This means the bootstrap process may +block if it tries to create a custom resource using a CRD that is not +installed. + +While most CRDs are created by the `cluster-version-operator`, some +CRDs are created later by the cluster operators. These CRDs from +cluster operators are not present during bootstrapping: -These are the CRDs that are not present during bootstrap: * clusternetworks.network.openshift.io * controllerconfigs.machineconfiguration.openshift.io * egressnetworkpolicies.network.openshift.io @@ -240,6 +284,20 @@ These are the CRDs that are not present during bootstrap: * volumesnapshotcontents.snapshot.storage.k8s.io * volumesnapshots.snapshot.storage.k8s.io +This limitation is unlikely to be triggered by manifests created by +the OpenShift installer, but we cannot control what extra manifests +users add to their deployment. Users need to be made aware of this +limitation and encouraged to avoid creating custom manifests using +CRDs installed by cluster operators instead of the +`cluster-version-operator`. + +## Design Details + +### Open Questions + +1. How will the user specify custom configuration, such as installation disk, or static IPs? +2. Number of revisions for the control plane - do we want to make changes to the bootstrap static pods to make them closer to the final ones? + ### Test Plan In order to claim full support for this configuration, we must have @@ -305,31 +363,81 @@ History`. ## Drawbacks 1. The API will be unavailable from time to time during the installation. -2. Coreos-installer cannot be used in the cloud environment. - +2. Coreos-installer cannot be used in a cloud environment. +3. We need to build new integration with RHACM and Hive for orchestration. + ## Alternatives ### Installing using remote bootstrap node -Run the bootstrap node in a HUB cluster as VM. -This approach is appealing because it keeps the current installation flow. -Requires external dependencies. -However, there are drawbacks: -1. It will require Load balancer and DNS per installation. -2. Deployments run remotely via L3 connection (high latency (up to 150ms), low BW in some cases), this isn't optimal for etcd cluster (one member is running on the bootstrap during the installation) -3. Running the bootstrap on the HUB cluster present a (resources) scale issue (~50*(8GB+4cores)), limits ACM capacity - -### Installing without liveISO +We could continue to run the bootstrap node in a HUB cluster as VM. -Run the bootstrap flow on the node disk and clean up all the bootstrap residues once the node fully configured. -This is very similar to the current enhancement installation approach but without the requirement to start from liveCD. -This approach advantage is that it will work on cloud environment. -The disadvantage is that it's more prune to result a single node deployment with bootstrap leftovers. +This approach is appealing because it keeps the current installation +flow. +However, there are drawbacks: -### Installing using a baked Ignition file. - -The installer will generate an ignition config. -This Ignition configuration includes all assets required for launching the single node cluster (including TLS certificates and keys). -When booting a machine with CoreOS and this Ignition configuration the Ignition config will lay down the control plane operator static pods. -The ignition config will also create a static pod that functions as cluster-bootstrap (this pod should delete itself once it’s done) and apply the OCP assets to the control plane. \ No newline at end of file +1. It will require configuring a Load balancer and DNS for each cluster. +2. In some cases, deployments run over L3 connection with high latency + (up to 150ms) and low bandwidth to sites where there is no + hypervisor. We would therefore need to run the bootstrap VM + remotely, and form the etcd cluster with members on both sides of + the poor connection. Since etcd has requirements for low latency, + high bandwidth, connections between all nodes, this is not ideal. +3. The bootstrap VM requires 8GB of RAM and 4 CPU cores. Running the + bootstrap VM on the hub cluster constrains the number of + simultaneous deployments that can be run based on the CPU and RAM + capacity of the hub cluster. + +### Installing without a live image + +We could run the bootstrap flow on the node's regular disk and clean +up all the bootstrap residue once the node is fully configured. This +is very similar to the current enhancement installation approach but +without the requirement to start from a live image. The advantage of +this approach is that it will work in a cloud environment as well as +on bare metal. The disadvantage is that it is more prone to result in +a single node deployment with bootstrapping leftovers in place, +potentially leading to confusion for users or support staff debugging +the instances. + + +### Installing using an Ignition config not built into the live image + +We could have the installer generate an Ignition config that includes +all of the assets required for launching the single node cluster +(including TLS certificates and keys). When booting a machine with +CoreOS and this Ignition configuration, the Ignition config would lay +down the control plane operator static pods and create a static pod +that functions as `cluster-bootstrap` This pod should delete itself +after it is done applying the OCP assets to the control plane. + +### Preserve etcd database instead of a snapshot + +Another option for preserving the etcd database when pivoting from +bootstrap to production is to copy the entire database, instead of +using a snapshot operation. When stopped, etcd will save its state +and exit. We can then add the `/var/lib/etcd` directory to the master +Ignition config. After the reboot, etcd should start with all the +data it had prior to the reboot. By using a snapshot, instead of +saving the entire database, we will have more flexibility to change +the production etcd configuration before restoring the content of the +database. + +### Creating a bootable installation artifact directly from the installer + +In order to embed the bootstrap-in-place-for-live-iso Ignition config +to the liveCD the user needs to download the liveCD image and the +`coreos-installer` binary. We considered adding an `openshift-install +create single-node-iso` command that that result a liveCD with the +`bootstrap-in-place-for-live-iso.ign` embeded. The installer command +could also include custom manifests, especially `MachineConfig` +instances for setting the realtime kernel, setting kernel args, +injecting network configuration as files, and choosing the target disk +drive for `coreos-installer`. Internally, `create single-node-iso` +would compile a single-node-iso-target.yaml into Ignition (much like +coreos/fcct) and include it along with the Ignition it generates and +embed it into the ISO. + +This approach has not been rejected entirely, and may be handled with +a future enhancement. From efbdf8d3e22e2aa96fe5303565b7fced504f87aa Mon Sep 17 00:00:00 2001 From: Eran Cohen Date: Thu, 14 Jan 2021 19:37:54 +0200 Subject: [PATCH 03/17] Updated POC Signed-off-by: Eran Cohen --- ...le-node-installation-bootstrap-in-place.md | 45 +++++++++---------- 1 file changed, 21 insertions(+), 24 deletions(-) diff --git a/enhancements/installer/single-node-installation-bootstrap-in-place.md b/enhancements/installer/single-node-installation-bootstrap-in-place.md index fa22216975..728bcd4517 100644 --- a/enhancements/installer/single-node-installation-bootstrap-in-place.md +++ b/enhancements/installer/single-node-installation-bootstrap-in-place.md @@ -67,7 +67,8 @@ The downsides of requiring a bootstrap node for Single Node OpenShift are: * Addressing a similar installation flow for multi-node clusters. * Single-node-developer (CRC) cluster-profile installation. -* Supporting cloud deployment for bootstrap in place. Using a live CD image is challenging in cloud environments, so this work is postponed to a future enhancement. +* Supporting cloud deployment for bootstrap in place. Using a live CD image is challenging in cloud environments, + so this work is postponed to a future enhancement. ## Proposal @@ -96,7 +97,8 @@ files, without affecting the persistent storage in the host. This capability makes the existing live ISO for RHCOS a good foundation on which to build this feature. A live ISO can serve as the "bootstrap environment", separate from the real OpenShift system on persistent -storage in the host. The BMC in the host can be used to automate +storage in the host, with just the master Ignition as the handoff point. +The BMC in the host can be used to automate deployment via a multi-cluster orchestration tool. The RHCOS live ISO image uses Ignition to configure the host, just as @@ -107,8 +109,8 @@ special-purpose image for performing the installation. We propose the following steps for deploying single-node instances of OpenShift: -1. Have the OpenShift installer generate a special Ignition config for - a single-node deployment. +1. Add a new create single-node-ignition-config command to openshift-installer + which generate a single node Ignition config for a single-node deployment. 2. Combine that Ignition config with an RHCOS live ISO image to build an image for deploying OpenShift on a single node. 3. Boot the new image on the host. @@ -149,10 +151,11 @@ kubernetes operations run during bootstrapping. #### OpenShift-installer -The OpenShift installer will be updated so that the `create -ignition-configs` command generates a new -`bootstrap-in-place-for-live-iso.ign` file when the number of replicas -for the control plane in the `install-config.yaml` is `1`. +Add new `create single-node-ignition-config` command to the installer to create + `bootstrap-in-place-for-live-iso.ign` Ignition config. +This new target will not output master.ign and worker.ign. +Allow the user to specify the target disk drive for coreos-installer using environment + variable `OPENSHIFT_INSTALL_EXPERIMENTAL_BOOTSTRAP_IN_PLACE_COREOS_INSTALLER_ARGS`. This Ignition config will have a different `bootkube.sh` from the default bootstrap Ignition. In addition to the standard rendering @@ -186,6 +189,13 @@ return an error. #### Bootstrap / Control plane static pods +We will review the list of revisions for apiserver/etcd and see if we can + reduce them by reducing revisions caused by observations of known conditions. + For example in a single node we know what the etcd endpoints will be in advance, + We can avoid a revision by observing this post install. + This work will go a long way to reducing disruption during install and improve + MTTR for upgrade re-deployments and failures. + The control plane components we will copy from `/etc/kubernetes/manifests` into the master Ignition are: @@ -230,20 +240,7 @@ Once the OCP control plane static pods are deployed we can delete the files as t A proof-of-concept implementation is available for experimenting with the design. -This POC uses the following services for mitigating some gaps: -- `patch.service` for allowing single node installation. This won't be required after [single-node production deployment](https://github.com/openshift/enhancements/pull/560) is implemented. -- `post_reboot.service` for approving the node CSR and bootstrap static pods resources cleanup post reboot. - -To try it out: - -1. Clone the installer branch: `iBIP_4_6` from https://github.com/eranco74/installer.git -2. Build the installer with the command: `TAGS=libvirt hack/build.sh` -3. Add your ssh key and pull secret to the `./install-config.yaml` -4. Generate the Ignition config with the command `make generate` -5. Set up DNS for `Cluster name: test-cluster, Base DNS: redhat.com` running: `make network` -6. Download an RHCOS live image and add the bootstrap Ignition config by running: `make embed` -7. Spin up a VM with the the liveCD with the command: `make start-iso` -8. Monitor the progress using `make ssh` and `journalctl -f -u bootkube.service` or `kubectl --kubeconfig ./mydir/auth/kubeconfig get clusterversion` +To try it out: [bootstrap-in-place-poc](https://github.com/eranco74/bootstrap-in-place-poc.git) ### Risks and Mitigations @@ -295,7 +292,7 @@ CRDs installed by cluster operators instead of the ### Open Questions -1. How will the user specify custom configuration, such as installation disk, or static IPs? +1. How will the user specify custom configurations, such as static IPs? 2. Number of revisions for the control plane - do we want to make changes to the bootstrap static pods to make them closer to the final ones? ### Test Plan @@ -440,4 +437,4 @@ coreos/fcct) and include it along with the Ignition it generates and embed it into the ISO. This approach has not been rejected entirely, and may be handled with -a future enhancement. +a future enhancement. \ No newline at end of file From 3e2a2daaf439eda9087845ba8af9894488a792e3 Mon Sep 17 00:00:00 2001 From: Eran Cohen Date: Thu, 14 Jan 2021 19:43:30 +0200 Subject: [PATCH 04/17] Added Requirements Signed-off-by: Eran Cohen --- .../single-node-installation-bootstrap-in-place.md | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/enhancements/installer/single-node-installation-bootstrap-in-place.md b/enhancements/installer/single-node-installation-bootstrap-in-place.md index 728bcd4517..e56b257f28 100644 --- a/enhancements/installer/single-node-installation-bootstrap-in-place.md +++ b/enhancements/installer/single-node-installation-bootstrap-in-place.md @@ -235,6 +235,15 @@ The files that we need to delete are under: These files are required for the bootstrap control plane to start before it is replaced by the control plane operators. Once the OCP control plane static pods are deployed we can delete the files as they are no longer required. +#### Prerequisites for a Single Node deployment with bootstrap-in-place +The requirements are a subset of the requirements for user-provisioned infrastructure installation. +1. Configure DHCP or set static IP addresses for the node. +The node IP should be persistent, otherwise TLS SAN will be invalidated and will cause the communications between apiserver and etcd to fail. +2. DNS records: +* api... +* api-int... +* *.apps... +* ... ### Initial Proof-of-Concept A proof-of-concept implementation is available for experimenting with From 79cdb261675677f4ff0514e002111c401f109134 Mon Sep 17 00:00:00 2001 From: Eran Cohen Date: Thu, 14 Jan 2021 19:41:50 +0200 Subject: [PATCH 05/17] Added alternatives clean up alternatives One alternative section listed 2 unrelated alternatives. Signed-off-by: Doug Hellmann --- ...le-node-installation-bootstrap-in-place.md | 52 +++++++++++++++---- 1 file changed, 43 insertions(+), 9 deletions(-) diff --git a/enhancements/installer/single-node-installation-bootstrap-in-place.md b/enhancements/installer/single-node-installation-bootstrap-in-place.md index e56b257f28..2e275993d7 100644 --- a/enhancements/installer/single-node-installation-bootstrap-in-place.md +++ b/enhancements/installer/single-node-installation-bootstrap-in-place.md @@ -189,12 +189,13 @@ return an error. #### Bootstrap / Control plane static pods -We will review the list of revisions for apiserver/etcd and see if we can - reduce them by reducing revisions caused by observations of known conditions. - For example in a single node we know what the etcd endpoints will be in advance, - We can avoid a revision by observing this post install. - This work will go a long way to reducing disruption during install and improve - MTTR for upgrade re-deployments and failures. +We will review the list of revisions for apiserver/etcd and see if we +can reduce them by reducing revisions caused by observations of known +conditions. For example in a single node we know what the etcd +endpoints will be in advance, We can avoid a revision by observing +this post install. This work will go a long way to reducing +disruption during install and improve MTTR for upgrade re-deployments +and failures. The control plane components we will copy from `/etc/kubernetes/manifests` into the master Ignition are: @@ -402,7 +403,7 @@ up all the bootstrap residue once the node is fully configured. This is very similar to the current enhancement installation approach but without the requirement to start from a live image. The advantage of this approach is that it will work in a cloud environment as well as -on bare metal. The disadvantage is that it is more prone to result in +on bare metal. The disadvantage is that it is more prone to result in a single node deployment with bootstrapping leftovers in place, potentially leading to confusion for users or support staff debugging the instances. @@ -412,11 +413,18 @@ the instances. We could have the installer generate an Ignition config that includes all of the assets required for launching the single node cluster -(including TLS certificates and keys). When booting a machine with +(including TLS certificates and keys). When booting a machine with CoreOS and this Ignition configuration, the Ignition config would lay down the control plane operator static pods and create a static pod that functions as `cluster-bootstrap` This pod should delete itself after it is done applying the OCP assets to the control plane. +The disadvantage in this approach is that it's very different than +the regular installation flow which involve a bootstrap node. +It also adds more challenges such as: +1. The installer machine (usually the Admin laptop) will need to pull +20 container images in order to render the Ignition. +2. The installer will need to know the node IP in advance for rendering +etcd certificates ### Preserve etcd database instead of a snapshot @@ -446,4 +454,30 @@ coreos/fcct) and include it along with the Ignition it generates and embed it into the ISO. This approach has not been rejected entirely, and may be handled with -a future enhancement. \ No newline at end of file +a future enhancement. + +### Allow bootstrap-in-place in cloud environment (future work) + +For bootstrap-in-place model to work in cloud environemnt we need to mitigate the following gaps: +1. The bootstrap-in-place model relay on the live ISO environment as a place to write bootstrapping files so that they don't end up on the real node. +Optional mitigation: We can mimic this environment by mounting some directories as tmpfs during the bootstrap phase. +2. The bootstrap-in-place model uses coreos-installer to write the final Ignition to disk along with the RHCOS image. +Optional mitigation: We can boot the machine with the right RHCOS image for the release. +Instead of writing the Ignition to disk we will use the cloud credentials to update the node Ignition config in the cloud provider. + +### Check control plane replica count in `create ignition-configs` + +Instead of adding a new installer command, we could use the current command for generating Ignition configs +`create ignition-configs` to generate the `bootstrap-in-place-for-live-iso.ign` file, +by adding logic to the installer that check the number of replicas for the control + plane (in the `install-config.yaml`) is `1`. +This approach might conflict with CRC/SNC which also run openshift-install with a 1-replica control plane. + +### Use `create ignition-configs` with environment variable to generate the `bootstrap-in-place-for-live-iso.ign`. + +We also considered adding a new environment variable `OPENSHIFT_INSTALL_EXPERIMENTAL_BOOTSTRAP_IN_PLACE` +for marking the new path under the `ignition-configs` target. +We decided to add `single-node-ignition-config` target to in order to gain: +1. Allow us to easily add different set of validations (e.g. ensure that the number of replicas for the control plane is 1). +2. We can avoid creating unnecessary assets (master.ign and worker.ign). +3. Less prune to user errors than environment variable. From bd7a78dec227475950843463e905630c6fa3c228 Mon Sep 17 00:00:00 2001 From: Eran Cohen Date: Sun, 17 Jan 2021 18:39:49 +0200 Subject: [PATCH 06/17] Added bootstrap logs and installer gather Signed-off-by: Eran Cohen --- ...le-node-installation-bootstrap-in-place.md | 37 ++++++++++++++++--- 1 file changed, 32 insertions(+), 5 deletions(-) diff --git a/enhancements/installer/single-node-installation-bootstrap-in-place.md b/enhancements/installer/single-node-installation-bootstrap-in-place.md index 2e275993d7..ee780810f6 100644 --- a/enhancements/installer/single-node-installation-bootstrap-in-place.md +++ b/enhancements/installer/single-node-installation-bootstrap-in-place.md @@ -53,14 +53,13 @@ The downsides of requiring a bootstrap node for Single Node OpenShift are: 2. Requires external dependencies: 1. Load balancer (only for bootstrap phase) 2. Preconfigured DNS (for each deployment) -3. Cannot use Bare Metal IPI: - 1. Adds additional dependencies - VIPs, keepalived, mdns ### Goals * Describe an approach for installing Single Node OpenShift in a BareMetal environment for production use. -* The implementation should require minimal changes to the OpenShift installer and the should not affect existing deployment flows. -* Installation should result a clean Single Node OpenShift without any bootstrap leftovers. +* The implementation should require minimal changes to the OpenShift installer, +it should strive to reuse existing code and should not affect existing deployment flows. +* Installation should result in a clean Single Node OpenShift without any bootstrap leftovers. * Describe an approach that can be carried out by a user manually or automated by an orchestration tool. ### Non-Goals @@ -194,7 +193,7 @@ can reduce them by reducing revisions caused by observations of known conditions. For example in a single node we know what the etcd endpoints will be in advance, We can avoid a revision by observing this post install. This work will go a long way to reducing -disruption during install and improve MTTR for upgrade re-deployments +disruption during install and improve mean time to recovery for upgrade re-deployments and failures. The control plane components we will copy from @@ -298,6 +297,34 @@ limitation and encouraged to avoid creating custom manifests using CRDs installed by cluster operators instead of the `cluster-version-operator`. +#### Bootstrap logs retention + + Due to the bootstrap-in-place behavior all bootstrap artifacts + will be lost once the bootstrap the node reboots. +In a regular installation flow the bootstrap node goes down only once + the control plane is running, and the bootstrap node served its purpose. +In case of bootstrap in place things can go wrong after the reboot. +The bootstrap logs can aid in troubleshooting a subsequently failed install. + +Mitigation by gathering the bootstrap logs before reboot. + bootstrap will gather logs from itself using /usr/local/bin/installer-gather.sh. + Once gathering is complete, the bundle will be added to the master ignition, + thus making the bootstrap logs available from the master after reboot. + The log bundle will be deleted once the installation completes. + +The installer `gather` command works as usual before the reboot. +We will add a new script called installer-master-bootstrap-in-place-gather.sh. + This script will be delivered to the master using via Ignition to the same + location where the bootstrap node usually has installer-gather.sh. +The installer-master-bootstrap-in-place-gather.sh, will be called by the + `openshift-install gather` command that believes its collecting logs from a bootstrap node. +The script however behaves slightly different, instead of collecting bootstrap logs +and then remotely running /usr/local/bin/installer-masters-gather.sh on all master nodes, + the script will collect the bootstrap logs from the bundle copied to via the master-ignition, + and collect master logs by running /usr/local/bin/installer-masters-gather.sh directly + on itself. The final archiving of all the logs into the home directory, + exactly the same as /usr/local/bin/installer-gather.sh. + ## Design Details ### Open Questions From d652815fd31ab79d8fb10d29c1bf20eb711a48fd Mon Sep 17 00:00:00 2001 From: Doug Hellmann Date: Thu, 14 Jan 2021 15:25:56 -0500 Subject: [PATCH 07/17] add risk about post-reboot service Signed-off-by: Doug Hellmann --- .../single-node-installation-bootstrap-in-place.md | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/enhancements/installer/single-node-installation-bootstrap-in-place.md b/enhancements/installer/single-node-installation-bootstrap-in-place.md index ee780810f6..e1afc35cb5 100644 --- a/enhancements/installer/single-node-installation-bootstrap-in-place.md +++ b/enhancements/installer/single-node-installation-bootstrap-in-place.md @@ -297,6 +297,14 @@ limitation and encouraged to avoid creating custom manifests using CRDs installed by cluster operators instead of the `cluster-version-operator`. +#### Post-reboot service makes single-node deployments different from multi-node clusters + +The `post-reboot` service is only used with single-node +deployments. This makes those deployments look different in a way that +may lead to confusion when debugging issues on the cluster. To +mitigate this, we can add documentation to the troubleshooting guide +to explain the service and its role in the cluster. + #### Bootstrap logs retention Due to the bootstrap-in-place behavior all bootstrap artifacts From 3ab928d02e24c2a4447a590757c68a8029277955 Mon Sep 17 00:00:00 2001 From: Eran Cohen Date: Mon, 25 Jan 2021 14:28:38 +0200 Subject: [PATCH 08/17] Apply suggestions from code review Co-authored-by: Doug Hellmann --- .../single-node-installation-bootstrap-in-place.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/enhancements/installer/single-node-installation-bootstrap-in-place.md b/enhancements/installer/single-node-installation-bootstrap-in-place.md index e1afc35cb5..76a354f1f4 100644 --- a/enhancements/installer/single-node-installation-bootstrap-in-place.md +++ b/enhancements/installer/single-node-installation-bootstrap-in-place.md @@ -108,8 +108,8 @@ special-purpose image for performing the installation. We propose the following steps for deploying single-node instances of OpenShift: -1. Add a new create single-node-ignition-config command to openshift-installer - which generate a single node Ignition config for a single-node deployment. +1. Add a new `create single-node-ignition-config` command to `openshift-installer` + which generates an Ignition config for a single-node deployment. 2. Combine that Ignition config with an RHCOS live ISO image to build an image for deploying OpenShift on a single node. 3. Boot the new image on the host. @@ -150,7 +150,7 @@ kubernetes operations run during bootstrapping. #### OpenShift-installer -Add new `create single-node-ignition-config` command to the installer to create +We will add a new `create single-node-ignition-config` command to the installer to create `bootstrap-in-place-for-live-iso.ign` Ignition config. This new target will not output master.ign and worker.ign. Allow the user to specify the target disk drive for coreos-installer using environment From c4490a3a423017c9361456a78d548f93f4a9ed6c Mon Sep 17 00:00:00 2001 From: Doug Hellmann Date: Mon, 25 Jan 2021 14:16:46 -0500 Subject: [PATCH 09/17] expand reasoning behind using vms for ci Signed-off-by: Doug Hellmann --- ...le-node-installation-bootstrap-in-place.md | 48 +++++++++++++++---- 1 file changed, 38 insertions(+), 10 deletions(-) diff --git a/enhancements/installer/single-node-installation-bootstrap-in-place.md b/enhancements/installer/single-node-installation-bootstrap-in-place.md index 76a354f1f4..b6b12be1fb 100644 --- a/enhancements/installer/single-node-installation-bootstrap-in-place.md +++ b/enhancements/installer/single-node-installation-bootstrap-in-place.md @@ -342,16 +342,44 @@ and then remotely running /usr/local/bin/installer-masters-gather.sh on all mast ### Test Plan -In order to claim full support for this configuration, we must have -CI coverage informing the release. -An end-to-end job using the bootstrap-in-place installation flow, -based on the [installer UPI CI](https://github.com/openshift/release/blob/master/ci-operator/templates/openshift/installer/cluster-launch-installer-metal-e2e.yaml#L507), - running an appropriate subset of the standard OpenShift tests -will be created and configured to block accepting release images unless it passes. -This job is a different CI from the Single node production edge CI that will run with a bootstrap vm on cloud environment. - -That end-to-end job should also be run against pull requests for -the control plane repos, installer and cluster-bootstrap. +In order to claim full support for this configuration, we must have CI +coverage informing the release. An end-to-end job using the +bootstrap-in-place installation flow, based on the [installer UPI +CI](https://github.com/openshift/release/blob/master/ci-operator/templates/openshift/installer/cluster-launch-installer-metal-e2e.yaml#L507) +and running an appropriate subset of the standard OpenShift tests will +be created. This job is a different CI from the Single node production +edge CI that will run with a bootstrap vm on cloud environment. + +The new end-to-end job will be configured to block accepting release +images, and be run against pull requests for the control plane +repositories, installer and cluster-bootstrap. + +Although the feature is primarily targeted at bare metal use cases, we +have to balance the desire to test in 100% accurate configurations +with the effort and cost of running CI on bare metal. + +Our bare metal CI environment runs on Packet. The hosts are not +necessarily similar to those used by edge or telco customers, and the +API for managing the host is completely different than the APIs +supported by the provisioning tools that will be used to deploy +single-node instances in production environments. Given these +differences, we would derive little benefit from running CI jobs +directly on the hardware. + +Each CI job will need to create the test ISO configured to create the +cluster, then boot it on a host. This cannot be done from within a +single host, because the ISO must be served up to the host during the +bootstrap process, while the installer is overwriting the internal +storage of the host. So either the code to create and serve the ISO +needs to run in the CI cluster, or on another host. + +Both of these constraints make it simpler, more economical, and faster +to implement the CI job using VMs on a Packet host. Gaps introduced by +using VMs in CI will be covered through other testing performed by the +QE team using hardware more similar to what customers are expected to +have in their production environments. Over time, we may be able to +move some of those tests to more automated systems, including Packet, +if it makes sense. ### Graduation Criteria From 924debe7321180c0f5d584ee5e771c6255385489 Mon Sep 17 00:00:00 2001 From: Doug Hellmann Date: Tue, 26 Jan 2021 14:49:02 -0500 Subject: [PATCH 10/17] remove trailing whitespace Signed-off-by: Doug Hellmann --- .../single-node-installation-bootstrap-in-place.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/enhancements/installer/single-node-installation-bootstrap-in-place.md b/enhancements/installer/single-node-installation-bootstrap-in-place.md index b6b12be1fb..19a3360225 100644 --- a/enhancements/installer/single-node-installation-bootstrap-in-place.md +++ b/enhancements/installer/single-node-installation-bootstrap-in-place.md @@ -306,20 +306,20 @@ mitigate this, we can add documentation to the troubleshooting guide to explain the service and its role in the cluster. #### Bootstrap logs retention - + Due to the bootstrap-in-place behavior all bootstrap artifacts will be lost once the bootstrap the node reboots. In a regular installation flow the bootstrap node goes down only once the control plane is running, and the bootstrap node served its purpose. In case of bootstrap in place things can go wrong after the reboot. -The bootstrap logs can aid in troubleshooting a subsequently failed install. +The bootstrap logs can aid in troubleshooting a subsequently failed install. Mitigation by gathering the bootstrap logs before reboot. bootstrap will gather logs from itself using /usr/local/bin/installer-gather.sh. Once gathering is complete, the bundle will be added to the master ignition, thus making the bootstrap logs available from the master after reboot. The log bundle will be deleted once the installation completes. - + The installer `gather` command works as usual before the reboot. We will add a new script called installer-master-bootstrap-in-place-gather.sh. This script will be delivered to the master using via Ignition to the same @@ -481,7 +481,7 @@ CoreOS and this Ignition configuration, the Ignition config would lay down the control plane operator static pods and create a static pod that functions as `cluster-bootstrap` This pod should delete itself after it is done applying the OCP assets to the control plane. -The disadvantage in this approach is that it's very different than +The disadvantage in this approach is that it's very different than the regular installation flow which involve a bootstrap node. It also adds more challenges such as: 1. The installer machine (usually the Admin laptop) will need to pull From 60a73f585b8e52129185cdb491f5e57f30ea11a8 Mon Sep 17 00:00:00 2001 From: Doug Hellmann Date: Tue, 26 Jan 2021 14:50:12 -0500 Subject: [PATCH 11/17] clarify use of experimental environment variable in installer Signed-off-by: Doug Hellmann --- ...gle-node-installation-bootstrap-in-place.md | 18 +++++++++++++----- 1 file changed, 13 insertions(+), 5 deletions(-) diff --git a/enhancements/installer/single-node-installation-bootstrap-in-place.md b/enhancements/installer/single-node-installation-bootstrap-in-place.md index 19a3360225..08038a8aa4 100644 --- a/enhancements/installer/single-node-installation-bootstrap-in-place.md +++ b/enhancements/installer/single-node-installation-bootstrap-in-place.md @@ -150,11 +150,16 @@ kubernetes operations run during bootstrapping. #### OpenShift-installer -We will add a new `create single-node-ignition-config` command to the installer to create - `bootstrap-in-place-for-live-iso.ign` Ignition config. -This new target will not output master.ign and worker.ign. -Allow the user to specify the target disk drive for coreos-installer using environment - variable `OPENSHIFT_INSTALL_EXPERIMENTAL_BOOTSTRAP_IN_PLACE_COREOS_INSTALLER_ARGS`. +We will add a new `create single-node-ignition-config` command to the +installer to create the `bootstrap-in-place-for-live-iso.ign` Ignition +config. This new target will not output `master.ign` and `worker.ign` +files. + +Users will specify the target disk drive for `coreos-installer` using +the environment variable +`OPENSHIFT_INSTALL_EXPERIMENTAL_BOOTSTRAP_IN_PLACE_COREOS_INSTALLER_ARGS`. +Before the feature graduates from preview, the environment variable +will be replaced with a field in the `install-config.yaml` schema. This Ignition config will have a different `bootkube.sh` from the default bootstrap Ignition. In addition to the standard rendering @@ -421,6 +426,9 @@ In general, we try to use the same stages (alpha, beta, GA), regardless how the - More testing (upgrade, downgrade, scale) - Sufficient time for feedback - Available by default +- Update the installer to replace + `OPENSHIFT_INSTALL_EXPERIMENTAL_BOOTSTRAP_IN_PLACE_COREOS_INSTALLER_ARGS` + with a field in the `install-config.yaml` schema. **For non-optional features moving to GA, the graduation criteria must include end to end tests.** From 09a6ef9f3bafd616067d049a1c7305bff1d5ea06 Mon Sep 17 00:00:00 2001 From: Doug Hellmann Date: Tue, 26 Jan 2021 15:04:24 -0500 Subject: [PATCH 12/17] clarify handling of bootstrap files Add a little more detail about the way bootstrap files are temporarily copied into the host and add a test plan for a test to ensure they do not stay. Signed-off-by: Doug Hellmann --- ...le-node-installation-bootstrap-in-place.md | 48 +++++++++++++++---- 1 file changed, 40 insertions(+), 8 deletions(-) diff --git a/enhancements/installer/single-node-installation-bootstrap-in-place.md b/enhancements/installer/single-node-installation-bootstrap-in-place.md index 08038a8aa4..c1a7b9684a 100644 --- a/enhancements/installer/single-node-installation-bootstrap-in-place.md +++ b/enhancements/installer/single-node-installation-bootstrap-in-place.md @@ -193,13 +193,20 @@ return an error. #### Bootstrap / Control plane static pods -We will review the list of revisions for apiserver/etcd and see if we -can reduce them by reducing revisions caused by observations of known -conditions. For example in a single node we know what the etcd -endpoints will be in advance, We can avoid a revision by observing -this post install. This work will go a long way to reducing -disruption during install and improve mean time to recovery for upgrade re-deployments -and failures. +We will review the list of revisions for apiserver and etcd to see if +we can reduce them by eliminating changes caused by observations of +known conditions. For example, in a single node we know what the etcd +endpoints will be in advance, so we can avoid a revision by observing +this after installation. This work will go a long way to reducing +disruption during install and improve mean time to recovery for +upgrade re-deployments and failures. + +While there is a goal to ensure that the final node state does not +include bootstrapping files, it is necessary to copy some of the files +into the host temporarily to allow bootstrapping to complete. These +files are copied by embedding them in the combined Ignition config, +and after OpenShift is running on the host the files are deleted by +the `post-reboot` service. The control plane components we will copy from `/etc/kubernetes/manifests` into the master Ignition are: @@ -216,7 +223,20 @@ These components also require other files generated during bootstrapping: 3. `/opt/openshift/tls/*` (`/etc/kubernetes/bootstrap-secrets`) 4. `/opt/openshift/auth/kubeconfig-loopback` (`/etc/kubernetes/bootstrap-secrets/kubeconfig`) -**Note**: `/etc/kubernetes/bootstrap-secrets` and `/etc/kubernetes/bootstrap-configs` will be deleted by the `post-reboot` service, after the node reboots (see below). +The bootstrap logs are also copied from `/var/log` to aid in +debugging. + +See [installer PR +#4482](https://github.com/openshift/installer/pull/4482/files#diff-d09d8f9e83a054002d5344223d496781ea603da7c52706dfcf33debf8ceb1df3) +for a detailed list of the files added to the Ignition config. + +After the node reboots, the temporary copies of the bootstrapping +files are deleted by the `post-reboot` service, including: + +1. `/etc/kubernetes/bootstrap-configs` +2. `/opt/openshift/tls/*` (`/etc/kubernetes/bootstrap-secrets`) +3. `/opt/openshift/auth/kubeconfig-loopback` (`/etc/kubernetes/bootstrap-secrets/kubeconfig`) +4. bootstrap logs The bootstrap static pods will be generated in a way that the control plane operators will be able to identify them and either continue in a @@ -347,6 +367,8 @@ and then remotely running /usr/local/bin/installer-masters-gather.sh on all mast ### Test Plan +#### End-to-end testing + In order to claim full support for this configuration, we must have CI coverage informing the release. An end-to-end job using the bootstrap-in-place installation flow, based on the [installer UPI @@ -386,6 +408,16 @@ have in their production environments. Over time, we may be able to move some of those tests to more automated systems, including Packet, if it makes sense. +#### Bootstrap cleanup tests + +A goal of this enhancement is to ensure that the host does not contain +any of the temporary bootstrapping files after OpenShift is +running. During bootstrapping, it is necessary to copy some of the +unwanted files into the host temporarily. A test will be created to +verify that the host does not retain those files after the cluster is +successfully launched. The test will run either as part of the +end-to-end job described above, or as part of a separate job. + ### Graduation Criteria **Note:** *Section not required until targeted at a release.* From 97a09245b2a874266c44ce722a9f659f67001500 Mon Sep 17 00:00:00 2001 From: Doug Hellmann Date: Tue, 26 Jan 2021 15:08:17 -0500 Subject: [PATCH 13/17] mention that post-reboot service removes itself Signed-off-by: Doug Hellmann --- .../single-node-installation-bootstrap-in-place.md | 14 +++++++++++--- 1 file changed, 11 insertions(+), 3 deletions(-) diff --git a/enhancements/installer/single-node-installation-bootstrap-in-place.md b/enhancements/installer/single-node-installation-bootstrap-in-place.md index c1a7b9684a..36ddb9c2da 100644 --- a/enhancements/installer/single-node-installation-bootstrap-in-place.md +++ b/enhancements/installer/single-node-installation-bootstrap-in-place.md @@ -250,15 +250,23 @@ the node Certificate Signing Requests. This service will also cleanup the bootstrap static pods resources when the OpenShift control plane is ready. -Since we start with a liveCD, the bootstrap services (`bootkube`, `approve-csr`, etc.), `/etc` and `/opt/openshift` temporary files are written to the ephemeral filesystem of the live image, and not to the node's real filesystem. +Since we start with a liveCD, the bootstrap services (`bootkube`, +`approve-csr`, etc.), `/etc` and `/opt/openshift` temporary files are +written to the ephemeral filesystem of the live image, and not to the +node's real filesystem. The files that we need to delete are under: * `/etc/kubernetes/bootstrap-secrets` * `/etc/kubernetes/bootstrap-configs` -These files are required for the bootstrap control plane to start before it is replaced by the control plane operators. -Once the OCP control plane static pods are deployed we can delete the files as they are no longer required. +These files are required for the bootstrap control plane to start +before it is replaced by the control plane operators. Once the OCP +control plane static pods are deployed we can delete the files as they +are no longer required. + +When the `post-reboot` service has completed its work, it removes +itself so it is not run the next time the host reboots. #### Prerequisites for a Single Node deployment with bootstrap-in-place The requirements are a subset of the requirements for user-provisioned infrastructure installation. From 2453d7bc6c7a958d80dbcf7df564017539438165 Mon Sep 17 00:00:00 2001 From: Doug Hellmann Date: Tue, 26 Jan 2021 15:09:55 -0500 Subject: [PATCH 14/17] point out static pod optimization can apply to all clusters Signed-off-by: Doug Hellmann --- .../single-node-installation-bootstrap-in-place.md | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/enhancements/installer/single-node-installation-bootstrap-in-place.md b/enhancements/installer/single-node-installation-bootstrap-in-place.md index 36ddb9c2da..d5c7c98601 100644 --- a/enhancements/installer/single-node-installation-bootstrap-in-place.md +++ b/enhancements/installer/single-node-installation-bootstrap-in-place.md @@ -371,7 +371,11 @@ and then remotely running /usr/local/bin/installer-masters-gather.sh on all mast ### Open Questions 1. How will the user specify custom configurations, such as static IPs? -2. Number of revisions for the control plane - do we want to make changes to the bootstrap static pods to make them closer to the final ones? +2. Number of revisions for the control plane - do we want to make + changes to the bootstrap static pods to make them closer to the + final ones? This would also benefit multi-node deployments, but is + especially helpful for single-node deployments where updating those + static pod definitions may be more disruptive. ### Test Plan From 89b11097da3b6f906ed1218489d7761117f0ee9b Mon Sep 17 00:00:00 2001 From: Eran Cohen Date: Wed, 27 Jan 2021 19:42:25 +0200 Subject: [PATCH 15/17] Fixed linter issues --- .../single-node-installation-bootstrap-in-place.md | 11 +++++------ 1 file changed, 5 insertions(+), 6 deletions(-) diff --git a/enhancements/installer/single-node-installation-bootstrap-in-place.md b/enhancements/installer/single-node-installation-bootstrap-in-place.md index d5c7c98601..fcbf623bb7 100644 --- a/enhancements/installer/single-node-installation-bootstrap-in-place.md +++ b/enhancements/installer/single-node-installation-bootstrap-in-place.md @@ -226,8 +226,7 @@ These components also require other files generated during bootstrapping: The bootstrap logs are also copied from `/var/log` to aid in debugging. -See [installer PR -#4482](https://github.com/openshift/installer/pull/4482/files#diff-d09d8f9e83a054002d5344223d496781ea603da7c52706dfcf33debf8ceb1df3) +See [installer PR#4482](https://github.com/openshift/installer/pull/4482/files#diff-d09d8f9e83a054002d5344223d496781ea603da7c52706dfcf33debf8ceb1df3) for a detailed list of the files added to the Ignition config. After the node reboots, the temporary copies of the bootstrapping @@ -340,7 +339,7 @@ to explain the service and its role in the cluster. #### Bootstrap logs retention - Due to the bootstrap-in-place behavior all bootstrap artifacts +Due to the bootstrap-in-place behavior all bootstrap artifacts will be lost once the bootstrap the node reboots. In a regular installation flow the bootstrap node goes down only once the control plane is running, and the bootstrap node served its purpose. @@ -444,8 +443,8 @@ Consider the following in developing the graduation criteria for this enhancement: - Maturity levels - - [`alpha`, `beta`, `stable` in upstream Kubernetes][maturity-levels] - - `Dev Preview`, `Tech Preview`, `GA` in OpenShift + - [`alpha`, `beta`, `stable` in upstream Kubernetes][maturity-levels] + - `Dev Preview`, `Tech Preview`, `GA` in OpenShift - [Deprecation policy][deprecation-policy] Clearly define what graduation means by either linking to the [API doc definition](https://kubernetes.io/docs/concepts/overview/kubernetes-api/#api-versioning), @@ -588,7 +587,7 @@ by adding logic to the installer that check the number of replicas for the contr plane (in the `install-config.yaml`) is `1`. This approach might conflict with CRC/SNC which also run openshift-install with a 1-replica control plane. -### Use `create ignition-configs` with environment variable to generate the `bootstrap-in-place-for-live-iso.ign`. +### Use `create ignition-configs` with environment variable to generate the `bootstrap-in-place-for-live-iso.ign` We also considered adding a new environment variable `OPENSHIFT_INSTALL_EXPERIMENTAL_BOOTSTRAP_IN_PLACE` for marking the new path under the `ignition-configs` target. From 514e207819e7b12f94d191065de8be385a31652e Mon Sep 17 00:00:00 2001 From: Doug Hellmann Date: Thu, 28 Jan 2021 14:05:57 -0500 Subject: [PATCH 16/17] update discussion of how OS is installed Signed-off-by: Doug Hellmann --- ...le-node-installation-bootstrap-in-place.md | 94 ++++++++++++++++--- 1 file changed, 83 insertions(+), 11 deletions(-) diff --git a/enhancements/installer/single-node-installation-bootstrap-in-place.md b/enhancements/installer/single-node-installation-bootstrap-in-place.md index fcbf623bb7..55b9caf12a 100644 --- a/enhancements/installer/single-node-installation-bootstrap-in-place.md +++ b/enhancements/installer/single-node-installation-bootstrap-in-place.md @@ -155,12 +155,6 @@ installer to create the `bootstrap-in-place-for-live-iso.ign` Ignition config. This new target will not output `master.ign` and `worker.ign` files. -Users will specify the target disk drive for `coreos-installer` using -the environment variable -`OPENSHIFT_INSTALL_EXPERIMENTAL_BOOTSTRAP_IN_PLACE_COREOS_INSTALLER_ARGS`. -Before the feature graduates from preview, the environment variable -will be replaced with a field in the `install-config.yaml` schema. - This Ignition config will have a different `bootkube.sh` from the default bootstrap Ignition. In addition to the standard rendering logic, the modified script will: @@ -171,8 +165,27 @@ logic, the modified script will: config, the control plane static pod manifests, the required kubernetes resources, and the bootstrap etcd database snapshot to create a new Ignition config for the host. -3. Write the RHCOS image and the combined Ignition config to disk. -4. Reboot the node. + +In order to successfully complete the installation, the user must tell +`coreos-install` where to write the operating system image on the +host's local storage. It may be necessary to provide other inputs to +`coreos-install` at the same time (network addresses, etc.), and we +have not yet completed the investigation and experimentation work +required to design a supportable API for the complex details that may +be needed. Therefore, to unblock progress on the bootstrap-in-place +feature, we will temporarily treat installing the operating system as +a manual step. Users running the OpenShift installer by hand will need +to login to the host via `ssh` and run `coreos-install`, passing the +appropriate arguments. Users relying on integration with Hive and the +assisted installer for multi-cluster management will not need to +perform this step by hand, since the agent driven by the assisted +installer service runs `coreos-install` already. The installer and +edge teams will work on this area further during this next release +cycle, and use another enhancement proposal to describe the +implementation of an API to replace the manual step. + +After the user runs the `coreos-install` and the RHCOS image is +written to disk, they also need to reboot the host. #### Cluster-bootstrap @@ -469,9 +482,12 @@ In general, we try to use the same stages (alpha, beta, GA), regardless how the - More testing (upgrade, downgrade, scale) - Sufficient time for feedback - Available by default -- Update the installer to replace - `OPENSHIFT_INSTALL_EXPERIMENTAL_BOOTSTRAP_IN_PLACE_COREOS_INSTALLER_ARGS` - with a field in the `install-config.yaml` schema. +- Either define a maintainable API for passing arguments to + `coreos-install` in the OpenShift installer so that the + bootstrapping services can install the operating system on the host, + or definitively decide that installing the operating system should + always be a manual step when the user is running the OpenShift + installer outside of our multi-cluster management integration. **For non-optional features moving to GA, the graduation criteria must include end to end tests.** @@ -595,3 +611,59 @@ We decided to add `single-node-ignition-config` target to in order to gain: 1. Allow us to easily add different set of validations (e.g. ensure that the number of replicas for the control plane is 1). 2. We can avoid creating unnecessary assets (master.ign and worker.ign). 3. Less prune to user errors than environment variable. + +### Alternatives for managing the operating system installation step + +Before settling on using a manual process, we identified several +options for installing the operating system during deployment. + +1. We could give the user instructions to ssh to the host and run a + command. + + This is the option we selected. It requires additional work for + users running the installer by hand, and will require some changes + to the CI jobs for single-node deployments, but is the most + flexible and does not commit us to maintaining and API we have + rushed to design. + +2. We could add an unsupported experimental environment variable to + all installer builds, so that the user could pass information + through the OpenShift installer to the CoreOS installer. + + The risk of using an environment variable is that it would become a + de facto API that we must support over a long period of time. This + is especially likely since we are collaborating closely with users + and cannot guarantee that they will not build automation of their + own around the installer. + +3. We could add an experimental environment variable to a special + installer build, so that it is not in all builds. + + Using a custom build would make it easier to control who has access + to the feature, but we would still need to share the binary with + our customers so this approach does not actually eliminate the risk + of having the feature implemented at all. + +4. We could pause now and spend time to design the install-config + fields needed. + + Because of the aggressive deadline for the project, we do not feel + we can afford to pause all implementation work until this API + design is finalized. + +5. We could assume that all single-node deployments would rely on the + assisted-installer, and not support manual installation. + + This assumption would put us at risk for delivering the single-node + feature to users on the timeline that has been promised. + +6. We could give the user instructions to edit the Ignition config + file produced by the OpenShift installer to add a service to run + `coreos-install`. + + While still a manual step, and still technically possible, editing + the Ignition config is more complicated to explain that having the + user run the installer by hand. Some users may still choose to use + this approach, but we should be careful that supporting it does not + cause problems with future work to make the OpenShift installer add + similar configuration to the same file. From c0d1e1c1e53a95154447d25312f53f34a8ad3aaa Mon Sep 17 00:00:00 2001 From: Doug Hellmann Date: Wed, 3 Feb 2021 14:47:00 -0500 Subject: [PATCH 17/17] switch 'create single-node-ignition-config' to take input and use systemd Signed-off-by: Doug Hellmann --- ...le-node-installation-bootstrap-in-place.md | 87 ++++++++++++------- 1 file changed, 56 insertions(+), 31 deletions(-) diff --git a/enhancements/installer/single-node-installation-bootstrap-in-place.md b/enhancements/installer/single-node-installation-bootstrap-in-place.md index 55b9caf12a..de2f2f08c9 100644 --- a/enhancements/installer/single-node-installation-bootstrap-in-place.md +++ b/enhancements/installer/single-node-installation-bootstrap-in-place.md @@ -152,12 +152,12 @@ kubernetes operations run during bootstrapping. We will add a new `create single-node-ignition-config` command to the installer to create the `bootstrap-in-place-for-live-iso.ign` Ignition -config. This new target will not output `master.ign` and `worker.ign` +config. This new target will *not* output `master.ign` and `worker.ign` files. -This Ignition config will have a different `bootkube.sh` from the -default bootstrap Ignition. In addition to the standard rendering -logic, the modified script will: +The `bootstrap-in-place-for-live-iso.ign` Ignition config will have a +different `bootkube.sh` from the default bootstrap Ignition. In +addition to the standard rendering logic, the modified script will: 1. Start `cluster-bootstrap` without required pods by setting `--required-pods=''` 2. Run `cluster-bootstrap` with the `--bootstrap-in-place` option. @@ -166,26 +166,37 @@ logic, the modified script will: kubernetes resources, and the bootstrap etcd database snapshot to create a new Ignition config for the host. -In order to successfully complete the installation, the user must tell -`coreos-install` where to write the operating system image on the -host's local storage. It may be necessary to provide other inputs to -`coreos-install` at the same time (network addresses, etc.), and we -have not yet completed the investigation and experimentation work -required to design a supportable API for the complex details that may -be needed. Therefore, to unblock progress on the bootstrap-in-place -feature, we will temporarily treat installing the operating system as -a manual step. Users running the OpenShift installer by hand will need -to login to the host via `ssh` and run `coreos-install`, passing the -appropriate arguments. Users relying on integration with Hive and the -assisted installer for multi-cluster management will not need to -perform this step by hand, since the agent driven by the assisted -installer service runs `coreos-install` already. The installer and -edge teams will work on this area further during this next release -cycle, and use another enhancement proposal to describe the -implementation of an API to replace the manual step. - -After the user runs the `coreos-install` and the RHCOS image is -written to disk, they also need to reboot the host. +In order to successfully complete the installation, we must know +where to write the operating system image on the host's local +storage. Initially this information will be passed as a single value +in the `install-config.yaml`, using a schema like + +```yaml +coreOSInstallation: + rootDevice: '/dev/vda' +``` + +We expect this section of the `install-config.yaml` schema to evolve +and add more fields. The precise name of the new section and this +first field will be worked out on the pull request to update +installer. Future updates, to pass additional data to the installer or +to support multiple values for multiple hosts, will be addressed in a +separate enhancement. + +The new `coreOSInstallation` section and the field with the root +device name will always be required for the `create +single-node-ignition-config`. If the details are omitted from the +`install-config.yaml`, the installer will report an error and exit. + +Using the data from the `coreOSInstallation` section, the OpenShift +installer will add a new `systemd` unit to the +`bootstrap-in-place-for-live-iso.ign` Ignition config to run +`coreos-install` to install the operating system onto the host's local +storage and reboot the host. This service will run when a single-node +cluster is bootstrapping without the assisted installer, but will be +skipped when the assisted installer is used so that we can retain the +existing features to monitor progress and control the OS installation +more completely. #### Cluster-bootstrap @@ -378,6 +389,21 @@ and then remotely running /usr/local/bin/installer-masters-gather.sh on all mast on itself. The final archiving of all the logs into the home directory, exactly the same as /usr/local/bin/installer-gather.sh. +#### Tight coupling of assisted installer and OpenShift installer implementation + +The assisted installer currently relies on implementation details of +the bootstrap process to change the way clusters are created. This +change introduces another dependency on the `bootkube.service` in the +Ignition config generated by the installer creating another Ignition +config during bootstrapping and placing it in +`/opt/openshift/master.ign`. Renaming that file within the OpenShift +installer will break the assisted installer. + +All of these dependencies on implementation details will be addressed +in future enhancements, either by documenting them as APIs or by +providing new APIs and having the assisted installer migrate to using +those. + ## Design Details ### Open Questions @@ -614,17 +640,16 @@ We decided to add `single-node-ignition-config` target to in order to gain: ### Alternatives for managing the operating system installation step -Before settling on using a manual process, we identified several -options for installing the operating system during deployment. +Before settling on adding a minimal set of fields to +`install-config.yaml`, we identified several options for installing +the operating system during deployment. 1. We could give the user instructions to ssh to the host and run a command. - This is the option we selected. It requires additional work for - users running the installer by hand, and will require some changes - to the CI jobs for single-node deployments, but is the most - flexible and does not commit us to maintaining and API we have - rushed to design. + This option requires extra manual steps by the user, which is + likely to lead to various implementations in different environments + when the steps are automated. 2. We could add an unsupported experimental environment variable to all installer builds, so that the user could pass information