diff --git a/enhancements/agent-installer/automated-workflow-kubeconfig.md b/enhancements/agent-installer/automated-workflow-kubeconfig.md new file mode 100644 index 0000000000..42ecb772e5 --- /dev/null +++ b/enhancements/agent-installer/automated-workflow-kubeconfig.md @@ -0,0 +1,275 @@ +--- +title: automated-workflow-kubeconfig +authors: + - "@zaneb" +reviewers: + - "@patrickdillon" + - "@sadasu" + - "@avishayt" + - "@mhrivnak" + - "@andfasano" + - "@bfournie" + - "@lranjbar" + - "@pawanpinjarkar" + - "@rwsu" +approvers: + - "@dhellmann" +api-approvers: + - None +creation-date: 2022-05-18 +last-updated: 2022-06-09 +tracking-link: + - https://issues.redhat.com/browse/AGENT-40 +see-also: + - "/enhancements/agent-installer/agent-based-installer.md" + - "/enhancements/agent-installer/automated-workflow-for-agent-based-installer.md" +replaces: N/A +superseded-by: N/A +--- + +# Pre-generating the admin Kubeconfig in the Agent-based installer + +## Summary + +We will generate the admin kubeconfig file in the agent installer at the same +time as the agent ISO. The CA keys used to sign the client certificate in the +kubeconfig will be added to the ISO and eventually installed into the cluster +so that the credentials in the admin kubeconfig will be accepted. + +## Motivation + +In the hosted assisted service, the admin kubeconfig is available for download +from the web UI only after assisted-service has run openshift-installer to +create the bootstrap ignition (i.e. after cluster installation has started). +Since the hosted service is always available, this config can be downloaded at +any time before the certificates in it expire. + +For the agent-based installer, doing the same would mean the kubeconfig was +only available for a brief window of time, since the assisted-service is +ephemeral. It runs on node 0 of the cluster and disappears after bootstrapping +when that node is rebooted to become a cluster member. + +Furthermore, the kubeconfig file needs to be available to the installer in +order for it to be used to monitor the cluster status in the `openshift-install +agent wait-for install-complete` command. + +### User Stories + +* As a cluster creator, I want to have a reliable, non-timing dependent way to + retrieve the admin kubeconfig for a new cluster created with the agent-based + installer. + +* As a cluster creator, I want to use the `agent wait-for install-complete` + command to wait for the cluster to complete, with the preceding steps in the + automated workflow limited to only running the `agent create image` command + and booting the ISO on the hosts. + +### Goals + +* Generate a working admin kubeconfig as part of the `openshift-install agent + create image` command. +* Remain as consistent as possible with the installer behaviour in the IPI and + UPI install methods. + +### Non-Goals + +* Make the admin kubeconfig available earlier in the hosted assisted UI. +* Generate the admin password for the console during creation of the ISO (this + will be added later). +* Implement key rotation for OpenShift clusters in general. + +## Proposal + +The admin kubeconfig will be generated by the `agent create image` command +using an Asset based on the existing `kubeconfig.AdminClient` one. This will +differ only in that it will get the cluster API address from the agent +cluster-manifests, rather than the install-config, because the install-config +is optional in the agent-based installer flow. + +The actual ignition files used for bootstrapping the cluster are not created by +the agent-based installer binary, but rather by the assisted-service running +openshift-installer again. In order to ensure that the cluster that is +installed accepts the credentials in the kubeconfig, then during the ignition +generation step of the installer run by assisted-service, we must overwrite or +modify on disk: + +* All assets that the admin kubeconfig depends on (directly or indirectly) + * Unless the asset can be regenerated deterministically from its own + dependencies. +* For each of *those* assets that cannot be loaded from disk, all other assets + that depend on it, directly or indirectly, until reaching assets that can be + loaded from disk + +The status quo is that amongst the writable assets (shown in color below), +there are some (shown in red) that cannot actually be read back by the +installer. As a result, both the nature and the sheer number of assets required +would preclude this approach: + +![Assets implicated by kubeconfig (status quo)](./kubeconfig_implications_without_changes.svg) + +However, by adding the ability to read back a handful of assets (shown in blue +below) that currently cannot be, this complexity is massively reduced: + +![Assets implicated by kubeconfig with cert/key loading](./kubeconfig_implications_with_cert_loading.svg) + +Accordingly, the installer will be modified to allow reading the following +certificate assets from disk: + +* `tls.KubeAPIServerLBSignerCertKey` +* `tls.KubeAPIServerLocalhostSignerCertKey` +* `tls.KubeAPIServerServiceNetworkSignerCertKey` +* `tls.AdminKubeConfigSignerCertKey` +* `tls.AdminKubeConfigClientCertKey` + +When generating the agent ISO, the certificate and key files for those same +assets will be built in to the ISO. + +The assisted-service will copy the certificates from a location provided by the +environment variable `EPHEMERAL_INSTALLER_CLUSTER_TLS_CERTS_OVERRIDE_DIR` to +the installer's work directory before generating the manifests/ignition. + +### Workflow Description + +```mermaid +sequenceDiagram +Cluster creator->>Agent installer: agent create image +Agent installer->>Certificate assets: Generate() +Agent installer->>Kubeconfig asset: Generate() +Certificate assets-->>Kubeconfig asset: certificates +Kubeconfig asset-->>Cluster creator: kubeconfig +Agent installer->>Agent ISO: Generate() +Certificate assets-->>Agent ISO: certificates +Agent ISO-->>Cluster creator: agent.iso +Agent ISO-->>Assisted Service: certificates +Assisted Service-->>OpenShift Installer: certificates +OpenShift Installer-->>Assisted Service: kubeconfig +OpenShift Installer-->>Bootstrap: ignition +``` + +### API Extensions + +N/A + +### Implementation Details/Notes/Constraints + +Note that the modifications to the installer to read certificates from disk +need to occur in the installer present in the payload of the release being +installed, i.e. on the `master` branch, not the `agent-installer` branch where +the agent-based installer is being developed. + +The work directory for the installer is a tmpdir created by assisted-service on +the fly with a randomised name. Therefore it is not possible to pre-populate +the certificates into the directory, and assisted-service must be involved in +copying the files. + +### Risks and Mitigations + +End users will be able to supply their own CA certificates to the installer +much more easily than they could before. If done incorrectly, this may leave +the cluster less secure than if the keys and certificates are generated inside +openshift-installer. To discourage users from relying on this, the ability to +read the certificates from disk will only be enabled when the environment +variable `OPENSHIFT_INSTALL_LOAD_CLUSTER_CERTS` is set to the value `true`. The +assisted-service will set this in the environment when running the installer, +but otherwise it should not be routinely set. + +A copy of the admin kubeconfig will remain accessible via the assisted-service +API, enabling an attacker with access to the network environment of the cluster +during bootstrapping to gain access to it. We could safely disable this API +when the `EPHEMERAL_INSTALLER_CLUSTER_TLS_CERTS_OVERRIDE_DIR` environment +variable is set. We expect to add authentication to the assisted-service API in +future, at which point this will cease to be an issue. + +The private key for the admin kubeconfig client certificate, as well as for the +API Server signing certificates, will be embedded in the generated ISO. This +means that the ISO must be treated as private, even after the cluster is +created. Care must be taken when exposing it over the network, e.g. for PXE +boot. However, the same is already true for e.g. the UPI Bootstrap Ignition. + +### Drawbacks + +The same mechanism will likely not be usable in the interactive flow, because +we won't know the cluster name (and therefore its API address) at the time of +generating the agent ISO. + +## Design Details + +### Open Questions + +### Test Plan + +Since the admin kubeconfig is required to verify that the cluster installation +is completed, this feature is expected to be tested on every CI run. + +### Graduation Criteria + +N/A + +#### Dev Preview -> Tech Preview + +N/A + +#### Tech Preview -> GA + +N/A + +#### Removing a deprecated feature + +N/A + +### Upgrade / Downgrade Strategy + +N/A + +### Version Skew Strategy + +N/A + +### Operational Aspects of API Extensions + +N/A + +#### Failure Modes + +N/A + +#### Support Procedures + +N/A + +## Implementation History + +The changes to the installer to read the certificates/keys from disk were +implemented in +[openshift/installer#5872](https://github.com/openshift/installer/pull/5872) +and are available in OpenShift 4.11 nightlies from Feature Freeze on. + +The changes to assisted-service to copy the certificates/keys into the +temporary work directory for the installer were implemented in +[openshift/assisted-service#3836](https://github.com/openshift/assisted-service/pull/3836). + +The changes to the agent installer to generate the kubeconfig and pass the +certificate/key files to the assisted-service were implemented in +[openshift/installer#5959](https://github.com/openshift/installer/pull/5959). + +## Alternatives + +The agent installer could attempt to download the kubeconfig from the +assisted-service during installation. However, since it is up to the user to +boot the ISO this would have to be a separate command that the user would have +to run at the right time. + +Similarly, the kubeconfig could be written to disk on one of the nodes and the +agent installer could attempt to download it over ssh. This suffers from +similar problems, although it could work at any time. It could also result in +nodes being tainted due to the ssh access. + +A service running in the cluster could provide access to the admin kubeconfig +using some sort of ad hoc authentication, not dependent on the admin kubeconfig +signing key. It could perhaps also be removed when no longer required. This +would not be consistent with the other installer workflows, and could require +complex work on the part of the user when the cluster is started in an +environment not connected to the one where the ISO was generated (so the +installer might not be available to provide automation). It would be better +with the automation implemented in `oc`, and applicable to clusters installed +by any method, but such a change is out of scope for this enhancement. diff --git a/enhancements/agent-installer/kubeconfig_implications_with_cert_loading.svg b/enhancements/agent-installer/kubeconfig_implications_with_cert_loading.svg new file mode 100644 index 0000000000..de8aab9d3a --- /dev/null +++ b/enhancements/agent-installer/kubeconfig_implications_with_cert_loading.svg @@ -0,0 +1,250 @@ + + + + + + +G + + +cluster_installconfig + +installconfig + + +cluster_kubeconfig + +kubeconfig + + +cluster_tls + +tls + + + +installconfig.InstallConfig + +installconfig.InstallConfig + + + +kubeconfig.AdminClient + +kubeconfig.AdminClient + + + +installconfig.InstallConfig->kubeconfig.AdminClient + + + + + +installconfig.sshPublicKey + +installconfig.sshPublicKey + + + +installconfig.sshPublicKey->installconfig.InstallConfig + + + + + +installconfig.baseDomain + +installconfig.baseDomain + + + +installconfig.baseDomain->installconfig.InstallConfig + + + + + +installconfig.clusterName + +installconfig.clusterName + + + +installconfig.baseDomain->installconfig.clusterName + + + + + +installconfig.platform + +installconfig.platform + + + +installconfig.platform->installconfig.InstallConfig + + + + + +installconfig.platform->installconfig.baseDomain + + + + + +installconfig.platform->installconfig.clusterName + + + + + +installconfig.networking + +installconfig.networking + + + +installconfig.platform->installconfig.networking + + + + + +installconfig.clusterName->installconfig.InstallConfig + + + + + +installconfig.networking->installconfig.InstallConfig + + + + + +installconfig.pullSecret + +installconfig.pullSecret + + + +installconfig.pullSecret->installconfig.InstallConfig + + + + + +tls.AdminKubeConfigClientCertKey + +tls.AdminKubeConfigClientCertKey + + + +tls.AdminKubeConfigClientCertKey->kubeconfig.AdminClient + + + + + +tls.AdminKubeConfigSignerCertKey + +tls.AdminKubeConfigSignerCertKey + + + +tls.AdminKubeConfigSignerCertKey->tls.AdminKubeConfigClientCertKey + + + + + +tls.KubeAPIServerLocalhostSignerCertKey + +tls.KubeAPIServerLocalhostSignerCertKey + + + +tls.KubeAPIServerLocalhostCABundle + +tls.KubeAPIServerLocalhostCABundle + + + +tls.KubeAPIServerLocalhostSignerCertKey->tls.KubeAPIServerLocalhostCABundle + + + + + +tls.KubeAPIServerServiceNetworkSignerCertKey + +tls.KubeAPIServerServiceNetworkSignerCertKey + + + +tls.KubeAPIServerServiceNetworkCABundle + +tls.KubeAPIServerServiceNetworkCABundle + + + +tls.KubeAPIServerServiceNetworkSignerCertKey->tls.KubeAPIServerServiceNetworkCABundle + + + + + +tls.KubeAPIServerLBSignerCertKey + +tls.KubeAPIServerLBSignerCertKey + + + +tls.KubeAPIServerLBCABundle + +tls.KubeAPIServerLBCABundle + + + +tls.KubeAPIServerLBSignerCertKey->tls.KubeAPIServerLBCABundle + + + + + +tls.KubeAPIServerCompleteCABundle + +tls.KubeAPIServerCompleteCABundle + + + +tls.KubeAPIServerCompleteCABundle->kubeconfig.AdminClient + + + + + +tls.KubeAPIServerLocalhostCABundle->tls.KubeAPIServerCompleteCABundle + + + + + +tls.KubeAPIServerServiceNetworkCABundle->tls.KubeAPIServerCompleteCABundle + + + + + +tls.KubeAPIServerLBCABundle->tls.KubeAPIServerCompleteCABundle + + + + + diff --git a/enhancements/agent-installer/kubeconfig_implications_without_changes.svg b/enhancements/agent-installer/kubeconfig_implications_without_changes.svg new file mode 100644 index 0000000000..45769500f6 --- /dev/null +++ b/enhancements/agent-installer/kubeconfig_implications_without_changes.svg @@ -0,0 +1,1703 @@ + + + + + + +G + + +cluster_baremetal + +baremetal + + +cluster_bootkube + +bootkube + + +cluster_bootstrap + +bootstrap + + +cluster_installconfig + +installconfig + + +cluster_kubeconfig + +kubeconfig + + +cluster_machine + +machine + + +cluster_machines + +machines + + +cluster_manifests + +manifests + + +cluster_openshift + +openshift + + +cluster_openshiftinstall + +openshiftinstall + + +cluster_password + +password + + +cluster_releaseimage + +releaseimage + + +cluster_rhcos + +rhcos + + +cluster_tls + +tls + + + +bootstrap.Bootstrap + +bootstrap.Bootstrap + + + +baremetal.IronicCreds + +baremetal.IronicCreds + + + +baremetal.IronicCreds->bootstrap.Bootstrap + + + + + +installconfig.InstallConfig + +installconfig.InstallConfig + + + +installconfig.InstallConfig->bootstrap.Bootstrap + + + + + +kubeconfig.AdminInternalClient + +kubeconfig.AdminInternalClient + + + +installconfig.InstallConfig->kubeconfig.AdminInternalClient + + + + + +kubeconfig.Kubelet + +kubeconfig.Kubelet + + + +installconfig.InstallConfig->kubeconfig.Kubelet + + + + + +kubeconfig.LoopbackClient + +kubeconfig.LoopbackClient + + + +installconfig.InstallConfig->kubeconfig.LoopbackClient + + + + + +machine.MasterIgnitionCustomizations + +machine.MasterIgnitionCustomizations + + + +installconfig.InstallConfig->machine.MasterIgnitionCustomizations + + + + + +machine.Master + +machine.Master + + + +installconfig.InstallConfig->machine.Master + + + + + +machine.WorkerIgnitionCustomizations + +machine.WorkerIgnitionCustomizations + + + +installconfig.InstallConfig->machine.WorkerIgnitionCustomizations + + + + + +machine.Worker + +machine.Worker + + + +installconfig.InstallConfig->machine.Worker + + + + + +machines.Master + +machines.Master + + + +installconfig.InstallConfig->machines.Master + + + + + +installconfig.ClusterID + +installconfig.ClusterID + + + +installconfig.InstallConfig->installconfig.ClusterID + + + + + +installconfig.PlatformCredsCheck + +installconfig.PlatformCredsCheck + + + +installconfig.InstallConfig->installconfig.PlatformCredsCheck + + + + + +rhcos.Image + +rhcos.Image + + + +installconfig.InstallConfig->rhcos.Image + + + + + +machines.Worker + +machines.Worker + + + +installconfig.InstallConfig->machines.Worker + + + + + +manifests.Manifests + +manifests.Manifests + + + +installconfig.InstallConfig->manifests.Manifests + + + + + +manifests.Ingress + +manifests.Ingress + + + +installconfig.InstallConfig->manifests.Ingress + + + + + +manifests.DNS + +manifests.DNS + + + +installconfig.InstallConfig->manifests.DNS + + + + + +manifests.Infrastructure + +manifests.Infrastructure + + + +installconfig.InstallConfig->manifests.Infrastructure + + + + + +manifests.CloudProviderConfig + +manifests.CloudProviderConfig + + + +installconfig.InstallConfig->manifests.CloudProviderConfig + + + + + +manifests.AdditionalTrustBundleConfig + +manifests.AdditionalTrustBundleConfig + + + +installconfig.InstallConfig->manifests.AdditionalTrustBundleConfig + + + + + +manifests.Networking + +manifests.Networking + + + +installconfig.InstallConfig->manifests.Networking + + + + + +manifests.Proxy + +manifests.Proxy + + + +installconfig.InstallConfig->manifests.Proxy + + + + + +manifests.Scheduler + +manifests.Scheduler + + + +installconfig.InstallConfig->manifests.Scheduler + + + + + +manifests.ImageContentSourcePolicy + +manifests.ImageContentSourcePolicy + + + +installconfig.InstallConfig->manifests.ImageContentSourcePolicy + + + + + +tls.MCSCertKey + +tls.MCSCertKey + + + +installconfig.InstallConfig->tls.MCSCertKey + + + + + +manifests.Openshift + +manifests.Openshift + + + +installconfig.InstallConfig->manifests.Openshift + + + + + +tls.CloudProviderCABundle + +tls.CloudProviderCABundle + + + +installconfig.InstallConfig->tls.CloudProviderCABundle + + + + + +tls.KubeAPIServerExternalLBServerCertKey + +tls.KubeAPIServerExternalLBServerCertKey + + + +installconfig.InstallConfig->tls.KubeAPIServerExternalLBServerCertKey + + + + + +tls.KubeAPIServerInternalLBServerCertKey + +tls.KubeAPIServerInternalLBServerCertKey + + + +installconfig.InstallConfig->tls.KubeAPIServerInternalLBServerCertKey + + + + + +tls.KubeAPIServerServiceNetworkServerCertKey + +tls.KubeAPIServerServiceNetworkServerCertKey + + + +installconfig.InstallConfig->tls.KubeAPIServerServiceNetworkServerCertKey + + + + + +kubeconfig.AdminClient + +kubeconfig.AdminClient + + + +installconfig.InstallConfig->kubeconfig.AdminClient + + + + + +installconfig.sshPublicKey + +installconfig.sshPublicKey + + + +installconfig.sshPublicKey->installconfig.InstallConfig + + + + + +installconfig.baseDomain + +installconfig.baseDomain + + + +installconfig.baseDomain->installconfig.InstallConfig + + + + + +installconfig.clusterName + +installconfig.clusterName + + + +installconfig.baseDomain->installconfig.clusterName + + + + + +installconfig.platform + +installconfig.platform + + + +installconfig.platform->installconfig.InstallConfig + + + + + +installconfig.platform->installconfig.baseDomain + + + + + +installconfig.platform->installconfig.clusterName + + + + + +installconfig.networking + +installconfig.networking + + + +installconfig.platform->installconfig.networking + + + + + +installconfig.clusterName->installconfig.InstallConfig + + + + + +installconfig.networking->installconfig.InstallConfig + + + + + +installconfig.pullSecret + +installconfig.pullSecret + + + +installconfig.pullSecret->installconfig.InstallConfig + + + + + +kubeconfig.AdminInternalClient->bootstrap.Bootstrap + + + + + +tls.AdminKubeConfigClientCertKey + +tls.AdminKubeConfigClientCertKey + + + +tls.AdminKubeConfigClientCertKey->kubeconfig.AdminInternalClient + + + + + +tls.AdminKubeConfigClientCertKey->kubeconfig.LoopbackClient + + + + + +tls.AdminKubeConfigClientCertKey->kubeconfig.AdminClient + + + + + +tls.AdminKubeConfigSignerCertKey + +tls.AdminKubeConfigSignerCertKey + + + +tls.AdminKubeConfigSignerCertKey->tls.AdminKubeConfigClientCertKey + + + + + +tls.AdminKubeConfigCABundle + +tls.AdminKubeConfigCABundle + + + +tls.AdminKubeConfigSignerCertKey->tls.AdminKubeConfigCABundle + + + + + +tls.KubeAPIServerCompleteCABundle + +tls.KubeAPIServerCompleteCABundle + + + +tls.KubeAPIServerCompleteCABundle->bootstrap.Bootstrap + + + + + +tls.KubeAPIServerCompleteCABundle->kubeconfig.AdminInternalClient + + + + + +tls.KubeAPIServerCompleteCABundle->kubeconfig.Kubelet + + + + + +tls.KubeAPIServerCompleteCABundle->kubeconfig.AdminClient + + + + + +tls.KubeAPIServerLocalhostCABundle + +tls.KubeAPIServerLocalhostCABundle + + + +tls.KubeAPIServerLocalhostCABundle->bootstrap.Bootstrap + + + + + +tls.KubeAPIServerLocalhostCABundle->tls.KubeAPIServerCompleteCABundle + + + + + +tls.KubeAPIServerLocalhostCABundle->kubeconfig.LoopbackClient + + + + + +tls.KubeAPIServerLocalhostSignerCertKey + +tls.KubeAPIServerLocalhostSignerCertKey + + + +tls.KubeAPIServerLocalhostSignerCertKey->bootstrap.Bootstrap + + + + + +tls.KubeAPIServerLocalhostSignerCertKey->tls.KubeAPIServerLocalhostCABundle + + + + + +tls.KubeAPIServerLocalhostServerCertKey + +tls.KubeAPIServerLocalhostServerCertKey + + + +tls.KubeAPIServerLocalhostSignerCertKey->tls.KubeAPIServerLocalhostServerCertKey + + + + + +tls.KubeControlPlaneCABundle + +tls.KubeControlPlaneCABundle + + + +tls.KubeAPIServerLocalhostSignerCertKey->tls.KubeControlPlaneCABundle + + + + + +tls.KubeAPIServerServiceNetworkCABundle + +tls.KubeAPIServerServiceNetworkCABundle + + + +tls.KubeAPIServerServiceNetworkCABundle->bootstrap.Bootstrap + + + + + +tls.KubeAPIServerServiceNetworkCABundle->tls.KubeAPIServerCompleteCABundle + + + + + +tls.KubeAPIServerServiceNetworkSignerCertKey + +tls.KubeAPIServerServiceNetworkSignerCertKey + + + +tls.KubeAPIServerServiceNetworkSignerCertKey->bootstrap.Bootstrap + + + + + +tls.KubeAPIServerServiceNetworkSignerCertKey->tls.KubeAPIServerServiceNetworkCABundle + + + + + +tls.KubeAPIServerServiceNetworkSignerCertKey->tls.KubeAPIServerServiceNetworkServerCertKey + + + + + +tls.KubeAPIServerServiceNetworkSignerCertKey->tls.KubeControlPlaneCABundle + + + + + +tls.KubeAPIServerLBCABundle + +tls.KubeAPIServerLBCABundle + + + +tls.KubeAPIServerLBCABundle->bootstrap.Bootstrap + + + + + +tls.KubeAPIServerLBCABundle->tls.KubeAPIServerCompleteCABundle + + + + + +tls.KubeAPIServerLBSignerCertKey + +tls.KubeAPIServerLBSignerCertKey + + + +tls.KubeAPIServerLBSignerCertKey->bootstrap.Bootstrap + + + + + +tls.KubeAPIServerLBSignerCertKey->tls.KubeAPIServerLBCABundle + + + + + +tls.KubeAPIServerLBSignerCertKey->tls.KubeAPIServerExternalLBServerCertKey + + + + + +tls.KubeAPIServerLBSignerCertKey->tls.KubeAPIServerInternalLBServerCertKey + + + + + +tls.KubeAPIServerLBSignerCertKey->tls.KubeControlPlaneCABundle + + + + + +kubeconfig.Kubelet->bootstrap.Bootstrap + + + + + +tls.KubeletClientCertKey + +tls.KubeletClientCertKey + + + +tls.KubeletClientCertKey->bootstrap.Bootstrap + + + + + +tls.KubeletClientCertKey->kubeconfig.Kubelet + + + + + +tls.KubeletBootstrapCertSigner + +tls.KubeletBootstrapCertSigner + + + +tls.KubeletBootstrapCertSigner->tls.KubeletClientCertKey + + + + + +tls.KubeletBootstrapCABundle + +tls.KubeletBootstrapCABundle + + + +tls.KubeletBootstrapCertSigner->tls.KubeletBootstrapCABundle + + + + + +kubeconfig.LoopbackClient->bootstrap.Bootstrap + + + + + +machine.MasterIgnitionCustomizations->bootstrap.Bootstrap + + + + + +tls.RootCA + +tls.RootCA + + + +tls.RootCA->bootstrap.Bootstrap + + + + + +tls.RootCA->machine.MasterIgnitionCustomizations + + + + + +tls.RootCA->machine.Master + + + + + +tls.RootCA->machine.WorkerIgnitionCustomizations + + + + + +tls.RootCA->machine.Worker + + + + + +tls.RootCA->manifests.Manifests + + + + + +tls.RootCA->tls.MCSCertKey + + + + + +tls.JournalCertKey + +tls.JournalCertKey + + + +tls.RootCA->tls.JournalCertKey + + + + + +machine.Master->machine.MasterIgnitionCustomizations + + + + + +machine.Master->machines.Master + + + + + +machine.WorkerIgnitionCustomizations->bootstrap.Bootstrap + + + + + +machine.Worker->machine.WorkerIgnitionCustomizations + + + + + +machine.Worker->machines.Worker + + + + + +machines.Master->bootstrap.Bootstrap + + + + + +installconfig.ClusterID->machines.Master + + + + + +installconfig.ClusterID->machines.Worker + + + + + +installconfig.ClusterID->manifests.Manifests + + + + + +installconfig.ClusterID->manifests.DNS + + + + + +installconfig.ClusterID->manifests.Infrastructure + + + + + +installconfig.ClusterID->manifests.CloudProviderConfig + + + + + +installconfig.ClusterID->manifests.Openshift + + + + + +installconfig.PlatformCredsCheck->machines.Master + + + + + +installconfig.PlatformCredsCheck->machines.Worker + + + + + +installconfig.PlatformCredsCheck->manifests.DNS + + + + + +installconfig.PlatformCredsCheck->manifests.CloudProviderConfig + + + + + +rhcos.Image->bootstrap.Bootstrap + + + + + +rhcos.Image->machines.Master + + + + + +rhcos.Image->machines.Worker + + + + + +rhcos.Image->manifests.Openshift + + + + + +machines.Worker->bootstrap.Bootstrap + + + + + +manifests.Manifests->bootstrap.Bootstrap + + + + + +bootstrap.CVOIgnore + +bootstrap.CVOIgnore + + + +manifests.Manifests->bootstrap.CVOIgnore + + + + + +manifests.Ingress->manifests.Manifests + + + + + +manifests.DNS->manifests.Manifests + + + + + +manifests.Infrastructure->manifests.Manifests + + + + + +manifests.CloudProviderConfig->manifests.Infrastructure + + + + + +manifests.AdditionalTrustBundleConfig->manifests.Infrastructure + + + + + +manifests.Networking->manifests.Manifests + + + + + +manifests.Networking->manifests.Proxy + + + + + +openshift.NetworkCRDs + +openshift.NetworkCRDs + + + +openshift.NetworkCRDs->manifests.Networking + + + + + +manifests.Proxy->bootstrap.Bootstrap + + + + + +manifests.Proxy->manifests.Manifests + + + + + +manifests.Scheduler->manifests.Manifests + + + + + +manifests.ImageContentSourcePolicy->manifests.Manifests + + + + + +tls.MCSCertKey->bootstrap.Bootstrap + + + + + +tls.MCSCertKey->manifests.Manifests + + + + + +bootkube.CVOOverrides + +bootkube.CVOOverrides + + + +bootkube.CVOOverrides->manifests.Manifests + + + + + +bootkube.KubeCloudConfig + +bootkube.KubeCloudConfig + + + +bootkube.KubeCloudConfig->manifests.Manifests + + + + + +bootkube.KubeSystemConfigmapRootCA + +bootkube.KubeSystemConfigmapRootCA + + + +bootkube.KubeSystemConfigmapRootCA->manifests.Manifests + + + + + +bootkube.MachineConfigServerTLSSecret + +bootkube.MachineConfigServerTLSSecret + + + +bootkube.MachineConfigServerTLSSecret->manifests.Manifests + + + + + +bootkube.OpenshiftConfigSecretPullSecret + +bootkube.OpenshiftConfigSecretPullSecret + + + +bootkube.OpenshiftConfigSecretPullSecret->manifests.Manifests + + + + + +manifests.Openshift->bootstrap.Bootstrap + + + + + +manifests.Openshift->bootstrap.CVOIgnore + + + + + +password.KubeadminPassword + +password.KubeadminPassword + + + +password.KubeadminPassword->manifests.Openshift + + + + + +openshiftinstall.Config + +openshiftinstall.Config + + + +openshiftinstall.Config->manifests.Openshift + + + + + +openshift.CloudCredsSecret + +openshift.CloudCredsSecret + + + +openshift.CloudCredsSecret->manifests.Openshift + + + + + +openshift.KubeadminPasswordSecret + +openshift.KubeadminPasswordSecret + + + +openshift.KubeadminPasswordSecret->manifests.Openshift + + + + + +openshift.RoleCloudCredsSecretReader + +openshift.RoleCloudCredsSecretReader + + + +openshift.RoleCloudCredsSecretReader->manifests.Openshift + + + + + +openshift.BaremetalConfig + +openshift.BaremetalConfig + + + +openshift.BaremetalConfig->manifests.Openshift + + + + + +openshift.AzureCloudProviderSecret + +openshift.AzureCloudProviderSecret + + + +openshift.AzureCloudProviderSecret->manifests.Openshift + + + + + +tls.AdminKubeConfigCABundle->bootstrap.Bootstrap + + + + + +tls.KubeAPIServerCompleteClientCABundle + +tls.KubeAPIServerCompleteClientCABundle + + + +tls.AdminKubeConfigCABundle->tls.KubeAPIServerCompleteClientCABundle + + + + + +tls.AggregatorCA + +tls.AggregatorCA + + + +tls.AggregatorCA->bootstrap.Bootstrap + + + + + +tls.APIServerProxyCertKey + +tls.APIServerProxyCertKey + + + +tls.AggregatorCA->tls.APIServerProxyCertKey + + + + + +tls.AggregatorCABundle + +tls.AggregatorCABundle + + + +tls.AggregatorCABundle->bootstrap.Bootstrap + + + + + +tls.AggregatorSignerCertKey + +tls.AggregatorSignerCertKey + + + +tls.AggregatorSignerCertKey->bootstrap.Bootstrap + + + + + +tls.AggregatorSignerCertKey->tls.AggregatorCABundle + + + + + +tls.AggregatorClientCertKey + +tls.AggregatorClientCertKey + + + +tls.AggregatorSignerCertKey->tls.AggregatorClientCertKey + + + + + +tls.AggregatorClientCertKey->bootstrap.Bootstrap + + + + + +tls.APIServerProxyCertKey->bootstrap.Bootstrap + + + + + +tls.BootstrapSSHKeyPair + +tls.BootstrapSSHKeyPair + + + +tls.BootstrapSSHKeyPair->bootstrap.Bootstrap + + + + + +tls.BoundSASigningKey + +tls.BoundSASigningKey + + + +tls.BoundSASigningKey->bootstrap.Bootstrap + + + + + +tls.CloudProviderCABundle->bootstrap.Bootstrap + + + + + +tls.JournalCertKey->bootstrap.Bootstrap + + + + + +tls.KubeAPIServerExternalLBServerCertKey->bootstrap.Bootstrap + + + + + +tls.KubeAPIServerInternalLBServerCertKey->bootstrap.Bootstrap + + + + + +tls.KubeAPIServerLocalhostServerCertKey->bootstrap.Bootstrap + + + + + +tls.KubeAPIServerServiceNetworkServerCertKey->bootstrap.Bootstrap + + + + + +tls.KubeAPIServerCompleteClientCABundle->bootstrap.Bootstrap + + + + + +tls.KubeletClientCABundle + +tls.KubeletClientCABundle + + + +tls.KubeletClientCABundle->bootstrap.Bootstrap + + + + + +tls.KubeletClientCABundle->tls.KubeAPIServerCompleteClientCABundle + + + + + +tls.KubeletCSRSignerCertKey + +tls.KubeletCSRSignerCertKey + + + +tls.KubeletCSRSignerCertKey->bootstrap.Bootstrap + + + + + +tls.KubeletCSRSignerCertKey->tls.KubeletClientCABundle + + + + + +tls.KubeletServingCABundle + +tls.KubeletServingCABundle + + + +tls.KubeletCSRSignerCertKey->tls.KubeletServingCABundle + + + + + +tls.KubeControlPlaneCABundle->bootstrap.Bootstrap + + + + + +tls.KubeControlPlaneCABundle->tls.KubeAPIServerCompleteClientCABundle + + + + + +tls.KubeControlPlaneSignerCertKey + +tls.KubeControlPlaneSignerCertKey + + + +tls.KubeControlPlaneSignerCertKey->bootstrap.Bootstrap + + + + + +tls.KubeControlPlaneSignerCertKey->tls.KubeControlPlaneCABundle + + + + + +tls.KubeControlPlaneKubeControllerManagerClientCertKey + +tls.KubeControlPlaneKubeControllerManagerClientCertKey + + + +tls.KubeControlPlaneSignerCertKey->tls.KubeControlPlaneKubeControllerManagerClientCertKey + + + + + +tls.KubeControlPlaneKubeSchedulerClientCertKey + +tls.KubeControlPlaneKubeSchedulerClientCertKey + + + +tls.KubeControlPlaneSignerCertKey->tls.KubeControlPlaneKubeSchedulerClientCertKey + + + + + +tls.KubeAPIServerToKubeletCABundle + +tls.KubeAPIServerToKubeletCABundle + + + +tls.KubeAPIServerToKubeletCABundle->bootstrap.Bootstrap + + + + + +tls.KubeAPIServerToKubeletCABundle->tls.KubeAPIServerCompleteClientCABundle + + + + + +tls.KubeAPIServerToKubeletSignerCertKey + +tls.KubeAPIServerToKubeletSignerCertKey + + + +tls.KubeAPIServerToKubeletSignerCertKey->bootstrap.Bootstrap + + + + + +tls.KubeAPIServerToKubeletSignerCertKey->tls.KubeAPIServerToKubeletCABundle + + + + + +tls.KubeAPIServerToKubeletClientCertKey + +tls.KubeAPIServerToKubeletClientCertKey + + + +tls.KubeAPIServerToKubeletSignerCertKey->tls.KubeAPIServerToKubeletClientCertKey + + + + + +tls.KubeletBootstrapCABundle->bootstrap.Bootstrap + + + + + +tls.KubeletBootstrapCABundle->tls.KubeAPIServerCompleteClientCABundle + + + + + +tls.KubeAPIServerToKubeletClientCertKey->bootstrap.Bootstrap + + + + + +tls.KubeControlPlaneKubeControllerManagerClientCertKey->bootstrap.Bootstrap + + + + + +tls.KubeControlPlaneKubeSchedulerClientCertKey->bootstrap.Bootstrap + + + + + +tls.KubeletServingCABundle->bootstrap.Bootstrap + + + + + +tls.ServiceAccountKeyPair + +tls.ServiceAccountKeyPair + + + +tls.ServiceAccountKeyPair->bootstrap.Bootstrap + + + + + +releaseimage.Image + +releaseimage.Image + + + +releaseimage.Image->bootstrap.Bootstrap + + + + + +bootstrap.CVOIgnore->bootstrap.Bootstrap + + + + +