This page covers architecture of the StackHPC release train. +It assumes familiarity with the overview.
+See the original release train design document for design & requirements.
+The following diagram shows the major components of the release train.
+
This diagram was created using Google Drawings.
+Pulp is a content server that manages repositories of software packages and facilitates their distribution to content consumers. +The core functionality of Pulp is to provide versioned collections of software packages for onward distribution to consumers. +Pulp's ability to host multiple snapshots of a repository make it a good choice for hosting the release train content.
+There are three types of Pulp service in the release train architecture.
+Ark is a public-facing production Pulp service hosted on a public cloud provider, leaf.cloud. + It is the master copy of development and released content, and makes the content available to clients.
+Clients access Ark via a Pulp service deployed on their local infrastructure. + Content is synced from Ark to the local Pulp service, and control plane hosts acquire the content from there.
+Test is an internal service running on the SMS lab cloud.
+ Content is synced from Ark to test, providing a local content mirror for testing in SMS lab.
+ In some respects, the test Pulp service may be considered a client.
Various different types of content are hosted by Pulp, including:
+Some of this content may be mirrored from upstream sources, while others are the result of release train build processes.
+Access to released Pulp content is restricted to clients with a support agreement. +Build and test processes also need access to content.
+Access to package repositories is controlled via Pulp RBAC content guards.
+Two content guards are in use - development and release.
+The development content guard is assigned to unreleased content, while the release content guard is assigned to released content.
+Clients are provided with a username and password which they use when syncing package repositories in their local Pulp service with Ark.
+Clients' credentials are authorised to access content protected by the release content guard.
+Build and test processes are provided with a user account that is authorised to access the development and release content guard.
Access to container images is controlled by token authentication, which uses Django users in the backend.
+Two container namespaces are in use - stackhpc-dev and stackhpc.
+The stackhpc-dev namespace is used for unreleased content, while the stackhpc namespace is used for released content.
+Clients are provided with a set of credentials, which they use when syncing container image repositories in their local Pulp service with Ark.
+Clients' credentials are authorised to pull from the stackhpc namespace.
+Build and test processes are provided with credentials that are authorised to push to the stackhpc-dev namespace.
At the top of the diagram above are the upstream sources. +Some of these may be mirrored/synced into Ark, including:
+The Sync package repositories GitHub Actions workflow runs nightly and on demand, ensuring that we have regular versioned snapshots of these repositories. +Synced content is immediately published and distributed, such that it is available to build & test processes. +After a successful sync in Ark, the content is synced to the test Pulp service.
+Mirrored content typically uses a policy of immediate, meaning that all content is downloaded from the upstream source during the sync.
+This avoids issues seen with the on_demand policy where content that is removed from the upstream source becomes inaccessible if it has not been previously requested by a client.
+For RPM content, we also use a sync_policy of mirror_complete, which removes content from the snapshots in line with upstream repositories (in contrast with the default additive sync_policy, which does not).
+There are a couple of repositories for which mirror_complete does not work, so we use mirror_content_only instead.
Package repositories are versioned based on the date/time stamp at the beginning of the sync workflow, e.g. 20211122T102435.
+This version string is used as the final component of the path at which the corresponding distribution is hosted.
+For example, a Rocky Linux 9 BaseOS snapshot may be hosted at https://ark.stackhpc.com/pulp/content/rocky/9/BaseOS/x86_64/os/20240105T044843/.
The rationale behind using a date/time stamp is that there is no sane way to version a large collection of content, such as a repository, in a way in which the version reflects changes in the content (e.g. SemVer). +While the timestamp used is fairly arbitrary, it does at least provide a reasonable guarantee of ordering, and is easily automated.
+Build processes may take as input the synced repository mirrors and other external sources, including:
+The outputs of these build processes are pushed to Ark.
+Build and test processes run on SMS cloud, to avoid excessive running costs. +All content in Ark that is required by the build and test processes is synced to the test Pulp service running in SMS cloud, minimising data egress from Ark.
+Kolla container images are built via Kayobe, using a ci-builder environment in StackHPC Kayobe config.
+The configuration uses the package repositories in Ark when building containers.
+Images are built using a manually triggered GitHub Actions workflow.
+The stackhpc-dev namespace in Ark contains container push repositories, which are pushed to using Kayobe.
The Sync container repositories GitHub Actions workflow runs demand, syncing container repositories in test Pulp service with those in Ark. +It also configures container image distributions to be private, since they are public by default.
+Kolla container images are versioned based on the OpenStack release name, OS distribution and the date/time stamp at the beginning of the build workflow, e.g. 2024.1-rocky-9-20240922T102435.
+This version string is used as the image tag.
+Unlike package repositories, container image tags allow multiple versions to be present in a distribution of a container repository simultaneously.
+We therefore use separate namespaces for development (stackhpc-dev) and release (stackhpc).
Overcloud host images are built via Kayobe, using the same ci-builder environment used to build Kolla container images.
+Overcloud images are built using a manually triggered GitHub Actions workflow.
+They are pushed to a Pulp file repository in Ark, and uploaded to the SMS lab Glance image service for all-in-one and multi-node testing.
Overcloud images are versioned based on the OpenStack release name, and the date/time stamp at the beginning of the build workflow, e.g. 2024.1-20240922T102435.
+This version string is included in the Pulp distribution base_path of the image, e.g. https://ark.stackhpc.com/pulp/content/kayobe-images/2024.1/rocky/9/2024.1-20240922T102435/overcloud-rocky-9.qcow2
Release train content is tested via a Kayobe deployment of OpenStack.
+A ci-aio environment in StackHPC Kayobe config provides a converged control/compute host for testing.
+Various all-in-one OpenStack test configurations are run against pull requests opened against the StackHPC Kayobe config repository.
Whether content is mirrored from an upstream source or built locally, it is not immediately released. +Promotion describes the process whereby release candidate content is made into a release that is available to clients.
+For package repositories and overcloud host images, promotion does not affect how content is accessed, only who may access it.
+Promotion involves changing the content guard for the distribution to be released from development to release.
+This makes the content accessible to clients using their client credentials.
The stackhpc container namespace contains regular container repositories, which cannot be pushed to via docker push.
+Instead, we use the Pulp API to sync specific tags from stackhpc-dev to stackhpc.
Clients access Ark via a Pulp service deployed on their local infrastructure. +Typically the Pulp service is deployed as a Pulp in one container running on the seed host. +Content is synced from Ark to the local Pulp service, and control plane hosts acquire the content from there. +This avoids excessive Internet bandwidth usage, both for the client and Ark.
+Content in the client Pulp service is synced using the on_demand policy.
+This avoids unnecessarily large storage requirements on the seed, and speeds up syncing.
+There should be no risk of content becoming permanently unavailable, so long as Ark continues to host sufficiently old versions.
+This approach does have a downside of requiring Ark to be available to provide any content which has not previously been downloaded.
In order to consume the release train, clients should migrate to StackHPC Kayobe config. +This repository provides configuration and playbooks to:
+This configuration is in active development and is expected to evolve over the coming releases.
+Further documentation of this configuration is out of scope here, but is available in the readme.
+The intention is to have as much as possible of the release train automated and run via CI. +Typically, workflows may go through the following stages as they evolve:
+This sequence discourages putting too much automation into the GitHub Actions workflows, ensuring it is possible to run them manually.
+The release train Ansible playbooks make heavy use of the stackhpc.pulp collection, which in turn uses modules from the pulp.squeezer collection.
+To maintain the various source code repositories used within the Release Train there exists a number of automated processes. A mixture of Github workflows and Ansible playbooks are used to achieve this
+CODEOWNERS which can be used to automatically assign the relevant individuals to a newly opened pull request.\n {translation(\"search.result.term.missing\")}: {...missing}\n
\n }\n \n