From 250db4400660cd6a602bcabff5634ca56410c7c5 Mon Sep 17 00:00:00 2001 From: Chris Gunn Date: Thu, 5 Sep 2024 11:37:22 -0700 Subject: [PATCH] Image Customizer: Add doc for cloning an RPM repo. (#10330) Provide instructions for cloning and RPM repo (e.g. PMC) and then using the clone with the image customizer tool. This is primarily intended for those who want reproducible builds. --- toolkit/tools/imagecustomizer/docs/cli.md | 4 + .../imagecustomizer/docs/clone-rpm-repo.md | 177 ++++++++++++++++++ 2 files changed, 181 insertions(+) create mode 100644 toolkit/tools/imagecustomizer/docs/clone-rpm-repo.md diff --git a/toolkit/tools/imagecustomizer/docs/cli.md b/toolkit/tools/imagecustomizer/docs/cli.md index ac955762154..27fc8ff14df 100644 --- a/toolkit/tools/imagecustomizer/docs/cli.md +++ b/toolkit/tools/imagecustomizer/docs/cli.md @@ -97,6 +97,10 @@ given the lowest priority. See, [Building custom packages](building-packages.md) for a guide on how to build your own packages for Azure Linux. +See, [Cloning an RPM repo](clone-rpm-repo.md) for how to clone or download RPMs from a +existing RPM repo (such as packages.microsoft.com). Using a cloned repo with +`--rpm-source` can help your builds avoid dependencies on external resources. + ## --disable-base-image-rpm-repos Disable the base image's installed RPM repos as a source of RPMs during package diff --git a/toolkit/tools/imagecustomizer/docs/clone-rpm-repo.md b/toolkit/tools/imagecustomizer/docs/clone-rpm-repo.md new file mode 100644 index 00000000000..5e31c5e2302 --- /dev/null +++ b/toolkit/tools/imagecustomizer/docs/clone-rpm-repo.md @@ -0,0 +1,177 @@ +# Cloning an RPM repo + +By default, the image customizer tool uses the base image's inbuilt repo files for where +to source RPMs from. The Azure Linux default repos typically point to +packages.microsoft.com (PMC). + +PMC is regularly updated with bug fixes and feature updates for packages. So, if an +image customization config includes package install or updates, then a run on one day +may produce a different result than a run on another day since PMC might have been +updated in between runs. This behavior may be perfectly fine (or even desirable) for +some users. However, other users may require more stable builds that don't change based +on the state of an external resource (e.g. PMC). For such users, it can be useful to +make a clone of PMC. + +## Cloning a repo to a local directory + +1. Acquire the `dnf reposync` and `dnf download` commands. + + Azure Linux 2.0 and 3.0: + + ```bash + sudo tdnf -y install dnf-utils + ``` + + Ubuntu 24.04: + + ```bash + sudo apt update -y + sudo apt install -y dnf-plugins-core + ``` + +2. Select the repo URL: + + | Azure Linux Version | Arch | URL | + | ------------------- | ------ | ----------------------------------------------------------------- | + | 2.0 | x86_64 | https://packages.microsoft.com/cbl-mariner/2.0/prod/base/x86_64/ | + | 2.0 | ARM64 | https://packages.microsoft.com/cbl-mariner/2.0/prod/base/aarch64/ | + | 3.0 | x86_64 | https://packages.microsoft.com/azurelinux/3.0/prod/base/x86_64/ | + | 3.0 | ARM64 | https://packages.microsoft.com/azurelinux/3.0/prod/base/aarch64/ | + + For example: + + ```bash + REPO_URL="https://packages.microsoft.com/azurelinux/3.0/prod/base/x86_64/" + ``` + +3. Clone PMC. + + If you want to clone all of PMC, then run: + + ```bash + dnf reposync --repofrompath "azurelinux,$REPO_URL" --repo azurelinux --newest-only + ``` + + If you want to only clone a subset of packages (and their dependencies), then run: + + ```bash + PACKAGE_LIST="vim nano" + dnf download --repofrompath "azurelinux,$REPO_URL" --repo azurelinux --resolve --alldeps --destdir azurelinux $PACKAGE_LIST + ``` + + This will download the RPMs into a directory named `azurelinux`. + +4. Cache the downloaded RPMs somewhere. + +5. Use cached RPMs with the image customizer tool. + + ```bash + sudo ./imagecustomizer \ + --build-dir ./build \ + --image-file \ + --output-image-file ./out/image.vhdx \ + --output-image-format vhdx \ + --config-file \ + --disable-base-image-rpm-repos \ + --rpm-source + ``` + + where: + + - ``: The base image file. + - ``: The image customizer config file. + - ``: The local directory that contains the downloaded RPMs. + +## Hosting a cloned repo + +It may be desirable to host the downloaded RPMs in a common location so that it can be +used by both builds and developers. + +An RPM server is simply a HTTP server that hosts static files. There is no dynamic +content. So, pretty much any HTTP server application or provider can be used. The files +served by the HTTP server are the RPM files themselves and a few metadata files that +document what RPMs are available. + +Example RPM server using httpd/apache2: + +1. Install prerequisites: + + Azure Linux 2.0 and 3.0: + + ```bash + sudo tdnf -y install createrepo_c httpd + sudo systemctl enable --now httpd + ``` + + Ubuntu 22.04: + + ```bash + sudo apt update -y + sudo apt install -y createrepo-c apache2 + ```` + +2. Download the cached RPMs to a local directory. + +3. Create the metadata files: + + ```bash + createrepo_c --compatibility --update + ``` + + where: + + - ``: The directory you downloaded the RPMs to. + +4. Move the RPMs directory: + + ```bash + sudo mkdir -p /var/www + sudo mv -T /var/www/rpms + ``` + +5. Configure the HTTP server: + + Azure Linux 2.0 and 3.0: + + ```bash + sudo sed -i 's|"/etc/httpd/htdocs"|"/var/www/rpms"|' /etc/httpd/conf/httpd.conf + sudo systemctl reload httpd + ``` + + Ubuntu 22.04: + + ```bash + sudo sed -i 's|/var/www/html|/var/www/rpms|' /etc/apache2/sites-available/000-default.conf + sudo systemctl reload httpd + ``` + +6. Create a file called `rpms.repo` with the following contents: + + ```ini + [rpmshost] + name=rpmshost + baseurl=http:// + enabled=1 + ``` + + where: + + - ``: The IP address of the HTTP server hosting the RPM files. + +7. Use the `rpms.repo` file with the image customizer tool: + + ```bash + sudo ./imagecustomizer \ + --build-dir ./build \ + --image-file \ + --output-image-file ./out/image.vhdx \ + --output-image-format vhdx \ + --config-file \ + --disable-base-image-rpm-repos \ + --rpm-source rpms.repo + ``` + + where: + + - ``: The base image file. + - ``: The image customizer config file.