WIP - prefetching-dependencies: Add new tutorials section

Signed-off-by: Michal Šoltis <[email protected]>

docs/modules/ROOT/pages/how-tos/configuring/prefetching-dependencies.adoc

@@ -448,6 +448,163 @@ spec:
 ----
 <1> The `*prefetch-input*` parameter specifies the path to the directory that has the lockfile. In this example, the `.` indicates that the lockfile is in the repository root. Additionally, if you have multiple directories, you can provide the path to those directories in the JSON array format. For example, `[{"path": ".", "type": "generic"}, {"path": "subpath/to/the/other/directory", "type": "generic"}]`. Alternatively, if your lockfile is generated as part of your pipeline and is not commited to the repository, you can specify an absolute path to it, like this: `{"type": "generic", "path": ".", "lockfile": "/absolute/path/to/artifacts.lock.yaml"}`.
 
+== Tutorials
+
+The following tutorials provide detailed instructions on prefetching dependencies for specific package managers.
+
+=== Configuring `pip` to build from source (how to generate requirements-build.txt)
+
+This tutorial explains how to configure `pip` to build from source and generate the `requirements-build.txt` file. The `requirements-build.txt` file contains all the build system requirements for your project.
+
+. Install link:https://pypi.org/project/pip-tools[pip-tools] package.
+
+. Compile a fully resolved `requirements.txt` that contains all the transitive dependencies and pins them to a specific version and hash either from `requirements.in`, `pyproject.toml`, `setup.cfg`, or `setup.py` specs.
+
++
+[source,command]
+----
+$ pip-compile --generate-hashes --output-file=requirements.txt ...
+----
++
+[NOTE]
+To successfully run the previous command, your environment must be as close as possible to the environment in the container build. That is, the environment should have the same operating system and the same Python _$major.$minor_ version.
+
+. Create a `requirements-build.in` file with the build system requirements for example from the project metadata file.
+
+. Download the link:https://raw.githubusercontent.com/containerbuildsystem/cachito/master/bin/pip_find_builddeps.py[pip_find_builddeps.py] script directly from GitHub.
+
++
+NOTE: This script has no runtime dependency other than `pip`.
+
+. Run the `pip_find_builddeps.py` script to generate the `requirements-build.in` file by using the following command:
+
++
+[source,command]
+----
+$ pip_find_builddeps.py requirements.txt --append --only-write-on-update -o requirements-build.in
+----
+
+. Use again the `pip-compile` command to convert the `requirements-build.in` file into the `requirements-build.txt` file by using the following command:
+
++
+[source,command]
+----
+$ pip-compile --generate-hashes --output-file=requirements-build.txt requirements-build.in
+----
+
++
+NOTE: `pip` automatically installs the build dependencies when needed for explicit installation. The purpose of the `requirement-build.txt` file is to enable Cachi2 to fetch the build dependencies and provide them to `pip` for offline installation in a network-isolated environment.
+
+=== Prefetching `pip` dependencies from custom index servers
+
+Cachi2 supports the link:https://pip.pypa.io/en/stable/cli/pip_install/#install-index-url[--index-url] option. You can add this option to your `requirements.txt` file(s), instructing Cachi2 to download packages from the specified index server. For example:
+
+[source,text]
+----
+--index-url=https://example.pypi.org/simple/
+requests==2.32.2 \
+    --hash=sha256:dd951ff5ecf3e3b3aa26b40703ba77495dab41da839ae72ef3c8e5d8e2433289 \
+    --hash=sha256:fc06670dd0ed212426dfeb94fc1b983d917c4f9847c863f313c9dfaaffb7c23c
+
+# ...other packages
+----
+
+WARNING: Do not include credentials in the index url. If needed, provide authentication through a `.netrc` file (as described below).
+
+To provide `.netrc` authentication for the index server:
+
+. Create a key/value secret (see xref:/how-tos/configuring/creating-secrets.adoc[creating secrets for your builds]).
+  Set `.netrc` as the key. Upload the content of your `.netrc` file as the value. For more details on netrc files,
+  review the link:https://pip.pypa.io/en/stable/topics/authentication/#netrc-support[pip documentation for netrc support].
+
+. In your `.tekton/` PipelineRun files, add the newly created secret to the `.spec.workspaces` section:
+
++
+[source,yaml]
+----
+spec:
+  ...
+  workspaces:
+    ...
+    - name: netrc
+      secret:
+        secretName: my-netrc  # the name you assigned to the secret in step 1
+----
+
+. In the `.spec.pipelineSpec.tasks` section, find the entry with `name: prefetch-dependencies`.
+  Add the `netrc` workspace to the list of workspaces (if not present):
+
++
+[source,yaml]
+----
+tasks:
+  ...
+  - name: prefetch-dependencies
+    ...
+    workspaces:
+      ...
+      - name: netrc
+        workspace: netrc
+----
+
+=== Generating `artifacts.lock.yaml` file for `generic` artifacts prefetching
+
+If you need to prefetch arbitrary files for your build, Cachi2 supports `generic fetcher` for that purpose. It uses a custom file named `artifacts.lock.yaml` to achieve this. This file needs to be either committed in the source repository or explicitly specified as an absolute path. The latter is useful if you need the lockfile to be dynamic and committing it to the repository would be problematic. To
+
+NOTE: For more information about the supported types of artifacts and the file format, see link:https://github.com/containerbuildsystem/cachi2/blob/main/docs/generic.md[Cachi2 documentation].
+
+Example of the `artifacts.lock.yaml` file with a list of files to prefetch, their checksums, and optionally their filenames:
+
+[source,yaml]
+----
+---
+metadata: <1>
+  version: "1.0"
+artifacts:
+ - download_url: "https://github.com/jeremylong/DependencyCheck/releases/download/v11.1.0/dependency-check-11.1.0-release.zip"
+    checksum: "sha256:c5b5b9e592682b700e17c28f489fe50644ef54370edeb2c53d18b70824de1e22" <2>
+    filename: "dependency-check.zip" <3>
+----
+<1> `metadata` section is required and needs to specify lockfile version
+<2> `checksum` is expected to be specified as `algorith:hash`
+<3> If no `filename` is specified, it will be derived from the URL.
+
+=== Generating `rpms.lock.yml` file for `rpm` artifacts prefetching
+
+Cachi2 has a dev-preview package manager capable of fetching `rpm` dependencies. This requires the use of a pair of `rpms.in.yaml` and `rpms.lock.yaml` files to be committed to your repository. You write a `rpms.in.yaml` file and the link:https://github.com/konflux-ci/rpm-lockfile-prototype?tab=readme-ov-file#what-is-this[rpm-lockfile-prototype] CLI tool resolves it to produce a `rpms.lock.yaml` file. It's new technology and the link:https://github.com/rpm-software-management/dnf5/issues/833[conversation] about which way forward in the dnf community is still ongoing.
+
+Example of the `rpms.in.yaml` file:
+
+[source,yaml]
+----
+packages: [nethack] <1>
+contentOrigin:
+  repofiles: ["./fedora.repo"] <2>
+----
+<1> The `*packages*` list is the list of packages you want to install in your Container. You don't have to declare transitive dependencies here. The rpm-lockfile-prototype tool will resolve them for you.
+<2> This should be a reference to a repo file, like those found in `/etc/yum.repos.d/`. This tells the tooling where to find your rpm and its dependencies.
+
+Copy any necessary yum/dnf repo files into your git repository. If you are using a fedora rawhide base image, that looks like:
+
+[source,console]
+----
+$ BASE_IMAGE=quay.io/fedora/fedora:rawhide
+$ podman run -it $BASE_IMAGE cat /etc/yum.repos.d/fedora.repo > fedora.repo
+----
+
+NOTE: For every repository defined in your set of repo files, make sure to add the corresponding sources repo (or make sure to enable them, if they are already present). Otherwise, the lockfile generator will not include any SRPMs in your lockfile, Cachi2 will not download any SRPMs and the source container for your build will be incomplete.
+
+Run the following command to resolve your `rpms.in.yaml` file and produce a `rpms.lock.yaml` file.
+
+[source,console]
+----
+$ BASE_IMAGE=quay.io/fedora/fedora:rawhide
+$ rpm-lockfile-prototype --image $BASE_IMAGE rpms.in.yaml <1>
+----
+<1> The produced `rpms.lock.yaml` file will include only your requested dependency plus its transitive dependencies, minus any rpms that are already installed in the provided base image.
+
+NOTE: Konflux also supports prefetching RPM content which requires a Red Hat subscription. For more information see xref:./activation-keys-subscription.adoc#hermetic-network-isolated-builds[Using Red Hat activation keys to access subscription content].
+
 == Procedure
 To prefetch dependencies for a component build, complete the following steps: