diff --git a/buildpack.md b/buildpack.md index 696c95e8..893be98a 100644 --- a/buildpack.md +++ b/buildpack.md @@ -38,6 +38,9 @@ The `ENTRYPOINT` of the OCI image contains logic implemented by the lifecycle th - [Phase #2: Analysis](#phase-2-analysis) - [Purpose](#purpose-1) - [Process](#process-1) + - [Phase #3: Image Extension](#phase-3-image-extension) + - [Purpose](#purpose) + - [Process](#process) - [Phase #3: Build](#phase-3-build) - [Purpose](#purpose-2) - [Process](#process-2) @@ -202,7 +205,7 @@ The lifecycle MUST treat a layer with unset `types` as a `launch = false`, `buil The following table illustrates the behavior depending on the value of each flag. Note that the lifecycle only restores layers from the cache, never from the previous image. -`build` | `cache` | `launch` | Metadata Restored | Layer Restored +`build` | `cache` | `launch` | Metadata Restored | Layer Restored ----------|----------|----------|--------------------------|--------------------- true | true | true | Yes - from the app image | Yes* - from the cache true | true | false | Yes - from the cache | Yes - from the cache @@ -406,6 +409,14 @@ Note that buildpack IDs are expanded depth-first in left-to-right order. If a buildpack order entry within a group has the parameter `optional = true`, then a copy of the group without the entry MUST be repeated after the original group. +#### Image Extensions + +Image extensions participate in the buildpack [detection](#detector) process, with the same `UID`, `GID`, and interface for `/bin/detect`. However: +- Detection is optional for extensions, and they are assumed to pass detection when `/bin/detect` is not present. A `/bin/detect` that exits with a 0 exit code passes detection, and fails otherwise. +- Extensions MUST only output `provides` entries to the build plan. They MUST NOT output `requires`. +- Extensions MUST all precede buildpacks in [order](#ordertoml-toml) definitions. +- Extensions MUST always be optional. + ## Phase #2: Analysis ![Analysis](img/analysis.svg) @@ -435,6 +446,60 @@ For each buildpack in the group, the lifecycle After analysis, the lifecycle MUST proceed to the build phase. + +## Phase #3: Image Extension + +### Purpose + +The purpose of the image extension phase is to apply dynamically-generated or static build-time and runtime modules that mutate the pre-build base image. + +During the image extension phase, an image extension might: + +1. Execute a pre-defined `Dockerfile` to install files in specific directories that require privileged access. +1. Dyanmically generate a `Dockerfile` based on the phase inputs and apply it. +1. Execute a pre-defined `Dockerfile` to install system packages provided as input to the phase. + +### Process + +**GIVEN:** +- The final ordered group of image extensions determined during the detection phase, +- A directory containing application source code, +- An `` directory used to store generated artifacts +- A shell, if needed, + +For each image extension in the group in order, the lifecycle MUST execute `/bin/build`. + +1. **If** the exit status of `/bin/build` is non-zero, \ + **Then** the lifecycle MUST fail the phase. + +2. **If** the exit status of `/bin/build` is zero, + 1. **If** there are additional image extensions in the group, \ + **Then** the lifecycle MUST proceed to the next image extension's `/bin/build`. + + 2. **If** there are no additional image extension in the group, \ + **Then** the lifecycle MUST proceed to the build phase. + +For each `/bin/build` executable in each image extension, the lifecycle: + +- MUST provide path arguments to `/bin/build` as described in the [Image Extension Interface](image-extension.md) section. +- MUST configure the build environment as described in the [Environment](#environment) section. +- MUST provide all `` entries that were required by any buildpack in the group during the detection phase with names matching the names that the buildpack provided. + +Correspondingly, each `/bin/build` executable: + +- MAY read from the `` directory. +- MUST NOT write to the `` directory. +- MAY read the build environment as described in the [Environment](#environment) section. +- MAY read the Buildpack Plan. +- MAY log output from the build process to `stdout`. +- MAY emit error, warning, or debug messages to `stderr`. +- SHOULD write SBOM (Software-Bill-of-Materials) files as described in the [Software-Bill-of-Materials](#software-bill-of-materials) section describing any contributions to the app image. +- MAY write key-value pairs to `/launch.toml` that are provided as build args to run.Dockerfile or Dockerfile +- SHOULD write build SBOM (Software-Bill-of-Materials) files as described in the [Software-Bill-of-Materials](#software-bill-of-materials) section describing any contributions to the build environment. +- MAY modify or delete any existing `` directories. +- MAY create new `` directories. +- MAY name any new `` directories without restrictions except those imposed by the filesystem and the ones noted below. + ## Phase #3: Build ![Build](img/build.svg) diff --git a/image-extension.md b/image-extension.md new file mode 100644 index 00000000..34bacba5 --- /dev/null +++ b/image-extension.md @@ -0,0 +1,201 @@ +# Image Extension Interface Specification + +This document specifies the interface of image extensions. + +Image extensions are dynamically-generated or static build-time and runtime modules that mutate the pre-build base image. + +## Table of Contents + + + + +## Image Extension API Version + +This document specifies Image Extension API version `0.1`. + +Image Extension API versions: + - MUST be in form `.` or ``, where `` is equivalent to `.0` + - When `` is greater than `0` increments to `` SHALL exclusively indicate additive changes + +## Image Extension Interface + +An image extension is used to apply a `Dockerfile` at the platform's discretion. + +The image extensions MAY contain a `/bin/build` and `/bin/detect` executable. +Each extension MUST have an `extension.toml` file in its root directory. + +Extensions participate in the buildpack [detection](#detector) process, with the same `UID`, `GID`, and interface for `/bin/detect`. However: +- Detection is optional for extensions, and they are assumed to pass detection when `/bin/detect` is not present. A `/bin/detect` that exits with a 0 exit code passes detection, and fails otherwise. +- Extensions MUST only output `provides` entries to the build plan. They MUST NOT output `requires`. +- Extensions MUST all precede buildpacks in [order](#ordertoml-toml) definitions. +- Extensions MUST always be optional. + +Extensions MUST generate `Dockerfile`s before the buildpack [build](#builder) phase. To generate Dockerfiles, the lifecycle MUST execute the extension's `/bin/build` executable with the same `UID`, `GID`, and interface as buildpacks. However: + +- Extension's `/bin/build` MUST NOT write to the app directory. +- Extension's `/bin/build` MAY be executed in parallel. +- The buildpack `` directory MUST be replaced by an `` directory. +- If an extension is missing `/bin/build`, the extension root MUST be treated as a pre-populated `` directory. + +After `/bin/build` executes, the `` directory MAY contain +- `build.toml`, with the same contents as a normal buildpack's `build.toml`, but + - With an additional `args` table array with `name` and `value` fields that are provided as build args to `build.Dockerfile` or `Dockerfile` +- `launch.toml`, with the same contents as a normal buildpack's `launch.toml`, but + - Without the `processes` table array + - Without the `slices` table array + - With an additional `args` table array with `name` and `value` fields that are provided as build args to `run.Dockerfile` or `Dockerfile` +- Either `Dockerfile` or either or both of `build.Dockerfile` and `run.Dockerfile`. The `build.Dockerfile`, `run.Dockerfile`, and `Dockerfile` target the builder image, runtime base image, or both base images, respectively. + +If no Dockerfiles are present, `/bin/build` may still consume build plan entries and add metadata to `build.toml`/`launch.toml`. + +`Dockerfile`s MUST be applied to their corresponding base images after all extensions are executed and before any regular buildpacks are executed. `Dockerfile`s MUST be applied in the order determined during buildpack detection. + +All `Dockerfile`s MUST be provided with `base_image` and `build_id` args. +The `base_image` arg allows the Dockerfile to reference the original base image. +The `build_id` arg allows the Dockerfile to invalidate the cache after a certain layer and must be defaulted to `0`. The executor of the Dockerfile will provide the `build_id` as a UUID (this eliminates the need to track this variable). +When the `$build_id` arg is referenced in a `RUN` instruction, all subsequent layers will be rebuilt on the next build (as the value will change). + +Build args specified in `build.toml` MUST be provided to `build.Dockerfile` or `Dockerfile` (when applied to the build-time base image). +Build args specified in `launch.toml` MUST be provided to `run.Dockerfile` or `Dockerfile` (when applied to the runtime base image). + +A runtime base image extension MAY indicate that it preserves ABI compatibility by adding the label `io.buildpacks.rebasable=true`. The `image-extender` MUST set `io.buildpacks.rebasable=true` on the final image only if `io.buildpacks.rebasable=true` on the base image and for all image extensions. + +### Image Extensions Directory Layout + +The image extensions directory MUST contain unarchived extensions such that: + +- Each top-level directory is an image extension ID. +- Each second-level directory is an image extension version. + +## Image Extension Examples + +### Example: App-specified Dockerfile Extension + +This example extension would allow an app to provide runtime and build-time base image extensions as "run.Dockerfile" and "build.Dockerfile." +The app developer can decide whether the extensions are rebasable. + +##### `/cnb/ext/com.example.appext/bin/build` +``` +#!/bin/sh +[ -f build.Dockerfile ] && cp build.Dockerfile "$1/" +[ -f run.Dockerfile ] && cp run.Dockerfile "$1/" +``` + +### Example: RPM Dockerfile Extension (app-based) + +This example extension would allow a builder to install RPMs for each language runtime, based on the app directory. + +Note: The Dockerfiles referenced must disable rebasing, and build times will be slower compared to buildpack-provided runtimes. + +##### `/cnb/ext/com.example.rpmext/bin/build` +``` +#!/bin/sh +[ -f Gemfile.lock ] && cp "$CNB_BUILDPACK_DIR/Dockerfile-ruby" "$1/Dockerfile" +[ -f package.json ] && cp "$CNB_BUILDPACK_DIR/Dockerfile-node" "$1/Dockerfile" +``` + + +### Dockerfiles for Base Images + +The same Dockerfile format may be used to create new base images or modify existing base images outside of the app build process (e.g., before creating a builder). Any specified labels override existing values. + +Dockerfiles that are used to create a base image must create a `/cnb/image/genpkgs` executable that outputs a [CycloneDX](https://cyclonedx.org)-formatted list of packages in the image with PURL IDs when invoked. This executable is executed after all Dockerfiles are applied, and the output replaces the label `io.buildpacks.sbom`. This label doubles as a Software Bill-of-Materials for the base image. In the future, this label will serve as a starting point for the application SBoM. + +### Example Dockerfiles + +Dockerfile used to create a runtime base image: + +``` +ARG base_image +FROM ${base_image} +ARG build_id=0 + +LABEL io.buildpacks.image.distro=ubuntu +LABEL io.buildpacks.image.version=18.04 +LABEL io.buildpacks.rebasable=true + +ENV CNB_USER_ID=1234 +ENV CNB_GROUP_ID=1235 + +RUN groupadd cnb --gid ${CNB_GROUP_ID} && \ + useradd --uid ${CNB_USER_ID} --gid ${CNB_GROUP_ID} -m -s /bin/bash cnb + +USER ${CNB_USER_ID}:${CNB_GROUP_ID} + +COPY genpkgs /cnb/image/genpkgs +``` + +`run.Dockerfile` for use with the example `app.run.Dockerfile.out` extension that always installs the latest version of curl: +``` +ARG base_image +FROM ${base_image} +ARG build_id=0 + +RUN echo ${build_id} + +RUN apt-get update && apt-get install -y curl && rm -rf /var/lib/apt/lists/* +``` +(note: this Dockerfile disables rebasing, as OS package installation is not rebasable) + +`run.Dockerfile` for use with the example `app.run.Dockerfile.out` extension that installs a special package to /opt: +``` +ARG base_image +FROM ${base_image} +ARG build_id=0 + +LABEL io.buildpacks.rebasable=true + +RUN curl -L https://example.com/mypkg-install | sh # installs to /opt +``` +(note: rebasing is explicitly allowed because only a single directory in /opt is created) + + +## Data Format + +### Files + +#### `extension.toml` (TOML) + +```toml +api = "" + +[extension] +id = "" +name = "" +version = "" +homepage = "" +description = "" +keywords = [ "" ] + +[[extension.licenses]] +type = "" +uri = "" +``` + +Image extension authors MUST choose a globally unique ID, for example: "io.buildpacks.apt". + +**The image extension ID:** + +*Key: `id = ""`* +- MUST only contain numbers, letters, and the characters `.`, `/`, and `-`. +- MUST NOT be `config` or `app`. +- MUST NOT be identical to any other buildpack ID when using a case-insensitive comparison. + +The image extension version: +- MUST be in the form `..` where `X`, `Y`, and `Z` are non-negative integers and must not contain leading zeros. + - Each element MUST increase numerically. + - Buildpack authors will define what changes will increment `X`, `Y`, and `Z`. + +**The image extension API:** + +*Key: `api = ""`* + - MUST be in form `.` or ``, where `` is equivalent to `.0` + - MUST describe the implemented platform API. + - SHOULD indicate the lowest compatible `` if extension behavior is consistent with multiple `` versions of a given `` + +**The image extension licenses:** + +The `[[extension.licenses]]` table is optional and MAY contain a list of image extension licenses where: + +- `type` - This MAY use the SPDX 2.1 license expression, but is not limited to identifiers in the SPDX Licenses List. +- `uri` - If this buildpack is using a nonstandard license, then this key MAY be specified in lieu of or in addition to `type` to point to the license. diff --git a/platform.md b/platform.md index a0a9b494..9f26a7d5 100644 --- a/platform.md +++ b/platform.md @@ -122,6 +122,8 @@ A **launcher layer** refers to a layer in the app OCI image containing the **lau The **launcher** refers to a lifecycle executable packaged in the **app image** for the purpose of executing processes at runtime. +An **image extension** is a dynamically-generated build-time and/or runtime Dockerfile that act as a pre-build base image extension. Extensions participate in detection and execute before the buildpack build process. + #### Additional Terminology An **image reference** refers to either a **tag reference** or **digest reference**. @@ -349,6 +351,7 @@ Usage: /cnb/lifecycle/detector \ [-app ] \ [-buildpacks ] \ + [-exts ] \ [-group ] \ [-layers ] \ [-log-level ] \ @@ -362,6 +365,7 @@ Usage: |---------------|-------------------------|--------------------------------------------------------|------- | `` | `CNB_APP_DIR` | `/workspace` | Path to application directory | `` | `CNB_BUILDPACKS_DIR` | `/cnb/buildpacks` | Path to buildpacks directory (see [Buildpacks Directory Layout](#buildpacks-directory-layout)) +| `` | `CNB_IMAGE_EXTS_DIR` | `/cnb/exts` | Path to image extensions directory (see [Image Extensions | `` | `CNB_GROUP_PATH` | `/group.toml` | Path to output group definition | `` | `CNB_LAYERS_DIR` | `/layers` | Path to layers directory | `` | `CNB_LOG_LEVEL` | `info` | Log Level @@ -445,6 +449,47 @@ Usage: ##### Layer Restoration lifeycle MUST use the provided `cache-dir` or `cache-image` to retrieve cache contents. The [rules](https://github.com/buildpacks/spec/blob/main/buildpack.md#layer-types) for restoration MUST be followed when determining how and when to store cache layers. +#### `image-extender` + +The platform MAY execute `image-extender`. + +Usage: +``` +/cnb/lifecycle/image-extender \ + [-group ] \ + [-exts ] \ + [-output ] \ + [-log-level ] \ + [-plan ] +``` + +##### Inputs +| Input | Env | Default Value | Description +|----------------|-----------------------|-----------------------|---------------------- +| `` | `CNB_IMAGE_EXTS_DIR` | `/cnb/exts` | Path to image extensions directory (see [Image Extensions Directory Layout](image-extension.md)) +| `` | `CNB_GROUP_PATH` | `/group.toml` | Path to group definition (see [`group.toml`](#grouptoml-toml)) +| `` | `CNB_OUTPUT_DIR` | `/output` | Path to output directory +| `` | `CNB_LOG_LEVEL` | `info` | Log Level +| `` | `CNB_PLAN_PATH` | `/plan.toml` | Path to resolved build plan (see [`plan.toml`](#plantoml-toml)) + +##### Outputs +| Output | Description +|--------------------------------------------|---------------------------------------------- +| [exit status] | (see Exit Code table below for values) +| `/dev/stdout` | Logs (info) +| `/dev/stderr` | Logs (warnings, errors) + +| Exit Code | Result| +|-----------|-------| +| `0` | Success +| `11` | Platform API incompatibility error +| `12` | Buildpack API incompatibility error +| `1-10`, `13-19` | Generic lifecycle errors +| `91` | Error applying image extension +| `90`, `92-99`| Extension-specific lifecycle errors + +- The lifecycle SHALL execute all image extensions in the order defined in `` according to the process outlined in the [Image Extensions Interface Specification](image-extension.md). + #### `builder` The platform MUST execute `builder` in the **build environment** @@ -912,6 +957,11 @@ version = "" api = "" optional = false +[[image-extensions]] +id = "" +version = "" +api = "" + [[processes]] type = "" command = ""