Added spec changes for Image Extension

buildpack.md

@@ -38,6 +38,9 @@ The `ENTRYPOINT` of the OCI image contains logic implemented by the lifecycle th
   - [Phase #2: Analysis](#phase-2-analysis)
     - [Purpose](#purpose-1)
     - [Process](#process-1)
+  - [Phase #3: Image Extension](#phase-3-image-extension)
+    - [Purpose](#purpose)
+    - [Process](#process)
   - [Phase #3: Build](#phase-3-build)
     - [Purpose](#purpose-2)
     - [Process](#process-2)
@@ -202,7 +205,7 @@ The lifecycle MUST treat a layer with unset `types` as a `launch = false`, `buil
 The following table illustrates the behavior depending on the value of each flag.
 Note that the lifecycle only restores layers from the cache, never from the previous image.
 
-`build`   | `cache`  | `launch` | Metadata Restored        | Layer Restored      
+`build`   | `cache`  | `launch` | Metadata Restored        | Layer Restored
 ----------|----------|----------|--------------------------|---------------------
 true      | true     | true     | Yes - from the app image | Yes* - from the cache
 true      | true     | false    | Yes - from the cache     | Yes - from the cache
@@ -406,6 +409,14 @@ Note that buildpack IDs are expanded depth-first in left-to-right order.
 
 If a buildpack order entry within a group has the parameter `optional = true`, then a copy of the group without the entry MUST be repeated after the original group.
 
+#### Image Extensions
+
+Image extensions participate in the buildpack [detection](#detector) process, with the same `UID`, `GID`, and interface for `/bin/detect`. However:
+- Detection is optional for extensions, and they are assumed to pass detection when `/bin/detect` is not present. A `/bin/detect` that exits with a 0 exit code passes detection, and fails otherwise.
+- Extensions MUST only output `provides` entries to the build plan. They MUST NOT output `requires`.
+- Extensions MUST all precede buildpacks in [order](#ordertoml-toml) definitions.
+- Extensions MUST always be optional.
+
 ## Phase #2: Analysis
 
 ![Analysis](img/analysis.svg)
@@ -435,6 +446,60 @@ For each buildpack in the group, the lifecycle
 
 After analysis, the lifecycle MUST proceed to the build phase.
 
+
+## Phase #3: Image Extension
+
+### Purpose
+
+The purpose of the image extension phase is to apply dynamically-generated or static build-time and runtime modules that mutate the pre-build base image.
+
+During the image extension phase, an image extension might:
+
+1. Execute a pre-defined `Dockerfile` to install files in specific directories that require privileged access.
+1. Dyanmically generate a `Dockerfile` based on the phase inputs and apply it.
+1. Execute a pre-defined `Dockerfile` to install system packages provided as input to the phase.
+
+### Process
+
+**GIVEN:**
+- The final ordered group of image extensions determined during the detection phase,
+- A directory containing application source code,
+- An `<output>` directory used to store generated artifacts
+- A shell, if needed,
+
+For each image extension in the group in order, the lifecycle MUST execute `/bin/build`.
+
+1. **If** the exit status of `/bin/build` is non-zero, \
+   **Then** the lifecycle MUST fail the phase.
+
+2. **If** the exit status of `/bin/build` is zero,
+   1. **If** there are additional image extensions in the group, \
+      **Then** the lifecycle MUST proceed to the next image extension's `/bin/build`.
+
+   2. **If** there are no additional image extension in the group, \
+      **Then** the lifecycle MUST proceed to the build phase.
+
+For each `/bin/build` executable in each image extension, the lifecycle:
+
+- MUST provide path arguments to `/bin/build` as described in the [Image Extension Interface](image-extension.md) section.
+- MUST configure the build environment as described in the [Environment](#environment) section.
+- MUST provide all `<plan>` entries that were required by any buildpack in the group during the detection phase with names matching the names that the buildpack provided.
+
+Correspondingly, each `/bin/build` executable:
+
+- MAY read from the `<app>` directory.
+- MUST NOT write to the `<app>` directory.
+- MAY read the build environment as described in the [Environment](#environment) section.
+- MAY read the Buildpack Plan.
+- MAY log output from the build process to `stdout`.
+- MAY emit error, warning, or debug messages to `stderr`.
+- SHOULD write SBOM (Software-Bill-of-Materials) files as described in the [Software-Bill-of-Materials](#software-bill-of-materials) section describing any contributions to the app image.
+- MAY write key-value pairs to `<output>/launch.toml` that are provided as build args to run.Dockerfile or Dockerfile
+- SHOULD write build SBOM (Software-Bill-of-Materials) files as described in the [Software-Bill-of-Materials](#software-bill-of-materials) section describing any contributions to the build environment.
+- MAY modify or delete any existing `<output>` directories.
+- MAY create new `<output>` directories.
+- MAY name any new `<output>` directories without restrictions except those imposed by the filesystem and the ones noted below.
+
 ## Phase #3: Build
 
 ![Build](img/build.svg)

image-extension.md

@@ -0,0 +1,201 @@
+# Image Extension Interface Specification
+
+This document specifies the interface of image extensions.
+
+Image extensions are dynamically-generated or static build-time and runtime modules that mutate the pre-build base image.
+
+## Table of Contents
+
+<!-- Using https://github.com/yzhang-gh/vscode-markdown to manage toc -->
+
+
+## Image Extension API Version
+
+This document specifies Image Extension API version `0.1`.
+
+Image Extension API versions:
+ - MUST be in form `<major>.<minor>` or `<major>`, where `<major>` is equivalent to `<major>.0`
+ - When `<major>` is greater than `0` increments to `<minor>` SHALL exclusively indicate additive changes
+
+## Image Extension Interface
+
+An image extension is used to apply a `Dockerfile` at the platform's discretion.
+
+The image extensions MAY contain a `/bin/build` and `/bin/detect` executable.
+Each extension MUST have an `extension.toml` file in its root directory.
+
+Extensions participate in the buildpack [detection](#detector) process, with the same `UID`, `GID`, and interface for `/bin/detect`. However:
+- Detection is optional for extensions, and they are assumed to pass detection when `/bin/detect` is not present. A `/bin/detect` that exits with a 0 exit code passes detection, and fails otherwise.
+- Extensions MUST only output `provides` entries to the build plan. They MUST NOT output `requires`.
+- Extensions MUST all precede buildpacks in [order](#ordertoml-toml) definitions.
+- Extensions MUST always be optional.
+
+Extensions MUST generate `Dockerfile`s before the buildpack [build](#builder) phase. To generate Dockerfiles, the lifecycle MUST execute the extension's `/bin/build` executable with the same `UID`, `GID`, and interface as buildpacks. However:
+
+- Extension's `/bin/build` MUST NOT write to the app directory.
+- Extension's `/bin/build` MAY be executed in parallel.
+- The buildpack `<layers>` directory MUST be replaced by an `<output>` directory.
+- If an extension is missing `/bin/build`, the extension root MUST be treated as a pre-populated `<output>` directory.
+
+After `/bin/build` executes, the `<output>` directory MAY contain
+- `build.toml`, with the same contents as a normal buildpack's `build.toml`, but
+  - With an additional `args` table array with `name` and `value` fields that are provided as build args to `build.Dockerfile` or `Dockerfile`
+- `launch.toml`, with the same contents as a normal buildpack's `launch.toml`, but
+  - Without the `processes` table array
+  - Without the `slices` table array
+  - With an additional `args` table array with `name` and `value` fields that are provided as build args to `run.Dockerfile` or `Dockerfile`
+- Either `Dockerfile` or either or both of `build.Dockerfile` and `run.Dockerfile`. The `build.Dockerfile`, `run.Dockerfile`, and `Dockerfile` target the builder image, runtime base image, or both base images, respectively.
+
+If no Dockerfiles are present, `/bin/build` may still consume build plan entries and add metadata to `build.toml`/`launch.toml`.
+
+`Dockerfile`s MUST be applied to their corresponding base images after all extensions are executed and before any regular buildpacks are executed. `Dockerfile`s MUST be applied in the order determined during buildpack detection.
+
+All `Dockerfile`s MUST be provided with `base_image` and `build_id` args.
+The `base_image` arg allows the Dockerfile to reference the original base image.
+The `build_id` arg allows the Dockerfile to invalidate the cache after a certain layer and must be defaulted to `0`. The executor of the Dockerfile will provide the `build_id` as a UUID (this eliminates the need to track this variable).
+When the `$build_id` arg is referenced in a `RUN` instruction, all subsequent layers will be rebuilt on the next build (as the value will change).
+
+Build args specified in `build.toml` MUST be provided to `build.Dockerfile` or `Dockerfile` (when applied to the build-time base image).
+Build args specified in `launch.toml` MUST be provided to `run.Dockerfile` or `Dockerfile` (when applied to the runtime base image).
+
+A runtime base image extension MAY indicate that it preserves ABI compatibility by adding the label `io.buildpacks.rebasable=true`. The `image-extender` MUST set `io.buildpacks.rebasable=true` on the final image only if `io.buildpacks.rebasable=true` on the base image and for all image extensions.
+
+### Image Extensions Directory Layout
+
+The image extensions directory MUST contain unarchived extensions such that:
+
+- Each top-level directory is an image extension ID.
+- Each second-level directory is an image extension version.
+
+## Image Extension Examples
+
+### Example: App-specified Dockerfile Extension
+
+This example extension would allow an app to provide runtime and build-time base image extensions as "run.Dockerfile" and "build.Dockerfile."
+The app developer can decide whether the extensions are rebasable.
+
+##### `/cnb/ext/com.example.appext/bin/build`
+```
+#!/bin/sh
+[ -f build.Dockerfile ] && cp build.Dockerfile "$1/"
+[ -f run.Dockerfile ] && cp run.Dockerfile "$1/"
+```
+
+### Example: RPM Dockerfile Extension (app-based)
+
+This example extension would allow a builder to install RPMs for each language runtime, based on the app directory.
+
+Note: The Dockerfiles referenced must disable rebasing, and build times will be slower compared to buildpack-provided runtimes.
+
+##### `/cnb/ext/com.example.rpmext/bin/build`
+```
+#!/bin/sh
+[ -f Gemfile.lock ] && cp "$CNB_BUILDPACK_DIR/Dockerfile-ruby" "$1/Dockerfile"
+[ -f package.json ] && cp "$CNB_BUILDPACK_DIR/Dockerfile-node" "$1/Dockerfile"
+```
+
+
+### Dockerfiles for Base Images
+
+The same Dockerfile format may be used to create new base images or modify existing base images outside of the app build process (e.g., before creating a builder). Any specified labels override existing values.
+
+Dockerfiles that are used to create a base image must create a `/cnb/image/genpkgs` executable that outputs a [CycloneDX](https://cyclonedx.org)-formatted list of packages in the image with PURL IDs when invoked. This executable is executed after all Dockerfiles are applied, and the output replaces the label `io.buildpacks.sbom`. This label doubles as a Software Bill-of-Materials for the base image. In the future, this label will serve as a starting point for the application SBoM.
+
+### Example Dockerfiles
+
+Dockerfile used to create a runtime base image:
+
+```
+ARG base_image
+FROM ${base_image}
+ARG build_id=0
+
+LABEL io.buildpacks.image.distro=ubuntu
+LABEL io.buildpacks.image.version=18.04
+LABEL io.buildpacks.rebasable=true
+
+ENV CNB_USER_ID=1234
+ENV CNB_GROUP_ID=1235
+
+RUN groupadd cnb --gid ${CNB_GROUP_ID} && \
+  useradd --uid ${CNB_USER_ID} --gid ${CNB_GROUP_ID} -m -s /bin/bash cnb
+
+USER ${CNB_USER_ID}:${CNB_GROUP_ID}
+
+COPY genpkgs /cnb/image/genpkgs
+```
+
+`run.Dockerfile` for use with the example `app.run.Dockerfile.out` extension that always installs the latest version of curl:
+```
+ARG base_image
+FROM ${base_image}
+ARG build_id=0
+
+RUN echo ${build_id}
+
+RUN apt-get update && apt-get install -y curl && rm -rf /var/lib/apt/lists/*
+```
+(note: this Dockerfile disables rebasing, as OS package installation is not rebasable)
+
+`run.Dockerfile` for use with the example `app.run.Dockerfile.out` extension that installs a special package to /opt:
+```
+ARG base_image
+FROM ${base_image}
+ARG build_id=0
+
+LABEL io.buildpacks.rebasable=true
+
+RUN curl -L https://example.com/mypkg-install | sh # installs to /opt
+```
+(note: rebasing is explicitly allowed because only a single directory in /opt is created)
+
+
+## Data Format
+
+### Files
+
+#### `extension.toml` (TOML)
+
+```toml
+api = "<image extension API version>"
+
+[extension]
+id = "<extension ID>"
+name = "<extension name>"
+version = "<extension version>"
+homepage = "<extension homepage>"
+description = "<extension description>"
+keywords = [ "<string>" ]
+
+[[extension.licenses]]
+type = "<string>"
+uri = "<uri>"
+```
+
+Image extension authors MUST choose a globally unique ID, for example: "io.buildpacks.apt".
+
+**The image extension ID:**
+
+*Key: `id = "<extension ID>"`*
+- MUST only contain numbers, letters, and the characters `.`, `/`, and `-`.
+- MUST NOT be `config` or `app`.
+- MUST NOT be identical to any other buildpack ID when using a case-insensitive comparison.
+
+The image extension version:
+- MUST be in the form `<X>.<Y>.<Z>` where `X`, `Y`, and `Z` are non-negative integers and must not contain leading zeros.
+   - Each element MUST increase numerically.
+   - Buildpack authors will define what changes will increment `X`, `Y`, and `Z`.
+
+**The image extension API:**
+
+*Key: `api = "<image extension API version>"`*
+ - MUST be in form `<major>.<minor>` or `<major>`, where `<major>` is equivalent to `<major>.0`
+ - MUST describe the implemented platform API.
+ - SHOULD indicate the lowest compatible `<minor>` if extension behavior is consistent with multiple `<minor>` versions of a given `<major>`
+
+**The image extension licenses:**
+
+The `[[extension.licenses]]` table is optional and MAY contain a list of image extension licenses where:
+
+- `type` - This MAY use the SPDX 2.1 license expression, but is not limited to identifiers in the SPDX Licenses List.
+- `uri` - If this buildpack is using a nonstandard license, then this key MAY be specified in lieu of or in addition to `type` to point to the license.

platform.md

@@ -122,6 +122,8 @@ A **launcher layer** refers to a layer in the app OCI image containing the **lau
 
 The **launcher** refers to a lifecycle executable packaged in the **app image** for the purpose of executing processes at runtime.
 
+An **image extension** is a dynamically-generated build-time and/or runtime Dockerfile that act as a pre-build base image extension. Extensions participate in detection and execute before the buildpack build process.
+
 #### Additional Terminology
 An **image reference** refers to either a **tag reference** or **digest reference**.
 
@@ -349,6 +351,7 @@ Usage:
 /cnb/lifecycle/detector \
   [-app <app>] \
   [-buildpacks <buildpacks>] \
+  [-exts <exts>] \
   [-group <group>] \
   [-layers <layers>] \
   [-log-level <log-level>] \
@@ -362,6 +365,7 @@ Usage:
 |---------------|-------------------------|--------------------------------------------------------|-------
 | `<app>`         | `CNB_APP_DIR`         | `/workspace`                                           | Path to application directory
 | `<buildpacks>`  | `CNB_BUILDPACKS_DIR`  | `/cnb/buildpacks`                                      | Path to buildpacks directory (see [Buildpacks Directory Layout](#buildpacks-directory-layout))
+| `<ext>`         | `CNB_IMAGE_EXTS_DIR`  | `/cnb/exts`                                            | Path to image extensions directory (see [Image Extensions
 | `<group>`       | `CNB_GROUP_PATH`      | `<layers>/group.toml`                                  | Path to output group definition
 | `<layers>`      | `CNB_LAYERS_DIR`      | `/layers`                                              | Path to layers directory
 | `<log-level>`   | `CNB_LOG_LEVEL`       | `info`                                                 | Log Level
@@ -445,6 +449,47 @@ Usage:
 ##### Layer Restoration
 lifeycle MUST use the provided `cache-dir` or `cache-image` to retrieve cache contents. The [rules](https://github.com/buildpacks/spec/blob/main/buildpack.md#layer-types) for restoration MUST be followed when determining how and when to store cache layers.
 
+#### `image-extender`
+
+The platform MAY execute `image-extender`.
+
+Usage:
+```
+/cnb/lifecycle/image-extender \
+  [-group <group>] \
+  [-exts <exts>] \
+  [-output <output>] \
+  [-log-level <log-level>] \
+  [-plan <plan>]
+```
+
+##### Inputs
+| Input          | Env                   | Default Value         | Description
+|----------------|-----------------------|-----------------------|----------------------
+| `<exts>`       | `CNB_IMAGE_EXTS_DIR`  | `/cnb/exts`           | Path to image extensions directory (see [Image Extensions Directory Layout](image-extension.md))
+| `<group>`      | `CNB_GROUP_PATH`      | `<layers>/group.toml` | Path to group definition (see [`group.toml`](#grouptoml-toml))
+| `<output>`     | `CNB_OUTPUT_DIR`      | `/output`             | Path to output directory
+| `<log-level>`  | `CNB_LOG_LEVEL`       | `info`                | Log Level
+| `<plan>`       | `CNB_PLAN_PATH`       | `<output>/plan.toml`  | Path to resolved build plan (see [`plan.toml`](#plantoml-toml))
+
+##### Outputs
+| Output                                     | Description
+|--------------------------------------------|----------------------------------------------
+| [exit status]                              | (see Exit Code table below for values)
+| `/dev/stdout`                              | Logs (info)
+| `/dev/stderr`                              | Logs (warnings, errors)
+
+| Exit Code | Result|
+|-----------|-------|
+| `0`       | Success
+| `11`      | Platform API incompatibility error
+| `12`      | Buildpack API incompatibility error
+| `1-10`, `13-19` | Generic lifecycle errors
+| `91`     | Error applying image extension
+| `90`, `92-99`|  Extension-specific lifecycle errors
+
+- The lifecycle SHALL execute all image extensions in the order defined in `<group>` according to the process outlined in the [Image Extensions Interface Specification](image-extension.md).
+
 #### `builder`
 The platform MUST execute `builder` in the **build environment**
 
@@ -912,6 +957,11 @@ version = "<buildpack version>"
 api = "<buildpack API version>"
 optional = false
 
+[[image-extensions]]
+id = "<image extension ID>"
+version = "<image extension version>"
+api = "<image extension API version>"
+
 [[processes]]
 type = "<process type>"
 command = "<command>"