From 5c6680baf151dd670444e7e7e6104b4a87c40769 Mon Sep 17 00:00:00 2001 From: Natalie Arellano Date: Thu, 10 Aug 2023 17:30:00 -0400 Subject: [PATCH 1/4] WIP Signed-off-by: Natalie Arellano --- .../create-extension/advanced-extensions.md | 85 +++++++++++++++++++ .../building-blocks-extension.md | 6 +- 2 files changed, 86 insertions(+), 5 deletions(-) create mode 100644 content/docs/extension-guide/create-extension/advanced-extensions.md diff --git a/content/docs/extension-guide/create-extension/advanced-extensions.md b/content/docs/extension-guide/create-extension/advanced-extensions.md new file mode 100644 index 000000000..80376cd97 --- /dev/null +++ b/content/docs/extension-guide/create-extension/advanced-extensions.md @@ -0,0 +1,85 @@ ++++ +title="The finer points of image extensions" +weight=406 ++++ + +TODO - move this to extension-guide/conventions? + +# Guidance for extension authors + +## During detect + +### Expressing provided dependencies through the build plan + +### Configuring build args + +``` +├── bin +│ ├── detect +├── generate +│ ├── build.Dockerfile +│ ├── run.Dockerfile +│ ├── extend-config.toml +├── extension.toml +``` + +## During generate + +### Re-setting the user/group with build args + +### Invalidating the build cache with the UUID build arg + +### Making 'rebasable' changes + +## In general + +### Choosing an extension ID + +### Expressing information in extension.toml + +### Pre-populating output + +The root directory for a typical extension might look like the following: + +``` +. +├── bin +│ ├── detect <- similar to a buildpack ./bin/detect +│ ├── generate <- similar to a buildpack ./bin/build +├── extension.toml <- similar to a buildpack buildpack.toml +``` + +But it could also look like any of the following: + +#### ./bin/detect is optional! + +``` +. +├── bin +│ ├── generate +├── detect +│ ├── plan.toml +├── extension.toml +``` + +#### ./bin/generate is optional! + +``` +├── bin +│ ├── detect +├── generate +│ ├── build.Dockerfile +│ ├── run.Dockerfile +├── extension.toml +``` + +#### It's all optional! + +``` +├── detect +│ ├── plan.toml +├── generate +│ ├── build.Dockerfile +│ ├── run.Dockerfile +├── extension.toml +``` diff --git a/content/docs/extension-guide/create-extension/building-blocks-extension.md b/content/docs/extension-guide/create-extension/building-blocks-extension.md index b577e4d6e..c4f447702 100644 --- a/content/docs/extension-guide/create-extension/building-blocks-extension.md +++ b/content/docs/extension-guide/create-extension/building-blocks-extension.md @@ -38,13 +38,9 @@ You should see something akin to the following: * Only a limited set of Dockerfile instructions is supported - consult the [spec](https://github.com/buildpacks/spec/blob/main/image_extension.md) for further details. - * In the [initial implementation](/docs/features/dockerfiles#phased-approach), `run.Dockerfile` instructions are - limited to a single `FROM` instruction (effectively, it is only possible to switch the run-time base image to a - pre-created image i.e., no dynamic image modification is allowed). Consult - the [spec](https://github.com/buildpacks/spec/blob/main/image_extension.md) - for further details. We'll take a closer look at the executables for the `vim` extension in the next step. +For guidance around writing extensions and more advanced use cases, see [here](/docs/extension-guide/create-extension/advanced-extensions). --- From 25bf43f045db43da1f0cb9c7783b6cb5e85bf45b Mon Sep 17 00:00:00 2001 From: Natalie Arellano Date: Mon, 14 Aug 2023 17:16:49 -0400 Subject: [PATCH 2/4] Add more information Signed-off-by: Natalie Arellano --- .../create-extension/advanced-extensions.md | 94 ++++++++++++++++--- 1 file changed, 82 insertions(+), 12 deletions(-) diff --git a/content/docs/extension-guide/create-extension/advanced-extensions.md b/content/docs/extension-guide/create-extension/advanced-extensions.md index 80376cd97..8d09e42f2 100644 --- a/content/docs/extension-guide/create-extension/advanced-extensions.md +++ b/content/docs/extension-guide/create-extension/advanced-extensions.md @@ -3,40 +3,110 @@ title="The finer points of image extensions" weight=406 +++ -TODO - move this to extension-guide/conventions? - # Guidance for extension authors ## During detect ### Expressing provided dependencies through the build plan +The [build plan](/docs/reference/spec/buildpack-api#build-plan) is a mechanism for inter-buildpack communication. +Through the build plan, buildpacks may express the dependencies they require, as well as those they provide. +The lifecycle uses information from the build plan to determine whether a group of buildpacks is compatible - that is, whether for every buildpack in the group, its required dependencies are provided by a buildpack that comes before it. + +Extensions can use the build plan too - but they are only allowed to provide dependencies, they cannot require any. +Note that because there is a separate order for extensions that is prepended to each buildpack group during the `detect` phase, +all extension "provides" come before all buildpack "requires" in the build plan. + +During the `detect` phase, the lifecycle sets a `CNB_OUTPUT_DIR` environment variable when executing each `./bin/detect`. +If using a build plan, extensions should write the plan to `$CNB_OUTPUT_DIR/plan.toml`. + +## During generate + ### Configuring build args -``` -├── bin -│ ├── detect -├── generate -│ ├── build.Dockerfile -│ ├── run.Dockerfile -│ ├── extend-config.toml -├── extension.toml +During the `generate` phase, extensions may output (in addition to Dockerfiles) an `extend-config.toml` +containing build-time arguments for Dockerfiles. +(Not to be confused with the `build` phase, "build" here refers to the application of Dockerfiles to a base image, +similar to a `docker build`). + +Arguments may be configured for builder image extension or runtime base image extension or both, +according to the schema defined in the [spec](https://github.com/buildpacks/spec/blob/main/image_extension.md#extend-configtoml-toml). + +During the `generate` phase, the lifecycle sets a `CNB_OUTPUT_DIR` environment variable when executing each `./bin/generate`. +If using an `extend-config.toml`, extensions should write the config to `$CNB_OUTPUT_DIR/extend-config.toml`. + +### Invalidating the build cache with the UUID build arg + +Whenever possible, the application of Dockerfiles to a base image will use a caching mechanism +similar to that of a `docker build`. +The lifecycle, for example, uses [`kaniko`](https://github.com/GoogleContainerTools/kaniko) to implement the `extender`. + +However, there may be times when caching is not desired - for example, when fetching the "latest" available version of a package. +In such cases, Dockerfiles can use the `build_id` build argument to invalidate the cache for all instructions that follow. + +As an example: + +```bash +RUN echo "this instruction may be found in the cache" + +ARG build_id=0 +RUN echo ${build_id} + +RUN echo "this instruction should never be found in the cache, as the value above will change" ``` -## During generate +Note that `build_id` is defaulted to `0` as a best practice. ### Re-setting the user/group with build args -### Invalidating the build cache with the UUID build arg +Dockerfiles from image extensions may contain `USER root` instructions in order to perform actions that would not be possible +when running as a non-root user. + +However, for security reasons, the final user after all Dockerfiles have been applied should _not_ be root. +To reset the user to its original value (before the application of the current Dockerfile), +Dockerfiles should make use of `user_id` and `group_id` build arguments, as seen below: + +```bash +ARG user_id=1000 +ARG group_id=1000 +USER ${user_id}:${group_id} +``` ### Making 'rebasable' changes +Image layers generated from extensions are "above the rebasable seam" - that is, +after swapping the runtime image to an updated version through a `rebase` operation, +the extension layers will be persisted in the rebased application image. + +Unlike buildpack layers, extension layers are _not_ always safe to rebase. +Extension layers _may_ be safe to rebase if: +* the changes they introduce are purely additive (no modification of pre-existing files from the base image), or +* any modified pre-existing files are safe to exclude from rebase (the file from the extension layer will override any updated version of the file from the new runtime base image) + +By default, the lifecycle will assume that any extension layers are _not_ rebasable. +To indicate otherwise, `run.Dockerfile`s should include: + +```bash +LABEL io.buildpacks.rebasable=true +``` + +If all `run.Dockerfile`s set this label to `true`, the application image will contain the label `io.buildpacks.rebasable=true`. +Otherwise, the application image will contain the label `io.buildpacks.rebasable=false`. +`pack rebase` requires a `--force` flag if the application image contains `io.buildpacks.rebasable=false`. + +Extension authors should take great care (and perform testing) to ensure that any layers marked as rebasable are in fact rebasable. + ## In general ### Choosing an extension ID +Extension IDs must be globally unique to extensions, but extensions and buildpacks can share the same ID. + ### Expressing information in extension.toml +Just like `buildpack.toml`, an `extension.toml` can contain additional metadata to describe its behavior. +See the [spec](https://github.com/buildpacks/spec/blob/main/image_extension.md#extensiontoml-toml) for more information. + ### Pre-populating output The root directory for a typical extension might look like the following: From 9654ae5a0a0c09e1b10e61b166650d9405976f4a Mon Sep 17 00:00:00 2001 From: Natalie Arellano Date: Mon, 14 Aug 2023 17:38:46 -0400 Subject: [PATCH 3/4] Remove comment Signed-off-by: Natalie Arellano --- .../extension-guide/create-extension/advanced-extensions.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/content/docs/extension-guide/create-extension/advanced-extensions.md b/content/docs/extension-guide/create-extension/advanced-extensions.md index 8d09e42f2..5fc55e122 100644 --- a/content/docs/extension-guide/create-extension/advanced-extensions.md +++ b/content/docs/extension-guide/create-extension/advanced-extensions.md @@ -114,9 +114,9 @@ The root directory for a typical extension might look like the following: ``` . ├── bin -│ ├── detect <- similar to a buildpack ./bin/detect -│ ├── generate <- similar to a buildpack ./bin/build -├── extension.toml <- similar to a buildpack buildpack.toml +│ ├── detect +│ ├── generate +├── extension.toml ``` But it could also look like any of the following: From 59dc0cd9e39de379b8325146c2ae8032b9db5119 Mon Sep 17 00:00:00 2001 From: Natalie Arellano Date: Mon, 14 Aug 2023 17:40:59 -0400 Subject: [PATCH 4/4] Add more info Signed-off-by: Natalie Arellano --- .../create-extension/advanced-extensions.md | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/content/docs/extension-guide/create-extension/advanced-extensions.md b/content/docs/extension-guide/create-extension/advanced-extensions.md index 5fc55e122..06542f264 100644 --- a/content/docs/extension-guide/create-extension/advanced-extensions.md +++ b/content/docs/extension-guide/create-extension/advanced-extensions.md @@ -123,6 +123,10 @@ But it could also look like any of the following: #### ./bin/detect is optional! +If `./bin/detect` is missing, +the extension is assumed to pass detection and +the lifecycle will interpret the contents of `./detect` as the contents of `$CNB_OUTPUT_DIR`. + ``` . ├── bin @@ -134,6 +138,9 @@ But it could also look like any of the following: #### ./bin/generate is optional! +If `./bin/generate` is missing, +the lifecycle will interpret the contents of `./generate` as the contents of `$CNB_OUTPUT_DIR`. + ``` ├── bin │ ├── detect