From b95dc2ca6818bdb7ef1f363e8db131ac3fcb8078 Mon Sep 17 00:00:00 2001 From: Juan Bustamante Date: Wed, 23 Feb 2022 10:22:47 -0500 Subject: [PATCH 01/62] WIP - Draft version of the RFC Signed-off-by: Juan Bustamante --- text/0000-publish-operation | 121 ++++++++++++++++++++++++++++++++++++ 1 file changed, 121 insertions(+) create mode 100644 text/0000-publish-operation diff --git a/text/0000-publish-operation b/text/0000-publish-operation new file mode 100644 index 000000000..78f14571a --- /dev/null +++ b/text/0000-publish-operation @@ -0,0 +1,121 @@ +# Meta +[meta]: #meta +- Name: Publish Operation +- Start Date: 2022-02-22 +- Author(s): Juan Bustamante (@jbustamante) +- Status: Draft +- RFC Pull Request: (leave blank) +- CNB Pull Request: (leave blank) +- CNB Issue: (leave blank) +- Supersedes: (put "N/A" unless this replaces an existing RFC, then link to that RFC) + +# Summary +[summary]: #summary + +Split the process of generating and saving it into the final destination in two different operations. +- Export: will create the image and saving it to disk in OCI layout format. When the `daemon` flag is enabled it will save all the metadata that can't be saved into the daemon in a separate report +- Publish: will take the image generated by Export and save it into the Daemon or in the Registry using the metadata report when it's necessary. + +# Definitions +[definitions]: #definitions + + +# Motivation +[motivation]: #motivation + + + +Because the [Image.Digest](https://pkg.go.dev/github.com/google/go-containerregistry/pkg/v1#Image) method from the GCR library would compute the value before pushing it to a Registry, we can be sure to create the Image and add all the metadata require (annotations, cosign, etc) save it to disk and be sure that when the image is push to the registry the Image Digest will not change. (See the [thread](https://cloud-native.slack.com/archives/C033DV9EBDF/p1644523524402149) on Slack) + +Splitting the creation of the image and it's saving operation in two different process allows us the following: +- Implement an export phase with an standard output format (OCI Layout) that can be consumed by `Platforms` if they want to +- Keep consistency with the push operation behavior because the digest and the metadata can be saved linked to the Manifest +- Solve unsupported metadata cases by the Daemon, because we can save this information in a custom format together with the Image exported in OCI Layout + + +We can support the use cases: + +- OCI annotations. See [RFC](https://github.com/buildpacks/rfcs/pull/196) +- Cosign integration. See [RFC](https://github.com/buildpacks/rfcs/pull/195) + + + +# What it is +[what-it-is]: #what-it-is + +The idea is to modify the existing *Export* phase to save the image on disk in a path defined by **Platform**, in case of **Pack** the image +would be saved in a volumen shared by other phases, OCI Layout forma will be used to saved the image and a new operation called *Publish* +will be created. This operation will be invoked when the a new flag `--publish` is sent to Lifecycle and it will actually load +the image from disk generated by the export phase and then push to Daemon or Registry. + +Check the following image for a visual representation of the idea. + +```mermaid +graph LR + A[ANALIZE] --> B[RESTORE] + B --> C[BUILD] + C --> D[EXPORT] + D --> |when --publish| E[PUBLISH] + +``` + + + + + +# How it Works +[how-it-works]: #how-it-works + + + +# Migration +[migration]: #migration + + +# Drawbacks +[drawbacks]: #drawbacks + +Why should we *not* do this? + +# Alternatives +[alternatives]: #alternatives + +- What other designs have been considered? +- Why is this proposal the best? +- What is the impact of not doing this? + +# Prior Art +[prior-art]: #prior-art + +Discuss prior art, both the good and bad. + +# Unresolved Questions +[unresolved-questions]: #unresolved-questions + +- What parts of the design do you expect to be resolved before this gets merged? +- What parts of the design do you expect to be resolved through implementation of the feature? +- What related issues do you consider out of scope for this RFC that could be addressed in the future independently of the solution that comes out of this RFC? + +# Spec. Changes (OPTIONAL) +[spec-changes]: #spec-changes +Does this RFC entail any proposed changes to the core specifications or extensions? If so, please document changes here. +Examples of a spec. change might be new lifecycle flags, new `buildpack.toml` fields, new fields in the buildpackage label, etc. +This section is not intended to be binding, but as discussion of an RFC unfolds, if spec changes are necessary, they should be documented here. From ff921a6033eafe02c8f54b8eb3b9b983f1620363 Mon Sep 17 00:00:00 2001 From: Juan Bustamante Date: Wed, 23 Feb 2022 10:29:19 -0500 Subject: [PATCH 02/62] Fixing extension of the file Signed-off-by: Juan Bustamante --- text/{0000-publish-operation => 0000-publish-operation.md} | 0 1 file changed, 0 insertions(+), 0 deletions(-) rename text/{0000-publish-operation => 0000-publish-operation.md} (100%) diff --git a/text/0000-publish-operation b/text/0000-publish-operation.md similarity index 100% rename from text/0000-publish-operation rename to text/0000-publish-operation.md From 430ad6d13495cfcd0905cfe26f4e7f77be2db478 Mon Sep 17 00:00:00 2001 From: Juan Bustamante Date: Thu, 24 Feb 2022 09:30:16 -0500 Subject: [PATCH 03/62] more work Signed-off-by: Juan Bustamante --- text/0000-publish-operation.md | 65 +++++++++++++++++++++++----------- 1 file changed, 44 insertions(+), 21 deletions(-) diff --git a/text/0000-publish-operation.md b/text/0000-publish-operation.md index 78f14571a..9220b08d3 100644 --- a/text/0000-publish-operation.md +++ b/text/0000-publish-operation.md @@ -12,43 +12,58 @@ # Summary [summary]: #summary -Split the process of generating and saving it into the final destination in two different operations. -- Export: will create the image and saving it to disk in OCI layout format. When the `daemon` flag is enabled it will save all the metadata that can't be saved into the daemon in a separate report -- Publish: will take the image generated by Export and save it into the Daemon or in the Registry using the metadata report when it's necessary. +Split the process of creating and saving the Image in two different operations. +- Export: will create the Image and saves it to disk in [OCI Layout](https://github.com/opencontainers/image-spec/blob/main/image-layout.md) format. When the `daemon` flag is enabled it could save all the metadata that can't be saved into the daemon in a separate report. +- Publish: will take the image generated by Export and push it into the Daemon or in the Registry. # Definitions [definitions]: #definitions - + +- A [Platform](https://buildpacks.io/docs/concepts/components/platform/) uses a lifecycle, Buildpacks (packaged in a builder), and application source code to produce an OCI image. +- A [Lifecycle](https://buildpacks.io/docs/concepts/components/lifecycle/) orchestrates Buildpacks execution, then assembles the resulting artifacts into a final app image. +- A Daemon is a service, popularized by Docker, for downloading container images, and executing and managing containers from those images. +- A Registry is a long-running service used for storing and retrieving container images. +- An [OCI Image Layout](https://github.com/opencontainers/image-spec/blob/main/image-layout.md) is the directory structure for OCI content-addressable blobs and location-addressable references. +- An [image index](https://github.com/opencontainers/image-spec/blob/main/image-index.md) is a higher-level manifest which points to a list of manifests and descriptors. +- An [Image manifest](https://github.com/opencontainers/image-spec/blob/main/manifest.md) provides a configuration and set of layers for a single container image for a specific architecture and operating system. +- A [config](https://github.com/opencontainers/image-spec/blob/main/descriptor.md) is a property references a configuration object for a container, by digest. It must support the following media type `application/vnd.oci.image.config.v1+json` # Motivation [motivation]: #motivation +Because the [Image.Digest](https://pkg.go.dev/github.com/google/go-containerregistry/pkg/v1#Image) method from the GCR library would compute the Digest value before pushing it to a Registry, it allows us to create the image and add metadata require like annotations or even sign it and save it to disk and be sure that when the Image is push to the registry the Digest will not change. (See the [thread](https://cloud-native.slack.com/archives/C033DV9EBDF/p1644523524402149) on Slack). + + + +We should do this to unblock uses cases like + +- OCI annotations. See [RFC](https://github.com/buildpacks/rfcs/pull/196) +- Cosign integration. See [RFC](https://github.com/buildpacks/rfcs/pull/195) + + + + +We expect the `Export` phase to save the Image in disk in [OCI Image Layout](https://github.com/opencontainers/image-spec/blob/main/image-layout.md) format and the `Publish` operation to push it to Daemon or Registry. + + -We can support the use cases: - -- OCI annotations. See [RFC](https://github.com/buildpacks/rfcs/pull/196) -- Cosign integration. See [RFC](https://github.com/buildpacks/rfcs/pull/195) +--> - # What it is [what-it-is]: #what-it-is -The idea is to modify the existing *Export* phase to save the image on disk in a path defined by **Platform**, in case of **Pack** the image -would be saved in a volumen shared by other phases, OCI Layout forma will be used to saved the image and a new operation called *Publish* -will be created. This operation will be invoked when the a new flag `--publish` is sent to Lifecycle and it will actually load -the image from disk generated by the export phase and then push to Daemon or Registry. +An new operation called *Publish* will be added in the Lifecycle. `/cnb/lifecycle/publisher` is responsibly for pushing the Image into the Daemon or into an OCI Registry. + + +The *Export* phase will be modify to save the image on disk in a path defined by **Platform**, in case of **Pack**, for example, the image would be saved in a shared volume in the same way it's done right now, [OCI Image Layout](https://github.com/opencontainers/image-spec/blob/main/image-layout.md) format could be used to save the Image on disk. Check the following image for a visual representation of the idea. @@ -57,12 +72,10 @@ graph LR A[ANALIZE] --> B[RESTORE] B --> C[BUILD] C --> D[EXPORT] - D --> |when --publish| E[PUBLISH] + D --> |when --push| E[PUBLISH] ``` - - # How it Works From 945b9869ef01f1f42f0f6e29a03a68dbb17da08e Mon Sep 17 00:00:00 2001 From: Juan Bustamante Date: Thu, 24 Feb 2022 13:46:07 -0500 Subject: [PATCH 04/62] WIP - adding more diagrams Signed-off-by: Juan Bustamante --- text/0000-publish-operation.md | 65 +++++++++++++++++++++++++++------- 1 file changed, 53 insertions(+), 12 deletions(-) diff --git a/text/0000-publish-operation.md b/text/0000-publish-operation.md index 9220b08d3..c98ee64a1 100644 --- a/text/0000-publish-operation.md +++ b/text/0000-publish-operation.md @@ -12,9 +12,9 @@ # Summary [summary]: #summary -Split the process of creating and saving the Image in two different operations. -- Export: will create the Image and saves it to disk in [OCI Layout](https://github.com/opencontainers/image-spec/blob/main/image-layout.md) format. When the `daemon` flag is enabled it could save all the metadata that can't be saved into the daemon in a separate report. -- Publish: will take the image generated by Export and push it into the Daemon or in the Registry. +Split the process of creating and writing the Image in a final destination in two different operations. +- Export: will create the Image and saves it to disk in [OCI Layout](https://github.com/opencontainers/image-spec/blob/main/image-layout.md) format. +- Publish: will take the Image generated by Export and write it into the Daemon or in the Registry. # Definitions [definitions]: #definitions @@ -60,19 +60,57 @@ We expect the `Export` phase to save the Image in disk in [OCI Image Layout](htt # What it is [what-it-is]: #what-it-is -An new operation called *Publish* will be added in the Lifecycle. `/cnb/lifecycle/publisher` is responsibly for pushing the Image into the Daemon or into an OCI Registry. +Currently the *Exporter* writes either in an OCI image registry or a docker daemon, the idea is to change that behavior to write into disk in [OCI Image Layout](https://github.com/opencontainers/image-spec/blob/main/image-layout.md) format. Then a new operation called *Publish* will be added in the Lifecycle. `/cnb/lifecycle/publisher` is responsibly for writing the Image either in a OCI image registry or a docker daemon. - -The *Export* phase will be modify to save the image on disk in a path defined by **Platform**, in case of **Pack**, for example, the image would be saved in a shared volume in the same way it's done right now, [OCI Image Layout](https://github.com/opencontainers/image-spec/blob/main/image-layout.md) format could be used to save the Image on disk. - -Check the following image for a visual representation of the idea. +The new Lifecycle orchestration process will change to something like this: ```mermaid graph LR - A[ANALIZE] --> B[RESTORE] - B --> C[BUILD] - C --> D[EXPORT] - D --> |when --push| E[PUBLISH] + A[DETECT] --> B[ANALYZE] + B --> C[RESTORE] + C --> D[BUILD] + D --> E[EXPORT] + E --> |when --push| F[PUBLISH] + +``` + +This features affects the Platform implementor because they will have to include this new operation as part of the workflow to build the application image. + +Let's see some examples: + +## Examples + +### Using the Creator + +#### Pushing an image to daemon + +```=shell + /cnb/lifecycle/creator -daemon -push + +``` + +#### Pushing an image to an OCI registry + +```=shell + /cnb/lifecycle/creator -push + +``` + +### Not using the Creator + +#### Pushing an image to daemon + +```=shell + /cnb/lifecycle/exporter -output /path/to/save/the/image + /cnb/lifecycle/publisher -daemon -input /path/to/the/image [...] + +``` + +#### Pushing an image to an OCI registry + +```=shell + /cnb/lifecycle/exporter -output /path/to/save/the/image + /cnb/lifecycle/publisher -input /path/to/the/image [...] ``` @@ -100,6 +138,9 @@ When it's invoke from the Creator the current workflow is not affected because a # How it Works [how-it-works]: #how-it-works + +![](https://i.imgur.com/DLMMpGf.png) + +- RFC Pull Request: (leave blank) +- CNB Pull Request: (leave blank) +- CNB Issue: (leave blank) +- Supersedes: (put "N/A" unless this replaces an existing RFC, then link to that RFC) + +# Summary +[summary]: #summary + +When the `Exporter` phase is invoked passing the `-daemon` flag besides writing into the Daemon also save the image to disk in [OCI Layout](https://github.com/opencontainers/image-spec/blob/main/image-layout.md) format updating the [report.toml](https://github.com/buildpacks/spec/blob/main/platform.md#reporttoml-toml) file with all the metadata require to validate the consistency of the image when it is published to a Registry. + +# Definitions +[definitions]: #definitions + +- A [Platform](https://buildpacks.io/docs/concepts/components/platform/) uses a lifecycle, Buildpacks (packaged in a builder), and application source code to produce an OCI image. +- A [Lifecycle](https://buildpacks.io/docs/concepts/components/lifecycle/) orchestrates Buildpacks execution, then assembles the resulting artifacts into a final app image. +- A Daemon is a service, popularized by Docker, for downloading container images, and executing and managing containers from those images. +- A Registry is a long-running service used for storing and retrieving container images. +- A digest reference refers to a [content addressable](https://en.wikipedia.org/wiki/Content-addressable_storage) identifier of form /@ which locates an image manifest in an [OCI Distribution Specification](https://github.com/opencontainers/distribution-spec/blob/master/spec.md) compliant registry. +- A Image Manifest provides a configuration and set of layers for a single container image for a specific architecture and operating system. +- An [OCI Image Layout](https://github.com/opencontainers/image-spec/blob/main/image-layout.md) is the directory structure for OCI content-addressable blobs and location-addressable references. + +# Motivation +[motivation]: #motivation + +Implementing this new feature will help us to solve the problem of loosing information when the image is saved into the Daemon keeping the image on disk along with the metadata it can be used as input for other tools to offer more capabilities to the end users. + +This feature will help to unblock uses cases like +- OCI annotations. See [RFC](https://github.com/buildpacks/rfcs/pull/196) +- Cosign integration. See [RFC](https://github.com/buildpacks/rfcs/pull/195) + +# What it is +[what-it-is]: #what-it-is + +Currently the *Exporter* writes either in an OCI image registry or a docker daemon, the idea is to add the capability to write into disk in [OCI Image Layout](https://github.com/opencontainers/image-spec/blob/main/image-layout.md) format only when the `-daemon` flag is used as argument AND the feature is enable using a new flag `-layout` or the default environment variable `CNB_LAYOUT_DIR`. + +## Examples + +### Exporting using the environment variable + +```=shell +> export CNB_LAYOUT_DIR=oci +> /cnb/lifecycle/exporter -daemon my-app-image +> tree /oci +. +└── oci/ + └── my-app-image/ + ├── blobs/ + │ └── sha256/ + │ └── 01.. + ├── index.json + └── oci-layout + +``` + +### Exporting using the command line flag + +```=shell +> /cnb/lifecycle/exporter -daemon -layout oci my-app-image +> tree /oci +. +└── oci/ + └── my-app-image/ + ├── blobs/ + │ └── sha256/ + │ └── 01.. + ├── index.json + └── oci-layout + +``` +As we can see there is a new folder called `oci` and inside that folder we can find our application image in [OCI Image Layout](https://github.com/opencontainers/image-spec/blob/main/image-layout.md) format + +Attempts to use this feature when the `-daemon` flag is not used could be ignore or show some warnings messages. + +# How it Works +[how-it-works]: #how-it-works + +The lifecycle phases affected by this new behavior is: [Export](https://buildpacks.io/docs/concepts/components/lifecycle/export/) + +The following new input is proposed to be added to this phase + +| Input | Environment Variable | Default Value | Description +|-------------------|-----------------------|--------------------------|---------------------- +| `` | `CNB_LAYOUT_DIR` | "" | The root directory where the OCI image will be written. The presence of a none empty value for this environment variable will enable the feature. | + + +- When the exporter is executed WITH the flag `daemon` it will check for the presence of the flag `layout` or the environment variable `CNB_LAYOUT_DIR` +- IF any of the values are present THEN + - It will create a folder name `` located at the path defined `` or `CNB_LAYOUT_DIR` + - In parallel of exporting the Image to the Daemon it will save the final Image in [OCI Image Layout](https://github.com/opencontainers/image-spec/blob/main/image-layout.md) format in the directory created before. The image will be saved using **uncompressed** layers + - It will calculate the digest of the manifest of the compressed layers and write that value into the report.toml file + - It will update the report.toml file with all the tags and require information to verify the image once it is pushed into a registry +- OTHERWISE it will behave as it is right now + +# Migration +[migration]: #migration + + +# Drawbacks +[drawbacks]: #drawbacks + +- We could increase the disk space if we not managed the duplication of saving the layers on disk. Currently the Cache implementation (used when daemon is ON) saved the layers tarballs on disk, because the current proposal is exporting the whole image on disk it will also require more space to save the layers for the OCI format in the `blobs` folder. + +# Alternatives +[alternatives]: #alternatives + +## Redesign the current Cache + +Another potential solution could be to export the OCI image along with the current Cache implementation. The current implementation when the Daemon is enable can be describe with the following class diagram + +```mermaid +classDiagram + Cache <|-- VolumeCache + Image <|-- LocalImage + LocalImage <|-- CachingImage + CachingImage .. VolumeCache +class Cache { + <> +} + +class Image { + <> +} +``` + +When the Daemon is enabled, a `CachingImage` is created, this image implementation saves the layers tarballs in a `VolumeCache` to reuse them and increase the speed of the process. The idea could be to redesign this `Cache` implementation (VolumeCache) with a new one, maybe a `OCICache` which handles the details to avoid saving the layers tarballs duplicated. + +```mermaid +classDiagram + Cache <|-- OCICache + Image <|-- LocalImage + LocalImage <|-- CachingImage + CachingImage .. OCICache +class Cache { + <> + +} + +class Image { + <> + +} +``` +### Drawbacks + +- We will have to expose Cache implementation details to the outside world, probably spec those details, for other tools to interact with this exported data + + + +# Prior Art +[prior-art]: #prior-art + +Discuss prior art, both the good and bad. + +# Unresolved Questions +[unresolved-questions]: #unresolved-questions + + + +# Spec. Changes (OPTIONAL) +[spec-changes]: #spec-changes + + diff --git a/text/0000-publish-operation.md b/text/0000-publish-operation.md deleted file mode 100644 index c98ee64a1..000000000 --- a/text/0000-publish-operation.md +++ /dev/null @@ -1,185 +0,0 @@ -# Meta -[meta]: #meta -- Name: Publish Operation -- Start Date: 2022-02-22 -- Author(s): Juan Bustamante (@jbustamante) -- Status: Draft -- RFC Pull Request: (leave blank) -- CNB Pull Request: (leave blank) -- CNB Issue: (leave blank) -- Supersedes: (put "N/A" unless this replaces an existing RFC, then link to that RFC) - -# Summary -[summary]: #summary - -Split the process of creating and writing the Image in a final destination in two different operations. -- Export: will create the Image and saves it to disk in [OCI Layout](https://github.com/opencontainers/image-spec/blob/main/image-layout.md) format. -- Publish: will take the Image generated by Export and write it into the Daemon or in the Registry. - -# Definitions -[definitions]: #definitions - -- A [Platform](https://buildpacks.io/docs/concepts/components/platform/) uses a lifecycle, Buildpacks (packaged in a builder), and application source code to produce an OCI image. -- A [Lifecycle](https://buildpacks.io/docs/concepts/components/lifecycle/) orchestrates Buildpacks execution, then assembles the resulting artifacts into a final app image. -- A Daemon is a service, popularized by Docker, for downloading container images, and executing and managing containers from those images. -- A Registry is a long-running service used for storing and retrieving container images. -- An [OCI Image Layout](https://github.com/opencontainers/image-spec/blob/main/image-layout.md) is the directory structure for OCI content-addressable blobs and location-addressable references. -- An [image index](https://github.com/opencontainers/image-spec/blob/main/image-index.md) is a higher-level manifest which points to a list of manifests and descriptors. -- An [Image manifest](https://github.com/opencontainers/image-spec/blob/main/manifest.md) provides a configuration and set of layers for a single container image for a specific architecture and operating system. -- A [config](https://github.com/opencontainers/image-spec/blob/main/descriptor.md) is a property references a configuration object for a container, by digest. It must support the following media type `application/vnd.oci.image.config.v1+json` - -# Motivation -[motivation]: #motivation - -Because the [Image.Digest](https://pkg.go.dev/github.com/google/go-containerregistry/pkg/v1#Image) method from the GCR library would compute the Digest value before pushing it to a Registry, it allows us to create the image and add metadata require like annotations or even sign it and save it to disk and be sure that when the Image is push to the registry the Digest will not change. (See the [thread](https://cloud-native.slack.com/archives/C033DV9EBDF/p1644523524402149) on Slack). - - - - -We should do this to unblock uses cases like - -- OCI annotations. See [RFC](https://github.com/buildpacks/rfcs/pull/196) -- Cosign integration. See [RFC](https://github.com/buildpacks/rfcs/pull/195) - - - - -We expect the `Export` phase to save the Image in disk in [OCI Image Layout](https://github.com/opencontainers/image-spec/blob/main/image-layout.md) format and the `Publish` operation to push it to Daemon or Registry. - - - - -# What it is -[what-it-is]: #what-it-is - -Currently the *Exporter* writes either in an OCI image registry or a docker daemon, the idea is to change that behavior to write into disk in [OCI Image Layout](https://github.com/opencontainers/image-spec/blob/main/image-layout.md) format. Then a new operation called *Publish* will be added in the Lifecycle. `/cnb/lifecycle/publisher` is responsibly for writing the Image either in a OCI image registry or a docker daemon. - -The new Lifecycle orchestration process will change to something like this: - -```mermaid -graph LR - A[DETECT] --> B[ANALYZE] - B --> C[RESTORE] - C --> D[BUILD] - D --> E[EXPORT] - E --> |when --push| F[PUBLISH] - -``` - -This features affects the Platform implementor because they will have to include this new operation as part of the workflow to build the application image. - -Let's see some examples: - -## Examples - -### Using the Creator - -#### Pushing an image to daemon - -```=shell - /cnb/lifecycle/creator -daemon -push - -``` - -#### Pushing an image to an OCI registry - -```=shell - /cnb/lifecycle/creator -push - -``` - -### Not using the Creator - -#### Pushing an image to daemon - -```=shell - /cnb/lifecycle/exporter -output /path/to/save/the/image - /cnb/lifecycle/publisher -daemon -input /path/to/the/image [...] - -``` - -#### Pushing an image to an OCI registry - -```=shell - /cnb/lifecycle/exporter -output /path/to/save/the/image - /cnb/lifecycle/publisher -input /path/to/the/image [...] - -``` - - - -# How it Works -[how-it-works]: #how-it-works - - -![](https://i.imgur.com/DLMMpGf.png) - - - -# Migration -[migration]: #migration - - -# Drawbacks -[drawbacks]: #drawbacks - -Why should we *not* do this? - -# Alternatives -[alternatives]: #alternatives - -- What other designs have been considered? -- Why is this proposal the best? -- What is the impact of not doing this? - -# Prior Art -[prior-art]: #prior-art - -Discuss prior art, both the good and bad. - -# Unresolved Questions -[unresolved-questions]: #unresolved-questions - -- What parts of the design do you expect to be resolved before this gets merged? -- What parts of the design do you expect to be resolved through implementation of the feature? -- What related issues do you consider out of scope for this RFC that could be addressed in the future independently of the solution that comes out of this RFC? - -# Spec. Changes (OPTIONAL) -[spec-changes]: #spec-changes -Does this RFC entail any proposed changes to the core specifications or extensions? If so, please document changes here. -Examples of a spec. change might be new lifecycle flags, new `buildpack.toml` fields, new fields in the buildpackage label, etc. -This section is not intended to be binding, but as discussion of an RFC unfolds, if spec changes are necessary, they should be documented here. From 7150fa1cd475fab3673d3308433b1aa82cda1791 Mon Sep 17 00:00:00 2001 From: Juan Bustamante Date: Wed, 2 Mar 2022 11:04:14 -0500 Subject: [PATCH 06/62] WIP - Adding some C4 models diagrams to complement the idea Signed-off-by: Juan Bustamante --- text/0000-export-to-oci.md | 16 +++++++++++++++- 1 file changed, 15 insertions(+), 1 deletion(-) diff --git a/text/0000-export-to-oci.md b/text/0000-export-to-oci.md index 8d30d551d..37095b2db 100644 --- a/text/0000-export-to-oci.md +++ b/text/0000-export-to-oci.md @@ -82,6 +82,12 @@ Attempts to use this feature when the `-daemon` flag is not used could be ignore The lifecycle phases affected by this new behavior is: [Export](https://buildpacks.io/docs/concepts/components/lifecycle/export/) +At high level view of the propose solution can be summarize with the following container diagram from the C4 model + +![](https://i.imgur.com/XhtvT9j.png) + +Notice that we are relying on the OCI format Specification to expose the data for `Platforms` + The following new input is proposed to be added to this phase | Input | Environment Variable | Default Value | Description @@ -97,6 +103,8 @@ The following new input is proposed to be added to this phase - It will update the report.toml file with all the tags and require information to verify the image once it is pushed into a registry - OTHERWISE it will behave as it is right now + + # Migration [migration]: #migration @@ -113,7 +121,13 @@ This section should document breaks to public API and breaks in compatibility du ## Redesign the current Cache -Another potential solution could be to export the OCI image along with the current Cache implementation. The current implementation when the Daemon is enable can be describe with the following class diagram +Another potential solution could be to export the OCI image along with the current Cache concept and expose some contract for `Platform` to interact with this Cache and extract the final OCI image. At high level, the solution can be represented with the following container diagram from C4 model + +![](https://i.imgur.com/xl4gL1G.png) + +Notice that on this solution, because the Cache is an internal component from the Lifecycle implementation we will have to expose some kind of specification for `Platforms` to understand its format and been able to read the OCI image. + +The current implementation when the Daemon is enable can be describe with the following class diagram ```mermaid classDiagram From b56954e71887484b3c6660c1d89f68eda35d6004 Mon Sep 17 00:00:00 2001 From: Juan Bustamante Date: Thu, 3 Mar 2022 09:48:30 -0500 Subject: [PATCH 07/62] WIP - adding a draft idea of some changes to the report.toml file Signed-off-by: Juan Bustamante --- text/0000-export-to-oci.md | 23 ++++++++++++++++++++++- 1 file changed, 22 insertions(+), 1 deletion(-) diff --git a/text/0000-export-to-oci.md b/text/0000-export-to-oci.md index 37095b2db..a85869263 100644 --- a/text/0000-export-to-oci.md +++ b/text/0000-export-to-oci.md @@ -103,7 +103,28 @@ The following new input is proposed to be added to this phase - It will update the report.toml file with all the tags and require information to verify the image once it is pushed into a registry - OTHERWISE it will behave as it is right now - +#### `report.toml` (TOML) + +The new information to be added into the `report.toml` file can be summarize as follows: + +```toml +[export] +[[export.oci]] +digest = "" +manifest-size = "" +compression-algorithm = "" +library-url = "https://github.com/google/go-containerregistry/" +library-language="go" +``` +Where: +- **If** the app image was exported using the `-layer` flag, the export section will be added to the report + - `digest` MUST contain the image digest calculated based on compressed layers + - `manifest-size` MUST contain the manifest size in bytes + - `compression-algorithm` MUST contain the information of the algorithm used by GCR to compute the digest + - `library-url` MUST contain the url of the library used to export the image + - `library-language` MUST contain the client's library programming language used to export the image + +The main idea of this new section in the `report.toml` file is to provide the information of the **expected** digest of the image if it is exported to a registry using a particular implementation. This information can be used by other tools (like publish) to complete the operation and verify the consistency of the image. # Migration [migration]: #migration From 49c555cca4a8a8b4b6833a3e534c2cb5587a24f3 Mon Sep 17 00:00:00 2001 From: Juan Bustamante Date: Fri, 4 Mar 2022 16:29:16 -0500 Subject: [PATCH 08/62] Apply suggestions from code review Co-authored-by: Natalie Arellano Signed-off-by: Juan Bustamante --- text/0000-export-to-oci.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/text/0000-export-to-oci.md b/text/0000-export-to-oci.md index a85869263..83cca2bd8 100644 --- a/text/0000-export-to-oci.md +++ b/text/0000-export-to-oci.md @@ -140,13 +140,13 @@ This section should document breaks to public API and breaks in compatibility du # Alternatives [alternatives]: #alternatives -## Redesign the current Cache +## Redesign the current Launch Cache -Another potential solution could be to export the OCI image along with the current Cache concept and expose some contract for `Platform` to interact with this Cache and extract the final OCI image. At high level, the solution can be represented with the following container diagram from C4 model +Another potential solution could be to export the OCI image along with the current Launch Cache concept and expose some contract for `Platform` to interact with this Cache and extract the final OCI image. At high level, the solution can be represented with the following container diagram from C4 model ![](https://i.imgur.com/xl4gL1G.png) -Notice that on this solution, because the Cache is an internal component from the Lifecycle implementation we will have to expose some kind of specification for `Platforms` to understand its format and been able to read the OCI image. +Notice that on this solution, because the Launch Cache is an internal component from the Lifecycle implementation we will have to expose some kind of specification for `Platforms` to understand its format and been able to read the OCI image. The current implementation when the Daemon is enable can be describe with the following class diagram From 480896f6cbf09479c4b5694a70fc7905d62779ec Mon Sep 17 00:00:00 2001 From: Juan Bustamante Date: Fri, 4 Mar 2022 16:55:16 -0500 Subject: [PATCH 09/62] Apply suggestions from code review Co-authored-by: Natalie Arellano Signed-off-by: Juan Bustamante --- text/0000-export-to-oci.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/text/0000-export-to-oci.md b/text/0000-export-to-oci.md index 83cca2bd8..9c4fd1535 100644 --- a/text/0000-export-to-oci.md +++ b/text/0000-export-to-oci.md @@ -185,7 +185,7 @@ class Image { ``` ### Drawbacks -- We will have to expose Cache implementation details to the outside world, probably spec those details, for other tools to interact with this exported data +- We will have to expose Launch Cache implementation details to the outside world, probably spec those details, for other tools to interact with this exported data |Yes| B + A -->|No| END + B{IS -launch-cache defined?} -->|Yes|D + B -->|No| E + E{DOES layout-dir/image exists?} --> |Yes| L + L[...] + E --> |No| M + M[Create layout-dir/image directory] --> O[export-dir = layout-dir/image] + O --> I + D[/Warn: will export to launch cache dir/] --> F + F{DOES launch-cache/image dir exists?} -->|Yes| G + G[ ...] + F -->|No| H + H[Create launch-cache/image directory] --> N[export-dir = launch-cache/image] + N --> I[Write image to $export-dir in OCI format **] + I --> J[Calculate manifest's digest] + J --> K[/Write digest into report.toml/] + K -->END((End)) + +``` + +Notes: + - WHEN `-launch-cache` flow is executed + - The content of `blobs//` MAY contain symbolic links to content saved in the launch cache to avoid duplicating files. + - The content of `blobs//` MAY reference tar files in **uncompressed** format because that's how they are saved in the cache + - WHEN `-launch-cache` IS NOT defined + - The content of `blobs//` MAY be saved in **compressed** format + #### `report.toml` (TOML) From 6ab321df0a0ae2d64e8daf8073644ea6f7dacf6f Mon Sep 17 00:00:00 2001 From: Juan Bustamante Date: Thu, 10 Mar 2022 17:43:36 -0500 Subject: [PATCH 12/62] WIP - formating issues Signed-off-by: Juan Bustamante --- text/0000-export-to-oci.md | 19 ++++++++++++------- 1 file changed, 12 insertions(+), 7 deletions(-) diff --git a/text/0000-export-to-oci.md b/text/0000-export-to-oci.md index f4d92f625..6007980fc 100644 --- a/text/0000-export-to-oci.md +++ b/text/0000-export-to-oci.md @@ -166,22 +166,28 @@ The following new input is proposed to be added to this phase |-------------------|-----------------------|--------------------------|---------------------- | `` | `CNB_LAYOUT_DIR` | "" | The root directory where the OCI image will be written. The presence of a none empty value for this environment variable will enable the feature. | - -The following flowchart explains the flow proposed +Let's see the propose flow ```mermaid flowchart - A{IS -layout OR CNB_LAYOUT_DIR defined?} -->|Yes| B + A{"IS -layout OR + CNB_LAYOUT_DIR + defined?"} -->|Yes| B A -->|No| END - B{IS -launch-cache defined?} -->|Yes|D + B{"IS -launch-cache + defined?"} -->|Yes|D B -->|No| E - E{DOES layout-dir/image exists?} --> |Yes| L + E{"DOES + layout-dir/image + exist?"} --> |Yes| L L[...] E --> |No| M M[Create layout-dir/image directory] --> O[export-dir = layout-dir/image] O --> I D[/Warn: will export to launch cache dir/] --> F - F{DOES launch-cache/image dir exists?} -->|Yes| G + F{"DOES + launch-cache/image + dir exist?"} -->|Yes| G G[ ...] F -->|No| H H[Create launch-cache/image directory] --> N[export-dir = launch-cache/image] @@ -189,7 +195,6 @@ flowchart I --> J[Calculate manifest's digest] J --> K[/Write digest into report.toml/] K -->END((End)) - ``` Notes: From 710ed11b08044fe0feb74d24b3abcb34f53486e8 Mon Sep 17 00:00:00 2001 From: Juan Bustamante Date: Wed, 16 Mar 2022 16:46:23 -0500 Subject: [PATCH 13/62] Improvements to the flowchart - Adding feedback from review - Simplify the flowchart diagram to avoid duplicated steps - Added the pending flows in the flowchart diagram Signed-off-by: Juan Bustamante --- text/0000-export-to-oci.md | 52 ++++++++++++++++++++------------------ 1 file changed, 27 insertions(+), 25 deletions(-) diff --git a/text/0000-export-to-oci.md b/text/0000-export-to-oci.md index 6007980fc..6e0bd8d41 100644 --- a/text/0000-export-to-oci.md +++ b/text/0000-export-to-oci.md @@ -63,7 +63,7 @@ Or > /cnb/lifecycle/exporter -daemon -launch-cache /launch-cache -layout /oci my-app-image ``` -The expected output is the `my-app-image` exported in [OCI Image Layout](https://github.com/opencontainers/image-spec/blob/main/image-layout.md) format into the `/launch-cache/my-app-image/` folder. +The expected output is the `my-app-image` exported in [OCI Image Layout](https://github.com/opencontainers/image-spec/blob/main/image-layout.md) format into the `/launch-cache/oci/my-app-image/` folder. ```=shell > cd /launch-cache @@ -76,14 +76,15 @@ The expected output is the `my-app-image` exported in [OCI Image Layout](https:/ │ ├── sha256:6905011516dcf4...tar │ └── sha256:83d85471d9f8a3...tar ├── staging - └── my-app-image/ - ├── blobs/ - │ └── sha256/ - │ ├── 65d9067f915e01...tar -> /launch-cache/committed/sha256:65d9067f915e01...tar - │ ├── 6905011516dcf4...tar -> /launch-cache/committed/sha256:6905011516dcf4...tar - │ └── 83d85471d9f8a3...tar -> /launch-cache/committed/sha256:83d85471d9f8a3...tar - ├── index.json - └── oci-layout + └── oci/ + └── my-app-image/ + ├── blobs/ + │ └── sha256/ + │ ├── 65d9067f915e01...tar -> /launch-cache/committed/sha256:65d9067f915e01...tar + │ ├── 6905011516dcf4...tar -> /launch-cache/committed/sha256:6905011516dcf4...tar + │ └── 83d85471d9f8a3...tar -> /launch-cache/committed/sha256:83d85471d9f8a3...tar + ├── index.json + └── oci-layout ``` @@ -172,38 +173,39 @@ Let's see the propose flow flowchart A{"IS -layout OR CNB_LAYOUT_DIR - defined?"} -->|Yes| B + defined?"} -->|Yes| BB A -->|No| END + BB["root = "] + BB --> B B{"IS -launch-cache defined?"} -->|Yes|D - B -->|No| E - E{"DOES - layout-dir/image - exist?"} --> |Yes| L - L[...] - E --> |No| M - M[Create layout-dir/image directory] --> O[export-dir = layout-dir/image] - O --> I - D[/Warn: will export to launch cache dir/] --> F + B -->|No| GG + D[/Warn: will export to launch cache dir/] --> FF + FF["root = $launch-cache"] --> GG + GG["export-dir = $root/$layout-dir/$image"] --> F F{"DOES - launch-cache/image - dir exist?"} -->|Yes| G - G[ ...] + $export-dir + exist?"} -->|Yes| P + P["Replace image at $export-dir ‡"] + P --> J F -->|No| H - H[Create launch-cache/image directory] --> N[export-dir = launch-cache/image] - N --> I[Write image to $export-dir in OCI format **] + H[Create $export-dir directory] --> I["Write image to $export-dir †"] I --> J[Calculate manifest's digest] J --> K[/Write digest into report.toml/] K -->END((End)) ``` -Notes: +Notes **†**: - WHEN `-launch-cache` flow is executed - The content of `blobs//` MAY contain symbolic links to content saved in the launch cache to avoid duplicating files. - The content of `blobs//` MAY reference tar files in **uncompressed** format because that's how they are saved in the cache - WHEN `-launch-cache` IS NOT defined - The content of `blobs//` MAY be saved in **compressed** format +Notes **‡**: + - WHEN `the image exists in the file system` + - The idea is to use [ReplaceImage](https://pkg.go.dev/github.com/google/go-containerregistry/pkg/v1/layout#Path.ReplaceImage) internally this method uses [WriteImage](https://pkg.go.dev/github.com/google/go-containerregistry/pkg/v1/layout#Path.WriteImage) which will skip to write blobs that already exits. It means the `blobs` folder MAY contain the blobs directory MAY contain blobs which are not referenced by any of the refs, which is valid according to the OCI image specification. + - `Platforms` could include a flag to clean the directory if the user desires it #### `report.toml` (TOML) From 95dc51789144826f0cc08548a1ba07eefb9435a5 Mon Sep 17 00:00:00 2001 From: Juan Bustamante Date: Thu, 28 Apr 2022 16:37:27 -0500 Subject: [PATCH 14/62] Some minor fixes Signed-off-by: Juan Bustamante --- text/0000-export-to-oci.md | 20 +++++++++++--------- 1 file changed, 11 insertions(+), 9 deletions(-) diff --git a/text/0000-export-to-oci.md b/text/0000-export-to-oci.md index 6e0bd8d41..ab88f5a82 100644 --- a/text/0000-export-to-oci.md +++ b/text/0000-export-to-oci.md @@ -39,7 +39,7 @@ This feature will help to unblock uses cases like Currently the *Exporter* writes either in an OCI image registry or a docker daemon, the idea is to add the capability to write into disk in [OCI Image Layout](https://github.com/opencontainers/image-spec/blob/main/image-layout.md) format when the new flag `-layout` or the default environment variable `CNB_LAYOUT_DIR` is set. -Let's see some examples for the propose behavior +Let's see some examples of the proposed behavior ## Examples @@ -152,9 +152,9 @@ cd oci # How it Works [how-it-works]: #how-it-works -The lifecycle phases affected by this new behavior is: [Export](https://buildpacks.io/docs/concepts/components/lifecycle/export/) +The lifecycle phase affected by this new behavior is: [Export](https://buildpacks.io/docs/concepts/components/lifecycle/export/) -At high level view the propose solution can be summarize with the following container diagram from the C4 model +At high level view the proposed solution can be summarize with the following container diagram from the C4 model ![](https://i.imgur.com/0OLSK8o.png) @@ -204,8 +204,8 @@ Notes **†**: Notes **‡**: - WHEN `the image exists in the file system` - - The idea is to use [ReplaceImage](https://pkg.go.dev/github.com/google/go-containerregistry/pkg/v1/layout#Path.ReplaceImage) internally this method uses [WriteImage](https://pkg.go.dev/github.com/google/go-containerregistry/pkg/v1/layout#Path.WriteImage) which will skip to write blobs that already exits. It means the `blobs` folder MAY contain the blobs directory MAY contain blobs which are not referenced by any of the refs, which is valid according to the OCI image specification. - - `Platforms` could include a flag to clean the directory if the user desires it + - The idea is to use the method [ReplaceImage](https://pkg.go.dev/github.com/google/go-containerregistry/pkg/v1/layout#Path.ReplaceImage). Internally this method uses [WriteImage](https://pkg.go.dev/github.com/google/go-containerregistry/pkg/v1/layout#Path.WriteImage) which will skip to write blobs that already exits. It means the `blobs` directory MAY contain blobs which are not referenced by any of the refs, which is valid according to the OCI image specification. + - `Platforms` could include a flag to clean the directory if the user desires it #### `report.toml` (TOML) @@ -225,13 +225,13 @@ Where: # Migration [migration]: #migration - +- No breaking changes were identified + # Drawbacks [drawbacks]: #drawbacks -- We could increase the disk space if we not managed the duplication of saving the layers on disk. Currently the Cache implementation (used when daemon is ON) saved the layers tarballs on disk, because the current proposal is exporting the whole image on disk it will also require more space to save the layers for the OCI format in the `blobs` folder. +- We could increase the disk space if we do not managed the duplication of saving the layers on disk. Currently the Cache implementation (used when daemon is ON) saved the layers tarballs on disk, the proposal is to references those layers in the image exporting on disk to avoid duplication. + # Alternatives [alternatives]: #alternatives @@ -249,6 +249,8 @@ Discuss prior art, both the good and bad. # Unresolved Questions [unresolved-questions]: #unresolved-questions +- Should be this change included on a previous RFC to handle multiple export targets int the Lifecycle? + END((End)) ``` +Notes **‡**: + - WHEN `the image exists in the file system` + - The idea is to use the method [ReplaceImage](https://pkg.go.dev/github.com/google/go-containerregistry/pkg/v1/layout#Path.ReplaceImage). Internally this method uses [WriteImage](https://pkg.go.dev/github.com/google/go-containerregistry/pkg/v1/layout#Path.WriteImage) which will skip to write blobs that already exists. It means the `blobs` directory MAY contain blobs which are not referenced by any of the refs, which is valid according to the OCI image specification. + - `Platforms` could include a flag to clean the directory if the user desires it + Notes **†**: - WHEN `-launch-cache` flow is executed - The content of `blobs//` MAY contain symbolic links to content saved in the launch cache to avoid duplicating files. @@ -202,11 +207,6 @@ Notes **†**: - WHEN `-launch-cache` IS NOT defined - The content of `blobs//` MAY be saved in **compressed** format -Notes **‡**: - - WHEN `the image exists in the file system` - - The idea is to use the method [ReplaceImage](https://pkg.go.dev/github.com/google/go-containerregistry/pkg/v1/layout#Path.ReplaceImage). Internally this method uses [WriteImage](https://pkg.go.dev/github.com/google/go-containerregistry/pkg/v1/layout#Path.WriteImage) which will skip to write blobs that already exits. It means the `blobs` directory MAY contain blobs which are not referenced by any of the refs, which is valid according to the OCI image specification. - - `Platforms` could include a flag to clean the directory if the user desires it - #### `report.toml` (TOML) The new information to be added into the `report.toml` file can be summarize as follows: From 460fae5cfd3bc1c96cf22248126ae34bf8806d14 Mon Sep 17 00:00:00 2001 From: Juan Bustamante Date: Mon, 6 Jun 2022 16:43:02 -0500 Subject: [PATCH 17/62] Updating the latest changes according to the latest PoC Signed-off-by: Juan Bustamante --- text/0000-export-to-oci.md | 240 +++++++++++++++++++------------------ 1 file changed, 122 insertions(+), 118 deletions(-) diff --git a/text/0000-export-to-oci.md b/text/0000-export-to-oci.md index c8e09b317..712d4f845 100644 --- a/text/0000-export-to-oci.md +++ b/text/0000-export-to-oci.md @@ -1,6 +1,6 @@ # Meta [meta]: #meta -- Name: Export to OCI format when daemon is enabled +- Name: Export to OCI format - Start Date: 2022-02-22 - Author(s): Juan Bustamante (@jbustamante) - Status: Draft @@ -12,7 +12,7 @@ # Summary [summary]: #summary -When the `Exporter` phase is invoked besides writing into the Daemon or a Registry add the capability (enable explicitly by the user) to save the image to disk in [OCI Layout](https://github.com/opencontainers/image-spec/blob/main/image-layout.md) format. +Add the capability to the `Exporter` phase to save the image to disk in [OCI Layout](https://github.com/opencontainers/image-spec/blob/main/image-layout.md) format. # Definitions [definitions]: #definitions @@ -28,16 +28,23 @@ When the `Exporter` phase is invoked besides writing into the Daemon or a Regist # Motivation [motivation]: #motivation -Implementing this new feature will help us to solve the problem of loosing information when the image is saved into the Daemon keeping the image on disk along with the metadata it can be used as input for other tools to offer more capabilities to the end users. - -This feature will help to unblock uses cases like -- OCI annotations. See [RFC](https://github.com/buildpacks/rfcs/pull/196) -- Cosign integration. See [RFC](https://github.com/buildpacks/rfcs/pull/195) +Exporting to [OCI Image Layout](https://github.com/opencontainers/image-spec/blob/main/image-layout.md) will enable new user's workflows in the top of this functionality. For example: + - Reduce the Lifecycle complexity removing the interaction with the Daemon. + - Solve the problem of loosing information when the image is saved into the Daemon keeping the image on disk along with the metadata generated by the Lifecycle. The OCI Image can be used as input for other tools to offer more capabilities to the end users. + - This feature will help to unblock uses cases like + - OCI annotations. See [RFC](https://github.com/buildpacks/rfcs/pull/196) + - Cosign integration. See [RFC](https://github.com/buildpacks/rfcs/pull/195) # What it is [what-it-is]: #what-it-is -Currently the *Exporter* writes either in an OCI image registry or a docker daemon, the idea is to add the capability to write into disk in [OCI Image Layout](https://github.com/opencontainers/image-spec/blob/main/image-layout.md) format when the new flag `-layout` or the default environment variable `CNB_LAYOUT_DIR` is set. +The target persona affected by this change are: + +- Platform implementors: because they want to offer this feature, they will have to take care of the responsibility of: + - Pull the require dependencies (runtime image for example) and pass it through the lifecycle + - Save the resulting image to the daemon + +The general idea is to produce an [OCI Image Layout](https://github.com/opencontainers/image-spec/blob/main/image-layout.md) and save it in a file system accesible from the lifecycle execution. Let's see some examples of the proposed behavior @@ -50,162 +57,159 @@ For each case, I will present two ways of invoking the new feature: For both ways the expected output is the same -### Exporting to Daemon with launch cache enabled +### Analyzing run-image from OCI layout format + +Let's suppose the `` or `CNB_OCI_PATH` environment variable points to a directory that contains the `run-image` with the following format ```=shell -> export CNB_LAYOUT=oci -> /cnb/lifecycle/exporter -daemon -launch-cache /launch-cache my-app-image +oci +├── cnbs +   └── sample-stack-run +   ├── blobs +   │   └── sha256 +   │   ├── 1f59171fcf9a8c2a78192d4dfbf88e6f0258e24f21798ffe015c07682d3a944a +   │   ├── 219480c73c20efd82d425207e7555eba09cb113c845250cef09ba376e1dd506e +   │   ├── 63882e64890d1adea5824dbb9bc6a812140d6b85752bf74a0a26fca78e1baf5a +   │   ├── 63cb781c9d22ed765584437fea6db568f79be248ce40fc6695017d3ef3e8caaa +   │   └── 6b534265b45a4c09a5e05dde5ccf2450ea505481658a1c9ae9e0aae32362e941 +   ├── index.json +   └── oci-layout ``` -Or +And the exporter is invoked as follows ```=shell -> /cnb/lifecycle/exporter -daemon -launch-cache /launch-cache -layout /oci my-app-image +> export CNB_USE_OCI=true +> /cnb/lifecycle/analyzer -run-image cnbs/sample-stack-run:bionic my-app-image ``` -The expected output is the `my-app-image` exported in [OCI Image Layout](https://github.com/opencontainers/image-spec/blob/main/image-layout.md) format into the `/launch-cache/oci/my-app-image/` folder. +Or ```=shell -> cd /launch-cache -> tree . -. -└── launch-cache/ - ├── committed/ - │ ├── io.buildpacks.lifecycle.cache.metadata - │ ├── sha256:65d9067f915e01...tar - │ ├── sha256:6905011516dcf4...tar - │ └── sha256:83d85471d9f8a3...tar - ├── staging - └── oci/ - └── my-app-image/ - ├── blobs/ - │ └── sha256/ - │ ├── 65d9067f915e01...tar -> /launch-cache/committed/sha256:65d9067f915e01...tar - │ ├── 6905011516dcf4...tar -> /launch-cache/committed/sha256:6905011516dcf4...tar - │ └── 83d85471d9f8a3...tar -> /launch-cache/committed/sha256:83d85471d9f8a3...tar - ├── index.json - └── oci-layout - +> /cnb/lifecycle/analyzer -oci -run-image cnbs/sample-stack-run:bionic my-app-image ``` -### Exporting to Daemon without launch cache enabled +The expected output is the analysis metadata [analyzed.toml](https://github.com/buildpacks/spec/blob/main/platform.md#analyzedtoml-toml) with a content similar to this one: -```=shell -> export CNB_LAYOUT=oci -> /cnb/lifecycle/exporter -daemon my-app-image +```=TOML +[run-image] + reference = "sha256:eed6d69be53111ad1d7d3f5d1c037350e6807986feb479a67f36b15f9205a56d" ``` -Or +Where the [ImageID](https://github.com/opencontainers/image-spec/blob/main/config.md#imageid) is calculated according to the OCI specification -```=shell -> /cnb/lifecycle/exporter -daemon -layout oci my-app-image -``` -The expected output is the `my-app-image` exported in [OCI Image Layout](https://github.com/opencontainers/image-spec/blob/main/image-layout.md) format into the `/oci/` folder +### Exporting to OCI layout format + +Similar to the previous example, let's suppose the `` or `CNB_OCI_PATH` environment variable points to a directory that contains the `run-image` with the following format ```=shell -cd oci -> tree . -. -└── oci/ - └── my-app-image/ - ├── blobs/ - │ └── sha256/ - │ ├── 65d9067f915e01...tar - │ ├── 6905011516dcf4...tar - │ └── 83d85471d9f8a3...tar - ├── index.json - └── oci-layout +oci +├── cnbs +   └── sample-stack-run +   ├── blobs +   │   └── sha256 +   │   ├── 1f59171fcf9a8c2a78192d4dfbf88e6f0258e24f21798ffe015c07682d3a944a +   │   ├── 219480c73c20efd82d425207e7555eba09cb113c845250cef09ba376e1dd506e +   │   ├── 63882e64890d1adea5824dbb9bc6a812140d6b85752bf74a0a26fca78e1baf5a +   │   ├── 63cb781c9d22ed765584437fea6db568f79be248ce40fc6695017d3ef3e8caaa +   │   └── 6b534265b45a4c09a5e05dde5ccf2450ea505481658a1c9ae9e0aae32362e941 +   ├── index.json +   └── oci-layout ``` -### Exporting to a Registry +And the exporter is invoked as follows ```=shell > export CNB_LAYOUT=oci -> /cnb/lifecycle/exporter gcr.io/my-repo/my-app-image +> /cnb/lifecycle/exporter -run-image cnbs/sample-stack-run:bionic my-app-image ``` Or ```=shell -> /cnb/lifecycle/exporter -layout oci gcr.io/my-repo/my-app-image +> /cnb/lifecycle/exporter -oci -run-image cnbs/sample-stack-run:bionic my-app-image ``` -The expected output is the `my-app-image` exported in [OCI Image Layout](https://github.com/opencontainers/image-spec/blob/main/image-layout.md) format into the `/oci/` folder +The expected output is the application `` exported in [OCI Image Layout](https://github.com/opencontainers/image-spec/blob/main/image-layout.md) format ```=shell -cd oci -> tree . -. -└── oci/ - └── my-app-image/ - ├── blobs/ - │ └── sha256/ - │ ├── 65d9067f915e01...tar - │ ├── 6905011516dcf4...tar - │ └── 83d85471d9f8a3...tar - ├── index.json - └── oci-layout +├── cnbs +│   └── sample-stack-run +│   ├── blobs +│   │   └── sha256 +│   │   ├── 1f59171fcf9a8c2a78192d4dfbf88e6f0258e24f21798ffe015c07682d3a944a +│   │   ├── 219480c73c20efd82d425207e7555eba09cb113c845250cef09ba376e1dd506e +│   │   ├── 63882e64890d1adea5824dbb9bc6a812140d6b85752bf74a0a26fca78e1baf5a +│   │   ├── 63cb781c9d22ed765584437fea6db568f79be248ce40fc6695017d3ef3e8caaa +│   │   └── 6b534265b45a4c09a5e05dde5ccf2450ea505481658a1c9ae9e0aae32362e941 +│   ├── index.json +│   └── oci-layout +└── my-app-image + ├── blobs + │   └── sha256 + │   ├── 14eaea7168b1fc4b8b30f7a20f7609335cc3dbcfb6d4c1afeb1e5daefd26cdf9 + │   ├── 219480c73c20efd82d425207e7555eba09cb113c845250cef09ba376e1dd506e -> ../../../cnbs/sample-stack-run/blobs/sha256/219480c73c20efd82d425207e7555eba09cb113c845250cef09ba376e1dd506e + │   ├── 410ce2030625414163d565a56bddc6587dbc49fa2a815af5e26bb08968dec7d2 + │   ├── 54890b0245f8b234be46817f731b7b981d9ee2ea8d5a76380d91bb9abb001cdb + │   ├── 58b81a67d77e732ef07c4b995b6a7e099e5aa772d1805da7cf929d80a3fa044e + │   ├── 63cb781c9d22ed765584437fea6db568f79be248ce40fc6695017d3ef3e8caaa -> ../../../cnbs/sample-stack-run/blobs/sha256/63cb781c9d22ed765584437fea6db568f79be248ce40fc6695017d3ef3e8caaa + │   ├── 68224c8aa806b9ec8cecf2282b7b10cdf1c615785652bcc85f3a79bd06e60384 + │   ├── 6977801a3b8ac0c4a76ea22fa4dc0541fe8fc6ba964883ee0a21b5736f8acee9 + │   ├── 6b534265b45a4c09a5e05dde5ccf2450ea505481658a1c9ae9e0aae32362e941 -> ../../../cnbs/sample-stack-run/blobs/sha256/6b534265b45a4c09a5e05dde5ccf2450ea505481658a1c9ae9e0aae32362e941 + │   ├── 7c83d7d24ac43dbdd419d15abb849b6d155d0bc361eca0908a8ae5aefcd55557 + │   └── addd2357f3a0175410ab8f9303e747cc72af9aaeb0e7402d3fd3f144adb29db5 + ├── index.json + └── oci-layout +``` + +Also, we are proposing to use symbolic links to reference the blobs from the `run-image` this will help to save some hard drive space for the end user. + +### Using -oci flag in combination with --daemon or --publish flags + +Any combination of using multiple sources or sinks in the Lifecycle invocation of phases should throw an error to the user. For example: +```=shell +> /cnb/lifecycle/exporter -oci -daemon -run-image cnbs/sample-stack-run:bionic my-app-image + +ERROR: exporting to multiples target is not allowed ``` # How it Works [how-it-works]: #how-it-works -The lifecycle phase affected by this new behavior is: [Export](https://buildpacks.io/docs/concepts/components/lifecycle/export/) +Before defining the detail behavior, lets remember some required terminology + +**Terminology** -At high level view the proposed solution can be summarize with the following container diagram from the C4 model +- An **image reference** refers to either a tag reference or digest reference. +- A **tag reference** refers to an identifier of form `/:` which locates an image manifest in an [OCI Distribution Specification](https://github.com/opencontainers/distribution-spec/blob/master/spec.md) compliant registry. +- A **digest reference** refers to a [content addressable](https://en.wikipedia.org/wiki/Content-addressable_storage) identifier of form `/@` which locates an image manifest in an [OCI Distribution Specification](https://github.com/opencontainers/distribution-spec/blob/master/spec.md) compliant registry. -![](https://i.imgur.com/0OLSK8o.png) +The lifecycle phases affected by this new behavior are: + - [Analyze](https://buildpacks.io/docs/concepts/components/lifecycle/analyze/) + - [Export](https://buildpacks.io/docs/concepts/components/lifecycle/export/) + - [Create](https://buildpacks.io/docs/concepts/components/lifecycle/create/) +At high level view the proposed solution can be summarize with the following system landscape diagram from the C4 model + +![](https://i.imgur.com/y972lTD.png) Notice that we are relying on the OCI format Specification to expose the data for `Platforms` -The following new input is proposed to be added to this phase - -| Input | Environment Variable | Default Value | Description -|-------------------|-----------------------|--------------------------|---------------------- -| `` | `CNB_LAYOUT_DIR` | "" | The root directory where the OCI image will be written. The presence of a none empty value for this environment variable will enable the feature. | - -Let's see the proposed flow - -```mermaid -flowchart - A{"IS -layout OR - CNB_LAYOUT_DIR - defined?"} -->|Yes| BB - A -->|No| END - BB["root = "] - BB --> B - B{"IS -launch-cache - defined?"} -->|Yes|D - B -->|No| GG - D[/Warn: will export to launch cache dir/] --> FF - FF["root = $launch-cache"] --> GG - GG["export-dir = $root/$layout-dir/$image"] --> F - F{"DOES - $export-dir - exist?"} -->|Yes| P - P["Replace image at $export-dir ‡"] - P --> J - F -->|No| H - H[Create $export-dir directory] --> I["Write image to $export-dir †"] - I --> J[Calculate manifest's digest] - J --> K[/Write digest into report.toml/] - K -->END((End)) -``` +The following new input are proposed to be added to these phases -Notes **‡**: - - WHEN `the image exists in the file system` - - The idea is to use the method [ReplaceImage](https://pkg.go.dev/github.com/google/go-containerregistry/pkg/v1/layout#Path.ReplaceImage). Internally this method uses [WriteImage](https://pkg.go.dev/github.com/google/go-containerregistry/pkg/v1/layout#Path.WriteImage) which will skip to write blobs that already exists. It means the `blobs` directory MAY contain blobs which are not referenced by any of the refs, which is valid according to the OCI image specification. - - `Platforms` could include a flag to clean the directory if the user desires it + | Input | Environment Variable | Default Value | Description + |-------------------|-----------------------|--------------------------|---------------------- + | `` | `CNB_USE_OCI` | false | Analyze or Export image from/to OCI layout format on disk | + | `` | `CNB_OCI_PATH` | /oci | Path to oci directory where the images are saved | -Notes **†**: - - WHEN `-launch-cache` flow is executed - - The content of `blobs//` MAY contain symbolic links to content saved in the launch cache to avoid duplicating files. - - The content of `blobs//` MAY reference tar files in **uncompressed** format because that's how they are saved in the cache - - WHEN `-launch-cache` IS NOT defined - - The content of `blobs//` MAY be saved in **compressed** format +- WHEN `the new flag -oci or the default environment variable CNB_USE_OCI are set to true` during the phase invocation THEN the feature will be enabled. +The image look up will be done following this rules: + - WHEN `the image points to a tag reference` + - Lifecycle will load the image from disk in [OCI Image Layout](https://github.com/opencontainers/image-spec/blob/main/image-layout.md) format at `///` + - WHEN `the image points to a digest reference` + - Lifecycle will load the image from disk in [OCI Image Layout](https://github.com/opencontainers/image-spec/blob/main/image-layout.md) format at `///` #### `report.toml` (TOML) @@ -218,7 +222,7 @@ digest = "" manifest-size = "" ``` Where: -- **If** the app image was exported using the `-layer` flag, the export section will be added to the report +- **If** the app image was exported to OCI format on disk, the export section will be added to the report - `digest` MUST contain the image digest calculated based on compressed layers - `manifest-size` MUST contain the manifest size in bytes From 580c8ba85b5ab67a9bf5bade7198c16f945fc8cc Mon Sep 17 00:00:00 2001 From: Juan Bustamante Date: Mon, 6 Jun 2022 16:54:56 -0500 Subject: [PATCH 18/62] Formatting changes Signed-off-by: Juan Bustamante --- text/0000-export-to-oci.md | 23 ++++++++--------------- 1 file changed, 8 insertions(+), 15 deletions(-) diff --git a/text/0000-export-to-oci.md b/text/0000-export-to-oci.md index 712d4f845..1f4bc4d9c 100644 --- a/text/0000-export-to-oci.md +++ b/text/0000-export-to-oci.md @@ -19,10 +19,12 @@ Add the capability to the `Exporter` phase to save the image to disk in [OCI Lay - A [Platform](https://buildpacks.io/docs/concepts/components/platform/) uses a lifecycle, Buildpacks (packaged in a builder), and application source code to produce an OCI image. - A [Lifecycle](https://buildpacks.io/docs/concepts/components/lifecycle/) orchestrates Buildpacks execution, then assembles the resulting artifacts into a final app image. -- A Daemon is a service, popularized by Docker, for downloading container images, and executing and managing containers from those images. -- A Registry is a long-running service used for storing and retrieving container images. -- A digest reference refers to a [content addressable](https://en.wikipedia.org/wiki/Content-addressable_storage) identifier of form /@ which locates an image manifest in an [OCI Distribution Specification](https://github.com/opencontainers/distribution-spec/blob/master/spec.md) compliant registry. -- A Image Manifest provides a configuration and set of layers for a single container image for a specific architecture and operating system. +- A **Daemon** is a service, popularized by Docker, for downloading container images, and executing and managing containers from those images. +- A **Registry** is a long-running service used for storing and retrieving container images. +- An **image reference** refers to either a tag reference or digest reference. +- A **tag reference** refers to an identifier of form `/:` which locates an image manifest in an [OCI Distribution Specification](https://github.com/opencontainers/distribution-spec/blob/master/spec.md) compliant registry. +- A **digest reference** refers to a [content addressable](https://en.wikipedia.org/wiki/Content-addressable_storage) identifier of form `/@` which locates an image manifest in an [OCI Distribution Specification](https://github.com/opencontainers/distribution-spec/blob/master/spec.md) compliant registry. +- A **image Manifest** provides a configuration and set of layers for a single container image for a specific architecture and operating system. - An [OCI Image Layout](https://github.com/opencontainers/image-spec/blob/main/image-layout.md) is the directory structure for OCI content-addressable blobs and location-addressable references. # Motivation @@ -40,11 +42,10 @@ Exporting to [OCI Image Layout](https://github.com/opencontainers/image-spec/blo The target persona affected by this change are: -- Platform implementors: because they want to offer this feature, they will have to take care of the responsibility of: +- **Platform implementors**: they will have to take care of the responsibility of: - Pull the require dependencies (runtime image for example) and pass it through the lifecycle - - Save the resulting image to the daemon -The general idea is to produce an [OCI Image Layout](https://github.com/opencontainers/image-spec/blob/main/image-layout.md) and save it in a file system accesible from the lifecycle execution. +The general idea is to produce an [OCI Image Layout](https://github.com/opencontainers/image-spec/blob/main/image-layout.md) and save it in a file system accesible from the lifecycle execution. Let's see some examples of the proposed behavior @@ -178,14 +179,6 @@ ERROR: exporting to multiples target is not allowed # How it Works [how-it-works]: #how-it-works -Before defining the detail behavior, lets remember some required terminology - -**Terminology** - -- An **image reference** refers to either a tag reference or digest reference. -- A **tag reference** refers to an identifier of form `/:` which locates an image manifest in an [OCI Distribution Specification](https://github.com/opencontainers/distribution-spec/blob/master/spec.md) compliant registry. -- A **digest reference** refers to a [content addressable](https://en.wikipedia.org/wiki/Content-addressable_storage) identifier of form `/@` which locates an image manifest in an [OCI Distribution Specification](https://github.com/opencontainers/distribution-spec/blob/master/spec.md) compliant registry. - The lifecycle phases affected by this new behavior are: - [Analyze](https://buildpacks.io/docs/concepts/components/lifecycle/analyze/) - [Export](https://buildpacks.io/docs/concepts/components/lifecycle/export/) From d7f4ec80bd6c937b5eb4db0faa6a56a4944f1d1c Mon Sep 17 00:00:00 2001 From: Juan Bustamante Date: Mon, 6 Jun 2022 17:12:44 -0500 Subject: [PATCH 19/62] Some other changes Signed-off-by: Juan Bustamante --- text/0000-export-to-oci.md | 22 ++++++++++------------ 1 file changed, 10 insertions(+), 12 deletions(-) diff --git a/text/0000-export-to-oci.md b/text/0000-export-to-oci.md index 1f4bc4d9c..28b59c83c 100644 --- a/text/0000-export-to-oci.md +++ b/text/0000-export-to-oci.md @@ -45,7 +45,7 @@ The target persona affected by this change are: - **Platform implementors**: they will have to take care of the responsibility of: - Pull the require dependencies (runtime image for example) and pass it through the lifecycle -The general idea is to produce an [OCI Image Layout](https://github.com/opencontainers/image-spec/blob/main/image-layout.md) and save it in a file system accesible from the lifecycle execution. +The general idea is to produce an image in [OCI Image Layout](https://github.com/opencontainers/image-spec/blob/main/image-layout.md) format and save it in a file system accesible from the lifecycle execution. Let's see some examples of the proposed behavior @@ -77,7 +77,7 @@ oci    └── oci-layout ``` -And the exporter is invoked as follows +And the analyzer is invoked as follows ```=shell > export CNB_USE_OCI=true @@ -203,10 +203,11 @@ The image look up will be done following this rules: - Lifecycle will load the image from disk in [OCI Image Layout](https://github.com/opencontainers/image-spec/blob/main/image-layout.md) format at `///` - WHEN `the image points to a digest reference` - Lifecycle will load the image from disk in [OCI Image Layout](https://github.com/opencontainers/image-spec/blob/main/image-layout.md) format at `///` + - A part from the look up, the logic for each phase should remain the same #### `report.toml` (TOML) -The new information to be added into the `report.toml` file can be summarize as follows: +The new information to be added into the `report.toml` file can be summarize as follows: ```toml [export] @@ -227,27 +228,24 @@ Where: # Drawbacks [drawbacks]: #drawbacks -- We could increase the disk space if we do not managed the duplication of saving the layers on disk. Currently the Cache implementation (used when daemon is ON) saved the layers tarballs on disk, the proposal is to references those layers in the image exporting on disk to avoid duplication. - +- We could increase the disk space if we do not managed the duplication of saving the layers on disk. The proposal suggest to use symbolic links to reference layers on disk and avoid duplication. # Alternatives [alternatives]: #alternatives - - +- What other designs have been considered? + - Doing nothing, just keep exporting only to Daemon or Registry +- Why is this proposal the best? [OCI Image Layout](https://github.com/opencontainers/image-spec/blob/main/image-layout.md) format is an standard from which other tools can create a [OCI Runtime Specification bundle](https://github.com/opencontainers/runtime-spec/blob/v1.0.0/bundle.md) exporting to this format enables Platforms to implement any feature in the top of this format +- What is the impact of not doing this? Probably will never remove the Daemon support in the Lifecycle # Prior Art [prior-art]: #prior-art -Discuss prior art, both the good and bad. +- Discussion around removing the Daemon support [RFC](https://github.com/buildpacks/rfcs/blob/jjbustamante/feature/deprecate-daemon/text/0000-deprecate-daemon.md) # Unresolved Questions [unresolved-questions]: #unresolved-questions -- Should this change be included on a previous RFC to handle multiple export targets int the Lifecycle? -- Currently the *Launch Cache* saves uncompressed tarballs. Is this by design? Is there any reason for those tarballs to do not be saved compressed? -Let's suppose the `` or `CNB_OCI_PATH` environment variable points to a directory that contains the `run-image` with the following format +Let's suppose a directory exits and it has the following structure: ```=shell oci -├── cnbs -   └── sample-stack-run -   ├── blobs -   │   └── sha256 -   │   ├── 1f59171fcf9a8c2a78192d4dfbf88e6f0258e24f21798ffe015c07682d3a944a -   │   ├── 219480c73c20efd82d425207e7555eba09cb113c845250cef09ba376e1dd506e -   │   ├── 63882e64890d1adea5824dbb9bc6a812140d6b85752bf74a0a26fca78e1baf5a -   │   ├── 63cb781c9d22ed765584437fea6db568f79be248ce40fc6695017d3ef3e8caaa -   │   └── 6b534265b45a4c09a5e05dde5ccf2450ea505481658a1c9ae9e0aae32362e941 -   ├── index.json -   └── oci-layout +├── cnb +│ └── my-stack-run +│ ├── blobs +│ │ └── sha256 +│ │ └── 1bcd..x +│ ├── index.json +│ └── oci-layout +├── foo +│ └── my-cache +│ ├── blobs +│ │ └── sha256 +│ │ └── 1f591..a +│ ├── index.json +│ └── oci-layout +└── bar + └── my-previous-app + ├── blobs + │ └── sha256 + │ └── 1efg..w + ├── index.json + └── oci-layout ``` -And the analyzer is invoked as follows +For each case, I will present two ways of enabling the new capability: + +- Using an environment Variable +- Using a new `oci` flag + +In any case the expected output is the same. + +#### Analyze phase + +##### Analyzing run-image ```=shell > export CNB_USE_OCI=true -> /cnb/lifecycle/analyzer -run-image cnbs/sample-stack-run:bionic my-app-image -``` +> /cnb/lifecycle/analyzer -run-image cnb/my-stack-run:bionic my-app-image -Or +# OR -```=shell -> /cnb/lifecycle/analyzer -oci -run-image cnbs/sample-stack-run:bionic my-app-image -``` +> /cnb/lifecycle/analyzer -oci -run-image cnb/my-stack-run:bionic my-app-image -The expected output is the analysis metadata [analyzed.toml](https://github.com/buildpacks/spec/blob/main/platform.md#analyzedtoml-toml) with a content similar to this one: +# expected output +# analyzed.toml file with the usual `run-image.reference` -```=TOML -[run-image] - reference = "sha256:eed6d69be53111ad1d7d3f5d1c037350e6807986feb479a67f36b15f9205a56d" ``` -Where the [ImageID](https://github.com/opencontainers/image-spec/blob/main/config.md#imageid) is calculated according to the OCI specification +##### Analyzing previous-image + ```=shell +> export CNB_USE_OCI=true +> /cnb/lifecycle/analyzer -run-image cnb/my-stack-run:bionic -previous-image bar/my-previous-app my-app-image + +# OR -### Exporting to OCI layout format +> /cnb/lifecycle/analyzer -oci -run-image cnb/my-stack-run:bionic-previous-image bar/my-previous-app my-app-image -Similar to the previous example, let's suppose the `` or `CNB_OCI_PATH` environment variable points to a directory that contains the `run-image` with the following format +# expected output +# analyzed.toml file with the usual `run-image.reference` and `previos-image.reference` +``` + + +##### Analyzing run-image not saved on disk ```=shell > export CNB_USE_OCI=true -> /cnb/lifecycle/exporter -run-image cnbs/sample-stack-run:bionic my-app-image +> /cnb/lifecycle/analyzer -run-image cnb/bad-run-image my-app-image + +# OR + +> /cnb/lifecycle/analyzer -oci -run-image cnb/bad-run-image my-app-image + +# expected output + +ERROR: the run-image could not be found at path: /oci/cnb/bad-run-image ``` -Or +##### Analyzing without run-image argument ```=shell -> /cnb/lifecycle/exporter -oci -run-image cnbs/sample-stack-run:bionic my-app-image +> export CNB_USE_OCI=true +> /cnb/lifecycle/analyzer my-app-image + +# OR + +> /cnb/lifecycle/analyzer -oci my-app-image + +# expected output + +ERROR: -run-image is required when OCI feature is enabled ``` -The expected output is the application `` exported in [OCI Image Layout](https://github.com/opencontainers/image-spec/blob/main/image-layout.md) format +#### Export phase + +##### Export to OCI + ```=shell +> export CNB_USE_OCI=true +> /cnb/lifecycle/exporter -run-image cnb/my-stack-run:bionic my-app-image + +# OR + +> /cnb/lifecycle/exporter -oci -run-image cnb/my-stack-run:bionic my-app-image + +# expected output +# the store folder will be updated with the application image as follows + oci -├── cnbs - │   └── sample-stack-run - │   ├── blobs - │   │   └── sha256 - │   │   ├── 1f59171fcf9a8c2a78192d4dfbf88e6f0258e24f21798ffe015c07682d3a944a - │   │   ├── 219480c73c20efd82d425207e7555eba09cb113c845250cef09ba376e1dd506e - │   │   ├── 63882e64890d1adea5824dbb9bc6a812140d6b85752bf74a0a26fca78e1baf5a - │   │   ├── 63cb781c9d22ed765584437fea6db568f79be248ce40fc6695017d3ef3e8caaa - │   │   └── 6b534265b45a4c09a5e05dde5ccf2450ea505481658a1c9ae9e0aae32362e941 - │   ├── index.json - │   └── oci-layout - └── my-app-image - ├── blobs - │   └── sha256 - │   ├── 14eaea7168b1fc4b8b30f7a20f7609335cc3dbcfb6d4c1afeb1e5daefd26cdf9 - │   ├── 219480c73c20efd82d425207e7555eba09cb113c845250cef09ba376e1dd506e -> ../../../cnbs/sample-stack-run/blobs/sha256/219480c73c20efd82d425207e7555eba09cb113c845250cef09ba376e1dd506e - │   ├── 410ce2030625414163d565a56bddc6587dbc49fa2a815af5e26bb08968dec7d2 - │   ├── 54890b0245f8b234be46817f731b7b981d9ee2ea8d5a76380d91bb9abb001cdb - │   ├── 58b81a67d77e732ef07c4b995b6a7e099e5aa772d1805da7cf929d80a3fa044e - │   ├── 63cb781c9d22ed765584437fea6db568f79be248ce40fc6695017d3ef3e8caaa -> ../../../cnbs/sample-stack-run/blobs/sha256/63cb781c9d22ed765584437fea6db568f79be248ce40fc6695017d3ef3e8caaa - │   ├── 68224c8aa806b9ec8cecf2282b7b10cdf1c615785652bcc85f3a79bd06e60384 - │   ├── 6977801a3b8ac0c4a76ea22fa4dc0541fe8fc6ba964883ee0a21b5736f8acee9 - │   ├── 6b534265b45a4c09a5e05dde5ccf2450ea505481658a1c9ae9e0aae32362e941 -> ../../../cnbs/sample-stack-run/blobs/sha256/6b534265b45a4c09a5e05dde5ccf2450ea505481658a1c9ae9e0aae32362e941 - │   ├── 7c83d7d24ac43dbdd419d15abb849b6d155d0bc361eca0908a8ae5aefcd55557 - │   └── addd2357f3a0175410ab8f9303e747cc72af9aaeb0e7402d3fd3f144adb29db5 - ├── index.json - └── oci-layout -``` +├── cnb +│ └── my-stack-run +│ ├── blobs +│ │ └── sha256 +│ │ └── 1bcd..x +│ ├── index.json +│ └── oci-layout +├── foo +│ └── // omiting for simplicity +├── bar +│ └── // omiting for simplicity +└── my-app-image + └── blobs + ├── sha256 + │ ├── 1bcd..x + │ ├── 2f789..d + │ └── 3g234..f + ├── index.json + └── oci-layout -Also, we are proposing to use symbolic links to reference the blobs from the `run-image` as this will help to save some hard drive space for the end user. +``` -### Using -oci flag in combination with --daemon or --publish flags +##### Using -oci flag in combination with --daemon or --publish flags Any combination of using multiple sources or sinks in the Lifecycle invocation of phases should throw an error to the user. For example: ```=shell -> /cnb/lifecycle/exporter -oci -daemon -run-image cnbs/sample-stack-run:bionic my-app-image +> export CNB_USE_OCI=true +> /cnb/lifecycle/exporter -daemon -run-image cnb/my-stack-run:bionic my-app-image + +# OR + +> /cnb/lifecycle/exporter -oci -daemon -run-image cnb/my-stack-run:bionic my-app-image ERROR: exporting to multiples target is not allowed ``` @@ -197,8 +264,8 @@ Notice that we are relying on the OCI format Specification to expose the data fo The following new inputs are proposed to be added to these phases - | Input | Environment Variable | Default Value | Description - |-------------------|-----------------------|--------------------------|---------------------- + | Input | Environment Variable | Default Value | Description + |-------|-----------------------|---------------|-------------- | `` | `CNB_USE_OCI` | false | Analyze or Export image from/to OCI layout format on disk | | `` | `CNB_OCI_PATH` | /oci | Path to oci directory where the images are saved | From f9e225f8108b8496a6a77d81fd079a73ef3ab7e7 Mon Sep 17 00:00:00 2001 From: Juan Bustamante Date: Mon, 31 Oct 2022 12:09:03 -0500 Subject: [PATCH 45/62] removing the plan for daemon removal Signed-off-by: Juan Bustamante --- text/0000-export-to-oci.md | 22 ++++------------------ 1 file changed, 4 insertions(+), 18 deletions(-) diff --git a/text/0000-export-to-oci.md b/text/0000-export-to-oci.md index a4ac5cc82..6fb8a9346 100644 --- a/text/0000-export-to-oci.md +++ b/text/0000-export-to-oci.md @@ -2,7 +2,7 @@ [meta]: #meta - Name: Export to OCI format - Start Date: 2022-02-22 -- Author(s): Juan Bustamante (@jbustamante) +- Author(s): Juan Bustamante (@jjbustamante) - Status: Draft - RFC Pull Request: (leave blank) - CNB Pull Request: (leave blank) @@ -277,6 +277,9 @@ The image look up will be done following these rules: - Lifecycle will load the image from disk in [OCI Image Layout](https://github.com/opencontainers/image-spec/blob/main/image-layout.md) format at `///` - Apart from the look up, the logic for each phase should remain the same + + + ## Proof of concept In order to validate the feasibility of the proposed feature, we developed a proof of concept with one of the most important side effects this capability can add into the project: **Removing the Daemon Support**. You can also check a recording with the demo in the following [link](https://drive.google.com/file/d/1W1125OHuyUlx88BRroUTLBfrFHhFM5A9/view?usp=sharing) @@ -334,23 +337,6 @@ I think, this PoC demonstrate that adding the exporting to OCI layout format is - No breaking changes were identified -## For removing Daemon support in near future - -I propose the following high level strategy to accomplish the goal - -- Approve this RFC and add the export to OCI layout format feature in Lifecycle -- Implement in Pack a new workflow when the `-daemon` flag is enabled based on what we did with the PoC. The workflow could be enabled by the user in **experimental mode** - - Make noise in the community so people can help us using it - - Collect feedback about it - - Correct issues until we feel confortable -- Turn off the actual `-daemon` implementation and replace it with the `-oci` by default in Pack - - Maybe we can also add a new flag to **re-use the old implementation** in case the user needs it - - Keep doing noise in the community - - Collect feedback about it - - Correct issues until we feel confortable -- Schedule the removal of the Daemon support in Lifecycle version X.Y.Z. -- Remove the actual daemon support from the code base for Lifecycle version X.Y.Z - # Drawbacks [drawbacks]: #drawbacks From f79bb237770c38fbaeec5a6915ffa13787c6b691 Mon Sep 17 00:00:00 2001 From: Juan Bustamante Date: Thu, 3 Nov 2022 09:46:41 -0500 Subject: [PATCH 46/62] adding the possibility to allow partial images on disk Signed-off-by: Juan Bustamante --- text/0000-export-to-oci.md | 10 ++++++++-- 1 file changed, 8 insertions(+), 2 deletions(-) diff --git a/text/0000-export-to-oci.md b/text/0000-export-to-oci.md index 6fb8a9346..7c922ac74 100644 --- a/text/0000-export-to-oci.md +++ b/text/0000-export-to-oci.md @@ -43,7 +43,7 @@ The current process, executed by the lifecycle, does not take into consideration ### What use cases does it support? Exporting to [OCI Image Layout](https://github.com/opencontainers/image-spec/blob/main/image-layout.md) will enable new user's workflows on top of this functionality. For example: - - Reduce the Lifecycle complexity removing the interaction with the Daemon. + - It provides a mechanism to reduce the Lifecycle complexity by removing the interaction with the Daemon in the future. - Solve the problem of losing information when the image is saved into the Daemon, keeping the image on disk along with the metadata generated by the Lifecycle. The OCI Image can be used as input for other tools to offer more capabilities to the end users. - This feature will help to unblock uses cases like - OCI annotations. See [RFC](https://github.com/buildpacks/rfcs/pull/196) @@ -52,7 +52,11 @@ Exporting to [OCI Image Layout](https://github.com/opencontainers/image-spec/blo ### What is the expected outcome? -Lifecycle will be capable of exporting application image into disk in [OCI Image Layout](https://github.com/opencontainers/image-spec/blob/main/image-layout.md) format. +Lifecycle will be capable of exporting the application image into disk in [OCI Image Layout](https://github.com/opencontainers/image-spec/blob/main/image-layout.md) format. The image saved on disk could have the following considerations: + +- The `blobs` directory MAY be missing `base image` or `run image` blobs. These layers may not be needed on disk as they could be already accessible in a blob store. +- The `blobs` directory SHOULD always have buildpacks generated `blobs`. + # What it is [what-it-is]: #what-it-is @@ -63,6 +67,8 @@ The target personas affected by this change are: - **Platform implementors**: they will have to take care of the responsibility of creating a store resource on disk in [OCI Image Layout](https://github.com/opencontainers/image-spec/blob/main/image-layout.md) format and pass it through the lifecycle during the phases execution. +The process of writing any image on disk in [OCI Image Layout](https://github.com/opencontainers/image-spec/blob/main/image-layout.md) format could be expensive in terms of hard drive space or IO operation (compressing or uncompressing layers). In order to provide flexibility for the implementation, the `analyzer` or `exporter` binaries only require the *Image Manifest* and the *Image Config* to execute their operations, based on this, we proposed the Lifecycle can be configured to work with a partial representation of the images on disk, meaning that some blobs MAY be missing (which is ok according to the [OCI Image Layout](https://github.com/opencontainers/image-spec/blob/main/image-layout.md) format). The missing blobs COULD be those that are already available in a daemon or registry. + Let's see some examples of the proposed behavior ## Examples From 966488a11a5690c8112f6794ac2a8c6caa34a556 Mon Sep 17 00:00:00 2001 From: Juan Bustamante Date: Thu, 3 Nov 2022 18:19:29 -0500 Subject: [PATCH 47/62] extending the examples section Signed-off-by: Juan Bustamante --- text/0000-export-to-oci.md | 256 +++++++++++++++++++++++++++++++------ 1 file changed, 215 insertions(+), 41 deletions(-) diff --git a/text/0000-export-to-oci.md b/text/0000-export-to-oci.md index 7c922ac74..be61b28dc 100644 --- a/text/0000-export-to-oci.md +++ b/text/0000-export-to-oci.md @@ -67,7 +67,7 @@ The target personas affected by this change are: - **Platform implementors**: they will have to take care of the responsibility of creating a store resource on disk in [OCI Image Layout](https://github.com/opencontainers/image-spec/blob/main/image-layout.md) format and pass it through the lifecycle during the phases execution. -The process of writing any image on disk in [OCI Image Layout](https://github.com/opencontainers/image-spec/blob/main/image-layout.md) format could be expensive in terms of hard drive space or IO operation (compressing or uncompressing layers). In order to provide flexibility for the implementation, the `analyzer` or `exporter` binaries only require the *Image Manifest* and the *Image Config* to execute their operations, based on this, we proposed the Lifecycle can be configured to work with a partial representation of the images on disk, meaning that some blobs MAY be missing (which is ok according to the [OCI Image Layout](https://github.com/opencontainers/image-spec/blob/main/image-layout.md) format). The missing blobs COULD be those that are already available in a daemon or registry. +The process of writing any image on disk in [OCI Image Layout](https://github.com/opencontainers/image-spec/blob/main/image-layout.md) format could be expensive in terms of hard drive space or IO operation (compressing or uncompressing layers). In order to provide flexibility for the implementation, the `analyzer` or `exporter` binaries only require the *Image Manifest* and the *Image Config* to execute their operations, based on this, we proposed the Lifecycle can be configured to work with a partial representation of the images on disk, meaning that some blobs MAY be missing (which is ok according to the [OCI Image Layout](https://github.com/opencontainers/image-spec/blob/main/image-layout.md) format). The missing blobs SHOULD be those that are already available in a daemon or registry. Let's see some examples of the proposed behavior @@ -92,10 +92,21 @@ Let's suppose a directory exits and it has the following structure: ```=shell oci ├── cnb -│ └── my-stack-run +│ ├── my-full-stack-run +│ │ ├── blobs +│ │ │ └── sha256 +│ │ │ ├── 1f59...944a // manifest +│ │ │ ├── 6388...af5a // config +│ │ │ ├── 824b...f984e +│ │ │ ├── f5f7...5b38 +│ │ │ └── 870e...f1b09 +│ │ ├── index.json +│ │ └── oci-layout +│ └── my-partial-stack-run │ ├── blobs │ │ └── sha256 -│ │ └── 1bcd..x +│ │ ├── 1f59...944a // manifest +│ │ └── 6388...af5a // config │ ├── index.json │ └── oci-layout ├── foo @@ -105,16 +116,29 @@ oci │ │ └── 1f591..a │ ├── index.json │ └── oci-layout -└── bar - └── my-previous-app - ├── blobs - │ └── sha256 - │ └── 1efg..w +├── bar +│ └── my-previous-app +│ ├── blobs +│ │ └── sha256 +│ │ └── 1efg..w +│ ├── index.json +│ └── oci-layout +└── my-app-image + └── blobs + ├── sha256 + │ ├── 1bcd5..x // app image manifest + │ ├── 2f789..d // app image config + │ ├── 824b...f984e // run layer + │ ├── f5f7...5b38 // run layer + │ ├── 870e...f1b09 // run layer + │ └── 3g234..f // buildpack layer ├── index.json └── oci-layout ``` -For each case, I will present two ways of enabling the new capability: +The images named **cnb/my-full-stack-run** and **cnb/my-partial-stack-run** represents the same image but the partial one has missing `blobs`, those `blobs` are the layers that are already available in the store from it came from. + +For each example case, I will present two ways of enabling the new capability: - Using an environment Variable - Using a new `oci` flag @@ -123,52 +147,51 @@ In any case the expected output is the same. #### Analyze phase -##### Analyzing run-image +##### Analyzing run-image full saved on disk ```=shell > export CNB_USE_OCI=true -> /cnb/lifecycle/analyzer -run-image cnb/my-stack-run:bionic my-app-image +> /cnb/lifecycle/analyzer -run-image cnb/my-full-stack-run:bionic my-app-image # OR -> /cnb/lifecycle/analyzer -oci -run-image cnb/my-stack-run:bionic my-app-image +> /cnb/lifecycle/analyzer -oci -run-image cnb/my-full-stack-run:bionic my-app-image # expected output # analyzed.toml file with the usual `run-image.reference` ``` -##### Analyzing previous-image +##### Analyzing run-image partial saved on disk - ```=shell +```=shell > export CNB_USE_OCI=true -> /cnb/lifecycle/analyzer -run-image cnb/my-stack-run:bionic -previous-image bar/my-previous-app my-app-image +> /cnb/lifecycle/analyzer -run-image cnb/cnb/my-partial-stack-run:bionic my-app-image # OR -> /cnb/lifecycle/analyzer -oci -run-image cnb/my-stack-run:bionic-previous-image bar/my-previous-app my-app-image +> /cnb/lifecycle/analyzer -oci -run-image cnb/cnb/my-partial-stack-run:bionic my-app-image # expected output -# analyzed.toml file with the usual `run-image.reference` and `previos-image.reference` +# analyzed.toml file with the usual `run-image.reference` +# there should be no change behavior in the phase even thought the image has missing blobs + ``` - - ##### Analyzing run-image not saved on disk ```=shell @@ -201,54 +224,96 @@ ERROR: -run-image is required when OCI feature is enabled #### Export phase -##### Export to OCI +##### Export to OCI using run-image full saved on disk +```=shell +> export CNB_USE_OCI=true +> /cnb/lifecycle/exporter -run-image cnb/my-full-stack-run:bionic my-app-image + +# OR + +> /cnb/lifecycle/exporter -oci -run-image cnb/my-full-stack-run:bionic my-app-image + +# expected output +# the store folder will be updated with the application image as follows + +oci +├── cnb +│ └── my-full-stack-run +│ ├── blobs +│ │ └── sha256 +│ │ ├── 1f59...944a // manifest +│ │ ├── 6388...af5a // config +│ │ ├── 824b...f984e +│ │ ├── f5f7...5b38 +│ │ └── 870e...f1b09 +│ ├── index.json +│ └── oci-layout +├── // some folders omitted for simplicity +└── my-app-image + └── blobs + ├── sha256 + │ ├── 1bcd5..x // app image manifest + │ ├── 2f789..d // app image config + │ ├── 824b...f984e // run layer + │ ├── f5f7...5b38 // run layer + │ ├── 870e...f1b09 // run layer + │ └── 3g234..f // buildpack layer + ├── index.json + └── oci-layout + +``` + +As we can see, the application image `my-app-image` contains a **full** copy of the layers in its `blobs` folder. + + +##### Export to OCI using run-image partially saved on disk ```=shell > export CNB_USE_OCI=true -> /cnb/lifecycle/exporter -run-image cnb/my-stack-run:bionic my-app-image +> /cnb/lifecycle/exporter -run-image cnb/my-partial-stack-run:bionic my-app-image # OR -> /cnb/lifecycle/exporter -oci -run-image cnb/my-stack-run:bionic my-app-image +> /cnb/lifecycle/exporter -oci -run-image cnb/my-partial-stack-run:bionic my-app-image # expected output # the store folder will be updated with the application image as follows oci ├── cnb -│ └── my-stack-run +│ └── my-partial-stack-run │ ├── blobs │ │ └── sha256 -│ │ └── 1bcd..x +│ │ ├── 1f59...944a // manifest +│ │ ├── 6388...af5a // config │ ├── index.json │ └── oci-layout -├── foo -│ └── // omiting for simplicity -├── bar -│ └── // omiting for simplicity +├── // some folders omitted for simplicity └── my-app-image └── blobs ├── sha256 - │ ├── 1bcd..x - │ ├── 2f789..d - │ └── 3g234..f + │ ├── 1bcd5..x // app image manifest + │ ├── 2f789..d // app image config + │ └── 3g234..f // buildpack layer ├── index.json └── oci-layout ``` +As we can see, the application image `my-app-image` has missing `blobs` because they were not provided as input and the lifecycle just **skip writing** those layers on disk. + ##### Using -oci flag in combination with --daemon or --publish flags Any combination of using multiple sources or sinks in the Lifecycle invocation of phases should throw an error to the user. For example: ```=shell > export CNB_USE_OCI=true -> /cnb/lifecycle/exporter -daemon -run-image cnb/my-stack-run:bionic my-app-image +> /cnb/lifecycle/exporter -daemon -run-image cnb/my-full-stack-run:bionic my-app-image # OR -> /cnb/lifecycle/exporter -oci -daemon -run-image cnb/my-stack-run:bionic my-app-image +> /cnb/lifecycle/exporter -oci -daemon -run-image cnb/my-full-stack-run:bionic my-app-image ERROR: exporting to multiples target is not allowed ``` @@ -281,10 +346,119 @@ The image look up will be done following these rules: - Lifecycle will load the image from disk in [OCI Image Layout](https://github.com/opencontainers/image-spec/blob/main/image-layout.md) format at `///` - WHEN `the image points to a digest reference` - Lifecycle will load the image from disk in [OCI Image Layout](https://github.com/opencontainers/image-spec/blob/main/image-layout.md) format at `///` - - Apart from the look up, the logic for each phase should remain the same + - Apart for the image look up, the logic for each phase should remain the same and the lifecycle will write the layers on disk according to the following rules: + - Any layer generated by a buildpack SHOULD always be written to disk + - Layers from `base-image` or `run-image` MAY be written on disk if they were provided + +Let's review our previous examples. + +## Examples + +In all the examples the new feature is enabled by the use of the new flag `-oci` or by setting the new environment variable `CNB_USE_OCI` to true. + +#### Analyze phase + +##### Analyzing run-image full saved on disk + +Arguments received: + + - `run-image`: `cnb/my-full-stack-run:bionic` + - `image`: `my-app-image` + +The `` is set with the default value `/oci` + +Lifecycle applies the rules for looking up the images: + - It takes the **tag reference** `cnb/my-full-stack-run:bionic` and look at path `/oci/cnb/my-full-stack-run` for an image saved on disk in [OCI Image Layout](https://github.com/opencontainers/image-spec/blob/main/image-layout.md) format. + - Lifecycle validates the `annotations` map in the **Image Manifest** contains a key-value paired `"org.opencontainers.image.ref.name" : "bionic"` + - In case of the *application image* it will look at path `/oci/my-app-image` + + Because both images are found, the phase is executed as usual and the `analyzed.toml` file will be updated + +##### Analyzing run-image partial saved on disk + +Arguments received: + + - `run-image`: `cnb/my-full-partial-run:bionic` + - `image`: `my-app-image` + +The `` is set with the default value `/oci` + +Noticed the structure of the `run-image` provided + +```=shell +oci +├── cnb + └── my-partial-stack-run + ├── blobs + │ └── sha256 + │ ├── 1f59...944a // manifest + │ └── 6388...af5a // config + ├── index.json + └── oci-layout +``` + +Similar to the previous example, Lifecycle applies the rules for looking up the images and look at path `/oci/cnb/my-partial-stack-run` and it determines a partial image was provided and execute the phase logic with the information from the **Image Manifest** and the **Image Config** + +##### Analyzing previous-image + +Arguments received: + +- `run-image`: `cnb/my-full-stack-run:bionic` +- `previous-image`: `bar/my-previous-app` +- `image`: `my-app-image` + +The `` is set with the default value `/oci` + +`run-image` and `image` arguments are treated in the same way as previous examples, and for `previous-image` argument the looking up images rules are applied and Lifecycle will look at path `/oci/bar/my-previous-app` for a image in [OCI Image Layout](https://github.com/opencontainers/image-spec/blob/main/image-layout.md) format. + +##### Analyzing run-image not saved on disk + +Arguments received: + +- `run-image`: `cnb/bad-run-image` +- `image`: `my-app-image` + +The `` is set with the default value `/oci` + +In this case, Lifecycle will will look at path `/oci/bad-run-image` and because the path doesn't exists then an error must be thrown. + +##### Analyzing without run-image argument + +Arguments received: + +- `image`: `my-app-image` + +The `` is set with the default value `/oci` + +When the feature is enabled, Lifecycle requires any input image must be available on disk, because the `run-image` reference was not provided Lifecycle must thrown an error. + +#### Export phase + +##### Export to OCI using run-image full saved on disk + +Arguments received: + +- `run-image`: `cnb/my-full-stack-run:bionic` +- `image`: `my-app-image` + +The `` is set with the default value `/oci` + +Lifecycle applies the rules for looking up the images: + - It takes the **tag reference** `cnb/my-full-stack-run:bionic` and look at path `/oci/cnb/my-full-stack-run` for an image saved on disk in [OCI Image Layout](https://github.com/opencontainers/image-spec/blob/main/image-layout.md) format. + - Lifecycle validates the `annotations` map in the **Image Manifest** contains a key-value paired `"org.opencontainers.image.ref.name" : "bionic"` + - Lifecycle determines the `run-image` is a full image reference (it contains all the blobs) + - Lifecycle writes the *application image* at path `/oci/my-app-image` in [OCI Image Layout](https://github.com/opencontainers/image-spec/blob/main/image-layout.md) format including the layers from the `run-image` + +##### Export to OCI using run-image partially saved on disk + +Arguments received: +- `run-image`: `cnb/my-partial-stack-run:bionic` +- `image`: `my-app-image` +The `` is set with the default value `/oci` +Lifecycle behaves similar to the previous example, but during the process of writing the output application image to disk it will skip the layers that are missing in the `blobs` folder at path `/oci/cnb/my-partial-stack-run` ## Proof of concept From 9eb99ad29e79ddf2cef7339d085cd3dda4d9dbb9 Mon Sep 17 00:00:00 2001 From: Juan Bustamante Date: Tue, 8 Nov 2022 08:23:40 -0500 Subject: [PATCH 48/62] Update text/0000-export-to-oci.md Fixes suggested by Natalie Co-authored-by: Natalie Arellano Signed-off-by: Juan Bustamante --- text/0000-export-to-oci.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/text/0000-export-to-oci.md b/text/0000-export-to-oci.md index be61b28dc..e7fab3b00 100644 --- a/text/0000-export-to-oci.md +++ b/text/0000-export-to-oci.md @@ -67,7 +67,7 @@ The target personas affected by this change are: - **Platform implementors**: they will have to take care of the responsibility of creating a store resource on disk in [OCI Image Layout](https://github.com/opencontainers/image-spec/blob/main/image-layout.md) format and pass it through the lifecycle during the phases execution. -The process of writing any image on disk in [OCI Image Layout](https://github.com/opencontainers/image-spec/blob/main/image-layout.md) format could be expensive in terms of hard drive space or IO operation (compressing or uncompressing layers). In order to provide flexibility for the implementation, the `analyzer` or `exporter` binaries only require the *Image Manifest* and the *Image Config* to execute their operations, based on this, we proposed the Lifecycle can be configured to work with a partial representation of the images on disk, meaning that some blobs MAY be missing (which is ok according to the [OCI Image Layout](https://github.com/opencontainers/image-spec/blob/main/image-layout.md) format). The missing blobs SHOULD be those that are already available in a daemon or registry. +The process of writing any image on disk in [OCI Image Layout](https://github.com/opencontainers/image-spec/blob/main/image-layout.md) format could be expensive in terms of hard drive space or IO operation (compressing or uncompressing layers). In order to provide flexibility for the implementation, the `analyzer` or `exporter` binaries only require the *Image Manifest* and the *Image Config* to execute their operations on the previous image and run image; based on this, we proposed the Lifecycle can be configured to work with a partial representation of the images on disk, meaning that some blobs MAY be missing (which is ok according to the [OCI Image Layout](https://github.com/opencontainers/image-spec/blob/main/image-layout.md) format). The missing blobs SHOULD be those that are already available in a daemon or registry. Let's see some examples of the proposed behavior From e6e01fdf0cb80982718214fee3bd7169ad80b3ea Mon Sep 17 00:00:00 2001 From: Juan Bustamante Date: Wed, 9 Nov 2022 17:16:12 -0500 Subject: [PATCH 49/62] renaming flags to use layout instead of oci Signed-off-by: Juan Bustamante --- text/0000-export-to-oci.md | 64 +++++++++++++++++++------------------- 1 file changed, 32 insertions(+), 32 deletions(-) diff --git a/text/0000-export-to-oci.md b/text/0000-export-to-oci.md index e7fab3b00..ba0c4ec61 100644 --- a/text/0000-export-to-oci.md +++ b/text/0000-export-to-oci.md @@ -141,7 +141,7 @@ The images named **cnb/my-full-stack-run** and **cnb/my-partial-stack-run** repr For each example case, I will present two ways of enabling the new capability: - Using an environment Variable -- Using a new `oci` flag +- Using a new `-layout` flag In any case the expected output is the same. @@ -150,12 +150,12 @@ In any case the expected output is the same. ##### Analyzing run-image full saved on disk ```=shell -> export CNB_USE_OCI=true +> export CNB_USE_OCI_LAYOUT=true > /cnb/lifecycle/analyzer -run-image cnb/my-full-stack-run:bionic my-app-image # OR -> /cnb/lifecycle/analyzer -oci -run-image cnb/my-full-stack-run:bionic my-app-image +> /cnb/lifecycle/analyzer -layout -run-image cnb/my-full-stack-run:bionic my-app-image # expected output # analyzed.toml file with the usual `run-image.reference` @@ -165,12 +165,12 @@ In any case the expected output is the same. ##### Analyzing run-image partial saved on disk ```=shell -> export CNB_USE_OCI=true +> export CNB_USE_OCI_LAYOUT=true > /cnb/lifecycle/analyzer -run-image cnb/cnb/my-partial-stack-run:bionic my-app-image # OR -> /cnb/lifecycle/analyzer -oci -run-image cnb/cnb/my-partial-stack-run:bionic my-app-image +> /cnb/lifecycle/analyzer -layout -run-image cnb/cnb/my-partial-stack-run:bionic my-app-image # expected output # analyzed.toml file with the usual `run-image.reference` @@ -181,12 +181,12 @@ In any case the expected output is the same. ##### Analyzing previous-image ```=shell -> export CNB_USE_OCI=true +> export CNB_USE_OCI_LAYOUT=true > /cnb/lifecycle/analyzer -run-image cnb/my-full-stack-run:bionic -previous-image bar/my-previous-app my-app-image # OR -> /cnb/lifecycle/analyzer -oci -run-image cnb/my-full-stack-run:bionic-previous-image bar/my-previous-app my-app-image +> /cnb/lifecycle/analyzer -layout -run-image cnb/my-full-stack-run:bionic-previous-image bar/my-previous-app my-app-image # expected output # analyzed.toml file with the usual `run-image.reference` and `previos-image.reference` @@ -195,12 +195,12 @@ In any case the expected output is the same. ##### Analyzing run-image not saved on disk ```=shell -> export CNB_USE_OCI=true +> export CNB_USE_OCI_LAYOUT=true > /cnb/lifecycle/analyzer -run-image cnb/bad-run-image my-app-image # OR -> /cnb/lifecycle/analyzer -oci -run-image cnb/bad-run-image my-app-image +> /cnb/lifecycle/analyzer -layout -run-image cnb/bad-run-image my-app-image # expected output @@ -210,12 +210,12 @@ ERROR: the run-image could not be found at path: /oci/cnb/bad-run-image ##### Analyzing without run-image argument ```=shell -> export CNB_USE_OCI=true +> export CNB_USE_OCI_LAYOUT=true > /cnb/lifecycle/analyzer my-app-image # OR -> /cnb/lifecycle/analyzer -oci my-app-image +> /cnb/lifecycle/analyzer -layout my-app-image # expected output @@ -227,12 +227,12 @@ ERROR: -run-image is required when OCI feature is enabled ##### Export to OCI using run-image full saved on disk ```=shell -> export CNB_USE_OCI=true +> export CNB_USE_OCI_LAYOUT=true > /cnb/lifecycle/exporter -run-image cnb/my-full-stack-run:bionic my-app-image # OR -> /cnb/lifecycle/exporter -oci -run-image cnb/my-full-stack-run:bionic my-app-image +> /cnb/lifecycle/exporter -layout -run-image cnb/my-full-stack-run:bionic my-app-image # expected output # the store folder will be updated with the application image as follows @@ -270,12 +270,12 @@ As we can see, the application image `my-app-image` contains a **full** copy of ##### Export to OCI using run-image partially saved on disk ```=shell -> export CNB_USE_OCI=true +> export CNB_USE_OCI_LAYOUT=true > /cnb/lifecycle/exporter -run-image cnb/my-partial-stack-run:bionic my-app-image # OR -> /cnb/lifecycle/exporter -oci -run-image cnb/my-partial-stack-run:bionic my-app-image +> /cnb/lifecycle/exporter -layout -run-image cnb/my-partial-stack-run:bionic my-app-image # expected output # the store folder will be updated with the application image as follows @@ -303,17 +303,17 @@ oci As we can see, the application image `my-app-image` has missing `blobs` because they were not provided as input and the lifecycle just **skip writing** those layers on disk. -##### Using -oci flag in combination with --daemon or --publish flags +##### Using -layout flag in combination with --daemon or --publish flags Any combination of using multiple sources or sinks in the Lifecycle invocation of phases should throw an error to the user. For example: ```=shell -> export CNB_USE_OCI=true +> export CNB_USE_OCI_LAYOUT=true > /cnb/lifecycle/exporter -daemon -run-image cnb/my-full-stack-run:bionic my-app-image # OR -> /cnb/lifecycle/exporter -oci -daemon -run-image cnb/my-full-stack-run:bionic my-app-image +> /cnb/lifecycle/exporter -layout -daemon -run-image cnb/my-full-stack-run:bionic my-app-image ERROR: exporting to multiples target is not allowed ``` @@ -337,15 +337,15 @@ The following new inputs are proposed to be added to these phases | Input | Environment Variable | Default Value | Description |-------|-----------------------|---------------|-------------- - | `` | `CNB_USE_OCI` | false | Analyze or Export image from/to OCI layout format on disk | - | `` | `CNB_OCI_PATH` | /oci | Path to oci directory where the images are saved | + | `` | `CNB_USE_OCI_LAYOUT` | false | Enables the capability of resolving image from/to in OCI layout format on disk | + | `` | `CNB_OCI_LAYOUT_PATH` | /oci | Path to oci directory where the images are saved | -- WHEN `the new flag -oci or the default environment variable CNB_USE_OCI are set to true` during the phase invocation THEN the feature will be enabled. +- WHEN `the new flag -layout or the default environment variable CNB_USE_OCI_LAYOUT are set to true` during the phase invocation THEN the feature will be enabled. The image look up will be done following these rules: - WHEN `the image points to a tag reference` - - Lifecycle will load the image from disk in [OCI Image Layout](https://github.com/opencontainers/image-spec/blob/main/image-layout.md) format at `///` + - Lifecycle will load the image from disk in [OCI Image Layout](https://github.com/opencontainers/image-spec/blob/main/image-layout.md) format at `///` - WHEN `the image points to a digest reference` - - Lifecycle will load the image from disk in [OCI Image Layout](https://github.com/opencontainers/image-spec/blob/main/image-layout.md) format at `///` + - Lifecycle will load the image from disk in [OCI Image Layout](https://github.com/opencontainers/image-spec/blob/main/image-layout.md) format at `///` - Apart for the image look up, the logic for each phase should remain the same and the lifecycle will write the layers on disk according to the following rules: - Any layer generated by a buildpack SHOULD always be written to disk - Layers from `base-image` or `run-image` MAY be written on disk if they were provided @@ -354,7 +354,7 @@ Let's review our previous examples. ## Examples -In all the examples the new feature is enabled by the use of the new flag `-oci` or by setting the new environment variable `CNB_USE_OCI` to true. +In all the examples the new feature is enabled by the use of the new flag `-layout` or by setting the new environment variable `CNB_USE_OCI_LAYOUT` to true. #### Analyze phase @@ -365,7 +365,7 @@ Arguments received: - `run-image`: `cnb/my-full-stack-run:bionic` - `image`: `my-app-image` -The `` is set with the default value `/oci` +The `` is set with the default value `/oci` Lifecycle applies the rules for looking up the images: - It takes the **tag reference** `cnb/my-full-stack-run:bionic` and look at path `/oci/cnb/my-full-stack-run` for an image saved on disk in [OCI Image Layout](https://github.com/opencontainers/image-spec/blob/main/image-layout.md) format. @@ -381,7 +381,7 @@ Arguments received: - `run-image`: `cnb/my-full-partial-run:bionic` - `image`: `my-app-image` -The `` is set with the default value `/oci` +The `` is set with the default value `/oci` Noticed the structure of the `run-image` provided @@ -407,7 +407,7 @@ Arguments received: - `previous-image`: `bar/my-previous-app` - `image`: `my-app-image` -The `` is set with the default value `/oci` +The `` is set with the default value `/oci` `run-image` and `image` arguments are treated in the same way as previous examples, and for `previous-image` argument the looking up images rules are applied and Lifecycle will look at path `/oci/bar/my-previous-app` for a image in [OCI Image Layout](https://github.com/opencontainers/image-spec/blob/main/image-layout.md) format. @@ -418,7 +418,7 @@ Arguments received: - `run-image`: `cnb/bad-run-image` - `image`: `my-app-image` -The `` is set with the default value `/oci` +The `` is set with the default value `/oci` In this case, Lifecycle will will look at path `/oci/bad-run-image` and because the path doesn't exists then an error must be thrown. @@ -428,7 +428,7 @@ Arguments received: - `image`: `my-app-image` -The `` is set with the default value `/oci` +The `` is set with the default value `/oci` When the feature is enabled, Lifecycle requires any input image must be available on disk, because the `run-image` reference was not provided Lifecycle must thrown an error. @@ -441,7 +441,7 @@ Arguments received: - `run-image`: `cnb/my-full-stack-run:bionic` - `image`: `my-app-image` -The `` is set with the default value `/oci` +The `` is set with the default value `/oci` Lifecycle applies the rules for looking up the images: - It takes the **tag reference** `cnb/my-full-stack-run:bionic` and look at path `/oci/cnb/my-full-stack-run` for an image saved on disk in [OCI Image Layout](https://github.com/opencontainers/image-spec/blob/main/image-layout.md) format. @@ -456,7 +456,7 @@ Arguments received: - `run-image`: `cnb/my-partial-stack-run:bionic` - `image`: `my-app-image` -The `` is set with the default value `/oci` +The `` is set with the default value `/oci` Lifecycle behaves similar to the previous example, but during the process of writing the output application image to disk it will skip the layers that are missing in the `blobs` folder at path `/oci/cnb/my-partial-stack-run` @@ -465,7 +465,7 @@ Lifecycle behaves similar to the previous example, but during the process of wri In order to validate the feasibility of the proposed feature, we developed a proof of concept with one of the most important side effects this capability can add into the project: **Removing the Daemon Support**. You can also check a recording with the demo in the following [link](https://drive.google.com/file/d/1W1125OHuyUlx88BRroUTLBfrFHhFM5A9/view?usp=sharing) As mentioned earlier, if we want to remove the daemon support in the Lifecycle, then all the responsibility to deal with it goes into the platforms implementors, that means, for example: -- Pull the require dependencies (runtime image for example), save them on disk in OCI layout format and pass it through the lifecycle using the `` parameter +- Pull the require dependencies (runtime image for example), save them on disk in OCI layout format and pass it through the lifecycle using the `` parameter - Push the application image (exported in OCI layout format) into the Daemon, because that is what users are expecting. During the proof of concept implementation I choose to use [skopeo](https://github.com/containers/skopeo) tool to solve the problem of interacting with the Daemon. The reason to do it was **simplicity** for the PoC developed but we believe this is a good subject to talk about with the community. From a38c474d4604851ae92f35538264d4f537d0f907 Mon Sep 17 00:00:00 2001 From: Juan Bustamante Date: Mon, 23 Jan 2023 12:37:49 -0500 Subject: [PATCH 50/62] Adding updates after the implementation Signed-off-by: Juan Bustamante --- text/0000-export-to-oci.md | 481 +++++++++++++++++++++++-------------- 1 file changed, 307 insertions(+), 174 deletions(-) diff --git a/text/0000-export-to-oci.md b/text/0000-export-to-oci.md index ba0c4ec61..dbbc93eca 100644 --- a/text/0000-export-to-oci.md +++ b/text/0000-export-to-oci.md @@ -77,63 +77,42 @@ Let's see some examples of the proposed behavior A folder on disk (accessible by the lifecycle) is required to execute the feature, this new folder works as a local registry and the content must be in [OCI Image Layout](https://github.com/opencontainers/image-spec/blob/main/image-layout.md) format. -Lifecycle phases accepts arguments pointing to `image reference`, those arguments are: - -| Input | Description -|-------|------------ -| `` | Tag reference to which the app image will be written | -| `` | Image reference to be analyzed (usually the result of the previous build) | -| `` | Run image reference | - - - -Let's suppose a directory exits and it has the following structure: +Let's suppose a directory exists and it has the following structure: ```=shell -oci -├── cnb -│ ├── my-full-stack-run -│ │ ├── blobs -│ │ │ └── sha256 -│ │ │ ├── 1f59...944a // manifest -│ │ │ ├── 6388...af5a // config -│ │ │ ├── 824b...f984e -│ │ │ ├── f5f7...5b38 -│ │ │ └── 870e...f1b09 -│ │ ├── index.json -│ │ └── oci-layout -│ └── my-partial-stack-run -│ ├── blobs -│ │ └── sha256 -│ │ ├── 1f59...944a // manifest -│ │ └── 6388...af5a // config -│ ├── index.json -│ └── oci-layout -├── foo -│ └── my-cache -│ ├── blobs -│ │ └── sha256 -│ │ └── 1f591..a -│ ├── index.json -│ └── oci-layout -├── bar -│ └── my-previous-app -│ ├── blobs -│ │ └── sha256 -│ │ └── 1efg..w -│ ├── index.json -│ └── oci-layout -└── my-app-image - └── blobs - ├── sha256 - │ ├── 1bcd5..x // app image manifest - │ ├── 2f789..d // app image config - │ ├── 824b...f984e // run layer - │ ├── f5f7...5b38 // run layer - │ ├── 870e...f1b09 // run layer - │ └── 3g234..f // buildpack layer - ├── index.json - └── oci-layout +layout-repo +└── index.docker.io + ├── cnb + │ ├── my-full-stack-run:bionic + │ │ └── bionic + │ │ └── blobs + │ │ ├── sha256 + │ │ │ ├── 1f59...944a // manifest + │ │ │ ├── 6388...af5a // config + │ │ │ ├── 824b...f984e + │ │ │ ├── f5f7...5b38 + │ │ │ └── 870e...f1b09 + │ │ ├── index.json + │ │ └── oci-layout + │ └── my-partial-stack-run:bionic + │ └── bionic + │ ├── blobs + │ │ └── sha256 + │ │ ├── 1f59...944a // manifest + │ │ └── 6388...af5a // config + │ ├── index.json + │ └── oci-layout + └── bar + └── my-previous-app + └── latest + ├── blobs + │ └── sha256 + │ ├── 4bcd5..x // app image manifest + │ ├── 5f789..d // app image config + │ ├── 624b...f984e // run layer + │ └── 4g234..f // buildpack layer + ├── index.json + └── oci-layout ``` The images named **cnb/my-full-stack-run** and **cnb/my-partial-stack-run** represents the same image but the partial one has missing `blobs`, those `blobs` are the layers that are already available in the store from it came from. @@ -150,15 +129,20 @@ In any case the expected output is the same. ##### Analyzing run-image full saved on disk ```=shell -> export CNB_USE_OCI_LAYOUT=true +> export CNB_USE_LAYOUT=true > /cnb/lifecycle/analyzer -run-image cnb/my-full-stack-run:bionic my-app-image # OR > /cnb/lifecycle/analyzer -layout -run-image cnb/my-full-stack-run:bionic my-app-image +``` -# expected output -# analyzed.toml file with the usual `run-image.reference` +expected analyzed.toml output + +```=toml +[run-image] + reference = "" + name = "/layout-repo/index.docker.io/cnb/my-full-stack-run/bionic" ``` @@ -171,10 +155,14 @@ In any case the expected output is the same. # OR > /cnb/lifecycle/analyzer -layout -run-image cnb/cnb/my-partial-stack-run:bionic my-app-image +``` -# expected output -# analyzed.toml file with the usual `run-image.reference` -# there should be no change behavior in the phase even thought the image has missing blobs +expected analyzed.toml output + +```=toml +[run-image] + reference = "" + name = "/layout-repo/index.docker.io/cnb/my-partial-stack-run/bionic" ``` @@ -187,9 +175,19 @@ In any case the expected output is the same. # OR > /cnb/lifecycle/analyzer -layout -run-image cnb/my-full-stack-run:bionic-previous-image bar/my-previous-app my-app-image +``` + +expected analyzed.toml output + +```=toml +[run-image] + reference = "" + name = "/layout-repo/index.docker.io/cnb/my-partial-stack-run/bionic" + +[previous-image] + reference = "" + name = "/layout-repo/index.docker.io/bar/my-previous-app/latest" -# expected output -# analyzed.toml file with the usual `run-image.reference` and `previos-image.reference` ``` ##### Analyzing run-image not saved on disk @@ -204,7 +202,7 @@ In any case the expected output is the same. # expected output -ERROR: the run-image could not be found at path: /oci/cnb/bad-run-image +ERROR: the run-image could not be found at path: /layout-repo/index.docker.io/cnb/bad-run-image/latest ``` ##### Analyzing without run-image argument @@ -219,48 +217,54 @@ ERROR: the run-image could not be found at path: /oci/cnb/bad-run-image # expected output -ERROR: -run-image is required when OCI feature is enabled +ERROR: -run-image is required when OCI Layout feature is enabled ``` +Let's also check some examples when the export phase is executed + #### Export phase ##### Export to OCI using run-image full saved on disk ```=shell > export CNB_USE_OCI_LAYOUT=true -> /cnb/lifecycle/exporter -run-image cnb/my-full-stack-run:bionic my-app-image +> /cnb/lifecycle/exporter my-app-image # OR -> /cnb/lifecycle/exporter -layout -run-image cnb/my-full-stack-run:bionic my-app-image +> /cnb/lifecycle/exporter -layout my-app-image +``` -# expected output -# the store folder will be updated with the application image as follows - -oci -├── cnb -│ └── my-full-stack-run -│ ├── blobs -│ │ └── sha256 -│ │ ├── 1f59...944a // manifest -│ │ ├── 6388...af5a // config -│ │ ├── 824b...f984e -│ │ ├── f5f7...5b38 -│ │ └── 870e...f1b09 -│ ├── index.json -│ └── oci-layout -├── // some folders omitted for simplicity -└── my-app-image - └── blobs - ├── sha256 - │ ├── 1bcd5..x // app image manifest - │ ├── 2f789..d // app image config - │ ├── 824b...f984e // run layer - │ ├── f5f7...5b38 // run layer - │ ├── 870e...f1b09 // run layer - │ └── 3g234..f // buildpack layer - ├── index.json - └── oci-layout +The output will be written into the repository folder described above and it should looks like this: + +```=shell +layout-repo +└── index.docker.io + ├── cnb + │ └── my-full-stack-run:bionic + │ └── bionic + │ └── blobs + │ ├── sha256 + │ │ ├── 1f59...944a // manifest + │ │ ├── 6388...af5a // config + │ │ ├── 824b...f984e + │ │ ├── f5f7...5b38 + │ │ └── 870e...f1b09 + │ ├── index.json + │ └── oci-layout + └── library + └── my-app-image + └── latest + ├── blobs + │ └── sha256 + │ ├── 1bcd5..x // app image manifest + │ ├── 2f789..d // app image config + │ ├── 824b...f984e // run layer + │ ├── f5f7...5b38 // run layer + │ ├── 870e...f1b09 // run layer + │ └── 3g234..f // buildpack layer + ├── index.json + └── oci-layout ``` @@ -271,33 +275,37 @@ As we can see, the application image `my-app-image` contains a **full** copy of ```=shell > export CNB_USE_OCI_LAYOUT=true -> /cnb/lifecycle/exporter -run-image cnb/my-partial-stack-run:bionic my-app-image +> /cnb/lifecycle/exporter my-app-image # OR -> /cnb/lifecycle/exporter -layout -run-image cnb/my-partial-stack-run:bionic my-app-image +> /cnb/lifecycle/exporter -layout my-app-image +``` -# expected output -# the store folder will be updated with the application image as follows - -oci -├── cnb -│ └── my-partial-stack-run -│ ├── blobs -│ │ └── sha256 -│ │ ├── 1f59...944a // manifest -│ │ ├── 6388...af5a // config -│ ├── index.json -│ └── oci-layout -├── // some folders omitted for simplicity -└── my-app-image - └── blobs - ├── sha256 - │ ├── 1bcd5..x // app image manifest - │ ├── 2f789..d // app image config - │ └── 3g234..f // buildpack layer - ├── index.json - └── oci-layout +Expected output: + +```=shell +layout-repo +└── index.docker.io + ├── cnb + │ └── my-partial-stack-run:bionic + │ └── bionic + │ ├── blobs + │ │ └── sha256 + │ │ ├── 1f59...944a // manifest + │ │ └── 6388...af5a // config + │ ├── index.json + │ └── oci-layout + └── library + └── my-app-image + └── latest + ├── blobs + │ └── sha256 + │ ├── 1bcd5..x // app image manifest + │ ├── 2f789..d // app image config + │ └── 3g234..f // buildpack layer + ├── index.json + └── oci-layout ``` @@ -338,127 +346,230 @@ The following new inputs are proposed to be added to these phases | Input | Environment Variable | Default Value | Description |-------|-----------------------|---------------|-------------- | `` | `CNB_USE_OCI_LAYOUT` | false | Enables the capability of resolving image from/to in OCI layout format on disk | - | `` | `CNB_OCI_LAYOUT_PATH` | /oci | Path to oci directory where the images are saved | + | `` | `CNB_OCI_LAYOUT_PATH` | /layout-repo | Path to a directory where the images are saved in OCI layout format| + +## How to map an image reference into a path in the layout repository +[rules]: #rules + +In the previous examples one key element was how to translate an image reference into a path to look for in the ``, let's define those rules. + +Considering an **image reference** refers to either a tag reference or digest reference. It could have the following formats +- A tag reference refers to an identifier of form `/:` +- A digest reference refers to a content addressable identifier of form `/@:` -- WHEN `the new flag -layout or the default environment variable CNB_USE_OCI_LAYOUT are set to true` during the phase invocation THEN the feature will be enabled. The image look up will be done following these rules: - WHEN `the image points to a tag reference` - - Lifecycle will load the image from disk in [OCI Image Layout](https://github.com/opencontainers/image-spec/blob/main/image-layout.md) format at `///` + - Lifecycle will load/save the image from/to disk in [OCI Image Layout](https://github.com/opencontainers/image-spec/blob/main/image-layout.md) format at `///` - WHEN `the image points to a digest reference` - Lifecycle will load the image from disk in [OCI Image Layout](https://github.com/opencontainers/image-spec/blob/main/image-layout.md) format at `///` - - Apart for the image look up, the logic for each phase should remain the same and the lifecycle will write the layers on disk according to the following rules: - - Any layer generated by a buildpack SHOULD always be written to disk - - Layers from `base-image` or `run-image` MAY be written on disk if they were provided + - WHEN `` is not provided default value will be **index.docker.io** + - IF `` is not also provided, then default value will be **library** -Let's review our previous examples. ## Examples -In all the examples the new feature is enabled by the use of the new flag `-layout` or by setting the new environment variable `CNB_USE_OCI_LAYOUT` to true. +In all the examples the new feature is enabled by the use of the new flag `-layout` or by setting +the new environment variable `CNB_USE_OCI_LAYOUT` to true. + +Let's review some of the previous examples #### Analyze phase ##### Analyzing run-image full saved on disk +Command: + +```=shell +> export CNB_USE_LAYOUT=true +> /cnb/lifecycle/analyzer -run-image cnb/my-full-stack-run:bionic my-app-image + +# OR + +> /cnb/lifecycle/analyzer -layout -run-image cnb/my-full-stack-run:bionic my-app-image +``` + Arguments received: - `run-image`: `cnb/my-full-stack-run:bionic` - `image`: `my-app-image` -The `` is set with the default value `/oci` +The `` is set with the default value `/layout-repo` Lifecycle applies the rules for looking up the images: - - It takes the **tag reference** `cnb/my-full-stack-run:bionic` and look at path `/oci/cnb/my-full-stack-run` for an image saved on disk in [OCI Image Layout](https://github.com/opencontainers/image-spec/blob/main/image-layout.md) format. - - Lifecycle validates the `annotations` map in the **Image Manifest** contains a key-value paired `"org.opencontainers.image.ref.name" : "bionic"` - - In case of the *application image* it will look at path `/oci/my-app-image` + - It takes the **tag reference** `cnb/my-full-stack-run:bionic`, applies the conversion rules and look at path `/layout-repo/index.docker.io/cnb/my-full-stack-run/bionic` for an image saved on disk in [OCI Image Layout](https://github.com/opencontainers/image-spec/blob/main/image-layout.md) format. + + - In case of the *application image* it will look at path `/layout-repo/index.docker.io/library/my-app-image/latest` - Because both images are found, the phase is executed as usual and the `analyzed.toml` file will be updated +Because both images are found, the phase is executed as usual and the `analyzed.toml` file will be updated. A new field `Name` was added into the `analyzed.toml` that will contain the path resolved by the lifecycle, in these cases: + +```=toml +[run-image] + reference = "" + name = "/layout-repo/index.docker.io/cnb/my-partial-stack-run/bionic" + +``` ##### Analyzing run-image partial saved on disk +Command received: + +```=shell +> export CNB_USE_OCI_LAYOUT=true +> /cnb/lifecycle/analyzer -run-image cnb/cnb/my-partial-stack-run:bionic my-app-image + +# OR + +> /cnb/lifecycle/analyzer -layout -run-image cnb/cnb/my-partial-stack-run:bionic my-app-image +``` + Arguments received: - `run-image`: `cnb/my-full-partial-run:bionic` - `image`: `my-app-image` -The `` is set with the default value `/oci` +The `` is set with the default value `/layout-repo` Noticed the structure of the `run-image` provided ```=shell -oci -├── cnb - └── my-partial-stack-run - ├── blobs - │ └── sha256 - │ ├── 1f59...944a // manifest - │ └── 6388...af5a // config - ├── index.json - └── oci-layout +layout-repo +└── index.docker.io + └── cnb + └── my-partial-stack-run:bionic + └── bionic + ├── blobs + │ └── sha256 + │ ├── 1f59...944a // manifest + │ └── 6388...af5a // config + ├── index.json + └── oci-layout ``` -Similar to the previous example, Lifecycle applies the rules for looking up the images and look at path `/oci/cnb/my-partial-stack-run` and it determines a partial image was provided and execute the phase logic with the information from the **Image Manifest** and the **Image Config** +Similar to the previous example, Lifecycle applies the rules for looking up the images and look at path `/layout-repo/index.docker.io/cnb/my-partial-stack-run/bionic` and it determines a partial image was provided and execute the phase logic with the information from the **Image Manifest** and the **Image Config** + +The output `analyzed.toml` will also include the new `name` field with the path where the image was located. + +```=toml +[run-image] + reference = "" + name = "/layout-repo/index.docker.io/cnb/my-partial-stack-run/bionic" + +``` ##### Analyzing previous-image +Command received: + +```=shell +> export CNB_USE_OCI_LAYOUT=true +> /cnb/lifecycle/analyzer -run-image cnb/my-full-stack-run:bionic -previous-image bar/my-previous-app my-app-image + +# OR + +> /cnb/lifecycle/analyzer -layout -run-image cnb/my-full-stack-run:bionic -previous-image bar/my-previous-app my-app-image +``` + Arguments received: - `run-image`: `cnb/my-full-stack-run:bionic` - `previous-image`: `bar/my-previous-app` - `image`: `my-app-image` -The `` is set with the default value `/oci` +The `` is set with the default value `/layout-repo` -`run-image` and `image` arguments are treated in the same way as previous examples, and for `previous-image` argument the looking up images rules are applied and Lifecycle will look at path `/oci/bar/my-previous-app` for a image in [OCI Image Layout](https://github.com/opencontainers/image-spec/blob/main/image-layout.md) format. +`run-image` and `image` arguments are treated in the same way as previous examples, and for `previous-image` argument the looking up images rules are applied and Lifecycle will look at path `/layout-repo/index.docker.io/bar/my-previous-app` for a image in [OCI Image Layout](https://github.com/opencontainers/image-spec/blob/main/image-layout.md) format. -##### Analyzing run-image not saved on disk +The `analyzed.toml` file es expected to be updated with the previous image section and the new label `name` will be also be there with the path to the `previous-image` -Arguments received: +```=toml +[run-image] + reference = "" + name = "/layout-repo/index.docker.io/cnb/my-full-stack-run/bionic" -- `run-image`: `cnb/bad-run-image` -- `image`: `my-app-image` +[previous-image] + reference = "" + name = "/layout-repo/index.docker.io/bar/my-previous-app/latest" -The `` is set with the default value `/oci` +``` -In this case, Lifecycle will will look at path `/oci/bad-run-image` and because the path doesn't exists then an error must be thrown. +Let's check how the `export` examples works on detailed -##### Analyzing without run-image argument +##### Export to OCI using run-image full saved on disk -Arguments received: +Pre-conditions: -- `image`: `my-app-image` +The following directories are accessible by the lifecycle +```=shell +/ +├── layout-repo +│ └── index.docker.io +│ └── cnb +│ └── my-full-stack-run:bionic +│ └── bionic +│ └── blobs +│ ├── sha256 +│ │ ├── 1f59...944a // manifest +│ │ ├── 6388...af5a // config +│ │ ├── 824b...f984e +│ │ ├── f5f7...5b38 +│ │ └── 870e...f1b09 +│ ├── index.json +│ └── oci-layout +└── layers + └── analyzed.tom +``` -The `` is set with the default value `/oci` +The `/layers/analyzed.toml` file contains the following data: -When the feature is enabled, Lifecycle requires any input image must be available on disk, because the `run-image` reference was not provided Lifecycle must thrown an error. +```=toml +[run-image] + reference = "" + name = "/layout-repo/index.docker.io/cnb/my-full-stack-run/bionic" -#### Export phase +``` -##### Export to OCI using run-image full saved on disk +Command executed: + +```=shell +> export CNB_USE_OCI_LAYOUT=true +> /cnb/lifecycle/exporter my-app-image + +# OR + +> /cnb/lifecycle/exporter -layout my-app-image +``` Arguments received: -- `run-image`: `cnb/my-full-stack-run:bionic` - `image`: `my-app-image` -The `` is set with the default value `/oci` - -Lifecycle applies the rules for looking up the images: - - It takes the **tag reference** `cnb/my-full-stack-run:bionic` and look at path `/oci/cnb/my-full-stack-run` for an image saved on disk in [OCI Image Layout](https://github.com/opencontainers/image-spec/blob/main/image-layout.md) format. - - Lifecycle validates the `annotations` map in the **Image Manifest** contains a key-value paired `"org.opencontainers.image.ref.name" : "bionic"` - - Lifecycle determines the `run-image` is a full image reference (it contains all the blobs) - - Lifecycle writes the *application image* at path `/oci/my-app-image` in [OCI Image Layout](https://github.com/opencontainers/image-spec/blob/main/image-layout.md) format including the layers from the `run-image` +The `` is set with the default value `/layout-repo` -##### Export to OCI using run-image partially saved on disk +Lifecycle: + - It will read the `[run-image]` section in the `analyzed.toml` and read the `name` attribute to load the `run-image` image saved on disk in [OCI Image Layout](https://github.com/opencontainers/image-spec/blob/main/image-layout.md) format at path `/layout-repo/index.docker.io/cnb/my-full-stack-run/bionic`. + - Lifecycle will execute the export steps and at the end of the process it will write the *application image* at path `/layout-repo/index.docker.io/library/my-app-image/latest` in [OCI Image Layout](https://github.com/opencontainers/image-spec/blob/main/image-layout.md) format -Arguments received: -- `run-image`: `cnb/my-partial-stack-run:bionic` -- `image`: `my-app-image` +The output image will be written at: -The `` is set with the default value `/oci` +```=shell +layout-repo +└── index.docker.io + └── library + └── my-app-image + └── latest + ├── blobs + │ └── sha256 + │ ├── 1bcd5..x // app image manifest + │ ├── 2f789..d // app image config + │ ├── 824b...f984e // run layer + │ ├── f5f7...5b38 // run layer + │ ├── 870e...f1b09 // run layer + │ └── 3g234..f // buildpack layer + ├── index.json + └── oci-layout -Lifecycle behaves similar to the previous example, but during the process of writing the output application image to disk it will skip the layers that are missing in the `blobs` folder at path `/oci/cnb/my-partial-stack-run` +``` ## Proof of concept @@ -570,7 +681,29 @@ I think, this PoC demonstrate that adding the exporting to OCI layout format is # Spec. Changes (OPTIONAL) [spec-changes]: #spec-changes - +The [Platform Interface Specification](https://github.com/buildpacks/spec/blob/platform/0.11/platform.md#inputs-5) must be updated to include the following inputs to the [Create](https://buildpacks.io/docs/concepts/components/lifecycle/create/), [Analyze](https://buildpacks.io/docs/concepts/components/lifecycle/analyze/) and [Export](https://buildpacks.io/docs/concepts/components/lifecycle/export/) phases + +| Input | Environment Variable | Default Value | Description +|-------|-----------------------|---------------|-------------- +| `` | `CNB_USE_OCI_LAYOUT` | false | Enables the capability of resolving image from/to in OCI layout format on disk | +| `` | `CNB_OCI_LAYOUT_PATH` | /layout-repo | Path to a directory where the images are saved in OCI layout format| + +Also the `analyzed.toml` [file](https://github.com/buildpacks/spec/blob/platform/0.11/platform.md#analyzedtoml-toml) will be updated to include the new `name` field + +```=toml +[image] + reference = "" + name = "" + +[run-image] + reference = "" + name = "" + +[previous-image] + reference = "" + name = "" +``` + +Where + +* `[image|run-image|previos-image].name` MUST point to the path of the image in OCI layout format following the rules describe [previously](#rules) From f498b0ac99ced52b7ca04335f41f22571d82b2de Mon Sep 17 00:00:00 2001 From: Juan Bustamante Date: Mon, 23 Jan 2023 12:41:24 -0500 Subject: [PATCH 51/62] Adding updates after the implementation Signed-off-by: Juan Bustamante --- text/0000-export-to-oci.md | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/text/0000-export-to-oci.md b/text/0000-export-to-oci.md index dbbc93eca..169a414f3 100644 --- a/text/0000-export-to-oci.md +++ b/text/0000-export-to-oci.md @@ -349,7 +349,6 @@ The following new inputs are proposed to be added to these phases | `` | `CNB_OCI_LAYOUT_PATH` | /layout-repo | Path to a directory where the images are saved in OCI layout format| ## How to map an image reference into a path in the layout repository -[rules]: #rules In the previous examples one key element was how to translate an image reference into a path to look for in the ``, let's define those rules. @@ -706,4 +705,4 @@ Also the `analyzed.toml` [file](https://github.com/buildpacks/spec/blob/platform Where -* `[image|run-image|previos-image].name` MUST point to the path of the image in OCI layout format following the rules describe [previously](#rules) +* `[image|run-image|previos-image].name` MUST point to the path of the image in OCI layout format following the rules describe [previously](#how-to-map-an-image-reference-into-a-path-in-the-layout-repository) From 5d9b008f3f6db79993cbcf0846f52aee8bb2236f Mon Sep 17 00:00:00 2001 From: Juan Bustamante Date: Mon, 23 Jan 2023 12:56:53 -0500 Subject: [PATCH 52/62] Open questions Signed-off-by: Juan Bustamante --- text/0000-export-to-oci.md | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-) diff --git a/text/0000-export-to-oci.md b/text/0000-export-to-oci.md index 169a414f3..97690fd39 100644 --- a/text/0000-export-to-oci.md +++ b/text/0000-export-to-oci.md @@ -672,9 +672,15 @@ I think, this PoC demonstrate that adding the exporting to OCI layout format is # Unresolved Questions [unresolved-questions]: #unresolved-questions - # Spec. Changes (OPTIONAL) From 6fc05ff497934e1e2baa9a56900e2030394fa8d3 Mon Sep 17 00:00:00 2001 From: Juan Bustamante Date: Mon, 23 Jan 2023 15:22:25 -0500 Subject: [PATCH 53/62] Unresolved question about exporting to tarball Signed-off-by: Juan Bustamante --- text/0000-export-to-oci.md | 1 + 1 file changed, 1 insertion(+) diff --git a/text/0000-export-to-oci.md b/text/0000-export-to-oci.md index 97690fd39..fd2b1e3fb 100644 --- a/text/0000-export-to-oci.md +++ b/text/0000-export-to-oci.md @@ -676,6 +676,7 @@ I think, this PoC demonstrate that adding the exporting to OCI layout format is - Tools like [umoci](https://umo.ci/) used to create a runtime bundle from an image in OCI layout format, requires the [annotation](https://github.com/opencontainers/image-spec/blob/main/annotations.md#pre-defined-annotation-keys) `org.opencontainers.image.ref.name` to be present. Also tools like [skopeo](https://github.com/containers/skopeo) when an image is `copy` in oci format the annotation is included. We are not adding the annotation as part of the Buildpacks Specification, but in this case this could make our output incompatible with some other tooling. - Depending on the previous question, the rules for mapping an image reference into a path could change, also validations that the Lifecycle could do to verify the `org.opencontainers.image.ref.name` against the value provided in the image reference. + - Exporting to a tarball can be handle on this RFC or a new one must be created? - What parts of the design do you expect to be resolved through implementation of the feature? - Handle symbolic links to the blobs in the `` repository, this could be more efficient on hard drive space From 40cd649d67b3611e18160744821f5baa665a54aa Mon Sep 17 00:00:00 2001 From: Juan Bustamante Date: Tue, 24 Jan 2023 09:39:52 -0500 Subject: [PATCH 54/62] updating the rules to map image ref to path Signed-off-by: Juan Bustamante --- text/0000-export-to-oci.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/text/0000-export-to-oci.md b/text/0000-export-to-oci.md index fd2b1e3fb..82724153f 100644 --- a/text/0000-export-to-oci.md +++ b/text/0000-export-to-oci.md @@ -353,14 +353,14 @@ The following new inputs are proposed to be added to these phases In the previous examples one key element was how to translate an image reference into a path to look for in the ``, let's define those rules. Considering an **image reference** refers to either a tag reference or digest reference. It could have the following formats -- A tag reference refers to an identifier of form `/:` -- A digest reference refers to a content addressable identifier of form `/@:` +- A tag reference refers to an identifier of form `//:` +- A digest reference refers to a content addressable identifier of form `//@:` The image look up will be done following these rules: - WHEN `the image points to a tag reference` - - Lifecycle will load/save the image from/to disk in [OCI Image Layout](https://github.com/opencontainers/image-spec/blob/main/image-layout.md) format at `///` + - Lifecycle will load/save the image from/to disk in [OCI Image Layout](https://github.com/opencontainers/image-spec/blob/main/image-layout.md) format at `////` - WHEN `the image points to a digest reference` - - Lifecycle will load the image from disk in [OCI Image Layout](https://github.com/opencontainers/image-spec/blob/main/image-layout.md) format at `///` + - Lifecycle will load the image from disk in [OCI Image Layout](https://github.com/opencontainers/image-spec/blob/main/image-layout.md) format at `/////` - WHEN `` is not provided default value will be **index.docker.io** - IF `` is not also provided, then default value will be **library** From 1771f9ff51157c185d6eaa6eb5eb341d174ce31d Mon Sep 17 00:00:00 2001 From: Juan Bustamante Date: Mon, 30 Jan 2023 11:55:33 -0500 Subject: [PATCH 55/62] Removing the Name field from the analyzed.toml Signed-off-by: Juan Bustamante --- text/0000-export-to-oci.md | 48 ++++++++++++++++---------------------- 1 file changed, 20 insertions(+), 28 deletions(-) diff --git a/text/0000-export-to-oci.md b/text/0000-export-to-oci.md index 82724153f..cfbc869fb 100644 --- a/text/0000-export-to-oci.md +++ b/text/0000-export-to-oci.md @@ -141,8 +141,7 @@ expected analyzed.toml output ```=toml [run-image] - reference = "" - name = "/layout-repo/index.docker.io/cnb/my-full-stack-run/bionic" + reference = "/layout-repo/index.docker.io/cnb/my-full-stack-run/bionic@sha256:fab3bb83de466ed29d7e9dcfdbee5b5fb2ff90e91bc849af85b261b4c2062a7a" ``` @@ -161,8 +160,7 @@ expected analyzed.toml output ```=toml [run-image] - reference = "" - name = "/layout-repo/index.docker.io/cnb/my-partial-stack-run/bionic" + reference = "/layout-repo/index.docker.io/cnb/my-full-stack-run/bionic@sha256:fab3bb83de466ed29d7e9dcfdbee5b5fb2ff90e91bc849af85b261b4c2062a7a" ``` @@ -181,12 +179,10 @@ expected analyzed.toml output ```=toml [run-image] - reference = "" - name = "/layout-repo/index.docker.io/cnb/my-partial-stack-run/bionic" + reference = "/layout-repo/index.docker.io/cnb/my-partial-stack-run/bionic@sha256:fab3bb83de466ed29d7e9dcfdbee5b5fb2ff90e91bc849af85b261b4c2062a7a" [previous-image] - reference = "" - name = "/layout-repo/index.docker.io/bar/my-previous-app/latest" + reference = "/layout-repo/index.docker.io/bar/my-previous-app/latest@sha256:aa0cf7fc8f161bdb96166c1644174affacd70d17f372373ca72c8e91116e2d43" ``` @@ -399,12 +395,11 @@ Lifecycle applies the rules for looking up the images: - In case of the *application image* it will look at path `/layout-repo/index.docker.io/library/my-app-image/latest` -Because both images are found, the phase is executed as usual and the `analyzed.toml` file will be updated. A new field `Name` was added into the `analyzed.toml` that will contain the path resolved by the lifecycle, in these cases: +Because both images are found, the phase is executed as usual and the `analyzed.toml` file will be updated. The `run-image.reference` added into the `analyzed.toml` will contain the path resolved by the lifecycle plus the digest reference to the image with the following format `[path]@[digest]`. In case of this example, it will look like this: ```=toml [run-image] - reference = "" - name = "/layout-repo/index.docker.io/cnb/my-partial-stack-run/bionic" + reference = "/layout-repo/index.docker.io/cnb/my-partial-stack-run/bionic@sha256:fab3bb83de466ed29d7e9dcfdbee5b5fb2ff90e91bc849af85b261b4c2062a7a" ``` @@ -444,14 +439,13 @@ layout-repo └── oci-layout ``` -Similar to the previous example, Lifecycle applies the rules for looking up the images and look at path `/layout-repo/index.docker.io/cnb/my-partial-stack-run/bionic` and it determines a partial image was provided and execute the phase logic with the information from the **Image Manifest** and the **Image Config** +Similar to the previous example, Lifecycle applies the rules for looking up the images and look at path `/layout-repo/index.docker.io/cnb/my-partial-stack-run/bionic` and it determines a partial image was provided and execute the phase with the information from the **Image Manifest** and the **Image Config** -The output `analyzed.toml` will also include the new `name` field with the path where the image was located. +The output `analyzed.toml` will also include the new `run-image.reference` field the path and the digest of the run image. ```=toml [run-image] - reference = "" - name = "/layout-repo/index.docker.io/cnb/my-partial-stack-run/bionic" + reference = "/layout-repo/index.docker.io/cnb/my-partial-stack-run/bionic@sha256:fab3bb83de466ed29d7e9dcfdbee5b5fb2ff90e91bc849af85b261b4c2062a7a" ``` @@ -478,16 +472,14 @@ The `` is set with the default value `/layout-repo` `run-image` and `image` arguments are treated in the same way as previous examples, and for `previous-image` argument the looking up images rules are applied and Lifecycle will look at path `/layout-repo/index.docker.io/bar/my-previous-app` for a image in [OCI Image Layout](https://github.com/opencontainers/image-spec/blob/main/image-layout.md) format. -The `analyzed.toml` file es expected to be updated with the previous image section and the new label `name` will be also be there with the path to the `previous-image` +The `analyzed.toml` file es expected to be updated with the `previous-image.reference` containing the path and the digest of the `previous-image` ```=toml [run-image] - reference = "" - name = "/layout-repo/index.docker.io/cnb/my-full-stack-run/bionic" + reference = "/layout-repo/index.docker.io/cnb/my-full-stack-run/bionic@sha256:fab3bb83de466ed29d7e9dcfdbee5b5fb2ff90e91bc849af85b261b4c2062a7a" [previous-image] - reference = "" - name = "/layout-repo/index.docker.io/bar/my-previous-app/latest" + reference = "/layout-repo/index.docker.io/bar/my-previous-app/latest@sha256:aa0cf7fc8f161bdb96166c1644174affacd70d17f372373ca72c8e91116e2d43" ``` @@ -522,8 +514,7 @@ The `/layers/analyzed.toml` file contains the following data: ```=toml [run-image] - reference = "" - name = "/layout-repo/index.docker.io/cnb/my-full-stack-run/bionic" + reference = "/layout-repo/index.docker.io/cnb/my-full-stack-run/bionic@sha256:fab3bb83de466ed29d7e9dcfdbee5b5fb2ff90e91bc849af85b261b4c2062a7a" ``` @@ -545,7 +536,8 @@ Arguments received: The `` is set with the default value `/layout-repo` Lifecycle: - - It will read the `[run-image]` section in the `analyzed.toml` and read the `name` attribute to load the `run-image` image saved on disk in [OCI Image Layout](https://github.com/opencontainers/image-spec/blob/main/image-layout.md) format at path `/layout-repo/index.docker.io/cnb/my-full-stack-run/bionic`. + - It will read the `[run-image]` section in the `analyzed.toml`, it will parse `reference` attribute using the `@` separator and load the `run-image` image saved on disk in [OCI Image Layout](https://github.com/opencontainers/image-spec/blob/main/image-layout.md) format at path `/layout-repo/index.docker.io/cnb/my-full-stack-run/bionic`. + - Lifecycle could also validate the digest of the image loaded is the same as the one established by the `reference`. - Lifecycle will execute the export steps and at the end of the process it will write the *application image* at path `/layout-repo/index.docker.io/library/my-app-image/latest` in [OCI Image Layout](https://github.com/opencontainers/image-spec/blob/main/image-layout.md) format @@ -694,22 +686,22 @@ The [Platform Interface Specification](https://github.com/buildpacks/spec/blob/p | `` | `CNB_USE_OCI_LAYOUT` | false | Enables the capability of resolving image from/to in OCI layout format on disk | | `` | `CNB_OCI_LAYOUT_PATH` | /layout-repo | Path to a directory where the images are saved in OCI layout format| -Also the `analyzed.toml` [file](https://github.com/buildpacks/spec/blob/platform/0.11/platform.md#analyzedtoml-toml) will be updated to include the new `name` field +Also the `analyzed.toml` [file](https://github.com/buildpacks/spec/blob/platform/0.11/platform.md#analyzedtoml-toml) will be updated to include the `reference` format in case of layout is being used. ```=toml [image] reference = "" - name = "" [run-image] reference = "" - name = "" [previous-image] reference = "" - name = "" ``` Where -* `[image|run-image|previos-image].name` MUST point to the path of the image in OCI layout format following the rules describe [previously](#how-to-map-an-image-reference-into-a-path-in-the-layout-repository) +- `[image|run-image|previos-image].reference` MUST be either a digest reference to an image in an OCI registry or the ID of an image in a docker daemon. + - In case an image in [OCI Image Layout](https://github.com/opencontainers/image-spec/blob/main/image-layout.md) format is being used, it will also include the path of the image in OCI layout format following the rules describe [previously](#how-to-map-an-image-reference-into-a-path-in-the-layout-repository) + - The format MUST be as follows: `[path]@[digest]` + From 35bd3e9513454b17c5f29bb7fb5564fd17cff0b8 Mon Sep 17 00:00:00 2001 From: Juan Bustamante Date: Mon, 6 Feb 2023 12:49:10 -0500 Subject: [PATCH 56/62] removing the layout-repo flag Signed-off-by: Juan Bustamante --- text/0000-export-to-oci.md | 78 +++++++++++++++----------------------- 1 file changed, 31 insertions(+), 47 deletions(-) diff --git a/text/0000-export-to-oci.md b/text/0000-export-to-oci.md index cfbc869fb..06e7247b8 100644 --- a/text/0000-export-to-oci.md +++ b/text/0000-export-to-oci.md @@ -75,13 +75,12 @@ Let's see some examples of the proposed behavior ### Requirements -A folder on disk (accessible by the lifecycle) is required to execute the feature, this new folder works as a local registry and the content must be in [OCI Image Layout](https://github.com/opencontainers/image-spec/blob/main/image-layout.md) format. +Lifecycle will converts image references into local paths following define [rules](#how-to-map-an-image-reference-into-a-path-in-the-layout-repository) and the content must be in [OCI Image Layout](https://github.com/opencontainers/image-spec/blob/main/image-layout.md) format. -Let's suppose a directory exists and it has the following structure: +Let's suppose a *platform implementor* creates directories with the following structure: ```=shell -layout-repo -└── index.docker.io +index.docker.io ├── cnb │ ├── my-full-stack-run:bionic │ │ └── bionic @@ -141,7 +140,7 @@ expected analyzed.toml output ```=toml [run-image] - reference = "/layout-repo/index.docker.io/cnb/my-full-stack-run/bionic@sha256:fab3bb83de466ed29d7e9dcfdbee5b5fb2ff90e91bc849af85b261b4c2062a7a" + reference = "/index.docker.io/cnb/my-full-stack-run/bionic@sha256:fab3bb83de466ed29d7e9dcfdbee5b5fb2ff90e91bc849af85b261b4c2062a7a" ``` @@ -160,7 +159,7 @@ expected analyzed.toml output ```=toml [run-image] - reference = "/layout-repo/index.docker.io/cnb/my-full-stack-run/bionic@sha256:fab3bb83de466ed29d7e9dcfdbee5b5fb2ff90e91bc849af85b261b4c2062a7a" + reference = "/index.docker.io/cnb/my-partial-stack-run@sha256:fab3bb83de466ed29d7e9dcfdbee5b5fb2ff90e91bc849af85b261b4c2062a7a" ``` @@ -179,10 +178,10 @@ expected analyzed.toml output ```=toml [run-image] - reference = "/layout-repo/index.docker.io/cnb/my-partial-stack-run/bionic@sha256:fab3bb83de466ed29d7e9dcfdbee5b5fb2ff90e91bc849af85b261b4c2062a7a" + reference = "/index.docker.io/cnb/my-partial-stack-run/bionic@sha256:fab3bb83de466ed29d7e9dcfdbee5b5fb2ff90e91bc849af85b261b4c2062a7a" [previous-image] - reference = "/layout-repo/index.docker.io/bar/my-previous-app/latest@sha256:aa0cf7fc8f161bdb96166c1644174affacd70d17f372373ca72c8e91116e2d43" + reference = "/index.docker.io/bar/my-previous-app/latest@sha256:aa0cf7fc8f161bdb96166c1644174affacd70d17f372373ca72c8e91116e2d43" ``` @@ -198,7 +197,7 @@ expected analyzed.toml output # expected output -ERROR: the run-image could not be found at path: /layout-repo/index.docker.io/cnb/bad-run-image/latest +ERROR: the run-image could not be found at path: /index.docker.io/cnb/bad-run-image/latest ``` ##### Analyzing without run-image argument @@ -234,8 +233,7 @@ Let's also check some examples when the export phase is executed The output will be written into the repository folder described above and it should looks like this: ```=shell -layout-repo -└── index.docker.io +index.docker.io ├── cnb │ └── my-full-stack-run:bionic │ └── bionic @@ -281,8 +279,7 @@ As we can see, the application image `my-app-image` contains a **full** copy of Expected output: ```=shell -layout-repo -└── index.docker.io +index.docker.io ├── cnb │ └── my-partial-stack-run:bionic │ └── bionic @@ -341,22 +338,21 @@ The following new inputs are proposed to be added to these phases | Input | Environment Variable | Default Value | Description |-------|-----------------------|---------------|-------------- - | `` | `CNB_USE_OCI_LAYOUT` | false | Enables the capability of resolving image from/to in OCI layout format on disk | - | `` | `CNB_OCI_LAYOUT_PATH` | /layout-repo | Path to a directory where the images are saved in OCI layout format| + | `` | `CNB_USE_OCI_LAYOUT` | false | Enables the capability of resolving image from/to in OCI layout format on disk | ## How to map an image reference into a path in the layout repository -In the previous examples one key element was how to translate an image reference into a path to look for in the ``, let's define those rules. +In the previous examples one key element was how to translate an image reference into a path, let's define those rules. Considering an **image reference** refers to either a tag reference or digest reference. It could have the following formats -- A tag reference refers to an identifier of form `//:` +- A name reference refers to an identifier of form `//:` - A digest reference refers to a content addressable identifier of form `//@:` The image look up will be done following these rules: - - WHEN `the image points to a tag reference` - - Lifecycle will load/save the image from/to disk in [OCI Image Layout](https://github.com/opencontainers/image-spec/blob/main/image-layout.md) format at `////` + - WHEN `the image points to a name reference` + - Lifecycle will load/save the image from/to disk in [OCI Image Layout](https://github.com/opencontainers/image-spec/blob/main/image-layout.md) format at `////` - WHEN `the image points to a digest reference` - - Lifecycle will load the image from disk in [OCI Image Layout](https://github.com/opencontainers/image-spec/blob/main/image-layout.md) format at `/////` + - Lifecycle will load the image from disk in [OCI Image Layout](https://github.com/opencontainers/image-spec/blob/main/image-layout.md) format at `/////` - WHEN `` is not provided default value will be **index.docker.io** - IF `` is not also provided, then default value will be **library** @@ -388,18 +384,16 @@ Arguments received: - `run-image`: `cnb/my-full-stack-run:bionic` - `image`: `my-app-image` -The `` is set with the default value `/layout-repo` - Lifecycle applies the rules for looking up the images: - - It takes the **tag reference** `cnb/my-full-stack-run:bionic`, applies the conversion rules and look at path `/layout-repo/index.docker.io/cnb/my-full-stack-run/bionic` for an image saved on disk in [OCI Image Layout](https://github.com/opencontainers/image-spec/blob/main/image-layout.md) format. + - It takes the **tag reference** `cnb/my-full-stack-run:bionic`, applies the conversion rules and look at path `/index.docker.io/cnb/my-full-stack-run/bionic` for an image saved on disk in [OCI Image Layout](https://github.com/opencontainers/image-spec/blob/main/image-layout.md) format. - - In case of the *application image* it will look at path `/layout-repo/index.docker.io/library/my-app-image/latest` + - In case of the *application image* it will look at path `/index.docker.io/library/my-app-image/latest` Because both images are found, the phase is executed as usual and the `analyzed.toml` file will be updated. The `run-image.reference` added into the `analyzed.toml` will contain the path resolved by the lifecycle plus the digest reference to the image with the following format `[path]@[digest]`. In case of this example, it will look like this: ```=toml [run-image] - reference = "/layout-repo/index.docker.io/cnb/my-partial-stack-run/bionic@sha256:fab3bb83de466ed29d7e9dcfdbee5b5fb2ff90e91bc849af85b261b4c2062a7a" + reference = "/index.docker.io/cnb/my-partial-stack-run/bionic@sha256:fab3bb83de466ed29d7e9dcfdbee5b5fb2ff90e91bc849af85b261b4c2062a7a" ``` @@ -421,13 +415,10 @@ Arguments received: - `run-image`: `cnb/my-full-partial-run:bionic` - `image`: `my-app-image` -The `` is set with the default value `/layout-repo` - Noticed the structure of the `run-image` provided ```=shell -layout-repo -└── index.docker.io +index.docker.io └── cnb └── my-partial-stack-run:bionic └── bionic @@ -439,13 +430,13 @@ layout-repo └── oci-layout ``` -Similar to the previous example, Lifecycle applies the rules for looking up the images and look at path `/layout-repo/index.docker.io/cnb/my-partial-stack-run/bionic` and it determines a partial image was provided and execute the phase with the information from the **Image Manifest** and the **Image Config** +Similar to the previous example, Lifecycle applies the rules for looking up the images and look at path `/index.docker.io/cnb/my-partial-stack-run/bionic` and it determines a partial image was provided and execute the phase with the information from the **Image Manifest** and the **Image Config** The output `analyzed.toml` will also include the new `run-image.reference` field the path and the digest of the run image. ```=toml [run-image] - reference = "/layout-repo/index.docker.io/cnb/my-partial-stack-run/bionic@sha256:fab3bb83de466ed29d7e9dcfdbee5b5fb2ff90e91bc849af85b261b4c2062a7a" + reference = "/index.docker.io/cnb/my-partial-stack-run/bionic@sha256:fab3bb83de466ed29d7e9dcfdbee5b5fb2ff90e91bc849af85b261b4c2062a7a" ``` @@ -468,18 +459,16 @@ Arguments received: - `previous-image`: `bar/my-previous-app` - `image`: `my-app-image` -The `` is set with the default value `/layout-repo` - -`run-image` and `image` arguments are treated in the same way as previous examples, and for `previous-image` argument the looking up images rules are applied and Lifecycle will look at path `/layout-repo/index.docker.io/bar/my-previous-app` for a image in [OCI Image Layout](https://github.com/opencontainers/image-spec/blob/main/image-layout.md) format. +`run-image` and `image` arguments are treated in the same way as previous examples, and for `previous-image` argument the looking up images rules are applied and Lifecycle will look at path `/index.docker.io/bar/my-previous-app` for a image in [OCI Image Layout](https://github.com/opencontainers/image-spec/blob/main/image-layout.md) format. The `analyzed.toml` file es expected to be updated with the `previous-image.reference` containing the path and the digest of the `previous-image` ```=toml [run-image] - reference = "/layout-repo/index.docker.io/cnb/my-full-stack-run/bionic@sha256:fab3bb83de466ed29d7e9dcfdbee5b5fb2ff90e91bc849af85b261b4c2062a7a" + reference = "/index.docker.io/cnb/my-full-stack-run/bionic@sha256:fab3bb83de466ed29d7e9dcfdbee5b5fb2ff90e91bc849af85b261b4c2062a7a" [previous-image] - reference = "/layout-repo/index.docker.io/bar/my-previous-app/latest@sha256:aa0cf7fc8f161bdb96166c1644174affacd70d17f372373ca72c8e91116e2d43" + reference = "/index.docker.io/bar/my-previous-app/latest@sha256:aa0cf7fc8f161bdb96166c1644174affacd70d17f372373ca72c8e91116e2d43" ``` @@ -492,8 +481,7 @@ Pre-conditions: The following directories are accessible by the lifecycle ```=shell / -├── layout-repo -│ └── index.docker.io +└── index.docker.io │ └── cnb │ └── my-full-stack-run:bionic │ └── bionic @@ -514,7 +502,7 @@ The `/layers/analyzed.toml` file contains the following data: ```=toml [run-image] - reference = "/layout-repo/index.docker.io/cnb/my-full-stack-run/bionic@sha256:fab3bb83de466ed29d7e9dcfdbee5b5fb2ff90e91bc849af85b261b4c2062a7a" + reference = "/index.docker.io/cnb/my-full-stack-run/bionic@sha256:fab3bb83de466ed29d7e9dcfdbee5b5fb2ff90e91bc849af85b261b4c2062a7a" ``` @@ -533,19 +521,16 @@ Arguments received: - `image`: `my-app-image` -The `` is set with the default value `/layout-repo` - Lifecycle: - - It will read the `[run-image]` section in the `analyzed.toml`, it will parse `reference` attribute using the `@` separator and load the `run-image` image saved on disk in [OCI Image Layout](https://github.com/opencontainers/image-spec/blob/main/image-layout.md) format at path `/layout-repo/index.docker.io/cnb/my-full-stack-run/bionic`. + - It will read the `[run-image]` section in the `analyzed.toml`, it will parse `reference` attribute using the `@` separator and load the `run-image` image saved on disk in [OCI Image Layout](https://github.com/opencontainers/image-spec/blob/main/image-layout.md) format at path `/index.docker.io/cnb/my-full-stack-run/bionic`. - Lifecycle could also validate the digest of the image loaded is the same as the one established by the `reference`. - - Lifecycle will execute the export steps and at the end of the process it will write the *application image* at path `/layout-repo/index.docker.io/library/my-app-image/latest` in [OCI Image Layout](https://github.com/opencontainers/image-spec/blob/main/image-layout.md) format + - Lifecycle will execute the export steps and at the end of the process it will write the *application image* at path `/index.docker.io/library/my-app-image/latest` in [OCI Image Layout](https://github.com/opencontainers/image-spec/blob/main/image-layout.md) format The output image will be written at: ```=shell -layout-repo -└── index.docker.io +index.docker.io └── library └── my-app-image └── latest @@ -671,7 +656,7 @@ I think, this PoC demonstrate that adding the exporting to OCI layout format is - Exporting to a tarball can be handle on this RFC or a new one must be created? - What parts of the design do you expect to be resolved through implementation of the feature? - - Handle symbolic links to the blobs in the `` repository, this could be more efficient on hard drive space + - Handle symbolic links to the blobs in the repository, this could be more efficient on hard drive space @@ -684,7 +669,6 @@ The [Platform Interface Specification](https://github.com/buildpacks/spec/blob/p | Input | Environment Variable | Default Value | Description |-------|-----------------------|---------------|-------------- | `` | `CNB_USE_OCI_LAYOUT` | false | Enables the capability of resolving image from/to in OCI layout format on disk | -| `` | `CNB_OCI_LAYOUT_PATH` | /layout-repo | Path to a directory where the images are saved in OCI layout format| Also the `analyzed.toml` [file](https://github.com/buildpacks/spec/blob/platform/0.11/platform.md#analyzedtoml-toml) will be updated to include the `reference` format in case of layout is being used. From eba5b7e9a9cc17fd7012df975edf4989bee951df Mon Sep 17 00:00:00 2001 From: Juan Bustamante Date: Wed, 8 Feb 2023 17:26:35 -0500 Subject: [PATCH 57/62] Revert "Removing the Name field from the analyzed.toml" This reverts commit 9e5c2d0ba198112d0e808b4f04a25937287e48cc. Signed-off-by: Juan Bustamante --- text/0000-export-to-oci.md | 48 +++++++++++++++++--------------------- 1 file changed, 21 insertions(+), 27 deletions(-) diff --git a/text/0000-export-to-oci.md b/text/0000-export-to-oci.md index 06e7247b8..53f70e8a6 100644 --- a/text/0000-export-to-oci.md +++ b/text/0000-export-to-oci.md @@ -140,7 +140,7 @@ expected analyzed.toml output ```=toml [run-image] - reference = "/index.docker.io/cnb/my-full-stack-run/bionic@sha256:fab3bb83de466ed29d7e9dcfdbee5b5fb2ff90e91bc849af85b261b4c2062a7a" + reference = "/layout-repo/index.docker.io/cnb/my-full-stack-run/bionic@sha256:fab3bb83de466ed29d7e9dcfdbee5b5fb2ff90e91bc849af85b261b4c2062a7a" ``` @@ -159,7 +159,7 @@ expected analyzed.toml output ```=toml [run-image] - reference = "/index.docker.io/cnb/my-partial-stack-run@sha256:fab3bb83de466ed29d7e9dcfdbee5b5fb2ff90e91bc849af85b261b4c2062a7a" + reference = "/layout-repo/index.docker.io/cnb/my-partial-stack-run@sha256:fab3bb83de466ed29d7e9dcfdbee5b5fb2ff90e91bc849af85b261b4c2062a7a" ``` @@ -178,10 +178,10 @@ expected analyzed.toml output ```=toml [run-image] - reference = "/index.docker.io/cnb/my-partial-stack-run/bionic@sha256:fab3bb83de466ed29d7e9dcfdbee5b5fb2ff90e91bc849af85b261b4c2062a7a" + reference = "/layout-repo/index.docker.io/cnb/my-partial-stack-run/bionic@sha256:fab3bb83de466ed29d7e9dcfdbee5b5fb2ff90e91bc849af85b261b4c2062a7a" [previous-image] - reference = "/index.docker.io/bar/my-previous-app/latest@sha256:aa0cf7fc8f161bdb96166c1644174affacd70d17f372373ca72c8e91116e2d43" + reference = "/layout-repo/index.docker.io/bar/my-previous-app/latest@sha256:aa0cf7fc8f161bdb96166c1644174affacd70d17f372373ca72c8e91116e2d43" ``` @@ -336,8 +336,8 @@ Notice that we are relying on the OCI format Specification to expose the data fo The following new inputs are proposed to be added to these phases - | Input | Environment Variable | Default Value | Description - |-------|-----------------------|---------------|-------------- + | Input | Environment Variable | Default Value | Description | + |-------|-----------------------|---------------|-------------| | `` | `CNB_USE_OCI_LAYOUT` | false | Enables the capability of resolving image from/to in OCI layout format on disk | ## How to map an image reference into a path in the layout repository @@ -389,12 +389,11 @@ Lifecycle applies the rules for looking up the images: - In case of the *application image* it will look at path `/index.docker.io/library/my-app-image/latest` -Because both images are found, the phase is executed as usual and the `analyzed.toml` file will be updated. The `run-image.reference` added into the `analyzed.toml` will contain the path resolved by the lifecycle plus the digest reference to the image with the following format `[path]@[digest]`. In case of this example, it will look like this: +Because both images are found, the phase is executed as usual and the `analyzed.toml` file will be updated. A new field `Name` was added into the `analyzed.toml` that will contain the path resolved by the lifecycle, in these cases: ```=toml [run-image] - reference = "/index.docker.io/cnb/my-partial-stack-run/bionic@sha256:fab3bb83de466ed29d7e9dcfdbee5b5fb2ff90e91bc849af85b261b4c2062a7a" - + reference = "/layout-repo/index.docker.io/cnb/my-partial-stack-run/bionic@sha256:fab3bb83de466ed29d7e9dcfdbee5b5fb2ff90e91bc849af85b261b4c2062a7a" ``` ##### Analyzing run-image partial saved on disk @@ -430,14 +429,12 @@ index.docker.io └── oci-layout ``` -Similar to the previous example, Lifecycle applies the rules for looking up the images and look at path `/index.docker.io/cnb/my-partial-stack-run/bionic` and it determines a partial image was provided and execute the phase with the information from the **Image Manifest** and the **Image Config** - -The output `analyzed.toml` will also include the new `run-image.reference` field the path and the digest of the run image. +Similar to the previous example, Lifecycle applies the rules for looking up the images and look at path `/layout-repo/index.docker.io/cnb/my-partial-stack-run/bionic` and it determines a partial image was provided and execute the phase with the information from the **Image Manifest** and the **Image Config** +The output `analyzed.toml` will also include the new `name` field with the path where the image was located. ```=toml [run-image] - reference = "/index.docker.io/cnb/my-partial-stack-run/bionic@sha256:fab3bb83de466ed29d7e9dcfdbee5b5fb2ff90e91bc849af85b261b4c2062a7a" - + reference = "/layout-repo/index.docker.io/cnb/my-partial-stack-run/bionic@sha256:fab3bb83de466ed29d7e9dcfdbee5b5fb2ff90e91bc849af85b261b4c2062a7a" ``` ##### Analyzing previous-image @@ -461,14 +458,14 @@ Arguments received: `run-image` and `image` arguments are treated in the same way as previous examples, and for `previous-image` argument the looking up images rules are applied and Lifecycle will look at path `/index.docker.io/bar/my-previous-app` for a image in [OCI Image Layout](https://github.com/opencontainers/image-spec/blob/main/image-layout.md) format. -The `analyzed.toml` file es expected to be updated with the `previous-image.reference` containing the path and the digest of the `previous-image` +The `analyzed.toml` file es expected to be updated with the previous image section and the new label `name` will be also be there with the path to the `previous-image` ```=toml [run-image] - reference = "/index.docker.io/cnb/my-full-stack-run/bionic@sha256:fab3bb83de466ed29d7e9dcfdbee5b5fb2ff90e91bc849af85b261b4c2062a7a" + reference = "/layout-repo/index.docker.io/cnb/my-full-stack-run/bionic@sha256:fab3bb83de466ed29d7e9dcfdbee5b5fb2ff90e91bc849af85b261b4c2062a7a" [previous-image] - reference = "/index.docker.io/bar/my-previous-app/latest@sha256:aa0cf7fc8f161bdb96166c1644174affacd70d17f372373ca72c8e91116e2d43" + reference = "/layout-repo/index.docker.io/bar/my-previous-app/latest@sha256:aa0cf7fc8f161bdb96166c1644174affacd70d17f372373ca72c8e91116e2d43" ``` @@ -502,7 +499,7 @@ The `/layers/analyzed.toml` file contains the following data: ```=toml [run-image] - reference = "/index.docker.io/cnb/my-full-stack-run/bionic@sha256:fab3bb83de466ed29d7e9dcfdbee5b5fb2ff90e91bc849af85b261b4c2062a7a" + reference = "/layout-repo/index.docker.io/cnb/my-full-stack-run/bionic@sha256:fab3bb83de466ed29d7e9dcfdbee5b5fb2ff90e91bc849af85b261b4c2062a7a" ``` @@ -522,9 +519,9 @@ Arguments received: - `image`: `my-app-image` Lifecycle: - - It will read the `[run-image]` section in the `analyzed.toml`, it will parse `reference` attribute using the `@` separator and load the `run-image` image saved on disk in [OCI Image Layout](https://github.com/opencontainers/image-spec/blob/main/image-layout.md) format at path `/index.docker.io/cnb/my-full-stack-run/bionic`. + - It will read the `[run-image]` section in the `analyzed.toml`, it will parse `reference` attribute using the `@` separator and load the `run-image` image saved on disk in [OCI Image Layout](https://github.com/opencontainers/image-spec/blob/main/image-layout.md) format at path `/layout-repo/index.docker.io/cnb/my-full-stack-run/bionic`. - Lifecycle could also validate the digest of the image loaded is the same as the one established by the `reference`. - - Lifecycle will execute the export steps and at the end of the process it will write the *application image* at path `/index.docker.io/library/my-app-image/latest` in [OCI Image Layout](https://github.com/opencontainers/image-spec/blob/main/image-layout.md) format + - Lifecycle will execute the export steps and at the end of the process it will write the *application image* at path `/layout-repo/index.docker.io/library/my-app-image/latest` in [OCI Image Layout](https://github.com/opencontainers/image-spec/blob/main/image-layout.md) format The output image will be written at: @@ -666,11 +663,11 @@ I think, this PoC demonstrate that adding the exporting to OCI layout format is The [Platform Interface Specification](https://github.com/buildpacks/spec/blob/platform/0.11/platform.md#inputs-5) must be updated to include the following inputs to the [Create](https://buildpacks.io/docs/concepts/components/lifecycle/create/), [Analyze](https://buildpacks.io/docs/concepts/components/lifecycle/analyze/) and [Export](https://buildpacks.io/docs/concepts/components/lifecycle/export/) phases -| Input | Environment Variable | Default Value | Description -|-------|-----------------------|---------------|-------------- +| Input | Environment Variable | Default Value | Description| +|-------|-----------------------|---------------|------------| | `` | `CNB_USE_OCI_LAYOUT` | false | Enables the capability of resolving image from/to in OCI layout format on disk | -Also the `analyzed.toml` [file](https://github.com/buildpacks/spec/blob/platform/0.11/platform.md#analyzedtoml-toml) will be updated to include the `reference` format in case of layout is being used. +Also the `analyzed.toml` [file](https://github.com/buildpacks/spec/blob/platform/0.11/platform.md#analyzedtoml-toml) will be updated to include the new `name` field ```=toml [image] @@ -685,7 +682,4 @@ Also the `analyzed.toml` [file](https://github.com/buildpacks/spec/blob/platform Where -- `[image|run-image|previos-image].reference` MUST be either a digest reference to an image in an OCI registry or the ID of an image in a docker daemon. - - In case an image in [OCI Image Layout](https://github.com/opencontainers/image-spec/blob/main/image-layout.md) format is being used, it will also include the path of the image in OCI layout format following the rules describe [previously](#how-to-map-an-image-reference-into-a-path-in-the-layout-repository) - - The format MUST be as follows: `[path]@[digest]` - +* `[image|run-image|previos-image].name` MUST point to the path of the image in OCI layout format following the rules describe [previously](#how-to-map-an-image-reference-into-a-path-in-the-layout-repository) From 1086f453a927260743f7d4a70a0e9e3c591d4e2e Mon Sep 17 00:00:00 2001 From: Juan Bustamante Date: Wed, 8 Feb 2023 18:27:57 -0500 Subject: [PATCH 58/62] adding Natalie's feedback Signed-off-by: Juan Bustamante Signed-off-by: Juan Bustamante --- text/0000-export-to-oci.md | 131 ++++++++++++++++++++++++------------- 1 file changed, 87 insertions(+), 44 deletions(-) diff --git a/text/0000-export-to-oci.md b/text/0000-export-to-oci.md index 53f70e8a6..3cb550a98 100644 --- a/text/0000-export-to-oci.md +++ b/text/0000-export-to-oci.md @@ -77,10 +77,11 @@ Let's see some examples of the proposed behavior Lifecycle will converts image references into local paths following define [rules](#how-to-map-an-image-reference-into-a-path-in-the-layout-repository) and the content must be in [OCI Image Layout](https://github.com/opencontainers/image-spec/blob/main/image-layout.md) format. -Let's suppose a *platform implementor* creates directories with the following structure: +Let's suppose a *platform implementor* creates a directory with the following structure: ```=shell -index.docker.io +layout-repo +└── index.docker.io ├── cnb │ ├── my-full-stack-run:bionic │ │ └── bionic @@ -118,8 +119,8 @@ The images named **cnb/my-full-stack-run** and **cnb/my-partial-stack-run** repr For each example case, I will present two ways of enabling the new capability: -- Using an environment Variable -- Using a new `-layout` flag +- Using an environment variables +- Using the new `-layout` and `layout-dir` flags In any case the expected output is the same. @@ -129,11 +130,12 @@ In any case the expected output is the same. ```=shell > export CNB_USE_LAYOUT=true +> export CNB_LAYOUT_DIR=/layout-repo > /cnb/lifecycle/analyzer -run-image cnb/my-full-stack-run:bionic my-app-image # OR -> /cnb/lifecycle/analyzer -layout -run-image cnb/my-full-stack-run:bionic my-app-image +> /cnb/lifecycle/analyzer -layout -layout-dir /layout-repo -run-image cnb/my-full-stack-run:bionic my-app-image ``` expected analyzed.toml output @@ -147,12 +149,13 @@ expected analyzed.toml output ##### Analyzing run-image partial saved on disk ```=shell -> export CNB_USE_OCI_LAYOUT=true +> export CNB_USE_LAYOUT=true +> export CNB_LAYOUT_DIR=/layout-repo > /cnb/lifecycle/analyzer -run-image cnb/cnb/my-partial-stack-run:bionic my-app-image # OR -> /cnb/lifecycle/analyzer -layout -run-image cnb/cnb/my-partial-stack-run:bionic my-app-image +> /cnb/lifecycle/analyzer -layout -layout-dir /layout-repo -run-image cnb/cnb/my-partial-stack-run:bionic my-app-image ``` expected analyzed.toml output @@ -166,12 +169,13 @@ expected analyzed.toml output ##### Analyzing previous-image ```=shell -> export CNB_USE_OCI_LAYOUT=true +> export CNB_USE_LAYOUT=true +> export CNB_LAYOUT_DIR=/layout-repo > /cnb/lifecycle/analyzer -run-image cnb/my-full-stack-run:bionic -previous-image bar/my-previous-app my-app-image # OR -> /cnb/lifecycle/analyzer -layout -run-image cnb/my-full-stack-run:bionic-previous-image bar/my-previous-app my-app-image +> /cnb/lifecycle/analyzer -layout -layout-dir /layout-repo -run-image cnb/my-full-stack-run:bionic-previous-image bar/my-previous-app my-app-image ``` expected analyzed.toml output @@ -188,33 +192,48 @@ expected analyzed.toml output ##### Analyzing run-image not saved on disk ```=shell -> export CNB_USE_OCI_LAYOUT=true +> export CNB_USE_LAYOUT=true +> export CNB_LAYOUT_DIR=/layout-repo > /cnb/lifecycle/analyzer -run-image cnb/bad-run-image my-app-image # OR -> /cnb/lifecycle/analyzer -layout -run-image cnb/bad-run-image my-app-image +> /cnb/lifecycle/analyzer -layout -layout-dir /layout-repo -run-image cnb/bad-run-image my-app-image # expected output -ERROR: the run-image could not be found at path: /index.docker.io/cnb/bad-run-image/latest +ERROR: the run-image could not be found at path: /layout-repo/index.docker.io/cnb/bad-run-image/latest ``` ##### Analyzing without run-image argument ```=shell -> export CNB_USE_OCI_LAYOUT=true +> export CNB_USE_LAYOUT=true +> export CNB_LAYOUT_DIR=/layout-repo > /cnb/lifecycle/analyzer my-app-image # OR -> /cnb/lifecycle/analyzer -layout my-app-image +> /cnb/lifecycle/analyzer -layout -layout-dir /layout-repo my-app-image # expected output ERROR: -run-image is required when OCI Layout feature is enabled ``` +```=shell +> export CNB_USE_LAYOUT=true +> /cnb/lifecycle/analyzer -run-image cnb/bad-run-image my-app-image + +# OR + +> /cnb/lifecycle/analyzer -layout -run-image cnb/bad-run-image my-app-image + +# expected output + +ERROR: defining a layout directory is required when OCI Layout feature is enabled. Use -layout-dir flag or CNB_LAYOUT_DIR environment variable +``` + Let's also check some examples when the export phase is executed #### Export phase @@ -222,18 +241,20 @@ Let's also check some examples when the export phase is executed ##### Export to OCI using run-image full saved on disk ```=shell -> export CNB_USE_OCI_LAYOUT=true +> export CNB_USE_LAYOUT=true +> export CNB_LAYOUT_DIR=/layout-repo > /cnb/lifecycle/exporter my-app-image # OR -> /cnb/lifecycle/exporter -layout my-app-image +> /cnb/lifecycle/exporter -layout -layout-dir /layout-repo my-app-image ``` The output will be written into the repository folder described above and it should looks like this: ```=shell -index.docker.io +layout-repo +└── index.docker.io ├── cnb │ └── my-full-stack-run:bionic │ └── bionic @@ -268,18 +289,20 @@ As we can see, the application image `my-app-image` contains a **full** copy of ##### Export to OCI using run-image partially saved on disk ```=shell -> export CNB_USE_OCI_LAYOUT=true +> export CNB_USE_LAYOUT=true +> export CNB_LAYOUT_DIR=/layout-repo > /cnb/lifecycle/exporter my-app-image # OR -> /cnb/lifecycle/exporter -layout my-app-image +> /cnb/lifecycle/exporter -layout -layout-dir /layout-repo my-app-image ``` Expected output: ```=shell -index.docker.io +layout-repo +└── index.docker.io ├── cnb │ └── my-partial-stack-run:bionic │ └── bionic @@ -309,12 +332,13 @@ As we can see, the application image `my-app-image` has missing `blobs` because Any combination of using multiple sources or sinks in the Lifecycle invocation of phases should throw an error to the user. For example: ```=shell -> export CNB_USE_OCI_LAYOUT=true +> export CNB_USE_LAYOUT=true +> export CNB_LAYOUT_DIR=/layout-repo > /cnb/lifecycle/exporter -daemon -run-image cnb/my-full-stack-run:bionic my-app-image # OR -> /cnb/lifecycle/exporter -layout -daemon -run-image cnb/my-full-stack-run:bionic my-app-image +> /cnb/lifecycle/exporter -layout -layout-dir /layout-repo -daemon -run-image cnb/my-full-stack-run:bionic my-app-image ERROR: exporting to multiples target is not allowed ``` @@ -336,9 +360,10 @@ Notice that we are relying on the OCI format Specification to expose the data fo The following new inputs are proposed to be added to these phases - | Input | Environment Variable | Default Value | Description | - |-------|-----------------------|---------------|-------------| - | `` | `CNB_USE_OCI_LAYOUT` | false | Enables the capability of resolving image from/to in OCI layout format on disk | + | Input | Environment Variable | Default Value | Description + |-------|-----------------------|---------------|-------------- + | `` | `CNB_USE_LAYOUT` | false | Enables the capability of resolving image from/to in OCI layout format on disk | + | `` | `CNB_LAYOUT_DIR` | | Path to a directory where the images are saved in OCI layout format| ## How to map an image reference into a path in the layout repository @@ -360,9 +385,9 @@ The image look up will be done following these rules: ## Examples In all the examples the new feature is enabled by the use of the new flag `-layout` or by setting -the new environment variable `CNB_USE_OCI_LAYOUT` to true. +the new environment variable `CNB_USE_LAYOUT` to true. -Let's review some of the previous examples +Let's review some previous examples #### Analyze phase @@ -372,11 +397,12 @@ Command: ```=shell > export CNB_USE_LAYOUT=true +> export CNB_LAYOUT_DIR=/layout-repo > /cnb/lifecycle/analyzer -run-image cnb/my-full-stack-run:bionic my-app-image # OR -> /cnb/lifecycle/analyzer -layout -run-image cnb/my-full-stack-run:bionic my-app-image +> /cnb/lifecycle/analyzer -layout -layout-dir /layout-repo -run-image cnb/my-full-stack-run:bionic my-app-image ``` Arguments received: @@ -384,12 +410,15 @@ Arguments received: - `run-image`: `cnb/my-full-stack-run:bionic` - `image`: `my-app-image` -Lifecycle applies the rules for looking up the images: - - It takes the **tag reference** `cnb/my-full-stack-run:bionic`, applies the conversion rules and look at path `/index.docker.io/cnb/my-full-stack-run/bionic` for an image saved on disk in [OCI Image Layout](https://github.com/opencontainers/image-spec/blob/main/image-layout.md) format. +The `` is set with the value `/layout-repo` - - In case of the *application image* it will look at path `/index.docker.io/library/my-app-image/latest` +Lifecycle applies the rules for looking up the images: + - It takes the **tag reference** `cnb/my-full-stack-run:bionic`, applies the conversion rules and gets `/index.docker.io/cnb/my-full-stack-run/bionic` + - It will append the `` at the beginning, getting the following path: `/layout-repo/index.docker.io/cnb/my-full-stack-run/bionic` + - It will look for an image saved on disk in [OCI Image Layout](https://github.com/opencontainers/image-spec/blob/main/image-layout.md) format at path `/layout-repo/index.docker.io/cnb/my-full-stack-run/bionic`. + - In case of the *application image* it will look at path `/layout-repo/index.docker.io/library/my-app-image/latest` -Because both images are found, the phase is executed as usual and the `analyzed.toml` file will be updated. A new field `Name` was added into the `analyzed.toml` that will contain the path resolved by the lifecycle, in these cases: +Because both images are found, the phase is executed as usual and the `analyzed.toml` file will be updated. The `run-image.reference` added into the `analyzed.toml` will contain the path resolved by the lifecycle plus the digest reference to the image with the following format `[path]@[digest]`. In case of this example, it will look like this: ```=toml [run-image] @@ -401,7 +430,7 @@ Because both images are found, the phase is executed as usual and the `analyzed. Command received: ```=shell -> export CNB_USE_OCI_LAYOUT=true +> export CNB_USE_LAYOUT=true > /cnb/lifecycle/analyzer -run-image cnb/cnb/my-partial-stack-run:bionic my-app-image # OR @@ -414,10 +443,13 @@ Arguments received: - `run-image`: `cnb/my-full-partial-run:bionic` - `image`: `my-app-image` +The `` is set with the default value `/layout-repo` + Noticed the structure of the `run-image` provided ```=shell -index.docker.io +layout-repo +└── index.docker.io └── cnb └── my-partial-stack-run:bionic └── bionic @@ -430,7 +462,8 @@ index.docker.io ``` Similar to the previous example, Lifecycle applies the rules for looking up the images and look at path `/layout-repo/index.docker.io/cnb/my-partial-stack-run/bionic` and it determines a partial image was provided and execute the phase with the information from the **Image Manifest** and the **Image Config** -The output `analyzed.toml` will also include the new `name` field with the path where the image was located. + +The output `analyzed.toml` will also include the new `run-image.reference` field the path and the digest of the run image. ```=toml [run-image] @@ -442,7 +475,7 @@ The output `analyzed.toml` will also include the new `name` field with the path Command received: ```=shell -> export CNB_USE_OCI_LAYOUT=true +> export CNB_USE_LAYOUT=true > /cnb/lifecycle/analyzer -run-image cnb/my-full-stack-run:bionic -previous-image bar/my-previous-app my-app-image # OR @@ -456,9 +489,11 @@ Arguments received: - `previous-image`: `bar/my-previous-app` - `image`: `my-app-image` +The `` is set with the default value `/layout-repo` + `run-image` and `image` arguments are treated in the same way as previous examples, and for `previous-image` argument the looking up images rules are applied and Lifecycle will look at path `/index.docker.io/bar/my-previous-app` for a image in [OCI Image Layout](https://github.com/opencontainers/image-spec/blob/main/image-layout.md) format. -The `analyzed.toml` file es expected to be updated with the previous image section and the new label `name` will be also be there with the path to the `previous-image` +The `analyzed.toml` file es expected to be updated with the `previous-image.reference` containing the path and the digest of the `previous-image` ```=toml [run-image] @@ -478,7 +513,8 @@ Pre-conditions: The following directories are accessible by the lifecycle ```=shell / -└── index.docker.io +├── layout-repo +│ └── index.docker.io │ └── cnb │ └── my-full-stack-run:bionic │ └── bionic @@ -506,7 +542,7 @@ The `/layers/analyzed.toml` file contains the following data: Command executed: ```=shell -> export CNB_USE_OCI_LAYOUT=true +> export CNB_USE_LAYOUT=true > /cnb/lifecycle/exporter my-app-image # OR @@ -518,6 +554,8 @@ Arguments received: - `image`: `my-app-image` +The `` is set with the default value `/layout-repo` + Lifecycle: - It will read the `[run-image]` section in the `analyzed.toml`, it will parse `reference` attribute using the `@` separator and load the `run-image` image saved on disk in [OCI Image Layout](https://github.com/opencontainers/image-spec/blob/main/image-layout.md) format at path `/layout-repo/index.docker.io/cnb/my-full-stack-run/bionic`. - Lifecycle could also validate the digest of the image loaded is the same as the one established by the `reference`. @@ -527,7 +565,8 @@ Lifecycle: The output image will be written at: ```=shell -index.docker.io +layout-repo +└── index.docker.io └── library └── my-app-image └── latest @@ -653,7 +692,7 @@ I think, this PoC demonstrate that adding the exporting to OCI layout format is - Exporting to a tarball can be handle on this RFC or a new one must be created? - What parts of the design do you expect to be resolved through implementation of the feature? - - Handle symbolic links to the blobs in the repository, this could be more efficient on hard drive space + - Handle symbolic links to the blobs in the `` repository, this could be more efficient on hard drive space @@ -665,9 +704,10 @@ The [Platform Interface Specification](https://github.com/buildpacks/spec/blob/p | Input | Environment Variable | Default Value | Description| |-------|-----------------------|---------------|------------| -| `` | `CNB_USE_OCI_LAYOUT` | false | Enables the capability of resolving image from/to in OCI layout format on disk | +| `` | `CNB_USE_LAYOUT` | false | Enables the capability of resolving image from/to in OCI layout format on disk | +| `` | `CNB_LAYOUT_DIR` | | Path to a directory where the images are saved in OCI layout format| -Also the `analyzed.toml` [file](https://github.com/buildpacks/spec/blob/platform/0.11/platform.md#analyzedtoml-toml) will be updated to include the new `name` field +Also the `analyzed.toml` [file](https://github.com/buildpacks/spec/blob/platform/0.11/platform.md#analyzedtoml-toml) will be updated to include the `reference` format in case of layout is being used. ```=toml [image] @@ -682,4 +722,7 @@ Also the `analyzed.toml` [file](https://github.com/buildpacks/spec/blob/platform Where -* `[image|run-image|previos-image].name` MUST point to the path of the image in OCI layout format following the rules describe [previously](#how-to-map-an-image-reference-into-a-path-in-the-layout-repository) +- `[image|run-image|previos-image].reference` MUST be either a digest reference to an image in an OCI registry or the ID of an image in a docker daemon. + - In case an image in [OCI Image Layout](https://github.com/opencontainers/image-spec/blob/main/image-layout.md) format is being used, it will also include the path of the image in OCI layout format following the rules describe [previously](#how-to-map-an-image-reference-into-a-path-in-the-layout-repository) + - The format MUST be as follows: `[path]@[digest]` + From b09fea05ebd6c039e5c2acb9cdfd03202f3e796d Mon Sep 17 00:00:00 2001 From: Juan Bustamante Date: Wed, 15 Feb 2023 11:58:46 -0500 Subject: [PATCH 59/62] adding layout-dir to some missing places Signed-off-by: Juan Bustamante --- text/0000-export-to-oci.md | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/text/0000-export-to-oci.md b/text/0000-export-to-oci.md index 3cb550a98..9bcd5fccd 100644 --- a/text/0000-export-to-oci.md +++ b/text/0000-export-to-oci.md @@ -221,6 +221,8 @@ ERROR: the run-image could not be found at path: /layout-repo/index.docker.io/cn ERROR: -run-image is required when OCI Layout feature is enabled ``` +##### Analyzing without layout-dir argument + ```=shell > export CNB_USE_LAYOUT=true > /cnb/lifecycle/analyzer -run-image cnb/bad-run-image my-app-image @@ -375,9 +377,9 @@ Considering an **image reference** refers to either a tag reference or digest re The image look up will be done following these rules: - WHEN `the image points to a name reference` - - Lifecycle will load/save the image from/to disk in [OCI Image Layout](https://github.com/opencontainers/image-spec/blob/main/image-layout.md) format at `////` + - Lifecycle will load/save the image from/to disk in [OCI Image Layout](https://github.com/opencontainers/image-spec/blob/main/image-layout.md) format at `////` - WHEN `the image points to a digest reference` - - Lifecycle will load the image from disk in [OCI Image Layout](https://github.com/opencontainers/image-spec/blob/main/image-layout.md) format at `/////` + - Lifecycle will load the image from disk in [OCI Image Layout](https://github.com/opencontainers/image-spec/blob/main/image-layout.md) format at `/////` - WHEN `` is not provided default value will be **index.docker.io** - IF `` is not also provided, then default value will be **library** From 76d034f4c6ff2470e85b38d2310d343f97d950f2 Mon Sep 17 00:00:00 2001 From: Juan Bustamante Date: Wed, 15 Feb 2023 16:37:34 -0500 Subject: [PATCH 60/62] updating the unresolved questions Signed-off-by: Juan Bustamante --- text/0000-export-to-oci.md | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/text/0000-export-to-oci.md b/text/0000-export-to-oci.md index 9bcd5fccd..c96850128 100644 --- a/text/0000-export-to-oci.md +++ b/text/0000-export-to-oci.md @@ -690,12 +690,13 @@ I think, this PoC demonstrate that adding the exporting to OCI layout format is - What parts of the design do you expect to be resolved before this gets merged? - Tools like [umoci](https://umo.ci/) used to create a runtime bundle from an image in OCI layout format, requires the [annotation](https://github.com/opencontainers/image-spec/blob/main/annotations.md#pre-defined-annotation-keys) `org.opencontainers.image.ref.name` to be present. Also tools like [skopeo](https://github.com/containers/skopeo) when an image is `copy` in oci format the annotation is included. We are not adding the annotation as part of the Buildpacks Specification, but in this case this could make our output incompatible with some other tooling. - - Depending on the previous question, the rules for mapping an image reference into a path could change, also validations that the Lifecycle could do to verify the `org.opencontainers.image.ref.name` against the value provided in the image reference. - - Exporting to a tarball can be handle on this RFC or a new one must be created? + - **Answer:** we agreed on adding `org.opencontainers.image.ref.name` annotation + - Exporting to a tarball can be handled on this RFC or a new one must be created? + - **Answer:** this can be handled on a different RFC - What parts of the design do you expect to be resolved through implementation of the feature? - Handle symbolic links to the blobs in the `` repository, this could be more efficient on hard drive space - + - **Answer:** this can be handled on the implementation side From 77c9fcce82e1504d29d119e6444e8d5d001e3ddd Mon Sep 17 00:00:00 2001 From: Juan Bustamante Date: Wed, 15 Feb 2023 16:44:18 -0500 Subject: [PATCH 61/62] updating the spec changes Signed-off-by: Juan Bustamante --- text/0000-export-to-oci.md | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/text/0000-export-to-oci.md b/text/0000-export-to-oci.md index c96850128..dc4e70dbd 100644 --- a/text/0000-export-to-oci.md +++ b/text/0000-export-to-oci.md @@ -725,7 +725,8 @@ Also the `analyzed.toml` [file](https://github.com/buildpacks/spec/blob/platform Where -- `[image|run-image|previos-image].reference` MUST be either a digest reference to an image in an OCI registry or the ID of an image in a docker daemon. - - In case an image in [OCI Image Layout](https://github.com/opencontainers/image-spec/blob/main/image-layout.md) format is being used, it will also include the path of the image in OCI layout format following the rules describe [previously](#how-to-map-an-image-reference-into-a-path-in-the-layout-repository) - - The format MUST be as follows: `[path]@[digest]` +- `[image|run-image|previos-image].reference` MUST be either: + - A digest reference to an image in an OCI registry + - The ID of an image in a docker daemon + - The path to an image in OCI layout format From 8beb9e83ba5efc77bb26985dbd2a2daccd6199ec Mon Sep 17 00:00:00 2001 From: Juan Bustamante Date: Fri, 17 Feb 2023 14:47:56 -0500 Subject: [PATCH 62/62] Update 0000-export-to-oci.md Fixing error message according to the implementation Signed-off-by: Juan Bustamante --- text/0000-export-to-oci.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/text/0000-export-to-oci.md b/text/0000-export-to-oci.md index dc4e70dbd..d1b806f22 100644 --- a/text/0000-export-to-oci.md +++ b/text/0000-export-to-oci.md @@ -342,7 +342,7 @@ Any combination of using multiple sources or sinks in the Lifecycle invocation o > /cnb/lifecycle/exporter -layout -layout-dir /layout-repo -daemon -run-image cnb/my-full-stack-run:bionic my-app-image -ERROR: exporting to multiples target is not allowed +ERROR: exporting to multiple targets is unsupported ``` # How it Works