Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

OCI layout documentation #574

Merged
merged 15 commits into from
Aug 18, 2023
Merged
Changes from 6 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
151 changes: 151 additions & 0 deletions content/docs/app-developer-guide/export-to-oci-layout.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,151 @@
+++
title="Export to OCI layout format"
weight=3
summary="Learn how to export your application image to disk in OCI layout format"
+++

<div class="quote mb-4">
The OCI Image Layout is the directory structure for OCI content-addressable blobs and location-addressable references.
<div class="author">See the <a href="https://github.com/opencontainers/image-spec/blob/main/image-layout.md">specification</a></div>
jjbustamante marked this conversation as resolved.
Show resolved Hide resolved
</div>

Exporting to OCI layout format is an **experimental** feature available on pack since version X.Y.Z

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we need a table somewhere comparing the cnb layout with the oci layout. The first question I'd ask as a user is "Which layout should I choose?", and such a table would contain enough "Why?" to answer which layout to choose.

### 1. Enable experimental feature

Verify your pack version is equal or greater than X.Y.Z
jjbustamante marked this conversation as resolved.
Show resolved Hide resolved

```bash
pack version
```

Enable the experimental features on pack

```bash
pack config experimental true
```

You can confirm everything is fine, checking the `config.toml` file in your `PACK_HOME` installation folder, for example:
jjbustamante marked this conversation as resolved.
Show resolved Hide resolved

```bash
cat ~/.pack/config.toml
experimental = true
layout-repo-dir = "<$HOME>/.pack/layout-repo"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does pack config experimental true add the "layout-repo-dir" to config.toml?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, that's correct

```

The configuration shows the experimental mode was **enabled** and a local directory to save images on disk was configured to path `<$HOME>/.pack/layout-repo`.

### 2. Build the app

If you haven't already, please follow the steps to [build an app](/docs/app-developer-guide/build-an-app), once you managed to build a sample application you can try exporting the sample application to disk in OCI layout format.
jjbustamante marked this conversation as resolved.
Show resolved Hide resolved
jjbustamante marked this conversation as resolved.
Show resolved Hide resolved

The OCI layout feature must be enabled using the convention `oci:<path/to/save/image>` in the `<image-name>` parameter when invoking `pack build`.

For example:

```bash
pack build oci:sample-app --path samples/apps/java-maven --builder cnbs/sample-builder:bionic
```

It will save the image in a folder `./sample-app` created in your current directory.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why do we need layout-repo-dir if the OCI layout is saved to the PWD?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We are using the layout-repo-dir to save the run-image on disk because actually invoking the lifecycle. When lifecycle is configured to use OCI layout it is expecting to have all the require inputs (like the run-image) on disk, because that should be transparent for users, we are saving those images in that particular folder.


### 3. Check your image

**Congratulations!**

You can verify your application image was saved on disk in a folder called `./sample-app` in your current directory in OCI layout format, for example:
jjbustamante marked this conversation as resolved.
Show resolved Hide resolved

```bash
tree sample-app

sample-app
├── blobs
│ └── sha256
│ ├── 141bfb0cd434d425bc70edb9e56ea11d07aed76450eb0e73e6110645f251a8d3
│ ├── 2fa192256ce255c6ea6c1296eadfe2feba8094f40e6aa85e699645caca2e85d8
│ ├── 5a44e4f7b58d74fe6f92dd7028075c91191128d1e2e7f39846fe061a9a98836e
│ ├── 72d9f18d70f395ff9bfae4d193077ccea3ca583e3da3dd66f5c84520c0100727
│ ├── 827746ec7ba80f4e4811b6c9195b6f810fbc2d58a6c9cc337bf0305791f24e97
│ ├── ad13830c92258c952f25d561d8bf7d9eb58b8a3003960db1502cbda8239130b5
│ ├── b97b58b190d5f731c879b0f7446a2bd554863b51851e03757199c74dd922ce61
│ ├── c44222730efa142cd5bedc0babf82a9a07d325494be7f5c3cfde56f43166b65f
│ ├── e1048fb89c3194a1f0542c0847aa086a7034dd7867c48fe8c93675cf36f90610
│ ├── f0a30c5bc44742065b1b4ffa95271a39994f05ba7a03dd7e7143d1d3e45fa0b1
│ └── f9d6350d0c44c0e7165a522155f53181ce8c163a6b8ead1f6baea22d1a8d8a78
├── index.json
└── oci-layout

3 directories, 13 files
```
If you are interested on keep playing with the image in OCI layout format, one tool you can take a look is [umoci](https://umo.ci/) it can help you to create a
jjbustamante marked this conversation as resolved.
Show resolved Hide resolved
[runtime bundler](https://github.com/opencontainers/runtime-spec) that can be executed with another tool like [runc](https://github.com/opencontainers/runc)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There a number of tools here that we might need to explain to end-users. I think we're using umoci because it is a strict-OCI compliant tool? That assembles an image. I presume that docker can also execute a strict-OCI image? If so, should we prefer to not introduce a new runtime there?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, I had some doubts about including those tool, maybe we can suppose users know what they can do with the OCI layout format


---

## Extra configuration

### Skip saving your run-image layers on disk

Before using this option we suggest to remove your local layout directory (the one configured in your pack config.toml with the key `layout-repo-dir`) and
your application image folder (if you are planning to use the same folder), the reason for this is pack doesn't remove the blobs saved in the `layout-repo-dir` if you use the `--sparse` flag
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Here it becomes more unclear why layout-repo-dir was necessary above.

jjbustamante marked this conversation as resolved.
Show resolved Hide resolved

If you don't need your `run-image` layers on disk, you can skip them using `--sparse` flag in your `pack build` command invocation.

For example:

```bash
pack build oci:sample-app --sparse --path samples/apps/java-maven --builder cnbs/sample-builder:bionic
```

Verify your application image

```bash
sample-app
├── blobs
│ └── sha256
│ ├── 2ebed3ab57806441e2bf814eaf0648ed77289e058340d2b76d32b422fbaac5d8
│ ├── 2fa192256ce255c6ea6c1296eadfe2feba8094f40e6aa85e699645caca2e85d8
│ ├── 5a44e4f7b58d74fe6f92dd7028075c91191128d1e2e7f39846fe061a9a98836e
│ ├── 741e558b7b807fea350b26b8152170a2463277cb3d1268b60de76ec12608518a
│ ├── 907c84671180d979a38affb62d9a6ea8e9a510e27639e0b60a34a42f1a846ddc
│ ├── ad13830c92258c952f25d561d8bf7d9eb58b8a3003960db1502cbda8239130b5
│ ├── c44222730efa142cd5bedc0babf82a9a07d325494be7f5c3cfde56f43166b65f
│ └── f9d6350d0c44c0e7165a522155f53181ce8c163a6b8ead1f6baea22d1a8d8a78
├── index.json
└── oci-layout

3 directories, 10 files
```

As you can see, there are 3 missing files at `sample-app/blobs/sha256` folder, the missing 3 blobs are the blobs from the
jjbustamante marked this conversation as resolved.
Show resolved Hide resolved
`run-image` that were not downloaded but if you check your config file you'll notice you have the same number of layers as
when you export the full image.

## Implementation notes

### Media Types

According to the OCI specification, the [compatibles media types](https://github.com/opencontainers/image-spec/blob/main/media-types.md#compatibility-matrix) for the index.json files must be:

- `application/vnd.oci.image.index.v1+json` or
- `application/vnd.docker.distribution.manifest.list.v2+json`

If you are trying to use the lifecycle directly without using `pack` to export your image, take on consideration that tools like:

[skopeo](https://github.com/containers/skopeo)
```bash
skopeo copy -a docker://<your-image> <dest>
```
It will give you `application/vnd.oci.image.index.v1+json` media type, which is currently working

But [crane](https://github.com/google/go-containerregistry/tree/main/cmd/crane)

```bash
crane pull <your-image> <dest> --format=oci
```
It will give you `application/vnd.docker.distribution.manifest.list.v2+json`, which will fail
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do these things pass/fail due to the state of our current implementation? Or because of a limitation in crane, or because the operation violates part of the OCI spec? Or for some other reason?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

agree, helpful to add a little more detail here

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It fails because of the state or our current implementation, it should be improve on new versions of the feature. I will add it

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let me explain this part a little better so you can guide me on how to document it.

According to the RFC if you want to export your application image on disk, you need to mount to the lifecycle a local storage with the images require by the build process, for example, the run-image. This storage must contain images in OCI layout format and name resolution will follow some rules defined also in the RFC.

The documented part was based on a feedback provided by @natalieparellano here, which is basically the following scenario: Suppose you want to execute the lifecycle directly without using pack, for example, a platform implementor. They need to create this local storage and save everything there before executing the build, right?, well the problem is: If they save in the local storage a manifest list and then they execute the lifecycle, right now, we are not smart enough to detect which manifest we need to pick to execute the build.

That is the actual comment we want to document so users can take it into consideration, now I am not sure this is the right place to do it, or maybe I didn't write it correctly, so any advice will be appreciate!