Seekable OCI and Docker Engine v24

[ollypom]

In Docker Engine v24 (Rel Notes) they announced experimental support for the containerd content store, unblocking the ability to use the soci snapshotter with the Docker Engine. Below you can see my findings on using the soci-snapshotter with the Docker Engine both from a building and a running perspective.

Test Environment

These tests were carried out on Amazon Linux 2023 as it provides a package for Docker Engine v24.

$ sudo yum install docker -y

$ docker --version
Docker version 24.0.5, build ced0996
  
$ sudo ctr --version
ctr github.com/containerd/containerd 1.7.2

$ uname -a
Linux ip-10-1-186-2.eu-west-1.compute.internal 6.1.61-85.141.amzn2023.x86_64 #1 SMP PREEMPT_DYNAMIC Wed Nov  8 00:39:18 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux

Indexing a Container Image built with Docker Build

Traditionally you have been unable to generate SOCI Indexes for container images built by docker build because:

• The soci CLI only indexes container image stored in the containerd content store (snapshotters)
• docker build stores container images in the docker engine image store (graph drivers).

But with Docker Engine v24, the second point is not longer accurate when using the experimental containerd content store support.

Prerequisites

1. Download the soci CLI, we don’t actually need to configure the soci-snapshotter and it does not need to be running to generate SOCI indexes.

# Retrieve the binaries
$ VERSION=0.4.0
$ wget https://github.com/awslabs/soci-snapshotter/releases/download/v$VERSION/soci-snapshotter-$VERSION-linux-amd64.tar.gz
$ sudo tar -C /usr/local/bin -xvf soci-snapshotter-$VERSION-linux-amd64.tar.gz soci soci-snapshotter-grpc

2. Configure Docker Engine’s containerd content store support

$ sudo vim /etc/docker/daemon.json
{
    "experimental": true,
    "features": {
        "containerd-snapshotter": true
    }
}

# Restart the Docker Daemon
$ sudo systemctl restart docker

# Check docker is using the containerd overlayfs snapshotter
$ docker info --format '{{json .Driver}}'
"overlayfs"

$ docker info --format '{{json .DriverStatus}}'
[["driver-type","io.containerd.snapshotter.v1"]]

Build a container image

Next we can build some container images with docker build and a local Dockerfile.

1. Create a Dockerfile

cat <<EOF > Dockerfile
FROM public.ecr.aws/amazonlinux/amazonlinux:2

RUN amazon-linux-extras enable epel && \
    yum install -y epel-release && \
    yum install -y nginx

CMD ["nginx", "-g", "daemon off; error_log /dev/stdout info;"]
EOF

2. Build the Image

$ IMAGE_URI=111222333444.dkr.ecr.eu-west-1.amazonaws.com/nginx-demo:latest

$ docker build -t $IMAGE_URI .
[+] Building 44.5s (6/6) FINISHED                                                                                                                                                                                                                                                                            docker:default
 => [internal] load build definition from Dockerfile                                                                                                                                                                                                                                                                   0.1s
 => => transferring dockerfile: 308B                                                                                                                                                                                                                                                                                   0.0s
 => [internal] load .dockerignore                                                                                                                                                                                                                                                                                      0.1s
 => => transferring context: 2B                                                                                                                                                                                                                                                                                        0.0s
 => [internal] load metadata for public.ecr.aws/amazonlinux/amazonlinux:2                                                                                                                                                                                                                                              1.1s
 => [1/2] FROM public.ecr.aws/amazonlinux/amazonlinux:2@sha256:a2a387c3b2019d37520269d0e04aab21bb2e4d9b50e41f3adce2229e36a5f40a                                                                                                                                                                                        3.1s
 => => resolve public.ecr.aws/amazonlinux/amazonlinux:2@sha256:a2a387c3b2019d37520269d0e04aab21bb2e4d9b50e41f3adce2229e36a5f40a                                                                                                                                                                                        0.0s
 => => sha256:6ebddf7084e9402a3ba6cf1360703e12633f9c49f51772f99c298cd1048d728e 62.65MB / 62.65MB                                                                                                                                                                                                                       0.8s
 => => extracting sha256:6ebddf7084e9402a3ba6cf1360703e12633f9c49f51772f99c298cd1048d728e                                                                                                                                                                                                                              2.2s
 => [2/2] RUN amazon-linux-extras enable epel &&     yum install -y epel-release &&     yum install -y nginx                                                                                                                                                                                                          18.5s
 => exporting to image                                                                                                                                                                                                                                                                                                21.7s
 => => exporting layers                                                                                                                                                                                                                                                                                               18.1s
 => => exporting manifest sha256:8a504b2c060d284bf3ef981a8a440c6e38ec3bfaa89b18eef572797cc6c7990e                                                                                                                                                                                                                      0.0s
 => => exporting config sha256:f2a03a7dcea003621f6c4bcf17cfdc06710a74b328ae3359644919c25736f0a8                                                                                                                                                                                                                        0.0s
 => => exporting attestation manifest sha256:0cc905ecc7a763d49cd38e6f661236a96e398645efbac123e0c792d5e849ebed                                                                                                                                                                                                          0.0s
 => => exporting manifest list sha256:6e3be4974edb10e7cd58711bc85133e3345b677ff7c95b9b9dd4e0169e454a26                                                                                                                                                                                                                 0.0s
 => => naming to 111222333444.dkr.ecr.eu-west-1.amazonaws.com/nginx-demo:latest                                                                                                                                                                                                                                        0.0s
 => => unpacking to 111222333444.dkr.ecr.eu-west-1.amazonaws.com/nginx-demo:latest   

3. Verify the container image is in the containerd content store

$ docker image list
REPOSITORY                                                TAG       IMAGE ID       CREATED         SIZE
111222333444.dkr.ecr.eu-west-1.amazonaws.com/nginx-demo   latest    6e3be4974edb   2 minutes ago   911MB

$ sudo ctr -n moby image list
REF                                                            TYPE                                    DIGEST                                                                  SIZE      PLATFORMS                   LABELS
111222333444.dkr.ecr.eu-west-1.amazonaws.com/nginx-demo:latest application/vnd.oci.image.index.v1+json sha256:6e3be4974edb10e7cd58711bc85133e3345b677ff7c95b9b9dd4e0169e454a26 241.6 MiB linux/amd64,unknown/unknown -

Create the SOCI index

Now we can use the soci CLI to index our container image

1. Build the SOCI Index

$ sudo soci -n moby create $IMAGE_URI
layer sha256:6ebddf7084e9402a3ba6cf1360703e12633f9c49f51772f99c298cd1048d728e -> ztoc sha256:9172ec7a2c5137776912e69ed694c8bcc9e7b41f16b4a77f0a4c4845976f71eb
layer sha256:4cb51dbecafe41e893eaf05e41bc6ca548e84b988cb90784509fdd2a09a6e2dd -> ztoc sha256:c88241a3a0de2066d33bad086514704b69a31093d6a27a7ce26a52c281d40924

2. Verify that the Index is created and a take a look at it.

$ sudo soci -n moby index ls
DIGEST                                                                     SIZE    IMAGE REF                                                         PLATFORM       MEDIA TYPE                                    CREATED
sha256:11c053930f7b1f4527c99b7bcea5e4e80f01d41dc85b66c9818423329cd23195    1189    111222333444.dkr.ecr.eu-west-1.amazonaws.com/nginx-demo:latest    linux/amd64    application/vnd.oci.image.manifest.v1+json    2m19s ago

$ sudo soci -n moby  index info $(sudo soci index ls -q) | jq -r '.'
{
  "schemaVersion": 2,
  "mediaType": "application/vnd.oci.image.manifest.v1+json",
  "config": {
    "mediaType": "application/vnd.amazon.soci.index.v1+json",
<snipped>
}

Push things up to the registry

1. Login in to the container registry

aws ecr get-login-password --region eu-west-1 | docker login --username AWS --password-stdin 111222333444.dkr.ecr.eu-west-1.amazonaws.com

2. Push the the container image

$ docker push $IMAGE_URI

3. Push the SOCI Index. Depending on which user you have been using for the docker commands, you may need to set the credential location (DOCKER_CONFIG) to the users credential file. The SOCI CLI by default looks in /root/.docker .

$ sudo DOCKER_CONFIG=/home/ec2-user/.docker soci -n moby push $IMAGE_URI
checking if a soci index already exists in remote repository...
pushing soci index with digest: sha256:11c053930f7b1f4527c99b7bcea5e4e80f01d41dc85b66c9818423329cd23195
skipped artifact with digest: sha256:9172ec7a2c5137776912e69ed694c8bcc9e7b41f16b4a77f0a4c4845976f71eb
skipped artifact with digest: sha256:44136fa355b3678a1146ad16f7e8649e94fb4fc21fe77e8310c060f61caaff8a
skipped artifact with digest: sha256:8a504b2c060d284bf3ef981a8a440c6e38ec3bfaa89b18eef572797cc6c7990e
pushing artifact with digest: sha256:c88241a3a0de2066d33bad086514704b69a31093d6a27a7ce26a52c281d40924
successfully pushed artifact with digest: sha256:c88241a3a0de2066d33bad086514704b69a31093d6a27a7ce26a52c281d40924
pushing artifact with digest: sha256:11c053930f7b1f4527c99b7bcea5e4e80f01d41dc85b66c9818423329cd23195
successfully pushed artifact with digest: sha256:11c053930f7b1f4527c99b7bcea5e4e80f01d41dc85b66c9818423329cd23195

Bugs

• Should we namespace SOCI Indexes? The soci CLI is unable to find the image reference if the container image is not in the default containerd namespace.

$ sudo soci index ls
DIGEST                                                                     SIZE    IMAGE REF    PLATFORM       MEDIA TYPE                                    CREATED
sha256:11c053930f7b1f4527c99b7bcea5e4e80f01d41dc85b66c9818423329cd23195    1189                 linux/amd64    application/vnd.oci.image.manifest.v1+json    8m16s ago

$ sudo soci -n moby index ls
DIGEST                                                                     SIZE    IMAGE REF                                                         PLATFORM       MEDIA TYPE                                    CREATED
sha256:11c053930f7b1f4527c99b7bcea5e4e80f01d41dc85b66c9818423329cd23195    1189    111222333444.dkr.ecr.eu-west-1.amazonaws.com/nginx-demo:latest    linux/amd64    application/vnd.oci.image.manifest.v1+json    8m20s ago

Lazy Loading Images

In this section we are going to lazy load container images using Docker Engine v24 and the SOCI Snapshotter.

Prerequisites

1. Download and start the SOCI Snapshotter

# Retrieve the binaries
$ VERSION=0.4.0
$ wget https://github.com/awslabs/soci-snapshotter/releases/download/v$VERSION/soci-snapshotter-$VERSION-linux-amd64.tar.gz
$ sudo tar -C /usr/local/bin -xvf soci-snapshotter-$VERSION-linux-amd64.tar.gz soci soci-snapshotter-grpc

# Install fuse
$ sudo yum install fuse -y

# Configure Systemd
$ wget https://github.com/awslabs/soci-snapshotter/blob/main/soci-snapshotter.service
$ sudo mv soci-snapshotter.service /usr/lib/systemd/system/soci-snapshotter.service
$ sudo systemctl daemon-reload
$ sudo systemctl enable --now soci-snapshotter

2. Register the SOCI Snapshotter as a remote snapshotter in containerd.

$ sudo vim /etc/containerd/config.toml
..<Existing Content>
[proxy_plugins]
  [proxy_plugins.soci]
    type = "snapshot"
    address = "/run/soci-snapshotter-grpc/soci-snapshotter-grpc.sock"

# Restart Containerd
sudo systemctl restart containerd

# Check the plugin has been found
sudo ctr plugin ls id==soci
TYPE                            ID      PLATFORMS    STATUS
io.containerd.snapshotter.v1    soci    -            o

3. Configuring the experimental containerd snapshotter support in the Docker Engine and set the soci snapshotter as the default snapshotter.

$ sudo vim /etc/docker/daemon.json
{
    "experimental": true,
    "features": {
        "containerd-snapshotter": true
    },
    "storage-driver": "soci"
}

# Restart the Docker Daemon
$ sudo systemctl restart docker

# Check SOCI is the default driver
$ docker info --format '{{json .Driver}}'
"soci"
$ docker info --format '{{json .DriverStatus}}'
[["driver-type","io.containerd.snapshotter.v1"]]

Lazy Load your container images

The assumption here is that you already have a container image that has been Indexed in a container image repository and that the container image you are attempting to lazy load does not exist locally. (If you have just built the container image using the first part of this post then you may need to remove the container image and the SOCI index from the local content stores).

1. Log in to the registry

$ aws ecr get-login-password --region eu-west-1 | docker login --username AWS --password-stdin 111222333444.dkr.ecr.eu-west-1.amazonaws.com

# If you are running your docker commands as a not root user such as ec2-user
# then you need to copy this credential file over to the root user for the soci snapshotter
$ sudo mkdir /root/.docker
$ sudo cp /home/ec2-user/.docker/config.json /root/.docker/config.json

2. Run the container image

$ docker run 111222333444.dkr.ecr.eu-west-1.amazonaws.com/nginx-demo:latest
Unable to find image '111222333444.dkr.ecr.eu-west-1.amazonaws.com/nginx-demo:latest' locally
8a504b2c060d: Download complete
6e3be4974edb: Download complete
f2a03a7dcea0: Download complete
2023/11/17 17:03:54 [notice] 1#1: using the "epoll" event method
2023/11/17 17:03:54 [notice] 1#1: nginx/1.20.1
2023/11/17 17:03:54 [notice] 1#1: built by gcc 4.8.5 20150623 (Red Hat 4.8.5-44) (GCC)
2023/11/17 17:03:54 [notice] 1#1: OS: Linux 6.1.61-85.141.amzn2023.x86_64
2023/11/17 17:03:54 [notice] 1#1: getrlimit(RLIMIT_NOFILE): 32768:65536
2023/11/17 17:03:54 [notice] 1#1: start worker processes
2023/11/17 17:03:54 [notice] 1#1: start worker process 7
2023/11/17 17:03:54 [notice] 1#1: start worker process 8

You’ll notice a few things are still reported as “Download Complete” rather then being lazy loaded (your shas will be different) but this was expected. The objects downloaded in full were:

• 8a504b2c060d - Container Image Manifest
• f2a03a7dcea0 - Container Image Config
• 6e3be4974edb - Container Image Manifest List / Image Index (this may not be there in your example, it depends on how the container image was built).

3. In a separate window you can verify that the container image was lazy loaded by checking the the SOCI Logs

$ mount
<snipped>
soci on /var/lib/soci-snapshotter-grpc/snapshotter/snapshots/1/fs type fuse.rawBridge (rw,nodev,relatime,user_id=0,group_id=0,allow_other,max_read=131072)
soci on /var/lib/soci-snapshotter-grpc/snapshotter/snapshots/2/fs type fuse.rawBridge (rw,nodev,relatime,user_id=0,group_id=0,allow_other,max_read=131072)
overlay on /var/lib/docker/rootfs/soci/00cea646a9edba4c95070e21134f70e6a040a3f44300e834bd9aade85325827b type overlay (rw,relatime,seclabel,lowerdir=/var/lib/soci-snapshotter-grpc/snapshotter/snapshots/2/fs:/var/lib/soci-snapshotter-grpc/snapshotter/snapshots/1/fs,upperdir=/var/lib/soci-snapshotter-grpc/snapshotter/snapshots/3/fs,workdir=/var/lib/soci-snapshotter-grpc/snapshotter/snapshots/3/work)

$ sudo journalctl -n 100 -u soci-snapshotter | grep "remote snapshot successfully prepared"
Nov 17 17:03:50 ip-10-1-186-2.eu-west-1.compute.internal soci-snapshotter-grpc[12169]: {"key":"moby/1/extract-994226060-elvg sha256:2c20b6340cf49456b92a112cfec50ffdc0f344bdb639ec8ba3765c246f5614e8","level":"info","msg":"remote snapshot successfully prepared.","parent":"","remote-snapshot-prepared":"true","time":"2023-11-17T17:03:50.916633258Z"}
Nov 17 17:03:51 ip-10-1-186-2.eu-west-1.compute.internal soci-snapshotter-grpc[12169]: {"key":"moby/2/extract-921657330-vEhb sha256:fc2d34903df5e8c0c501c9e9b656766da05d2a7b8206e0986f8f9f169efa7e3f","level":"info","msg":"remote snapshot successfully prepared.","parent":"sha256:2c20b6340cf49456b92a112cfec50ffdc0f344bdb639ec8ba3765c246f5614e8","remote-snapshot-prepared":"true","time":"2023-11-17T17:03:51.267874617Z"}

Bugs

• The Amazon ECR Credential Helper does not work yet with Docker Engine’s containerd integration. I believe the issue is fixed here: Default the auth config domain to the target image domain moby/moby#46779.

# Client Side
$ docker run 111222333444.dkr.ecr.eu-west-1.amazonaws.com/nginx-demo:latest
Unable to find image '111222333444.dkr.ecr.eu-west-1.amazonaws.com/nginx-demo:latest' locally
docker: Error response from daemon: failed to resolve reference "111222333444.dkr.ecr.eu-west-1.amazonaws.com/nginx-demo:latest": pulling from host 111222333444.dkr.ecr.eu-west-1.amazonaws.com failed with status code [manifests latest]: 401 Unauthorized.
See 'docker run --help'.

# Daemon Side
dockerd[2829]: time="2023-11-17T15:10:16.034962183Z" level=debug msg="Calling POST /v1.43/images/create?fromImage=111222333444.dkr.ecr.eu-west-1.amazonaws.com%2Fnginx-demo&tag=latest"
dockerd[2829]: time="2023-11-17T15:10:16.037724873Z" level=debug msg=resolving host=111222333444.dkr.ecr.eu-west-1.amazonaws.com
dockerd[2829]: time="2023-11-17T15:10:16.037752159Z" level=debug msg="do request" host=111222333444.dkr.ecr.eu-west-1.amazonaws.com request.header.accept="application/vnd.docker.distribution.manifest.v2+json, application/vnd.docker.distribution.manifest.list.v2+json, application/vnd.oci.image.manifest.v1+json, application/vnd.oci.image.index.v1+json, */*" request.header.user-agent="docker/24.0.5 go/go1.20.10 git-commit/a61e2b4 kernel/6.1.61-85.141.amzn2023.x86_64 os/linux arch/amd64 containerd-client/1.6.21+unknown storage-driver/soci UpstreamClient(Docker-Client/24.0.5 \\(linux\\))" request.method=HEAD url="https://111222333444.dkr.ecr.eu-west-1.amazonaws.com/v2/nginx-demo/manifests/latest"
dockerd[2829]: time="2023-11-17T15:10:16.043656269Z" level=debug msg="fetch response received" host=111222333444.dkr.ecr.eu-west-1.amazonaws.com response.header.content-length=15 response.header.content-type="text/plain; charset=utf-8" response.header.date="Fri, 17 Nov 2023 15:10:16 GMT" response.header.docker-distribution-api-version=registry/2.0 response.header.sizes= response.header.www-authenticate="Basic realm=\"https://111222333444.dkr.ecr.eu-west-1.amazonaws.com/\",service=\"ecr.amazonaws.com\"" response.status="401 Unauthorized" url="https://111222333444.dkr.ecr.eu-west-1.amazonaws.com/v2/nginx-demo/manifests/latest"
dockerd[2829]: time="2023-11-17T15:10:16.043698700Z" level=debug msg=Unauthorized header="Basic realm=\"https://111222333444.dkr.ecr.eu-west-1.amazonaws.com/\",service=\"ecr.amazonaws.com\"" host=111222333444.dkr.ecr.eu-west-1.amazonaws.com
dockerd[2829]: time="2023-11-17T15:10:16.043716364Z" level=warning msg="Host doesn't match" cfgHost=registry-1.docker.io host=111222333444.dkr.ecr.eu-west-1.amazonaws.com
dockerd[2829]: time="2023-11-17T15:10:16.046196987Z" level=debug msg="FIXME: Got an API for which error does not match any expected type!!!: failed to resolve reference \"111222333444.dkr.ecr.eu-west-1.amazonaws.com/nginx-demo:latest\": pulling from host 111222333444.dkr.ecr.eu-west-1.amazonaws.com failed with status code [manifests latest]: 401 Unauthorized" error_type="*fmt.wrapError" module=api
dockerd[2829]: time="2023-11-17T15:10:16.046220416Z" level=error msg="Handler for POST /v1.43/images/create returned error: failed to resolve reference \"111222333444.dkr.ecr.eu-west-1.amazonaws.com/nginx-demo:latest\": pulling from host 111222333444.dkr.ecr.eu-west-1.amazonaws.com failed with status code [manifests latest]: 401 Unauthorized"

• You need to manually create a soci rootfs directory for the Docker Engine before you can lazy load a container images. I haven’t bottomed out where this error is coming from, I am unable to recreate this error when using the stargz-snapshotter with the Docker Engine v24. I’m unsure if this is an upstream Moby bug or a SOCI Snapshotter bug but the easy workaround is to create the directory manually sudo mkdir -p /var/lib/docker/rootfs/soci

# Client side
$ docker run 111222333444.dkr.ecr.eu-west-1.amazonaws.com/nginx-demo:latest
Unable to find image '111222333444.dkr.ecr.eu-west-1.amazonaws.com/nginx-demo:latest' locally
3ebddb6ccdbb: Download complete
c40a625ed681: Download complete
daa29e58f81b: Download complete
docker: Error response from daemon: failed to mount : mkdir /var/lib/docker/rootfs/soci/7fb900fbc1847f818da84befaa49a6babf7ed271ade7ce7f105b9be7501de292: no such file or directory.
See 'docker run --help'.

# Daemon Side
dockerd[5483]: time="2023-11-17T15:20:19.797329143Z" level=debug msg="FIXME: Got an API for which error does not match any expected type!!!: failed to mount : mkdir /var/lib/docker/rootfs/soci/7fb900fbc1847f818da84befaa49a6babf7ed271ade7ce7f105b9be7501de292: no such file or directory" error_type="*fmt.wrapError" module=api
dockerd[5483]: time="2023-11-17T15:20:19.797357328Z" level=error msg="Handler for POST /v1.43/containers/create returned error: failed to mount : mkdir /var/lib/docker/rootfs/soci/7fb900fbc1847f818da84befaa49a6babf7ed271ade7ce7f105b9be7501de292: no such file or directory"
dockerd[5483]: time="2023-11-17T15:20:19.797378054Z" level=debug msg="FIXME: Got an API for which error does not match any expected type!!!: failed to mount : mkdir /var/lib/docker/rootfs/soci/7fb900fbc1847f818da84befaa49a6babf7ed271ade7ce7f105b9be7501de292: no such file or directory" error_type="*fmt.wrapError" module=api

• I also hit the odd “not found” error reported here every time I attempted to use a layer as a local snapshot.

{
    "error": "cannot unpack the layer: cannot fetch layer: size of descriptor is 0; unable to resolve: unable to resolve ref (111222333444.dkr.ecr.eu-west-1.amazonaws.com/nginx-demo@sha256:b70bf03ce15dc3f3f6370bc1239d7206483792abf99d34ff6319a390d0c7a501): 111222333444.dkr.ecr.eu-west-1.amazonaws.com/nginx-demo@sha256:b70bf03ce15dc3f3f6370bc1239d7206483792abf99d34ff6319a390d0c7a501: not found",
    "key": "moby/3/extract-919571228-5Oa3 sha256:2c28eb1d77a47da7c034479d5fe8e292448d4f2c49360431f324409b06d4fa59",
    "level": "warning",
    "msg": "failed to prepare snapshot; deferring to container runtime",
    "parent": "sha256:7aa918955da73e31e7bb1bbb445e0562a13d0aaec0b124bd56773ec2865a7faa",
    "time": "2023-11-17T15:42:14.994479325Z"
}

[nealef]

We just ran into the mkdir /var/lib/docker/rootfs... problem yesterday. We have our own content store & snapshotter plugins and just added docker to the containerd mix. Did you ever find out why this occurs? I had a scan through the stargz git log to see if would shed any light on the subject but to no avail.

[jrobison-sb]

The workflow seen with Docker v24 which builds and pushes a Docker image without needing to rely on ctr commands directly looks great and I'll be happy to use that when Codebuild eventually supports Docker v24. Thanks for putting this together.

Configure Docker Engine’s containerd content store support

Unfortunately changing the Docker image store to containerd and then restarting Docker engine isn't currently possible in Codebuild using the standard AL2 Codebuild images (AWS support ticket 13316810401). So when Codebuild eventually gains support for Docker v24, hopefully it also gains support for using the containerd image store and restarting Docker.