pack build fails with experimental containerd feature enabled in Docker Desktop (macos)

[pbusko]

Summary

pack build fails with the error ERROR: failed to build: failed to fetch builder image 'index.docker.io/paketobuildpacks/builder:base': image index.docker.io/paketobuildpacks/builder:base does not exist on the daemon: not found on MacOS with the experimental containerd feature enabled for docker.

---

Reproduction

Steps

1. Install pack
2. Install Docker Desktop for Mac (version >= 4.12.0)
3. Enable experimental containerd feature (Preferences -> Experimental features -> Use containerd for pulling and storing images)
4. Run pack build against any sample application.

Current behavior

Note: issue persists regardless of which builder is used.

Note: the builder image is pulled successfully, can be started with docker run

$ pack build --builder paketobuildpacks/builder:base --path . test-image
cd5c630c64e8: Download complete
36312f3c7f1f: Download complete
01c55d295d1f: Exists
a6cd17527f03: Exists
6a0c5265ea0c: Exists
365f6846cf94: Exists
b60529c10e24: Exists
dd61f634638a: Exists
04798f682aea: Exists
a817b1a53002: Exists
6c016ae9058d: Exists
4f4fb700ef54: Exists
3b0e22c8c879: Exists
30d845c0bb1d: Exists
e0b4f01ad7b7: Exists
47f4ce4d2f04: Exists
64651b4c8e32: Exists
88236dee0022: Exists
fcf29a1baad8: Exists
69b4bab08e3b: Exists
a06b476f6ec5: Exists
72ad9888618d: Exists
bda6f0e74fec: Exists
a7598a602211: Exists
900c10f916b3: Exists
5a42bbcf6576: Exists
34af1a2f8596: Exists
18ee4cd834fc: Exists
8f6bb17d0375: Exists
02f68249ddd3: Exists
d1337f81868d: Exists
1b620ae6bf17: Exists
e2d9aa217ebb: Exists
9fe21e36faa2: Exists
343726d75f2f: Exists
eeae65faff79: Exists
b5a1c8ec9c65: Exists
71dea8ea27aa: Exists
b6159da725e2: Exists
52d6fb36e3e3: Exists
613375ad7aaa: Exists
1623613fd3f1: Exists
497a711d323b: Exists
0ab3e873f7fa: Exists
f9180f5e7eee: Exists
27bd211db33f: Exists
c2500576a01c: Exists
01e1e34ec3c6: Exists
d4c7f094dd93: Exists
a038b82e3bbd: Exists
bc22ffea56cc: Exists
b9a98dcc6824: Exists
d849e601a9cb: Exists
7b48d0b8675d: Exists
d93f2c1f46f0: Exists
357fefdf9bc9: Exists
b2248d1a82a5: Exists
411f27cd255a: Exists
1c17179128b5: Exists
3dcb8ad3050f: Exists
fbb9179cbbb1: Exists
abf90eddb26c: Exists
0739122927f8: Exists
6451c72807f6: Exists
bb7e9d60b057: Exists
11bb404e7037: Exists
d7b02183eb06: Exists
fb38b561c15d: Exists
ad443f24b1ed: Exists
741bedb61220: Exists
91fb955a61bd: Download complete
780dcde9ba7c: Download complete
d47eb88483ed: Download complete
0e93cd8ef1ea: Download complete
9a5e304a1273: Download complete
6ab71b9cde83: Download complete
7a818b9017bf: Download complete
900f7db316c1: Download complete
9fc7bc58b7fd: Download complete
5f2fb8d984ed: Download complete
d73a1a46c022: Download complete
6ff58d53b60e: Download complete
12ddd1d90a49: Download complete
0b68da443b93: Download complete
eda215647626: Download complete
d7283610e7cb: Download complete
8034b0454db2: Download complete
68e9e49f7e69: Download complete
41869a3aefd1: Download complete
8cf4b87edd12: Download complete
f07d57f9e194: Download complete
ERROR: failed to build: failed to fetch builder image 'index.docker.io/paketobuildpacks/builder:base': image index.docker.io/paketobuildpacks/builder:base does not exist on the daemon: not found

$ docker images                                                                                                                                                                             
REPOSITORY                 TAG       IMAGE ID       CREATED         SIZE
paketobuildpacks/builder   base      cd5c630c64e8   3 minutes ago   491MB

Expected behavior

pack build should succeed

---

Environment

pack info

Pack:
  Version:  0.27.0+git-f4f5be1.build-3382
  OS/Arch:  darwin/amd64

Default Lifecycle Version:  0.14.1

Supported Platform APIs:  0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9

Config:
  default-builder-image = "[REDACTED]"

docker info

Client:
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc., v0.9.1)
  compose: Docker Compose (Docker Inc., v2.10.2)
  extension: Manages Docker extensions (Docker Inc., v0.2.9)
  sbom: View the packaged-based Software Bill Of Materials (SBOM) for an image (Anchore Inc., 0.6.0)
  scan: Docker Scan (Docker Inc., v0.19.0)

Server:
 Containers: 0
  Running: 0
  Paused: 0
  Stopped: 0
 Images: 1
 Server Version: 22.06.0-beta.0-372-gd3bb8227ce.m
 Storage Driver: stargz
  driver-type: io.containerd.snapshotter.v1
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 9cd3357b7fd7218e4aec3eae239db1f68a5a6ec6
 runc version: v1.1.4-0-g5fd4c4d
 init version: de40ad0
 Security Options:
  seccomp
   Profile: builtin
  cgroupns
 Kernel Version: 5.10.124-linuxkit
 Operating System: Docker Desktop
 OSType: linux
 Architecture: x86_64
 CPUs: 6
 Total Memory: 3.842GiB
 Name: docker-desktop
 ID: 7f8ed1ee-f6af-4cd1-a83d-bdc61b288cfe
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 HTTP Proxy: http.docker.internal:3128
 HTTPS Proxy: http.docker.internal:3128
 No Proxy: hubproxy.docker.internal
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  hubproxy.docker.internal:5000
  127.0.0.0/8
 Live Restore Enabled: false

[githubcdr]

Confirmed here, disabling the containerd option makes it work.

[jjbustamante]

This is related to #1853.
Support for containerd is experimental in Docker, that's why we haven't invest a lot of effort on it

[natalieparellano]

Cross-posting from #1853 (comment):

We want to research the reason for this bug. the idea will be:

• Confirm if there is a bug on pack or if this something related to docker.
• We had some ideas to use the export to OCI layout as a way to interact with containerd (see this Feature Request: Support exporting images to a containerd socket lifecycle#829 on lifecycle)
  Once we have done that, we can make some decisions if it worth it to work on this.

Also as a side note, containerd image storage is also experimental in Docker side, I don't want to invest a lot of effort on it until is ready.

[pbusko]

The problem seems to be in the buildpacks/imgutil library, the unit tests are failing with Docker Desktop + Containerd

[rgreig]

Just adding a note that this isn't only applicable to MacOS - Docker Desktop on Windows has the same issue when containerd is enabled. You get a slightly different error message, but it is also resolved by disabling containerd support.

Worth noting that on the version of docker desktop that I am using on windows (4.23) it seems that containerd support is the default.

[natalieparellano]

I played around with this a little today. I think in the short term this could be fixed with some small changes to imgutil, but we'll probably want a larger refactor to imgutil/local package at some point. I'll put up a PR shortly and link back to it here...

[jjbustamante]

Hi @pbusko @rgreig

The shipped a release candidate version of pack 0.32.0, in this version we included @natalieparellano pull request, which should be fixing the problem with containerd.

The bad news is, right now, there is a performance penalty on builds, compared to the default docker storage, that we are thinking on how to improve. We wanted to add at least a warning message for end users to let them know about this performance penalty, but that message is not there yet.

We will love if you can take a look and test this pack version to validate if your problem is solved! Let us know any feedback.

[pbusko]

Hi @jjbustamante!

After trying it out, the problem is still present but at a different stage in my case. Now the actual build starts and completes without issues, however it fails during export:

Timer: Saving sample-java... started at 2023-11-03T09:33:53Z
*** Images (abe3488d59be):
      sample-java - saving image "sample-java": Error response from daemon: No such image: sha256:878a15501fd46ffc6542b80bf8c99461a0f82d6e5ed77cbaf0a699a3c5c2b86f
Timer: Saving sample-java... ran for 7.118055516s and ended at 2023-11-03T09:34:00Z
Timer: Exporter ran for 8.112504274s and ended at 2023-11-03T09:34:00Z
ERROR: failed to export: failed to write image to the following tags: [sample-java: saving image "sample-java": Error response from daemon: No such image: sha256:878a15501fd46ffc6542b80bf8c99461a0f82d6e5ed77cbaf0a699a3c5c2b86f]
ERROR: failed to build: executing lifecycle: failed with status code: 62

The actual final image has been successfully saved and available, but with a different ID

Also pack fails to pull a builder/run image from a protected registry (no problems when running docker pull):

failed to authorize: failed to fetch anonymous token: ... : 401 Unauthorized

[natalieparellano]

@pbusko I think this makes sense - we need to update imgutil in the lifecycle, then the save should succeed.

[jjbustamante]

@pbusko I think this makes sense - we need to update imgutil in the lifecycle, then the save should succeed.

Oops! I forgot about the update on lifecycle! sorry @pbusko but thanks for testing it!!!

[natalieparellano]

We recently learned (see moby/moby#46746) that we could send an OCI layout formatted tar to the daemon instead of the current tar format. We should use imgutil layout/sparse images to preserve our optimization. In the meantime we could update imgutil in the lifecycle, but we should emit a warning that performance may be degraded if we detect containerd storage.