Image build with nerdctl failed "ERROR exporting to docker image format" , but docker works well (content digest sha256:...: not found)

[sunerpy]

❯ nerdctl --version
nerdctl version 1.1.0
❯ buildkitd --version
buildkitd github.com/moby/buildkit 0.11.0 830288a71f447b46ad44ad5f7bd45148ec450d44

OS info:

Archlinux
Linux k8saio 5.15.86-1-lts #1 SMP Sun, 01 Jan 2023 16:03:00 +0000 x86_64 GNU/Linux

k3s
❯ kl version --short
Flag --short has been deprecated, and will be removed in the future. The --short output will become the default.
Client Version: v1.26.0+k3s1
Kustomize Version: v4.5.7
Server Version: v1.26.0+k3s1

Dockerfile:

FROM python:3.9-slim
ENV PYTHONUNBUFFERED 1
COPY . /code
WORKDIR /code
RUN echo "deb http://mirrors.aliyun.com/debian/ bullseye main" > /etc/apt/sources.list && \
    echo "deb http://mirrors.aliyun.com/debian/ bullseye-updates main" >>/etc/apt/sources.list && \
    apt-get update && apt-get install libpq-dev gcc -y && \
    apt-get clean && rm -rf /var/lib/apt/lists/* && \
    pip install -r /code/requirements.txt -i https://mirrors.aliyun.com/pypi/simple/ --no-cache-dir && ln -s /usr/local/bin/django-admin /usr/local/bin/django-admin.py &&\
    mv docker-entrypoint.sh /docker-entrypoint.sh
EXPOSE 9000
CMD ["python", "manage.py", "runserver", "0.0.0.0:9000"]

nerdctl buiild info:

❯ nerdctl build -t django:latest .  --debug --namespace k8s.io
DEBU[0000] Choosing the buildkit host "buildkit-k8s.io/buildkitd.sock", candidates=[buildkit-k8s.io/buildkitd.sock buildkit-default/buildkitd.sock buildkit/buildkitd.sock] (in "/run/") 
DEBU[0000] Choosing the buildkit host "buildkit-default/buildkitd.sock", candidates=[buildkit-k8s.io/buildkitd.sock buildkit-default/buildkitd.sock buildkit/buildkitd.sock] (in "/run/") 
DEBU[0000] Choosing the buildkit host "buildkit/buildkitd.sock", candidates=[buildkit-k8s.io/buildkitd.sock buildkit-default/buildkitd.sock buildkit/buildkitd.sock] (in "/run/") 
DEBU[0000] Chosen buildkit host "unix:///run/buildkit/buildkitd.sock" 
DEBU[0000] worker labels: map[org.mobyproject.buildkit.worker.containerd.namespace:k8s.io org.mobyproject.buildkit.worker.containerd.uuid:2d3b4263-b0a5-4ddc-b006-e18ec8ee46f5 org.mobyproject.buildkit.worker.executor:containerd org.mobyproject.buildkit.worker.hostname:k8saio org.mobyproject.buildkit.worker.network:host org.mobyproject.buildkit.worker.selinux.enabled:false org.mobyproject.buildkit.worker.snapshotter:overlayfs] 
DEBU[0000] running /usr/bin/buildctl [--addr=unix:///run/buildkit/buildkitd.sock build --progress=auto --frontend=dockerfile.v0 --local=context=. --output=type=docker,name=docker.io/library/django:latest --local=dockerfile=/workdir/projects/kubeinstall/k8s/homenas/django --opt=filename=Dockerfile] 
[+] Building 379.8s (8/9)                                                                                                                                                                                       
 => [internal] load .dockerignore                                                                                                                                                                          0.0s
 => => transferring context: 91B                                                                                                                                                                           0.0s
[+] Building 379.9s (9/9) FINISHED                                                                                                                                                                              
 => [internal] load .dockerignore                                                                                                                                                                          0.0s
 => => transferring context: 91B                                                                                                                                                                           0.0s
 => [internal] load build definition from Dockerfile                                                                                                                                                       0.0s
 => => transferring dockerfile: 707B                                                                                                                                                                       0.0s
 => [internal] load metadata for docker.io/library/python:3.9-slim                                                                                                                                         4.8s
 => [internal] load build context                                                                                                                                                                          0.7s
 => => transferring context: 109B                                                                                                                                                                          0.0s
 => [1/4] FROM docker.io/library/python:3.9-slim@sha256:e3a284c2b358fefc161004d7c94ba2e64823296127b39f79ac0f3e1841c1b070                                                                                   0.0s
 => => resolve docker.io/library/python:3.9-slim@sha256:e3a284c2b358fefc161004d7c94ba2e64823296127b39f79ac0f3e1841c1b070                                                                                   0.0s
 => CACHED [2/4] COPY . /code                                                                                                                                                                              0.0s
 => [3/4] WORKDIR /code                                                                                                                                                                                    0.0s
 => [4/4] RUN echo "deb http://mirrors.aliyun.com/debian/ bullseye main" > /etc/apt/sources.list &&     echo "deb http://mirrors.aliyun.com/debian/ bullseye-updates main" >>/etc/apt/sources.list &&    367.1s 
 => ERROR exporting to docker image format                                                                                                                                                                 7.2s 
 => => exporting layers                                                                                                                                                                                    5.8s 
 => => exporting manifest sha256:310adb268f63e63d74b45badf9c4a339a505d577067f173cf75f45376451ed04                                                                                                          0.0s 
 => => exporting config sha256:156da4fd7e8ac126d15639d5b95aeb3e680952d8ed7f2737e60fbe773b00ac47                                                                                                            0.0s
 => => sending tarball                                                                                                                                                                                     1.5s
------
 > exporting to docker image format:
------
error: failed to solve: failed to get reader: content digest sha256:69038a8b17e6522ee257591b4fedd33c5a44012d99fa761409aa2753414fdfc5: not found

docker build info:

❯ docker build -t test:latest .
Sending build context to Docker daemon  8.704kB
Step 1/7 : FROM python:3.9-slim
3.9-slim: Pulling from library/python
3.9-slim: Pulling from library/python
a2abf6c4d29d: Pull complete 
27003db43ed4: Pull complete 
eba080ca5a70: Pull complete 
00ceeca3d803: Pull complete 
8980b900ecd2: Pull complete 
Digest: sha256:f4efbe5d1eb52c221fded79ddf18e4baa0606e7766afe2f07b0b330a9e79564a
Status: Downloaded newer image for python:3.9-slim
 ---> 8ace3a02b842
Step 2/7 : ENV PYTHONUNBUFFERED 1
 ---> Running in 06ec485e9598
Removing intermediate container 06ec485e9598
 ---> 0f895d25e204
Step 3/7 : COPY . /code
 ---> bed92fce0603
Step 4/7 : WORKDIR /code
 ---> Running in b995b944331f
Removing intermediate container b995b944331f
 ---> 2c8ea4633af2
Step 5/7 : RUN echo "deb http://mirrors.aliyun.com/debian/ bullseye main" > /etc/apt/sources.list &&     echo "deb http://mirrors.aliyun.com/debian/ bullseye-updates main" >>/etc/apt/sources.list &&     apt-get update && apt-get install libpq-dev gcc -y &&     apt-get clean && rm -rf /var/lib/apt/lists/* &&     pip install -r /code/requirements.txt -i https://mirrors.aliyun.com/pypi/simple/ --no-cache-dir && ln -s /usr/local/bin/django-admin /usr/local/bin/django-admin.py &&    mv docker-entrypoint.sh /docker-entrypoint.sh
 ---> Running in be5158fcc43c
Get:1 http://mirrors.aliyun.com/debian bullseye InRelease [116 kB]
Get:2 http://mirrors.aliyun.com/debian bullseye-updates InRelease [44.1 kB]
Get:3 http://mirrors.aliyun.com/debian bullseye/main amd64 Packages [8183 kB]
Get:4 http://mirrors.aliyun.com/debian bullseye-updates/main amd64 Packages [14.6 kB]
...
...
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
WARNING: You are using pip version 21.2.4; however, version 22.3.1 is available.
You should consider upgrading via the '/usr/local/bin/python -m pip install --upgrade pip' command.
Removing intermediate container be5158fcc43c
 ---> b91c75214caf
Step 6/7 : EXPOSE 9000
 ---> Running in 02b589004a37
Removing intermediate container 02b589004a37
 ---> a4331391554b
Step 7/7 : CMD ["python", "manage.py", "runserver", "0.0.0.0:9000"]
 ---> Running in 50c7c1d93198
Removing intermediate container 50c7c1d93198
 ---> f31046f6f098
Successfully built f31046f6f098
Successfully tagged test:latest

I have already restart containerd, k3s and reboot this host, but it no effect. Why docker build works well, nerdctl failed?

[sunerpy]

I also test an easy dockerfile:

❯ cat tests/Dockerfile
FROM busybox
RUN echo "hello"

But strangely, in this case, both docker and nerdctl works fun.

[davidjade]

I was also having this issue, but only for one image. A new image also worked fine.

However, after doing nerdctl builder prune, things started working again.