ImageVolume broken

[laurivosandi]

Environmental Info:
K3s Version:
k3s version v1.31.0+k3s1 (34be6d9)
go version go1.22.5

Node(s) CPU architecture, OS, and Version:
Linux kubetest 6.8.0-44-generic #44-Ubuntu SMP PREEMPT_DYNAMIC Tue Aug 13 13:35:26 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux

Cluster Configuration:
Single node

Describe the bug:
The freshly arrived ImageVolume feature seems broken

Steps To Reproduce:

Deployed k3s 1.31 with:

INSTALL_K3S_EXEC="--kubelet-arg feature-gates=ImageVolume=true --kube-apiserver-arg feature-gates=ImageVolume=true"
INSTALL_K3S_CHANNEL=latest

Applied manifest:

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: test
spec:
  replicas: 2
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
        - name: nginx
          image: nginx:1.14.2
          ports:
            - containerPort: 80
          volumeMounts:
            - name: model
              mountPath: /model
      volumes:
        - name: model
          image:
            pullPolicy: IfNotPresent
            reference: registry.k8s.io/conformance:v1.31.0

Expected behavior:

Pod starts with registry.k8s.io/conformance:v1.31.0 image contents mounted at /model

Actual behavior:

In Kubernetes Lens following error message is shown

Error: failed to generate container "558bb3f947d9e1a70c697143ba10b2795c58ee7bde9c4fa0e3b317acb02dbe7b" spec: failed to generate spec: failed to mkdir "": mkdir : no such file or directory

Additional context / logs:

Sep 12 07:33:18 kubetest k3s[2270]: E0912 07:33:18.926553    2270 log.go:32] "CreateContainer in sandbox from runtime service failed" err="rpc error: code = Unknown desc = failed to generate container \"536ef130eb99ae8a9f69c175a410b6400a576c514f07ea5faf68b6dc638a8860\" spec: failed to generate spec: failed to mkdir \"\": mkdir : no such file or directory" podSandboxID="f785734aa1099aa8557e8d4e6cabe4de5d889901d31880caf2a3f972946d7f78"
Sep 12 07:33:18 kubetest k3s[2270]: E0912 07:33:18.926664    2270 kuberuntime_manager.go:1272] "Unhandled Error" err="container &Container{Name:nginx,Image:nginx:1.14.2,Command:[],Args:[],WorkingDir:,Ports:[]ContainerPort{ContainerPort{Name:,HostPort:0,ContainerPort:80,Protocol:TCP,HostIP:,},},Env:[]EnvVar{},Resources:ResourceRequirements{Limits:ResourceList{},Requests:ResourceList{},Claims:[]ResourceClaim{},},VolumeMounts:[]VolumeMount{VolumeMount{Name:model,ReadOnly:false,MountPath:/model,SubPath:,MountPropagation:nil,SubPathExpr:,RecursiveReadOnly:nil,},VolumeMount{Name:kube-api-access-rjk5r,ReadOnly:true,MountPath:/var/run/secrets/kubernetes.io/serviceaccount,SubPath:,MountPropagation:nil,SubPathExpr:,RecursiveReadOnly:nil,},},LivenessProbe:nil,ReadinessProbe:nil,Lifecycle:nil,TerminationMessagePath:/dev/termination-log,ImagePullPolicy:IfNotPresent,SecurityContext:nil,Stdin:false,StdinOnce:false,TTY:false,EnvFrom:[]EnvFromSource{},TerminationMessagePolicy:File,VolumeDevices:[]VolumeDevice{},StartupProbe:nil,ResizePolicy:[]ContainerResizePolicy{},RestartPolicy:nil,} start failed in pod ai-test-67984f78cd-n8xqd_default(e55f166a-1f0f-4728-96e2-fde67da551e3): CreateContainerError: failed to generate container \"536ef130eb99ae8a9f69c175a410b6400a576c514f07ea5faf68b6dc638a8860\" spec: failed to generate spec: failed to mkdir \"\": mkdir : no such file or directory"
Sep 12 07:33:18 kubetest k3s[2270]: E0912 07:33:18.927733    2270 pod_workers.go:1301] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"nginx\" with CreateContainerError: \"failed to generate container \\\"536ef130eb99ae8a9f69c175a410b6400a576c514f07ea5faf68b6dc638a8860\\\" spec: failed to generate spec: failed to mkdir \\\"\\\": mkdir : no such file or directory\"" pod="default/ai-test-67984f78cd-n8xqd" podUID="e55f166a-1f0f-4728-96e2-fde67da551e3"

[brandond]

I don't know where this error is coming from, but it is not code that lives within this repo. I suspect that the problem is either in the kubelet, or containerd.

The error itself is coming back from containerd (the runtime service), but I don't know if the container spec generated by the kubelet is incorrect, or if there's a bug in containerd, or what.

[brandond]

Have you tried this with any other Kubernetes distro with containerd 1.7, or k3s with a different container runtime (cri-dockerd perhaps)?