K3s not working in container ( systemd-nspawn based )

[Alceatraz]

Before start

This issue a little bit too specific situations. If not accepting
such kind issue I will move this to Discussions.

---

Environmental Info:
K3s Version: k3s version v1.30.5+k3s1 (9b58670) go version go1.22.6

Node(s) CPU architecture, OS, and Version:
Linux develop 6.1.0-25-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.1.106-3 (2024-08-26) x86_64 GNU/Linux
debian 12

Cluster Configuration:
Single node

Describe the bug:

K3s not working in container ( systemd-nspawn based )

Steps To Reproduce:

• Start with brand new Debian 12 minimal install ( Expert install only base system )

• Prepare a rootfs

debootstrap --include=dbus stable /opt/container/debian

• Prepare Host OS

echo 1 > /proc/sys/net/ipv4/ip_forward
echo 1 > /proc/sys/vm/overcommit_memory
modprobe br_netfilter
modprobe overlay
modprobe iptable_nat
modprobe iptable_filter

• Start the container and attach it

systemd-run \
  --collect \
  --unit=debian \
  --working-directory=/opt/container/debian \
  --property=KillMode=mixed \
  --property=Type=notify \
  --property=Delegate=yes \
  --property=TasksMax=infinity \
  --property=SuccessExitStatus=133 \
  --property=RestartForceExitStatus=133 \
  --setenv=SYSTEMD_NSPAWN_LOCK=0 \
  -- \
  systemd-nspawn \
  --keep-unit \
  --quiet \
  --boot \
  --machine=debian \
  --directory=/opt/container/debian \
  --notify-ready=yes \
  --resolv-conf=bind-host \
  --network-macvlan=ens33 \
  --system-call-filter=bpf \
  --system-call-filter=keyctl \
  --system-call-filter=add_key \
  --inaccessible=/sys/module/apparmor \
  --bind-ro=/sys/module \
  --bind=/dev/kmsg

• Config the container network ( This should be completely unrelated with this issue )

cat << EOF > /etc/systemd/network/80-container-mv.network
[Match]
Virtualization=container
Name=mv-*

[Network]
DHCP=yes
LinkLocalAddressing=yes

[DHCP]
UseDNS=true
UseTimezone=yes
EOF

systemctl enable --now systemd-networkd

• Install K3s ( I'm using mirror )

curl -sfL https://rancher-mirror.rancher.cn/k3s/k3s-install.sh | INSTALL_K3S_MIRROR=cn sh -

Expected behavior:

A working K3s

Actual behavior:

K3s install script will stuck at [INFO] systemd: Starting k3s

k3s.service - Lightweight Kubernetes
     Loaded: loaded (/etc/systemd/system/k3s.service; enabled; preset: enabled)
     Active: inactive (dead) (Result: exit-code) since Sun 2024-09-29 18:02:26 CST; 18min ago
   Duration: 489ms
       Docs: https://k3s.io
    Process: 634 ExecStartPre=/bin/sh -xc ! /usr/bin/systemctl is-enabled --quiet nm-cloud-setup.service 2>/dev/null (code=exited, status=0/SUCCESS)
    Process: 636 ExecStartPre=/sbin/modprobe br_netfilter (code=exited, status=1/FAILURE)
    Process: 637 ExecStartPre=/sbin/modprobe overlay (code=exited, status=1/FAILURE)
    Process: 638 ExecStart=/usr/local/bin/k3s server (code=exited, status=1/FAILURE)
   Main PID: 638 (code=exited, status=1/FAILURE)
        CPU: 5.015s

Sep 29 18:02:26 develop systemd[1]: Stopped k3s.service - Lightweight Kubernetes.
Sep 29 18:02:26 develop systemd[1]: k3s.service: Consumed 5.015s CPU time.

Additional context / logs:

Log of k3s server --debug, Full log too long

W0929 18:21:35.405435     754 feature_gate.go:246] Setting GA feature gate CloudDualStackNodeIPs=true. It will be removed in a future release.
W0929 18:21:35.405799     754 feature_gate.go:246] Setting GA feature gate CloudDualStackNodeIPs=true. It will be removed in a future release.
I0929 18:21:35.414247     754 server.go:479] "Kubelet version" kubeletVersion="v1.30.5+k3s1"
I0929 18:21:35.414415     754 server.go:481] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
W0929 18:21:35.414706     754 feature_gate.go:246] Setting GA feature gate CloudDualStackNodeIPs=true. It will be removed in a future release.
W0929 18:21:35.415322     754 feature_gate.go:246] Setting GA feature gate CloudDualStackNodeIPs=true. It will be removed in a future release.
I0929 18:21:35.423615     754 dynamic_cafile_content.go:157] "Starting controller" name="client-ca-bundle::/var/lib/rancher/k3s/agent/client-ca.crt"
INFO[0002] Creating helm-controller event broadcaster   
INFO[0002] Starting /v1, Kind=Node controller           
DEBU[0002] DesiredSet - Patch apps/v1, Kind=Deployment kube-system/coredns for  kube-system/coredns -- [PATCH:{"spec":{"template":{"spec":{"$setElementOrder/containers":[{"name":"coredns"}],"containers":[{"name":"coredns","readinessProbe":{"initialDelaySeconds":0}}]}}}}, ORIGINAL:{"apiVersion":"apps/v1","kind":"Deployment","metadata":{"annotations":{"objectset.rio.cattle.io/applied":"H4sIAAAAAAAA/6xVQW/bOBP9Kx/mLMVW3DaugO/QjbPboq3XqJNeCqOgqZHFNcXhkiMnRuD/vhhJduw2TdrFniyTb4ZvHucN70F58xlDNOQgB+V9HGwySGBtXAE5TNBb2tboGBKokVWhWEF+D8o5YsWGXJS/tPwLNUfks2DoTCtmi2eGBkaSQPLDfbp1GNLVZg05rEfxaGeTJf97b1zx/zdFQe7ZFE7VCDloCli4+FPw6JWWmHWzxDRuI2MNuwSsWqJti1qPY6q830O6vPIZHDJGydYfe0kBJ9P5E8dWKlaQw1Lj+Xh0/no8zrKLFyM1HI1fqeXLbFiel68usLx4cf5iqF9eCJHvSnqCdPSohXLAjZG7fGsiU9h+MLVhyIcJRLSomYKAasW6+vBUmTtJyUExrrZtWrLWuNWNLxRjl+LuxqmNMlYtLUKe7RLgrRdmn06wso61t/u4oxZ6UujdUVGaHCvjMETIv9yDCiv5gFSTKyGBAbIe9CoN5CZKYxEWCZharYRRUE5XGAa1CUFgaQ/e/+bZWZadjaCPmDXWzsgavYUc3pVT4lnA2FnAmg06jHEWaNkWVCpjm4DXVcBYkS0gHyVQMfs/kGXfK5Z7H1SoLFeQgKfAkI+HY7kUXWF7x2+vr2cilXGGjbITtGo7R02uiJC/GibgMRgqDkuZBDdaY4xHJ2cJsKmRGn4APtZHQqGT8qDsrGX1cnRA98hATJos5HAzEYbPhKSs/WnY9eWjYa+zo8AaORgdHwlcJBBQFeZfSS6R2wfFs3H2s4p/L/j5L+gdMFITNLatbcWBsWv9moK0VHYx/GigBf7dYOx2tW9kazis20HbQzukWAF1EwxvL8kx3rVlKmvpdhbMxlhc4VXUyrbzGPJS2YgJaOXV0ljDpqOiikJsM726/vrbu+nk6/zq0+d3l1filCKQlz1lLSx2neh/Orv9RMS/G4v9oMk5NLhLYEO2qfEjNa7vo1o+Z73uR3aEo+5zpVmlXSQ8nLDP+eMcA91EpvooVfs/fSbjQpqncPHg5AmWqrFiYkcFzo/m4elIpwg5WOOaO7kjHwy1wlsV47Qj0KmRattExpDqYNhoZUGuKWyMxjdaSzHTb43HZDHsH80v97BGIXbZx7cPXWxLSIC8IIUfXN0ZaRLRCMsSNUMOU5rrCovGSuVdGqkqDWTx7LQecV4gm3qrHP6nmWsl9T+eciHVerK02s69XM0lOXlRzL5l2uk//+VXqVZ38zXedubrD3jfsjzlVlHktl8SuK3Q3bio2MTSdM8VTGhKfChU2HZ9dBiLpVl9VF6IGMb65Lr2L0yynzSHFRGyA02pwLckShxQD0ty3DdDefcDo/Sj84HNaVx68AZ5aStlDx59yiy7xW632/0TAAD//y2Oxzv8CQAA","objectset.rio.cattle.io/id":"","objectset.rio.cattle.io/owner-gvk":"k3s.cattle.io/v1, Kind=Addon","objectset.rio.cattle.io/owner-name":"coredns","objectset.rio.cattle.io/owner-namespace":"kube-system"},"labels":{"k8s-app":"kube-dns","kubernetes.io/name":"CoreDNS","objectset.rio.cattle.io/hash":"bce283298811743a0386ab510f2f67ef74240c57"},"name":"coredns","namespace":"kube-system"},"spec":{"revisionHistoryLimit":0,"selector":{"matchLabels":{"k8s-app":"kube-dns"}},"strategy":{"rollingUpdate":{"maxUnavailable":1},"type":"RollingUpdate"},"template":{"metadata":{"labels":{"k8s-app":"kube-dns"}},"spec":{"containers":[{"args":["-conf","/etc/coredns/Corefile"],"image":"rancher/mirrored-coredns-coredns:1.11.3","imagePullPolicy":"IfNotPresent","livenessProbe":{"failureThreshold":3,"httpGet":{"path":"/health","port":8080,"scheme":"HTTP"},"initialDelaySeconds":60,"periodSeconds":10,"successThreshold":1,"timeoutSeconds":1},"name":"coredns","ports":[{"containerPort":53,"name":"dns","protocol":"UDP"},{"containerPort":53,"name":"dns-tcp","protocol":"TCP"},{"containerPort":9153,"name":"metrics","protocol":"TCP"}],"readinessProbe":{"failureThreshold":3,"httpGet":{"path":"/ready","port":8181,"scheme":"HTTP"},"initialDelaySeconds":0,"periodSeconds":2,"successThreshold":1,"timeoutSeconds":1},"resources":{"limits":{"memory":"170Mi"},"requests":{"cpu":"100m","memory":"70Mi"}},"securityContext":{"allowPrivilegeEscalation":false,"capabilities":{"add":["NET_BIND_SERVICE"],"drop":["all"]},"readOnlyRootFilesystem":true},"volumeMounts":[{"mountPath":"/etc/coredns","name":"config-volume","readOnly":true},{"mountPath":"/etc/coredns/custom","name":"custom-config-volume","readOnly":true}]}],"dnsPolicy":"Default","nodeSelector":{"kubernetes.io/os":"linux"},"priorityClassName":"system-cluster-critical","serviceAccountName":"coredns","tolerations":[{"key":"CriticalAddonsOnly","operator":"Exists"},{"effect":"NoSchedule","key":"node-role.kubernetes.io/control-plane","operator":"Exists"},{"effect":"NoSchedule","key":"node-role.kubernetes.io/master","operator":"Exists"}],"topologySpreadConstraints":[{"labelSelector":{"matchLabels":{"k8s-app":"kube-dns"}},"maxSkew":1,"topologyKey":"kubernetes.io/hostname","whenUnsatisfiable":"DoNotSchedule"}],"volumes":[{"configMap":{"items":[{"key":"Corefile","path":"Corefile"},{"key":"NodeHosts","path":"NodeHosts"}],"name":"coredns"},"name":"config-volume"},{"configMap":{"name":"coredns-custom","optional":true},"name":"custom-config-volume"}]}}}}, MODIFIED:{"apiVersion":"apps/v1","kind":"Deployment","metadata":{"annotations":{"objectset.rio.cattle.io/applied":"H4sIAAAAAAAA/6xVQW/bOBP9Kx/mLMVW3DaugO/QjbPboq3XqJNeCqOgqZHFNcXhkiMnRuD/vhhJduw2TdrFniyTb4ZvHucN70F58xlDNOQgB+V9HGwySGBtXAE5TNBb2tboGBKokVWhWEF+D8o5YsWGXJS/tPwLNUfks2DoTCtmi2eGBkaSQPLDfbp1GNLVZg05rEfxaGeTJf97b1zx/zdFQe7ZFE7VCDloCli4+FPw6JWWmHWzxDRuI2MNuwSsWqJti1qPY6q830O6vPIZHDJGydYfe0kBJ9P5E8dWKlaQw1Lj+Xh0/no8zrKLFyM1HI1fqeXLbFiel68usLx4cf5iqF9eCJHvSnqCdPSohXLAjZG7fGsiU9h+MLVhyIcJRLSomYKAasW6+vBUmTtJyUExrrZtWrLWuNWNLxRjl+LuxqmNMlYtLUKe7RLgrRdmn06wso61t/u4oxZ6UujdUVGaHCvjMETIv9yDCiv5gFSTKyGBAbIe9CoN5CZKYxEWCZharYRRUE5XGAa1CUFgaQ/e/+bZWZadjaCPmDXWzsgavYUc3pVT4lnA2FnAmg06jHEWaNkWVCpjm4DXVcBYkS0gHyVQMfs/kGXfK5Z7H1SoLFeQgKfAkI+HY7kUXWF7x2+vr2cilXGGjbITtGo7R02uiJC/GibgMRgqDkuZBDdaY4xHJ2cJsKmRGn4APtZHQqGT8qDsrGX1cnRA98hATJos5HAzEYbPhKSs/WnY9eWjYa+zo8AaORgdHwlcJBBQFeZfSS6R2wfFs3H2s4p/L/j5L+gdMFITNLatbcWBsWv9moK0VHYx/GigBf7dYOx2tW9kazis20HbQzukWAF1EwxvL8kx3rVlKmvpdhbMxlhc4VXUyrbzGPJS2YgJaOXV0ljDpqOiikJsM726/vrbu+nk6/zq0+d3l1filCKQlz1lLSx2neh/Orv9RMS/G4v9oMk5NLhLYEO2qfEjNa7vo1o+Z73uR3aEo+5zpVmlXSQ8nLDP+eMcA91EpvooVfs/fSbjQpqncPHg5AmWqrFiYkcFzo/m4elIpwg5WOOaO7kjHwy1wlsV47Qj0KmRattExpDqYNhoZUGuKWyMxjdaSzHTb43HZDHsH80v97BGIXbZx7cPXWxLSIC8IIUfXN0ZaRLRCMsSNUMOU5rrCovGSuVdGqkqDWTx7LQecV4gm3qrHP6nmWsl9T+eciHVerK02s69XM0lOXlRzL5l2uk//+VXqVZ38zXedubrD3jfsjzlVlHktl8SuK3Q3bio2MTSdM8VTGhKfChU2HZ9dBiLpVl9VF6IGMb65Lr2L0yynzSHFRGyA02pwLckShxQD0ty3DdDefcDo/Sj84HNaVx68AZ5aStlDx59yiy7xW632/0TAAD//y2Oxzv8CQAA","objectset.rio.cattle.io/id":"","objectset.rio.cattle.io/owner-gvk":"k3s.cattle.io/v1, Kind=Addon","objectset.rio.cattle.io/owner-name":"coredns","objectset.rio.cattle.io/owner-namespace":"kube-system"},"labels":{"k8s-app":"kube-dns","kubernetes.io/name":"CoreDNS","objectset.rio.cattle.io/hash":"bce283298811743a0386ab510f2f67ef74240c57"},"name":"coredns","namespace":"kube-system"},"spec":{"revisionHistoryLimit":0,"selector":{"matchLabels":{"k8s-app":"kube-dns"}},"strategy":{"rollingUpdate":{"maxUnavailable":1},"type":"RollingUpdate"},"template":{"metadata":{"labels":{"k8s-app":"kube-dns"}},"spec":{"containers":[{"args":["-conf","/etc/coredns/Corefile"],"image":"rancher/mirrored-coredns-coredns:1.11.3","imagePullPolicy":"IfNotPresent","livenessProbe":{"failureThreshold":3,"httpGet":{"path":"/health","port":8080,"scheme":"HTTP"},"initialDelaySeconds":60,"periodSeconds":10,"successThreshold":1,"timeoutSeconds":1},"name":"coredns","ports":[{"containerPort":53,"name":"dns","protocol":"UDP"},{"containerPort":53,"name":"dns-tcp","protocol":"TCP"},{"containerPort":9153,"name":"metrics","protocol":"TCP"}],"readinessProbe":{"failureThreshold":3,"httpGet":{"path":"/ready","port":8181,"scheme":"HTTP"},"initialDelaySeconds":0,"periodSeconds":2,"successThreshold":1,"timeoutSeconds":1},"resources":{"limits":{"memory":"170Mi"},"requests":{"cpu":"100m","memory":"70Mi"}},"securityContext":{"allowPrivilegeEscalation":false,"capabilities":{"add":["NET_BIND_SERVICE"],"drop":["all"]},"readOnlyRootFilesystem":true},"volumeMounts":[{"mountPath":"/etc/coredns","name":"config-volume","readOnly":true},{"mountPath":"/etc/coredns/custom","name":"custom-config-volume","readOnly":true}]}],"dnsPolicy":"Default","nodeSelector":{"kubernetes.io/os":"linux"},"priorityClassName":"system-cluster-critical","serviceAccountName":"coredns","tolerations":[{"key":"CriticalAddonsOnly","operator":"Exists"},{"effect":"NoSchedule","key":"node-role.kubernetes.io/control-plane","operator":"Exists"},{"effect":"NoSchedule","key":"node-role.kubernetes.io/master","operator":"Exists"}],"topologySpreadConstraints":[{"labelSelector":{"matchLabels":{"k8s-app":"kube-dns"}},"maxSkew":1,"topologyKey":"kubernetes.io/hostname","whenUnsatisfiable":"DoNotSchedule"}],"volumes":[{"configMap":{"items":[{"key":"Corefile","path":"Corefile"},{"key":"NodeHosts","path":"NodeHosts"}],"name":"coredns"},"name":"config-volume"},{"configMap":{"name":"coredns-custom","optional":true},"name":"custom-config-volume"}]}}}}, CURRENT:{"apiVersion":"apps/v1","kind":"Deployment","metadata":{"annotations":{"objectset.rio.cattle.io/applied":"H4sIAAAAAAAA/6xVQW/bOBP9Kx/mLMVW3DaugO/QjbPboq3XqJNeCqOgqZHFNcXhkiMnRuD/vhhJduw2TdrFniyTb4ZvHucN70F58xlDNOQgB+V9HGwySGBtXAE5TNBb2tboGBKokVWhWEF+D8o5YsWGXJS/tPwLNUfks2DoTCtmi2eGBkaSQPLDfbp1GNLVZg05rEfxaGeTJf97b1zx/zdFQe7ZFE7VCDloCli4+FPw6JWWmHWzxDRuI2MNuwSsWqJti1qPY6q830O6vPIZHDJGydYfe0kBJ9P5E8dWKlaQw1Lj+Xh0/no8zrKLFyM1HI1fqeXLbFiel68usLx4cf5iqF9eCJHvSnqCdPSohXLAjZG7fGsiU9h+MLVhyIcJRLSomYKAasW6+vBUmTtJyUExrrZtWrLWuNWNLxRjl+LuxqmNMlYtLUKe7RLgrRdmn06wso61t/u4oxZ6UujdUVGaHCvjMETIv9yDCiv5gFSTKyGBAbIe9CoN5CZKYxEWCZharYRRUE5XGAa1CUFgaQ/e/+bZWZadjaCPmDXWzsgavYUc3pVT4lnA2FnAmg06jHEWaNkWVCpjm4DXVcBYkS0gHyVQMfs/kGXfK5Z7H1SoLFeQgKfAkI+HY7kUXWF7x2+vr2cilXGGjbITtGo7R02uiJC/GibgMRgqDkuZBDdaY4xHJ2cJsKmRGn4APtZHQqGT8qDsrGX1cnRA98hATJos5HAzEYbPhKSs/WnY9eWjYa+zo8AaORgdHwlcJBBQFeZfSS6R2wfFs3H2s4p/L/j5L+gdMFITNLatbcWBsWv9moK0VHYx/GigBf7dYOx2tW9kazis20HbQzukWAF1EwxvL8kx3rVlKmvpdhbMxlhc4VXUyrbzGPJS2YgJaOXV0ljDpqOiikJsM726/vrbu+nk6/zq0+d3l1filCKQlz1lLSx2neh/Orv9RMS/G4v9oMk5NLhLYEO2qfEjNa7vo1o+Z73uR3aEo+5zpVmlXSQ8nLDP+eMcA91EpvooVfs/fSbjQpqncPHg5AmWqrFiYkcFzo/m4elIpwg5WOOaO7kjHwy1wlsV47Qj0KmRattExpDqYNhoZUGuKWyMxjdaSzHTb43HZDHsH80v97BGIXbZx7cPXWxLSIC8IIUfXN0ZaRLRCMsSNUMOU5rrCovGSuVdGqkqDWTx7LQecV4gm3qrHP6nmWsl9T+eciHVerK02s69XM0lOXlRzL5l2uk//+VXqVZ38zXedubrD3jfsjzlVlHktl8SuK3Q3bio2MTSdM8VTGhKfChU2HZ9dBiLpVl9VF6IGMb65Lr2L0yynzSHFRGyA02pwLckShxQD0ty3DdDefcDo/Sj84HNaVx68AZ5aStlDx59yiy7xW632/0TAAD//y2Oxzv8CQAA","objectset.rio.cattle.io/id":"","objectset.rio.cattle.io/owner-gvk":"k3s.cattle.io/v1, Kind=Addon","objectset.rio.cattle.io/owner-name":"coredns","objectset.rio.cattle.io/owner-namespace":"kube-system"},"creationTimestamp":"2024-09-29T10:02:17Z","generation":1,"labels":{"k8s-app":"kube-dns","kubernetes.io/name":"CoreDNS","objectset.rio.cattle.io/hash":"bce283298811743a0386ab510f2f67ef74240c57"},"managedFields":[{"apiVersion":"apps/v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:objectset.rio.cattle.io/applied":{},"f:objectset.rio.cattle.io/id":{},"f:objectset.rio.cattle.io/owner-gvk":{},"f:objectset.rio.cattle.io/owner-name":{},"f:objectset.rio.cattle.io/owner-namespace":{}},"f:labels":{".":{},"f:k8s-app":{},"f:kubernetes.io/name":{},"f:objectset.rio.cattle.io/hash":{}}},"f:spec":{"f:progressDeadlineSeconds":{},"f:replicas":{},"f:revisionHistoryLimit":{},"f:selector":{},"f:strategy":{"f:rollingUpdate":{".":{},"f:maxSurge":{},"f:maxUnavailable":{}},"f:type":{}},"f:template":{"f:metadata":{"f:labels":{".":{},"f:k8s-app":{}}},"f:spec":{"f:containers":{"k:{\"name\":\"coredns\"}":{".":{},"f:args":{},"f:image":{},"f:imagePullPolicy":{},"f:livenessProbe":{".":{},"f:failureThreshold":{},"f:httpGet":{".":{},"f:path":{},"f:port":{},"f:scheme":{}},"f:initialDelaySeconds":{},"f:periodSeconds":{},"f:successThreshold":{},"f:timeoutSeconds":{}},"f:name":{},"f:ports":{".":{},"k:{\"containerPort\":53,\"protocol\":\"TCP\"}":{".":{},"f:containerPort":{},"f:name":{},"f:protocol":{}},"k:{\"containerPort\":53,\"protocol\":\"UDP\"}":{".":{},"f:containerPort":{},"f:name":{},"f:protocol":{}},"k:{\"containerPort\":9153,\"protocol\":\"TCP\"}":{".":{},"f:containerPort":{},"f:name":{},"f:protocol":{}}},"f:readinessProbe":{".":{},"f:failureThreshold":{},"f:httpGet":{".":{},"f:path":{},"f:port":{},"f:scheme":{}},"f:periodSeconds":{},"f:successThreshold":{},"f:timeoutSeconds":{}},"f:resources":{".":{},"f:limits":{".":{},"f:memory":{}},"f:requests":{".":{},"f:cpu":{},"f:memory":{}}},"f:securityContext":{".":{},"f:allowPrivilegeEscalation":{},"f:capabilities":{".":{},"f:add":{},"f:drop":{}},"f:readOnlyRootFilesystem":{}},"f:terminationMessagePath":{},"f:terminationMessagePolicy":{},"f:volumeMounts":{".":{},"k:{\"mountPath\":\"/etc/coredns\"}":{".":{},"f:mountPath":{},"f:name":{},"f:readOnly":{}},"k:{\"mountPath\":\"/etc/coredns/custom\"}":{".":{},"f:mountPath":{},"f:name":{},"f:readOnly":{}}}}},"f:dnsPolicy":{},"f:nodeSelector":{},"f:priorityClassName":{},"f:restartPolicy":{},"f:schedulerName":{},"f:securityContext":{},"f:serviceAccount":{},"f:serviceAccountName":{},"f:terminationGracePeriodSeconds":{},"f:tolerations":{},"f:topologySpreadConstraints":{".":{},"k:{\"topologyKey\":\"kubernetes.io/hostname\",\"whenUnsatisfiable\":\"DoNotSchedule\"}":{".":{},"f:labelSelector":{},"f:maxSkew":{},"f:topologyKey":{},"f:whenUnsatisfiable":{}}},"f:volumes":{".":{},"k:{\"name\":\"config-volume\"}":{".":{},"f:configMap":{".":{},"f:defaultMode":{},"f:items":{},"f:name":{}},"f:name":{}},"k:{\"name\":\"custom-config-volume\"}":{".":{},"f:configMap":{".":{},"f:defaultMode":{},"f:name":{},"f:optional":{}},"f:name":{}}}}}}},"manager":"deploy@develop","operation":"Update","time":"2024-09-29T10:02:17Z"}],"name":"coredns","namespace":"kube-system","resourceVersion":"234","uid":"297bd518-275f-4a29-a11a-2b5f7e97646a"},"spec":{"progressDeadlineSeconds":600,"replicas":1,"revisionHistoryLimit":0,"selector":{"matchLabels":{"k8s-app":"kube-dns"}},"strategy":{"rollingUpdate":{"maxSurge":"25%","maxUnavailable":1},"type":"RollingUpdate"},"template":{"metadata":{"creationTimestamp":null,"labels":{"k8s-app":"kube-dns"}},"spec":{"containers":[{"args":["-conf","/etc/coredns/Corefile"],"image":"rancher/mirrored-coredns-coredns:1.11.3","imagePullPolicy":"IfNotPresent","livenessProbe":{"failureThreshold":3,"httpGet":{"path":"/health","port":8080,"scheme":"HTTP"},"initialDelaySeconds":60,"periodSeconds":10,"successThreshold":1,"timeoutSeconds":1},"name":"coredns","ports":[{"containerPort":53,"name":"dns","protocol":"UDP"},{"containerPort":53,"name":"dns-tcp","protocol":"TCP"},{"containerPort":9153,"name":"metrics","protocol":"TCP"}],"readinessProbe":{"failureThreshold":3,"httpGet":{"path":"/ready","port":8181,"scheme":"HTTP"},"periodSeconds":2,"successThreshold":1,"timeoutSeconds":1},"resources":{"limits":{"memory":"170Mi"},"requests":{"cpu":"100m","memory":"70Mi"}},"securityContext":{"allowPrivilegeEscalation":false,"capabilities":{"add":["NET_BIND_SERVICE"],"drop":["all"]},"readOnlyRootFilesystem":true},"terminationMessagePath":"/dev/termination-log","terminationMessagePolicy":"File","volumeMounts":[{"mountPath":"/etc/coredns","name":"config-volume","readOnly":true},{"mountPath":"/etc/coredns/custom","name":"custom-config-volume","readOnly":true}]}],"dnsPolicy":"Default","nodeSelector":{"kubernetes.io/os":"linux"},"priorityClassName":"system-cluster-critical","restartPolicy":"Always","schedulerName":"default-scheduler","securityContext":{},"serviceAccount":"coredns","serviceAccountName":"coredns","terminationGracePeriodSeconds":30,"tolerations":[{"key":"CriticalAddonsOnly","operator":"Exists"},{"effect":"NoSchedule","key":"node-role.kubernetes.io/control-plane","operator":"Exists"},{"effect":"NoSchedule","key":"node-role.kubernetes.io/master","operator":"Exists"}],"topologySpreadConstraints":[{"labelSelector":{"matchLabels":{"k8s-app":"kube-dns"}},"maxSkew":1,"topologyKey":"kubernetes.io/hostname","whenUnsatisfiable":"DoNotSchedule"}],"volumes":[{"configMap":{"defaultMode":420,"items":[{"key":"Corefile","path":"Corefile"},{"key":"NodeHosts","path":"NodeHosts"}],"name":"coredns"},"name":"config-volume"},{"configMap":{"defaultMode":420,"name":"coredns-custom","optional":true},"name":"custom-config-volume"}]}}},"status":{}}] 
DEBU[0002] DesiredSet - Updated apps/v1, Kind=Deployment kube-system/coredns for  kube-system/coredns -- application/strategic-merge-patch+json {"spec":{"template":{"spec":{"$setElementOrder/containers":[{"name":"coredns"}],"containers":[{"name":"coredns","readinessProbe":{"initialDelaySeconds":0}}]}}}} 
I0929 18:21:35.441165     754 server.go:737] "--cgroups-per-qos enabled, but --cgroup-root was not specified.  defaulting to /"
I0929 18:21:35.441223     754 server.go:805] "NoSwap is set due to memorySwapBehavior not specified" memorySwapBehavior="" FailSwapOn=false
I0929 18:21:35.441650     754 container_manager_linux.go:265] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
I0929 18:21:35.441794     754 container_manager_linux.go:270] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"develop","RuntimeCgroupsName":"","SystemCgroupsName":"","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":false,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":null,"HardEvictionThresholds":[{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"ExperimentalMemoryManagerPolicy":"None","ExperimentalMemoryManagerReservedMemory":null,"PodPidsLimit":-1,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null}
I0929 18:21:35.446251     754 topology_manager.go:138] "Creating topology manager with none policy"
I0929 18:21:35.450128     754 container_manager_linux.go:301] "Creating device plugin manager"
I0929 18:21:35.450264     754 state_mem.go:36] "Initialized new in-memory state store"
I0929 18:21:35.450431     754 kubelet.go:401] "Attempting to sync node with API server"
I0929 18:21:35.451857     754 kubelet.go:302] "Adding static pod path" path="/var/lib/rancher/k3s/agent/pod-manifests"
I0929 18:21:35.452039     754 kubelet.go:313] "Adding apiserver pod source"
I0929 18:21:35.452167     754 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
INFO[0002] Waiting for control-plane node develop startup: nodes "develop" not found 
INFO[0002] Cluster dns configmap already exists         
I0929 18:21:35.460526     754 kuberuntime_manager.go:261] "Container runtime initialized" containerRuntime="containerd" version="v1.7.21-k3s2" apiVersion="v1"
I0929 18:21:35.461097     754 kubelet.go:816] "Not starting ClusterTrustBundle informer because we are in static kubelet mode"
I0929 18:21:35.461926     754 server.go:1259] "Started kubelet"
I0929 18:21:35.468082     754 controller.go:615] quota admission added evaluator for: deployments.apps
I0929 18:21:35.468953     754 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
E0929 18:21:35.487995     754 kubelet.go:1468] "Image garbage collection failed once. Stats initialization may not have completed yet" err="invalid capacity 0 on image filesystem"
I0929 18:21:35.494903     754 server.go:163] "Starting to listen" address="0.0.0.0" port=10250
I0929 18:21:35.496009     754 server.go:455] "Adding debug handlers to kubelet server"
I0929 18:21:35.496615     754 ratelimit.go:55] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
I0929 18:21:35.496843     754 server.go:227] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
DEBU[0002] DesiredSet - No change(2) /v1, Kind=Service kube-system/kube-dns for  kube-system/coredns 
I0929 18:21:35.498103     754 event.go:389] "Event occurred" object="kube-system/coredns" fieldPath="" kind="Addon" apiVersion="k3s.cattle.io/v1" type="Normal" reason="AppliedManifest" message="Applied manifest at \"/var/lib/rancher/k3s/server/manifests/coredns.yaml\""
I0929 18:21:35.500619     754 volume_manager.go:291] "Starting Kubelet Volume Manager"
I0929 18:21:35.501233     754 desired_state_of_world_populator.go:149] "Desired state populator starts to run"
I0929 18:21:35.501476     754 reconciler.go:26] "Reconciler: start to sync state"
I0929 18:21:35.507289     754 factory.go:219] Registration of the crio container factory failed: Get "http://%2Fvar%2Frun%2Fcrio%2Fcrio.sock/info": dial unix /var/run/crio/crio.sock: connect: no such file or directory
I0929 18:21:35.508924     754 kubelet_network_linux.go:50] "Initialized iptables rules." protocol="IPv4"
I0929 18:21:35.509782     754 factory.go:221] Registration of the containerd container factory successfully
I0929 18:21:35.509817     754 factory.go:221] Registration of the systemd container factory successfully
DEBU[0002] cgroupv2 io stats: skipping over unmappable dbytes=0 entry 
DEBU[0002] cgroupv2 io stats: skipping over unmappable dios=0 entry 
E0929 18:21:35.514715     754 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"develop\" not found" node="develop"
I0929 18:21:35.521039     754 event.go:389] "Event occurred" object="kube-system/local-storage" fieldPath="" kind="Addon" apiVersion="k3s.cattle.io/v1" type="Normal" reason="ApplyingManifest" message="Applying manifest at \"/var/lib/rancher/k3s/server/manifests/local-storage.yaml\""
I0929 18:21:35.521420     754 cpu_manager.go:214] "Starting CPU manager" policy="none"
I0929 18:21:35.521451     754 cpu_manager.go:215] "Reconciling" reconcilePeriod="10s"
I0929 18:21:35.521489     754 state_mem.go:36] "Initialized new in-memory state store"
I0929 18:21:35.521633     754 state_mem.go:88] "Updated default CPUSet" cpuSet=""
I0929 18:21:35.521696     754 state_mem.go:96] "Updated CPUSet assignments" assignments={}
I0929 18:21:35.521728     754 policy_none.go:49] "None policy: Start"
I0929 18:21:35.509799     754 kubelet_network_linux.go:50] "Initialized iptables rules." protocol="IPv6"
I0929 18:21:35.526753     754 status_manager.go:217] "Starting to sync pod status with apiserver"
I0929 18:21:35.526793     754 kubelet.go:2356] "Starting kubelet main sync loop"
E0929 18:21:35.526879     754 kubelet.go:2380] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
DEBU[0002] DesiredSet - No change(2) /v1, Kind=ServiceAccount kube-system/local-path-provisioner-service-account for  kube-system/local-storage 
I0929 18:21:35.534323     754 memory_manager.go:170] "Starting memorymanager" policy="None"
I0929 18:21:35.534722     754 state_mem.go:35] "Initializing new in-memory state store"
I0929 18:21:35.535039     754 state_mem.go:75] "Updated machine memory state"
E0929 18:21:35.535442     754 kubelet.go:1556] "Failed to start ContainerManager" err="[open /proc/sys/kernel/panic: read-only file system, open /proc/sys/kernel/panic_on_oops: read-only file system]"

It says ReadOnly, But here is mount.

/dev/nvme0n1p2 on / type xfs (rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota)
/dev/nvme0n1p2 on /etc/resolv.conf type xfs (ro,nosuid,nodev,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota)
/dev/nvme0n1p2 on /run/host/os-release type xfs (ro,nosuid,nodev,noexec,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota)
/dev/nvme0n1p2 on /run/host/os-release type xfs (rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota)
cgroup on /sys/fs/cgroup type cgroup2 (rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=666)
hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime,pagesize=2M)
mqueue on /dev/mqueue type mqueue (rw,nosuid,nodev,noexec,relatime)
proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
proc on /proc/acpi type proc (ro,nosuid,nodev,noexec,relatime)
proc on /proc/bus type proc (ro,nosuid,nodev,noexec,relatime)
proc on /proc/fs type proc (ro,nosuid,nodev,noexec,relatime)
proc on /proc/irq type proc (ro,nosuid,nodev,noexec,relatime)
proc on /proc/sys type proc (ro,nosuid,nodev,noexec,relatime)
proc on /proc/sys/net type proc (rw,nosuid,nodev,noexec,relatime)
proc on /proc/sys/net type proc (rw,nosuid,nodev,noexec,relatime)
ramfs on /run/credentials/systemd-sysctl.service type ramfs (ro,nosuid,nodev,noexec,relatime,mode=700)
ramfs on /run/credentials/systemd-sysusers.service type ramfs (ro,nosuid,nodev,noexec,relatime,mode=700)
ramfs on /run/credentials/systemd-tmpfiles-setup-dev.service type ramfs (ro,nosuid,nodev,noexec,relatime,mode=700)
ramfs on /run/credentials/systemd-tmpfiles-setup.service type ramfs (ro,nosuid,nodev,noexec,relatime,mode=700)
sysfs on /sys/block type sysfs (ro,nosuid,nodev,noexec,relatime)
sysfs on /sys/bus type sysfs (ro,nosuid,nodev,noexec,relatime)
sysfs on /sys/class type sysfs (ro,nosuid,nodev,noexec,relatime)
sysfs on /sys/dev type sysfs (ro,nosuid,nodev,noexec,relatime)
sysfs on /sys/devices type sysfs (ro,nosuid,nodev,noexec,relatime)
sysfs on /sys/kernel type sysfs (ro,nosuid,nodev,noexec,relatime)
sysfs on /sys/module type sysfs (ro,nosuid,nodev,noexec,relatime)
tmpfs on /dev type tmpfs (rw,nosuid,size=4096k,nr_inodes=65536,mode=755,inode64)
tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev,size=1634120k,nr_inodes=409600,inode64)
tmpfs on /proc/kmsg type tmpfs (rw,nosuid,nodev,size=3268236k,nr_inodes=819200,mode=755,inode64)
tmpfs on /proc/sys/kernel/random/boot_id type tmpfs (ro,nosuid,nodev,noexec,size=3268236k,nr_inodes=819200,mode=755,inode64)
tmpfs on /proc/sys/kernel/random/boot_id type tmpfs (rw,nosuid,nodev,size=3268236k,nr_inodes=819200,mode=755,inode64)
tmpfs on /run type tmpfs (rw,nosuid,nodev,size=3268236k,nr_inodes=819200,mode=755,inode64)
tmpfs on /run/host type tmpfs (ro,nosuid,nodev,noexec,size=3268236k,nr_inodes=819200,mode=755,inode64)
tmpfs on /run/host/incoming type tmpfs (ro,relatime,size=1634120k,mode=755,inode64)
tmpfs on /run/host/incoming type tmpfs (rw,nosuid,nodev,noexec,relatime,size=1634120k,mode=755,inode64)
tmpfs on /run/lock type tmpfs (rw,nosuid,nodev,noexec,relatime,size=5120k,inode64)
tmpfs on /sys type tmpfs (ro,nosuid,nodev,noexec,relatime,size=4096k,nr_inodes=1024,mode=555,inode64)
tmpfs on /sys/module/apparmor type tmpfs (ro,relatime,size=1634120k,mode=755,inode64)
tmpfs on /tmp type tmpfs (rw,nosuid,nodev,size=1634120k,nr_inodes=409600,inode64)
udev on /dev/kmsg type devtmpfs (rw,nosuid,relatime,size=8162492k,nr_inodes=2040623,mode=755,inode64)

I have done some research with container in container, But those posts seems like no helping:

• docker is unable to run inside systemd-nspawn container moby/moby#44402
• https://docs.docker.com/reference/cli/dockerd/#default-cgroup-parent
• https://github.com/systemd/systemd/blob/13a5ffa4777545ca18971bf23c6dfd12f005eef2/src/nspawn/nspawn-cgroup.c#L549

[brandond]

E0929 18:21:35.535442 754 kubelet.go:1556] "Failed to start ContainerManager" err="[open /proc/sys/kernel/panic: read-only file system, open /proc/sys/kernel/panic_on_oops: read-only file system]"

The container must be privileged, and allowed to write to these profcs paths.