kubelet error during installation: unable to setup user: chown /dev/stdout: permission denied: unknown

[hmz-walter]

When I installed Argo Workflows in my K8s cluster, I encountered an issue with the pods argo-server and workflow-controller. The error message was unable to setup user: chown /dev/stdout: permission denied: unknown. Here are my steps:

1. install yaml

curl -LO "https://github.com/argoproj/argo-workflows/releases/download/v3.5.10/install.yaml"

2. change nodeSelector of workflow-controller and argo-server

      nodeSelector:
        kubernetes.io/hostname: "master"

3. kubectl apply -n argo -f ./install.yaml

[kube@master argoWorkflow]$ kubectl apply -n argo -f ./install.yaml 
customresourcedefinition.apiextensions.k8s.io/clusterworkflowtemplates.argoproj.io created
customresourcedefinition.apiextensions.k8s.io/cronworkflows.argoproj.io created
customresourcedefinition.apiextensions.k8s.io/workflowartifactgctasks.argoproj.io created
customresourcedefinition.apiextensions.k8s.io/workfloweventbindings.argoproj.io created
customresourcedefinition.apiextensions.k8s.io/workflows.argoproj.io created
customresourcedefinition.apiextensions.k8s.io/workflowtaskresults.argoproj.io created
customresourcedefinition.apiextensions.k8s.io/workflowtasksets.argoproj.io created
customresourcedefinition.apiextensions.k8s.io/workflowtemplates.argoproj.io created
serviceaccount/argo created
serviceaccount/argo-server created
role.rbac.authorization.k8s.io/argo-role created
clusterrole.rbac.authorization.k8s.io/argo-aggregate-to-admin created
clusterrole.rbac.authorization.k8s.io/argo-aggregate-to-edit created
clusterrole.rbac.authorization.k8s.io/argo-aggregate-to-view created
clusterrole.rbac.authorization.k8s.io/argo-cluster-role created
clusterrole.rbac.authorization.k8s.io/argo-server-cluster-role created
rolebinding.rbac.authorization.k8s.io/argo-binding created
clusterrolebinding.rbac.authorization.k8s.io/argo-binding created
clusterrolebinding.rbac.authorization.k8s.io/argo-server-binding created
configmap/workflow-controller-configmap created
service/argo-server created
priorityclass.scheduling.k8s.io/workflow-controller created
deployment.apps/argo-server created
deployment.apps/workflow-controller created

4. but these two pods failed to start

[kube@master argoWorkflow]$ kubectl get pods -n argo
NAME                                   READY   STATUS             RESTARTS     AGE
argo-server-869dbfcdc4-trx59           0/1     CrashLoopBackOff   1 (7s ago)   8s
workflow-controller-5477889d45-gjdhd   0/1     CrashLoopBackOff   1 (7s ago)   8s

5. then I found the problem:
   Warning Failed 21m (x5 over 23m) kubelet Error: failed to create containerd task: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: unable to setup user: chown /dev/stdout: permission denied: unknown

[kube@master argoWorkflow]$ kubectl describe pod -n argo argo-server-869dbfcdc4-trx59 
Name:             argo-server-869dbfcdc4-trx59
Namespace:        argo
Priority:         0
Service Account:  argo-server
Node:             master/192.168.131.131
Start Time:       Mon, 14 Oct 2024 09:19:26 +0800
Labels:           app=argo-server
                  pod-template-hash=869dbfcdc4
Annotations:      <none>
Status:           Running
IP:               10.244.0.162
IPs:
  IP:           10.244.0.162
Controlled By:  ReplicaSet/argo-server-869dbfcdc4
Containers:
  argo-server:
    Container ID:  containerd://b8e7cd9f40231db61e018ddd6f6f21dc04e45a7a41f094c9844d6ecfdbaa295f
    Image:         quay.io/argoproj/argocli:v3.5.10
    Image ID:      quay.io/argoproj/argocli@sha256:ccc55a32c9739f6d3e7649fe8b896ea90980fae95c4d318a43610dc80d20ddf9
    Port:          2746/TCP
    Host Port:     0/TCP
    Args:
      server
    State:          Waiting
      Reason:       CrashLoopBackOff
    Last State:     Terminated
      Reason:       StartError
      Message:      failed to create containerd task: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: unable to setup user: chown /dev/stdout: permission denied: unknown
      Exit Code:    128
      Started:      Thu, 01 Jan 1970 08:00:00 +0800
      Finished:     Mon, 14 Oct 2024 09:40:40 +0800
    Ready:          False
    Restart Count:  9
    Readiness:      http-get https://:2746/ delay=10s timeout=1s period=20s #success=1 #failure=3
    Environment:    <none>
    Mounts:
      /tmp from tmp (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-m57ch (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             False 
  ContainersReady   False 
  PodScheduled      True 
Volumes:
  tmp:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:     
    SizeLimit:  <unset>
  kube-api-access-m57ch:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   BestEffort
Node-Selectors:              kubernetes.io/hostname=master
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason     Age                From               Message
  ----     ------     ----               ----               -------
  Normal   Scheduled  23m                default-scheduler  Successfully assigned argo/argo-server-869dbfcdc4-trx59 to master
  Normal   Pulled     21m (x5 over 23m)  kubelet            Container image "quay.io/argoproj/argocli:v3.5.10" already present on machine
  Normal   Created    21m (x5 over 23m)  kubelet            Created container argo-server
  Warning  Failed     21m (x5 over 23m)  kubelet            Error: failed to create containerd task: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: unable to setup user: chown /dev/stdout: permission denied: unknown
  Warning  BackOff    3m (x97 over 23m)  kubelet            Back-off restarting failed container argo-server in pod argo-server-869dbfcdc4-trx59_argo(bf6dffb1-9530-49bb-8c7c-8188ea432ed6)

6. I understand what happened to these pods, but I'm not sure how to resolve the issue. Any help would be appreciated.