Next Steps in example workflows not executed

[AvasDream]

Hello All,

I currently face a problem, and I'm not sure if it is a bug or If I'm missing something. So, before opening an issue I ask you here.

Setup:

I'm using Terraform to deploy a managed Kubernetes Cluster to DigitalOcean. This may be relevant because the Bug was not reproducible in all my Clusters. After the Cluster is created, and I added the kubectl config I executed the following commands from the documentation:

kubectl create ns argo
kubectl apply -n argo -f https://raw.githubusercontent.com/argoproj/argo-workflows/stable/manifests/quick-start-postgres.yaml
kubectl patch svc argo-server -n argo -p '{"spec": {"type": "LoadBalancer"}}'
kubectl get svc argo-server -n argo

The Terraform code

provider "digitalocean" {
  version = "~> 1.22"
  token   = var.do_access_key
}

resource "digitalocean_kubernetes_cluster" "security-testing-cluster" {
  name   = "kubernetes-argo"
  region = "fra1"
  version = "1.20.2-do.0"

  node_pool {
    name       = "worker-pool"
    size       = "s-2vcpu-2gb"
    node_count = 6
  }
  provisioner "local-exec" {
    command = "doctl kubernetes cluster kubeconfig save ${self.id}"
  }
}

I'm aware of the security implications. The cluster is destroyed after max 2h.

Versions in use:

Argo

argo: v2.12.11
  BuildDate: 2021-04-05T21:14:07Z
  GitCommit: 71d00c7878e2b904ad35ca25712bef7e84893ae2
  GitTreeState: clean
  GitTag: v2.12.11
  GoVersion: go1.16.3
  Compiler: gc
  Platform: darwin/amd64

Kubectl

clientVersion:
  buildDate: "2021-04-08T21:16:14Z"
  compiler: gc
  gitCommit: cb303e613a121a29364f75cc67d3d580833a7479
  gitTreeState: clean
  gitVersion: v1.21.0
  goVersion: go1.16.3
  major: "1"
  minor: "21"
  platform: darwin/amd64
serverVersion:
  buildDate: "2021-01-13T13:20:00Z"
  compiler: gc
  gitCommit: faecb196815e248d3ecfb03c680a4507229c2a56
  gitTreeState: clean
  gitVersion: v1.20.2
  goVersion: go1.15.5
  major: "1"
  minor: "20"
  platform: linux/amd64

The Problem:

When trying the example workflow for steps or dag only the first step in the workflow is executed. The execution is successful without any errors in the logs.

Steps Workflow:

apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  generateName: steps-
spec:
  entrypoint: hello-hello-hello

  # This spec contains two templates: hello-hello-hello and whalesay
  templates:
  - name: hello-hello-hello
    # Instead of just running a container
    # This template has a sequence of steps
    steps:
    - - name: hello1            # hello1 is run before the following steps
        template: whalesay
        arguments:
          parameters:
          - name: message
            value: "hello1"
    - - name: hello2a           # double dash => run after previous step
        template: whalesay
        arguments:
          parameters:
          - name: message
            value: "hello2a"
      - name: hello2b           # single dash => run in parallel with previous step
        template: whalesay
        arguments:
          parameters:
          - name: message
            value: "hello2b"

  # This is the same template as from the previous example
  - name: whalesay
    inputs:
      parameters:
      - name: message
    container:
      image: docker/whalesay
      command: [cowsay]
      args: ["{{inputs.parameters.message}}"]

logs hello1 main

 ________
< hello1 >
 --------
    \
     \
      \
                    ##        .
              ## ## ##       ==
           ## ## ## ##      ===
       /""""""""""""""""___/ ===
  ~~~ {~~ ~~~~ ~~~ ~~~~ ~~ ~ /  ===- ~~~
       \______ o          __/
        \    \        __/
          \____\______/

logs hello1 wait

...
kubernetes.pod.namespace=argo --filter=label=io.kubernetes.pod.name=steps-vs5jb-1691724592"
time="2021-04-18T12:01:12.537Z" level=info msg="docker ps --all --no-trunc --format={{.Status}}|{{.Label \"io.kubernetes.container.name\"}}|{{.ID}}|{{.CreatedAt}} --filter=label=io.
...

Commands used to execute the Workflow:

argo submit test/steps-example.yaml -n argo

> argo get steps-68465
Name:                steps-68465
Namespace:           argo
ServiceAccount:      default
Status:              Running
Conditions:          
 PodRunning          True
Created:             Sun Apr 18 14:05:14 +0200 (1 minute ago)
Started:             Sun Apr 18 14:05:14 +0200 (1 minute ago)
Duration:            1 minute 45 seconds
Progress:            0/1

STEP            TEMPLATE           PODNAME                 DURATION  MESSAGE
 ● steps-68465  hello-hello-hello                                      
 └───● hello1   whalesay           steps-68465-2208092209  1m 

I got the problem in multiple workflows. The workflow shown above and the example DAG workflow from the documentation.
My Question here is, did I do something wrong? I followed the documentation for the setup and this did work in a previous cluster. The Configuration of the Cluster did not change because I used the same terraform code. Any feedback is appreciated.

[AvasDream]

So, the solution for me was setting the containerRuntimeExecutor to pns.

kubectl -n argo patch cm workflow-controller-configmap -p '{"data": {"containerRuntimeExecutor": "pns"}}'

Reference: https://github.com/argoproj/argo-workflows/blob/master/docs/workflow-executors.md

[marcstreeter]

this was so helpful, many thanks!