level=fatal msg="key unsupported" when running artifacts passing example

[SimonDanisch]

So this is likely just a misconfiguration as I'm just getting started with argo, but the error message is so undescriptive, that it almost feels like a bug and it's not clear how to proceed from here.

I followed the getting started tutorial and did something like:

minikube start --cpus=2 --memory=4096 --disk-size=40g
kubectl create ns argo
helm repo update
helm install argo-artifacts minio/minio --set service.type=LoadBalancer --set fullnameOverride=argo-artifacts
kubectl create rolebinding default-admin --clusterrole=admin --serviceaccount=argo:default --namespace=argo
kubectl apply -n argo -f https://raw.githubusercontent.com/argoproj/argo-workflows/master/manifests/namespace-install.yaml
minikube service --url argo-artifacts # go to url + create my-bucket
argo submit -n argo https://raw.githubusercontent.com/argoproj/argo/master/examples/artifact-passing.yaml --watch

If I interpret the docs correctly, that's all I need to do to get started with minio...
From other tutorials I've taken, that I also need to run:

kubectl edit configmap workflow-controller-configmap -n argo

And put something like this in there:

data:
  config: |
    artifactRepository:
      s3:
        bucket: my-bucket
        endpoint: argo-artifacts:9000
        insecure: true
        # accessKeySecret and secretKeySecret are secret selectors.
        # It references the k8s secret named 'argo-artifacts'
        # which was created during the minio helm install. The keys,
        # 'accesskey' and 'secretkey', inside that secret are where the
        # actual minio credentials are stored.
        accessKeySecret:
          name: argo-artifacts
          key: accesskey
        secretKeySecret:
          name: argo-artifacts
          key: secretkey

I tried that as well and tried all kind of combinations for the endpoint (e.g. the url returned by minikube service --url argo-artifacts).
I may have borked my config with all these different sources of information (I deleted things and started from scratch a couple of times, but who knows where there are remnants of configs), and argo gives me very little informations about why it's failing.
Here are the logs of the failed pod:

$ kubectl logs artifact-passing-dpws4-1843541938 -n argo -c wait
time="2021-02-12T12:50:41.018Z" level=info msg="Starting Workflow Executor" version="{latest+3b7e373.dirty 2021-02-11T22:43:36Z 3b7e373eeeb486efa2bef8f722394ef279ba1606 3b7e373eeeb486efa2bef8f722394ef279ba1606 dirty go1.15.7 gc linux/amd64}"
time="2021-02-12T12:50:41.020Z" level=info msg="Creating a docker executor"
time="2021-02-12T12:50:41.020Z" level=info msg="Executor (version: latest+3b7e373.dirty, build_date: 2021-02-11T22:43:36Z) initialized (pod: argo/artifact-passing-dpws4-1843541938) with template:\n{\"name\":\"whalesay\",\"inputs\":{},\"outputs\":{\"artifacts\":[{\"name\":\"hello-art\",\"path\":\"/tmp/hello_world.txt\"}]},\"metadata\":{},\"container\":{\"name\":\"\",\"image\":\"docker/whalesay:latest\",\"command\":[\"sh\",\"-c\"],\"args\":[\"sleep 1; cowsay hello world | tee /tmp/hello_world.txt\"],\"resources\":{}},\"archiveLocation\":{}}"
time="2021-02-12T12:50:41.020Z" level=info msg="Waiting on main container"
time="2021-02-12T12:50:41.022Z" level=info msg="Watch pods 200"
time="2021-02-12T12:50:43.887Z" level=info msg="main container started with container ID: 3f7f738e1b1251e45dca46c6650a884428c55c4b8ed888063712ad94b29d2cc8"
time="2021-02-12T12:50:43.887Z" level=info msg="Starting annotations monitor"
time="2021-02-12T12:50:43.889Z" level=info msg="Get pods 200"
time="2021-02-12T12:50:43.889Z" level=info msg="Starting deadline monitor"
time="2021-02-12T12:50:43.889Z" level=info msg="docker wait 3f7f738e1b1251e45dca46c6650a884428c55c4b8ed888063712ad94b29d2cc8"
time="2021-02-12T12:50:44.528Z" level=info msg="Main container completed"
time="2021-02-12T12:50:44.528Z" level=info msg="No Script output reference in workflow. Capturing script output ignored"
time="2021-02-12T12:50:44.528Z" level=info msg="Capturing script exit code"
time="2021-02-12T12:50:44.528Z" level=info msg="Annotations monitor stopped"
time="2021-02-12T12:50:44.541Z" level=info msg="No output parameters"
time="2021-02-12T12:50:44.541Z" level=info msg="Saving output artifacts"
time="2021-02-12T12:50:44.541Z" level=info msg="Staging artifact: hello-art"
time="2021-02-12T12:50:44.541Z" level=info msg="Copying /tmp/hello_world.txt from container base image layer to /tmp/argo/outputs/artifacts/hello-art.tgz"
time="2021-02-12T12:50:44.541Z" level=info msg="Archiving 3f7f738e1b1251e45dca46c6650a884428c55c4b8ed888063712ad94b29d2cc8:/tmp/hello_world.txt to /tmp/argo/outputs/artifacts/hello-art.tgz"
time="2021-02-12T12:50:44.541Z" level=info msg="sh -c docker cp -a 3f7f738e1b1251e45dca46c6650a884428c55c4b8ed888063712ad94b29d2cc8:/tmp/hello_world.txt - | gzip  > /tmp/argo/outputs/artifacts/hello-art.tgz"
time="2021-02-12T12:50:44.579Z" level=info msg="Archiving completed"
time="2021-02-12T12:50:44.579Z" level=error msg="executor error: key unsupported"
time="2021-02-12T12:50:44.579Z" level=info msg="Killing sidecars"
time="2021-02-12T12:50:44.581Z" level=info msg="Get pods 200"
time="2021-02-12T12:50:44.581Z" level=info msg="Alloc=5188 TotalAlloc=10674 Sys=74833 NumGC=4 Goroutines=10"
time="2021-02-12T12:50:44.585Z" level=info msg="Patch pods 200"
time="2021-02-12T12:50:44.585Z" level=fatal msg="key unsupported"

It would be nice if someone can point me to clues of what I'm missing, and how to debug this situation (I would assume there must be some logs somewhere, where it tells me what endpoint it uses & with what secrets and with what kind of error it fails).

[jorenvs]

I had this as well when I didn't configure an ArtifactRepository. You need to put it in "artifact-repositories" config map (doesn't exist by default). I created one like this, which works:

apiVersion: v1
kind: ConfigMap
metadata:
    # if you want to use this config map by default - name it "artifact-repositories" 
  name: artifact-repositories
  namespace: argo
  annotations:
    # v3.0 and after - if you want to use a specific key, put that's key into this annotation 
    workflows.argoproj.io/default-artifact-repository: artifactRepository
data:
  artifactRepository: |
    archiveLogs: true
    gcs:
      bucket: argo-bucket
      serviceAccountKeySecret:
        name: gcs-secret-key
        key: gcsSecretKey

I agree that the error is not really indicative though.

[alexec]

we're updating v3.0.2 to have better messages