Cannot push to Amazon ECR in KafkaConnect spec.build.output

[brsolomon-deloitte]

Attempting to point spec.build.output to an Amazon ECR endpoint and, despite having working credentials in a secret called regred, the push fails in the xxx-connect-build pod with a single log line:

error checking push permissions -- make sure you entered the correct tag name, and that you are authenticated correctly, and try again: checking push permission for "REDACTED.dkr.ecr.REGION-REDACTED.amazonaws.com/kafka-connect-elasticsearch:latest": POST https://REDACTED.dkr.ecr.REGION-REDACTED.amazonaws.com/v2/kafka-connect-elasticsearch/blobs/uploads/: DENIED: Could not resolve registry id from host REDACTED.dkr.ecr.REGION-REDACTED.amazonaws.com

I do not have SSH access to worker nodes so this is proving extremely difficult to debug.

KafkaConnect config:

apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaConnect
metadata:
  name: my-connect-cluster
  annotations:
    # Enables KafkaConnectors for the Kafka Connect cluster.
    strimzi.io/use-connector-resources: "true"
 
spec:
  replicas: 3

  # The Kafka Connect version
  version: 2.8.1

  jvmOptions:
    "-Xmx": "1g"
    "-Xms": "1g"
  resources:
    requests:
      cpu: "2"
      memory: 2Gi
    limits:
      cpu: "4"
      memory: 4Gi

  # Bootstrap server for connection to the Kafka Connect cluster
  # We use the ClusterIP service here, not the NodePort
  bootstrapServers: my-cluster-kafka-bootstrap:9092

  config:
    # Kafka Connect configuration of workers (not connectors)
    # This corresponds to a worker.properties file for bin/connect-distributed.sh
    group.id: my-connect-cluster
    offset.storage.topic: my-connect-cluster-offsets
    config.storage.topic: my-connect-cluster-configs
    status.storage.topic: my-connect-cluster-status
    key.converter: org.apache.kafka.connect.json.JsonConverter
    value.converter: org.apache.kafka.connect.json.JsonConverter
    key.converter.schemas.enable: false
    value.converter.schemas.enable: false
    config.storage.replication.factor: 3
    offset.storage.replication.factor: 3
    status.storage.replication.factor: 3
    # plugin.path: /usr/local/share/kafka/plugins
  
  build:
    # Configures where should the newly built image be stored
    # TODO: this needs to be renamed, it is not specific to elasticsearch now.
    output:
      type: docker
      # image: REDACTED.dkr.ecr.REGION-REDACTED.amazonaws.com/kafka-connect-elasticsearch:11.1.3
      image: REDACTED.dkr.ecr.REGION-REDACTED.amazonaws.com/kafka-connect-elasticsearch:latest
      pushSecret: regcred
    # List of connector plugins which should be added to the Kafka Connect
    plugins:
      - name: kafka-connect-elasticsearch
        artifacts:
          - type: jar
            url: https://packages.confluent.io/maven/io/confluent/kafka-connect-elasticsearch/11.1.3/kafka-connect-elasticsearch-11.1.3.jar
      - name: kafka-connect-s3
        artifacts:
          - type: jar
            url: https://packages.confluent.io/maven/io/confluent/kafka-connect-s3/10.0.3/kafka-connect-s3-10.0.3.jar

Secret:

apiVersion: v1
data:
  .dockerconfigjson: eyJhdXR...
kind: Secret
metadata:
  creationTimestamp: "2021-11-09T18:26:54Z"
  name: regcred
  namespace: adega
  resourceVersion: "3665122"
  uid: e6d12f87-8e6c-4a40-9b55-825d4e4e6cc1
type: kubernetes.io/dockerconfigjson

The K8s (EKS) worker nodes have a NodeInstanceRole with an IAM policy that allows full access to ECR. (AmazonEC2ContainerRegistryFullAccess)

Secret was created via

kubectl create secret -n adega \
    docker-registry regcred \
    --docker-server=https://REDACTED.dkr.ecr.REGION-REDACTED.amazonaws.com \
    --docker-username=AWS \
    --docker-password=$(aws ecr get-login-password --region REGION-REDACTED)

I can also verify this login token actually works via

aws ecr get-login-password --region REGION-REDACTED | docker login --username AWS --password-stdin 5REDACTED.dkr.ecr.REGION-REDACTED.amazonaws.com

which will show a Login Succeeded.

What might I be missing here?

[brsolomon-deloitte]

Update - I'm an idiot, and forgot a single character in the image value as a result of a bad copy-paste job.

After fixing, push is successful.