Standalone Topic Operator - failed to connect to secured cluster #5712

itaydj · 2021-10-13T06:27:45Z

itaydj
Oct 13, 2021

Hi,
We have a secured cluster (TLS enabled) running outside of Kubernetes.
I would like to provide a tool for our devs to manage topics with and chose the standalone topic operator.
I configured it as seen below but sadly it tries to start but keeps failing showing it is disconnecting from the brokers.

I couldnt figure out why.

My setup:
kafka-keystore.jks - contains a certificate + key combined to a p12 file
kafka-truststore.jks - contains the ca.crt for the domain

both are mounted from a kubernetes secret

the standalone-topic-operator.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: strimzi-topic-operator
#  labels:
#    app: strimzi
spec:
  replicas: 1
  selector:
    matchLabels:
      name: strimzi-topic-operator
  template:
    metadata:
      labels:
        name: strimzi-topic-operator
    spec:
      serviceAccountName: strimzi-topic-operator
      containers:
        - name: strimzi-topic-operator
          image: quay.io/strimzi/operator:0.25.0
          args:
          - /opt/strimzi/bin/topic_operator_run.sh
          env:
            - name: STRIMZI_RESOURCE_LABELS
              value: "strimzi.io/cluster=test"
            - name: STRIMZI_KAFKA_BOOTSTRAP_SERVERS
              value: dig-poc-kafka01.eqx.solaredge.com:9093,esh-poc-kafka01.eqx.solaredge.com:9093,nt-poc-kafka01.eqx.solaredge.com:9093
            - name: STRIMZI_ZOOKEEPER_CONNECT
              value: dig-prod-zk01.eqx.solaredge.com:2181,nt-prod-zk01.eqx.solaredge.com:2181,esh-prod-zk01.eqx.solaredge.com:2181/poc-kafka
            - name: STRIMZI_ZOOKEEPER_SESSION_TIMEOUT_MS
              value: "18000"
            - name: STRIMZI_FULL_RECONCILIATION_INTERVAL_MS
              value: "120000"
            - name: STRIMZI_TOPIC_METADATA_MAX_ATTEMPTS
              value: "6"
            - name: STRIMZI_TLS_ENABLED
              value: "false"
            - name: STRIMZI_NAMESPACE
              valueFrom:
                fieldRef:
                  fieldPath: metadata.namespace
            - name: STRIMZI_TRUSTSTORE_LOCATION
              value: "/keys/kafka-truststore.jks"
            - name: STRIMZI_TRUSTSTORE_PASSWORD
              value: "123456"
            - name: STRIMZI_KEYSTORE_LOCATION
              value: "/keys/kafka-keystore.jks"
            - name: STRIMZI_KEYSTORE_PASSWORD
              value: "123456"
            - name: STRIMZI_LOG_LEVEL
              value: DEBUG
            - name: STRIMZI_JAVA_OPTS
              value: "-Xmx2048M -Xms1024M"
          livenessProbe:
            httpGet:
              path: /healthy
              port: 8080
            initialDelaySeconds: 10
            periodSeconds: 30
          readinessProbe:
            httpGet:
              path: /ready
              port: 8080
            initialDelaySeconds: 10
            periodSeconds: 30
          resources:
            limits:
#              memory: 96Mi
#              cpu: 100m
              memory: 6048Mi
              cpu: 1000m
            requests:
              memory: 2048Mi
              cpu: 100m
          volumeMounts:
          - name: kafka-keystore
            mountPath: "/keys/kafka-keystore.jks"
          - name: kafka-truststore
            mountPath:  "/keys/kafka-truststore.jks"
      volumes:
        - name: kafka-keystore
          secret:
            secretName: kafka-keystore
        - name: kafka-truststore
          secret:
            secretName: kafka-truststore
  strategy:
    type: Recreate

The log file:

 kubectl logs -f strimzi-topic-operator-79ff9f9c95-7sg4c 
+ export MALLOC_ARENA_MAX=2
+ MALLOC_ARENA_MAX=2
+ JAVA_OPTS=' -Xmx2048M -Xms1024M -Dvertx.cacheDirBase=/tmp/vertx-cache -Djava.security.egd=file:/dev/./urandom'
++ get_gc_opts
++ '[' '' == true ']'
++ echo ''
+ JAVA_OPTS=' -Xmx2048M -Xms1024M -Dvertx.cacheDirBase=/tmp/vertx-cache -Djava.security.egd=file:/dev/./urandom '
+ JAVA_OPTS=' -Xmx2048M -Xms1024M -Dvertx.cacheDirBase=/tmp/vertx-cache -Djava.security.egd=file:/dev/./urandom  --illegal-access=deny'
+ exec /usr/bin/tini -w -e 143 -- java -Xmx2048M -Xms1024M -Dvertx.cacheDirBase=/tmp/vertx-cache -Djava.security.egd=file:/dev/./urandom --illegal-access=deny -classpath lib/io.strimzi.topic-operator-0.25.0.jar:lib/org.apache.kafka.kafka-clients-2.8.0.jar:lib/io.netty.netty-handler-4.1.66.Final.jar:lib/io.prometheus.simpleclient_common-0.7.0.jar:lib/io.fabric8.openshift-model-whereabouts-5.6.0.jar:lib/io.apicurio.apicurio-registry-utils-streams-1.3.2.Final.jar:lib/io.fabric8.openshift-model-operator-5.6.0.jar:lib/io.fabric8.kubernetes-model-rbac-5.6.0.jar:lib/com.101tec.zkclient-0.11.jar:lib/org.apache.kafka.connect-json-2.8.0.jar:lib/io.strimzi.certificate-manager-0.25.0.jar:lib/com.squareup.okio.okio-1.15.0.jar:lib/io.apicurio.apicurio-registry-utils-kafka-1.3.2.Final.jar:lib/io.fabric8.openshift-model-clusterautoscaling-5.6.0.jar:lib/io.fabric8.openshift-model-tuned-5.6.0.jar:lib/io.fabric8.kubernetes-client-5.6.0.jar:lib/io.fabric8.kubernetes-model-coordination-5.6.0.jar:lib/jakarta.annotation.jakarta.annotation-api-1.3.5.jar:lib/org.hdrhistogram.HdrHistogram-2.1.11.jar:lib/io.fabric8.kubernetes-model-admissionregistration-5.6.0.jar:lib/io.netty.netty-codec-4.1.66.Final.jar:lib/com.fasterxml.jackson.core.jackson-databind-2.11.3.jar:lib/org.apache.logging.log4j.log4j-core-2.14.1.jar:lib/io.prometheus.simpleclient-0.7.0.jar:lib/io.fabric8.openshift-model-monitoring-5.6.0.jar:lib/com.squareup.okhttp3.logging-interceptor-3.12.12.jar:lib/io.fabric8.openshift-model-machine-5.6.0.jar:lib/org.apache.yetus.audience-annotations-0.5.0.jar:lib/io.strimzi.api-0.25.0.jar:lib/io.netty.netty-resolver-dns-4.1.66.Final.jar:lib/com.fasterxml.jackson.core.jackson-annotations-2.11.3.jar:lib/io.vertx.vertx-micrometer-metrics-4.1.2.jar:lib/com.github.luben.zstd-jni-1.4.9-1.jar:lib/io.micrometer.micrometer-core-1.3.1.jar:lib/io.netty.netty-codec-http2-4.1.66.Final.jar:lib/org.apache.zookeeper.zookeeper-jute-3.5.9.jar:lib/io.netty.netty-buffer-4.1.66.Final.jar:lib/com.fasterxml.jackson.dataformat.jackson-dataformat-yaml-2.10.5.jar:lib/io.netty.netty-codec-http-4.1.66.Final.jar:lib/io.fabric8.kubernetes-model-networking-5.6.0.jar:lib/com.github.mifmif.generex-1.0.2.jar:lib/io.fabric8.openshift-client-5.6.0.jar:lib/org.apache.zookeeper.zookeeper-3.5.9.jar:lib/io.fabric8.kubernetes-model-autoscaling-5.6.0.jar:lib/io.vertx.vertx-core-4.1.2.jar:lib/org.xerial.snappy.snappy-java-1.1.8.1.jar:lib/org.apache.kafka.connect-api-2.8.0.jar:lib/io.fabric8.kubernetes-model-storageclass-5.6.0.jar:lib/com.fasterxml.jackson.core.jackson-core-2.11.3.jar:lib/io.fabric8.openshift-model-machineconfig-5.6.0.jar:lib/io.fabric8.openshift-model-storageversionmigrator-5.6.0.jar:lib/com.fasterxml.jackson.datatype.jackson-datatype-jdk8-2.10.5.jar:lib/org.yaml.snakeyaml-1.26.jar:lib/io.apicurio.apicurio-registry-common-1.3.2.Final.jar:lib/io.fabric8.kubernetes-model-policy-5.6.0.jar:lib/io.fabric8.kubernetes-model-batch-5.6.0.jar:lib/io.netty.netty-transport-native-epoll-4.1.66.Final.jar:lib/org.lz4.lz4-java-1.7.1.jar:lib/io.netty.netty-transport-native-unix-common-4.1.66.Final.jar:lib/io.strimzi.operator-common-0.25.0.jar:lib/io.netty.netty-transport-4.1.66.Final.jar:lib/dk.brics.automaton.automaton-1.11-8.jar:lib/io.fabric8.kubernetes-model-certificates-5.6.0.jar:lib/org.eclipse.microprofile.config.microprofile-config-api-1.4.jar:lib/io.fabric8.openshift-model-operatorhub-5.6.0.jar:lib/io.fabric8.zjsonpatch-0.3.0.jar:lib/org.jboss.spec.javax.ws.rs.jboss-jaxrs-api_2.1_spec-2.0.1.Final.jar:lib/io.fabric8.kubernetes-model-discovery-5.6.0.jar:lib/io.netty.netty-resolver-4.1.66.Final.jar:lib/io.fabric8.openshift-model-console-5.6.0.jar:lib/com.squareup.okhttp3.okhttp-3.12.6.jar:lib/io.fabric8.kubernetes-model-extensions-5.6.0.jar:lib/io.netty.netty-common-4.1.66.Final.jar:lib/org.apache.logging.log4j.log4j-slf4j-impl-2.14.1.jar:lib/io.fabric8.kubernetes-model-metrics-5.6.0.jar:lib/io.fabric8.kubernetes-model-scheduling-5.6.0.jar:lib/io.fabric8.kubernetes-model-common-5.6.0.jar:lib/org.apache.logging.log4j.log4j-api-2.14.1.jar:lib/io.fabric8.openshift-model-miscellaneous-5.6.0.jar:lib/io.fabric8.kubernetes-model-apps-5.6.0.jar:lib/org.slf4j.slf4j-api-1.7.25.jar:lib/org.latencyutils.LatencyUtils-2.0.3.jar:lib/io.netty.netty-codec-dns-4.1.66.Final.jar:lib/io.strimzi.crd-annotations-0.25.0.jar:lib/org.apache.kafka.kafka-streams-2.8.0.jar:lib/io.fabric8.openshift-model-5.6.0.jar:lib/org.rocksdb.rocksdbjni-5.18.4.jar:lib/io.fabric8.kubernetes-model-flowcontrol-5.6.0.jar:lib/io.netty.netty-codec-socks-4.1.66.Final.jar:lib/io.fabric8.kubernetes-model-core-5.6.0.jar:lib/com.fasterxml.jackson.datatype.jackson-datatype-jsr310-2.11.2.jar:lib/io.fabric8.kubernetes-model-apiextensions-5.6.0.jar:lib/io.fabric8.kubernetes-model-node-5.6.0.jar:lib/io.netty.netty-handler-proxy-4.1.66.Final.jar:lib/io.micrometer.micrometer-registry-prometheus-1.3.1.jar:lib/io.fabric8.kubernetes-model-events-5.6.0.jar io.strimzi.operator.topic.Main
2021-10-13 06:15:39 INFO  Main:29 - TopicOperator 0.25.0 is starting
2021-10-13 06:15:39 DEBUG Config:537 - Trying to configure client from Kubernetes config...
2021-10-13 06:15:39 DEBUG Config:543 - Did not find Kubernetes config at: [/home/strimzi/.kube/config]. Ignoring.
2021-10-13 06:15:39 DEBUG Config:469 - Trying to configure client from service account...
2021-10-13 06:15:39 DEBUG Config:474 - Found service account host and port: 10.96.0.1:443
2021-10-13 06:15:39 DEBUG Config:480 - Found service account ca cert at: [/var/run/secrets/kubernetes.io/serviceaccount/ca.crt].
2021-10-13 06:15:39 DEBUG Config:487 - Found service account token at: [/var/run/secrets/kubernetes.io/serviceaccount/token].
2021-10-13 06:15:39 DEBUG Config:742 - Trying to configure client namespace from Kubernetes service account namespace path...
2021-10-13 06:15:39 DEBUG Config:747 - Found service account namespace at: [/var/run/secrets/kubernetes.io/serviceaccount/namespace].
2021-10-13 06:15:40 DEBUG LoggerFactory: - Using io.vertx.core.logging.SLF4JLogDelegateFactory
2021-10-13 06:15:40 DEBUG InternalLoggerFactory:63 - Using SLF4J as the default logging framework
2021-10-13 06:15:40 DEBUG PlatformDependent0:460 - -Dio.netty.noUnsafe: false
2021-10-13 06:15:40 DEBUG PlatformDependent0:954 - Java version: 11
2021-10-13 06:15:40 DEBUG PlatformDependent0:135 - sun.misc.Unsafe.theUnsafe: available
2021-10-13 06:15:40 DEBUG PlatformDependent0:159 - sun.misc.Unsafe.copyMemory: available
2021-10-13 06:15:40 DEBUG PlatformDependent0:202 - java.nio.Buffer.address: available
2021-10-13 06:15:41 DEBUG PlatformDependent0:285 - direct buffer constructor: unavailable: Reflective setAccessible(true) disabled
2021-10-13 06:15:41 DEBUG PlatformDependent0:350 - java.nio.Bits.unaligned: available, true
2021-10-13 06:15:41 DEBUG PlatformDependent0:417 - jdk.internal.misc.Unsafe.allocateUninitializedArray(int): unavailable: class io.netty.util.internal.PlatformDependent0$6 cannot access class jdk.internal.misc.Unsafe (in module java.base) because module java.base does not export jdk.internal.misc to unnamed module @2250b9f2
2021-10-13 06:15:41 DEBUG PlatformDependent0:446 - java.nio.DirectByteBuffer.<init>(long, int): unavailable
2021-10-13 06:15:41 DEBUG PlatformDependent:1103 - sun.misc.Unsafe: available
2021-10-13 06:15:41 DEBUG PlatformDependent:1205 - maxDirectMemory: 2075918336 bytes (maybe)
2021-10-13 06:15:41 DEBUG PlatformDependent:1224 - -Dio.netty.tmpdir: /tmp (java.io.tmpdir)
2021-10-13 06:15:41 DEBUG PlatformDependent:1303 - -Dio.netty.bitMode: 64 (sun.arch.data.model)
2021-10-13 06:15:41 DEBUG PlatformDependent:178 - -Dio.netty.maxDirectMemory: -1 bytes
2021-10-13 06:15:41 DEBUG PlatformDependent:185 - -Dio.netty.uninitializedArrayAllocationThreshold: -1
2021-10-13 06:15:41 DEBUG CleanerJava9:71 - java.nio.ByteBuffer.cleaner(): available
2021-10-13 06:15:41 DEBUG PlatformDependent:205 - -Dio.netty.noPreferDirect: false
2021-10-13 06:15:41 DEBUG InternalLoggerFactory:61 - Using SLF4J as the default logging framework
2021-10-13 06:15:41 DEBUG ResourceLeakDetector:129 - -Dio.netty.leakDetection.level: simple
2021-10-13 06:15:41 DEBUG ResourceLeakDetector:130 - -Dio.netty.leakDetection.targetRecords: 4
2021-10-13 06:15:41 DEBUG MultithreadEventLoopGroup:44 - -Dio.netty.eventLoopThreads: 2
2021-10-13 06:15:41 DEBUG InternalThreadLocalMap:83 - -Dio.netty.threadLocalMap.stringBuilder.initialSize: 1024
2021-10-13 06:15:41 DEBUG InternalThreadLocalMap:86 - -Dio.netty.threadLocalMap.stringBuilder.maxSize: 4096
2021-10-13 06:15:41 DEBUG NioEventLoop:106 - -Dio.netty.noKeySetOptimization: false
2021-10-13 06:15:41 DEBUG NioEventLoop:107 - -Dio.netty.selectorAutoRebuildThreshold: 512
2021-10-13 06:15:41 DEBUG PlatformDependent:967 - org.jctools-core.MpscChunkedArrayQueue: available
2021-10-13 06:15:41 DEBUG DefaultDnsServerAddressStreamProvider:80 - Default DNS servers: [/10.96.0.10:53] (sun.net.dns.ResolverConfiguration)
2021-10-13 06:15:41 DEBUG NetUtil:135 - -Djava.net.preferIPv4Stack: false
2021-10-13 06:15:41 DEBUG NetUtil:136 - -Djava.net.preferIPv6Addresses: false
2021-10-13 06:15:41 DEBUG NetUtilInitializations:129 - Loopback interface: lo (lo, 0:0:0:0:0:0:0:1%lo)
2021-10-13 06:15:41 DEBUG NetUtil:169 - /proc/sys/net/core/somaxconn: 4096
2021-10-13 06:15:41 INFO  Session:77 - Using config:
	STRIMZI_TRUSTSTORE_LOCATION: /keys/kafka-truststore.jks
	STRIMZI_RESOURCE_LABELS: strimzi.io/cluster=test
	STRIMZI_SASL_MECHANISM: 
	STRIMZI_SASL_USERNAME: 
	STRIMZI_FULL_RECONCILIATION_INTERVAL_MS: 120000
	STRIMZI_CLIENT_ID: strimzi-topic-operator-60617a5d-b08a-46bd-a3e9-c11e83b3e762
	STRIMZI_SECURITY_PROTOCOL: 
	STRIMZI_STALE_RESULT_TIMEOUT_MS: 5000
	STRIMZI_TOPIC_METADATA_MAX_ATTEMPTS: 6
	STRIMZI_KEYSTORE_LOCATION: /keys/kafka-keystore.jks
	STRIMZI_REASSIGN_THROTTLE: 9223372036854775807
	STRIMZI_USE_ZOOKEEPER_TOPIC_STORE: false
	STRIMZI_SASL_PASSWORD: ********
	STRIMZI_KAFKA_BOOTSTRAP_SERVERS: dc1-kafka01.mydomain.com:9093,dc2-kafka01.mydomain.com:9093,dc3-kafka01.mydomain.com:9093
	STRIMZI_SASL_ENABLED: false
	STRIMZI_SSL_ENDPOINT_IDENTIFICATION_ALGORITHM: HTTPS
	STRIMZI_NAMESPACE: default
	STRIMZI_APPLICATION_ID: __strimzi-topic-operator-kstreams
	STRIMZI_APPLICATION_SERVER: localhost:9000
	STRIMZI_ZOOKEEPER_SESSION_TIMEOUT_MS: 18000
	STRIMZI_TOPICS_PATH: /strimzi/topics
	STRIMZI_ZOOKEEPER_CONNECT: dc1-prod-zk01.mydomain.com:2181,dc3-prod-zk01.mydomain.com:2181,dc2-prod-zk01.mydomain.com:2181/poc-kafka
	STRIMZI_TLS_ENABLED: false
	STRIMZI_KEYSTORE_PASSWORD: ********
	STRIMZI_STORE_NAME: topic-store
	STRIMZI_REASSIGN_VERIFY_INTERVAL_MS: 120000
	TC_ZK_CONNECTION_TIMEOUT_MS: 18000
	STRIMZI_TRUSTSTORE_PASSWORD: ********
	STRIMZI_STORE_TOPIC: __strimzi_store_topic

2021-10-13 06:15:41 INFO  Session:155 - Starting
2021-10-13 06:15:41 INFO  AdminClientConfig:372 - AdminClientConfig values: 
	bootstrap.servers = [dc1-kafka01.mydomain.com:9093, dc2-kafka01.mydomain.com:9093, dc3-kafka01.mydomain.com:9093]
	client.dns.lookup = use_all_dns_ips
	client.id = 
	connections.max.idle.ms = 300000
	default.api.timeout.ms = 60000
	metadata.max.age.ms = 300000
	metric.reporters = []
	metrics.num.samples = 2
	metrics.recording.level = INFO
	metrics.sample.window.ms = 30000
	receive.buffer.bytes = 65536
	reconnect.backoff.max.ms = 1000
	reconnect.backoff.ms = 50
	request.timeout.ms = 30000
	retries = 2147483647
	retry.backoff.ms = 100
	sasl.client.callback.handler.class = null
	sasl.jaas.config = null
	sasl.kerberos.kinit.cmd = /usr/bin/kinit
	sasl.kerberos.min.time.before.relogin = 60000
	sasl.kerberos.service.name = null
	sasl.kerberos.ticket.renew.jitter = 0.05
	sasl.kerberos.ticket.renew.window.factor = 0.8
	sasl.login.callback.handler.class = null
	sasl.login.class = null
	sasl.login.refresh.buffer.seconds = 300
	sasl.login.refresh.min.period.seconds = 60
	sasl.login.refresh.window.factor = 0.8
	sasl.login.refresh.window.jitter = 0.05
	sasl.mechanism = GSSAPI
	security.protocol = PLAINTEXT
	security.providers = null
	send.buffer.bytes = 131072
	socket.connection.setup.timeout.max.ms = 30000
	socket.connection.setup.timeout.ms = 10000
	ssl.cipher.suites = null
	ssl.enabled.protocols = [TLSv1.2, TLSv1.3]
	ssl.endpoint.identification.algorithm = https
	ssl.engine.factory.class = null
	ssl.key.password = null
	ssl.keymanager.algorithm = SunX509
	ssl.keystore.certificate.chain = null
	ssl.keystore.key = null
	ssl.keystore.location = null
	ssl.keystore.password = null
	ssl.keystore.type = JKS
	ssl.protocol = TLSv1.3
	ssl.provider = null
	ssl.secure.random.implementation = null
	ssl.trustmanager.algorithm = PKIX
	ssl.truststore.certificates = null
	ssl.truststore.location = null
	ssl.truststore.password = null
	ssl.truststore.type = JKS

2021-10-13 06:15:41 DEBUG AdminMetadataManager:245 - [AdminClient clientId=adminclient-1] Setting bootstrap cluster metadata Cluster(id = null, nodes = [dc1-kafka01.mydomain.com:9093 (id: -1 rack: null), dc2-kafka01.mydomain.com:9093 (id: -2 rack: null), dc3-kafka01.mydomain.com:9093 (id: -3 rack: null)], partitions = [], controller = null).
2021-10-13 06:15:42 WARN  AdminClientConfig:380 - The configuration 'application.id' was supplied but isn't a known config.
2021-10-13 06:15:42 INFO  AppInfoParser:119 - Kafka version: 2.8.0
2021-10-13 06:15:42 INFO  AppInfoParser:120 - Kafka commitId: ebb1d6e21cc92130
2021-10-13 06:15:42 INFO  AppInfoParser:121 - Kafka startTimeMs: 1634105742013
2021-10-13 06:15:42 DEBUG KafkaAdminClient:591 - [AdminClient clientId=adminclient-1] Kafka admin client initialized
2021-10-13 06:15:42 DEBUG Session:161 - Using AdminClient org.apache.kafka.clients.admin.KafkaAdminClient@4f74bf6c
2021-10-13 06:15:42 DEBUG Session:163 - Using Kafka io.strimzi.operator.topic.KafkaImpl@2a25999f
2021-10-13 06:15:42 DEBUG Session:167 - Using namespace default
2021-10-13 06:15:42 DEBUG NetworkClient:986 - [AdminClient clientId=adminclient-1] Initiating connection to node dc3-kafka01.mydomain.com:9093 (id: -3 rack: null) using address dc3-kafka01.mydomain.com/10.15.14.70
2021-10-13 06:15:42 DEBUG Session:169 - Using k8s io.strimzi.operator.topic.K8sImpl@36ae343a
2021-10-13 06:15:42 DEBUG Session:172 - Using client-Id strimzi-topic-operator-60617a5d-b08a-46bd-a3e9-c11e83b3e762
2021-10-13 06:15:42 DEBUG Selector:531 - [AdminClient clientId=adminclient-1] Created socket with SO_RCVBUF = 65536, SO_SNDBUF = 131072, SO_TIMEOUT = 0 to node -3
2021-10-13 06:15:42 DEBUG ZkConnection:69 - Creating new ZookKeeper instance to connect to dc1-prod-zk01.mydomain.com:2181,dc3-prod-zk01.mydomain.com:2181,dc2-prod-zk01.mydomain.com:2181/poc-kafka.
2021-10-13 06:15:42 INFO  ZkEventThread:65 - Starting ZkClient event thread.
2021-10-13 06:15:42 INFO  ZooKeeper:109 - Client environment:zookeeper.version=3.5.9-83df9301aa5c2a5d284a9940177808c01bc35cef, built on 01/06/2021 20:03 GMT
2021-10-13 06:15:42 INFO  ZooKeeper:109 - Client environment:host.name=strimzi-topic-operator-79ff9f9c95-7sg4c
2021-10-13 06:15:42 INFO  ZooKeeper:109 - Client environment:java.version=11.0.12
2021-10-13 06:15:42 INFO  ZooKeeper:109 - Client environment:java.vendor=Red Hat, Inc.
2021-10-13 06:15:42 INFO  ZooKeeper:109 - Client environment:java.home=/usr/lib/jvm/java-11-openjdk-11.0.12.0.7-0.el8_4.x86_64
2021-10-13 06:15:42 INFO  ZooKeeper:109 - Client environment:java.class.path=lib/io.strimzi.topic-operator-0.25.0.jar:lib/org.apache.kafka.kafka-clients-2.8.0.jar:lib/io.netty.netty-handler-4.1.66.Final.jar:lib/io.prometheus.simpleclient_common-0.7.0.jar:lib/io.fabric8.openshift-model-whereabouts-5.6.0.jar:lib/io.apicurio.apicurio-registry-utils-streams-1.3.2.Final.jar:lib/io.fabric8.openshift-model-operator-5.6.0.jar:lib/io.fabric8.kubernetes-model-rbac-5.6.0.jar:lib/com.101tec.zkclient-0.11.jar:lib/org.apache.kafka.connect-json-2.8.0.jar:lib/io.strimzi.certificate-manager-0.25.0.jar:lib/com.squareup.okio.okio-1.15.0.jar:lib/io.apicurio.apicurio-registry-utils-kafka-1.3.2.Final.jar:lib/io.fabric8.openshift-model-clusterautoscaling-5.6.0.jar:lib/io.fabric8.openshift-model-tuned-5.6.0.jar:lib/io.fabric8.kubernetes-client-5.6.0.jar:lib/io.fabric8.kubernetes-model-coordination-5.6.0.jar:lib/jakarta.annotation.jakarta.annotation-api-1.3.5.jar:lib/org.hdrhistogram.HdrHistogram-2.1.11.jar:lib/io.fabric8.kubernetes-model-admissionregistration-5.6.0.jar:lib/io.netty.netty-codec-4.1.66.Final.jar:lib/com.fasterxml.jackson.core.jackson-databind-2.11.3.jar:lib/org.apache.logging.log4j.log4j-core-2.14.1.jar:lib/io.prometheus.simpleclient-0.7.0.jar:lib/io.fabric8.openshift-model-monitoring-5.6.0.jar:lib/com.squareup.okhttp3.logging-interceptor-3.12.12.jar:lib/io.fabric8.openshift-model-machine-5.6.0.jar:lib/org.apache.yetus.audience-annotations-0.5.0.jar:lib/io.strimzi.api-0.25.0.jar:lib/io.netty.netty-resolver-dns-4.1.66.Final.jar:lib/com.fasterxml.jackson.core.jackson-annotations-2.11.3.jar:lib/io.vertx.vertx-micrometer-metrics-4.1.2.jar:lib/com.github.luben.zstd-jni-1.4.9-1.jar:lib/io.micrometer.micrometer-core-1.3.1.jar:lib/io.netty.netty-codec-http2-4.1.66.Final.jar:lib/org.apache.zookeeper.zookeeper-jute-3.5.9.jar:lib/io.netty.netty-buffer-4.1.66.Final.jar:lib/com.fasterxml.jackson.dataformat.jackson-dataformat-yaml-2.10.5.jar:lib/io.netty.netty-codec-http-4.1.66.Final.jar:lib/io.fabric8.kubernetes-model-networking-5.6.0.jar:lib/com.github.mifmif.generex-1.0.2.jar:lib/io.fabric8.openshift-client-5.6.0.jar:lib/org.apache.zookeeper.zookeeper-3.5.9.jar:lib/io.fabric8.kubernetes-model-autoscaling-5.6.0.jar:lib/io.vertx.vertx-core-4.1.2.jar:lib/org.xerial.snappy.snappy-java-1.1.8.1.jar:lib/org.apache.kafka.connect-api-2.8.0.jar:lib/io.fabric8.kubernetes-model-storageclass-5.6.0.jar:lib/com.fasterxml.jackson.core.jackson-core-2.11.3.jar:lib/io.fabric8.openshift-model-machineconfig-5.6.0.jar:lib/io.fabric8.openshift-model-storageversionmigrator-5.6.0.jar:lib/com.fasterxml.jackson.datatype.jackson-datatype-jdk8-2.10.5.jar:lib/org.yaml.snakeyaml-1.26.jar:lib/io.apicurio.apicurio-registry-common-1.3.2.Final.jar:lib/io.fabric8.kubernetes-model-policy-5.6.0.jar:lib/io.fabric8.kubernetes-model-batch-5.6.0.jar:lib/io.netty.netty-transport-native-epoll-4.1.66.Final.jar:lib/org.lz4.lz4-java-1.7.1.jar:lib/io.netty.netty-transport-native-unix-common-4.1.66.Final.jar:lib/io.strimzi.operator-common-0.25.0.jar:lib/io.netty.netty-transport-4.1.66.Final.jar:lib/dk.brics.automaton.automaton-1.11-8.jar:lib/io.fabric8.kubernetes-model-certificates-5.6.0.jar:lib/org.eclipse.microprofile.config.microprofile-config-api-1.4.jar:lib/io.fabric8.openshift-model-operatorhub-5.6.0.jar:lib/io.fabric8.zjsonpatch-0.3.0.jar:lib/org.jboss.spec.javax.ws.rs.jboss-jaxrs-api_2.1_spec-2.0.1.Final.jar:lib/io.fabric8.kubernetes-model-discovery-5.6.0.jar:lib/io.netty.netty-resolver-4.1.66.Final.jar:lib/io.fabric8.openshift-model-console-5.6.0.jar:lib/com.squareup.okhttp3.okhttp-3.12.6.jar:lib/io.fabric8.kubernetes-model-extensions-5.6.0.jar:lib/io.netty.netty-common-4.1.66.Final.jar:lib/org.apache.logging.log4j.log4j-slf4j-impl-2.14.1.jar:lib/io.fabric8.kubernetes-model-metrics-5.6.0.jar:lib/io.fabric8.kubernetes-model-scheduling-5.6.0.jar:lib/io.fabric8.kubernetes-model-common-5.6.0.jar:lib/org.apache.logging.log4j.log4j-api-2.14.1.jar:lib/io.fabric8.openshift-model-miscellaneous-5.6.0.jar:lib/io.fabric8.kubernetes-model-apps-5.6.0.jar:lib/org.slf4j.slf4j-api-1.7.25.jar:lib/org.latencyutils.LatencyUtils-2.0.3.jar:lib/io.netty.netty-codec-dns-4.1.66.Final.jar:lib/io.strimzi.crd-annotations-0.25.0.jar:lib/org.apache.kafka.kafka-streams-2.8.0.jar:lib/io.fabric8.openshift-model-5.6.0.jar:lib/org.rocksdb.rocksdbjni-5.18.4.jar:lib/io.fabric8.kubernetes-model-flowcontrol-5.6.0.jar:lib/io.netty.netty-codec-socks-4.1.66.Final.jar:lib/io.fabric8.kubernetes-model-core-5.6.0.jar:lib/com.fasterxml.jackson.datatype.jackson-datatype-jsr310-2.11.2.jar:lib/io.fabric8.kubernetes-model-apiextensions-5.6.0.jar:lib/io.fabric8.kubernetes-model-node-5.6.0.jar:lib/io.netty.netty-handler-proxy-4.1.66.Final.jar:lib/io.micrometer.micrometer-registry-prometheus-1.3.1.jar:lib/io.fabric8.kubernetes-model-events-5.6.0.jar
2021-10-13 06:15:42 INFO  ZooKeeper:109 - Client environment:java.library.path=/usr/java/packages/lib:/usr/lib64:/lib64:/lib:/usr/lib
2021-10-13 06:15:42 INFO  ZooKeeper:109 - Client environment:java.io.tmpdir=/tmp
2021-10-13 06:15:42 INFO  ZooKeeper:109 - Client environment:java.compiler=<NA>
2021-10-13 06:15:42 INFO  ZooKeeper:109 - Client environment:os.name=Linux
2021-10-13 06:15:42 INFO  ZooKeeper:109 - Client environment:os.arch=amd64
2021-10-13 06:15:42 INFO  ZooKeeper:109 - Client environment:os.version=5.11.0-37-generic
2021-10-13 06:15:42 INFO  ZooKeeper:109 - Client environment:user.name=strimzi
2021-10-13 06:15:42 INFO  ZooKeeper:109 - Client environment:user.home=/home/strimzi
2021-10-13 06:15:42 INFO  ZooKeeper:109 - Client environment:user.dir=/opt/strimzi
2021-10-13 06:15:42 INFO  ZooKeeper:109 - Client environment:os.memory.free=864MB
2021-10-13 06:15:42 INFO  ZooKeeper:109 - Client environment:os.memory.max=1979MB
2021-10-13 06:15:42 INFO  ZooKeeper:109 - Client environment:os.memory.total=989MB
2021-10-13 06:15:42 INFO  ZooKeeper:868 - Initiating client connection, connectString=dc1-prod-zk01.mydomain.com:2181,dc3-prod-zk01.mydomain.com:2181,dc2-prod-zk01.mydomain.com:2181/poc-kafka sessionTimeout=18000 watcher=org.I0Itec.zkclient.ZkClient@212abcb6
2021-10-13 06:15:42 INFO  X509Util:79 - Setting -D jdk.tls.rejectClientInitiatedRenegotiation=true to disable client-initiated TLS renegotiation
2021-10-13 06:15:42 INFO  ClientCnxnSocket:237 - jute.maxbuffer value is 4194304 Bytes
2021-10-13 06:15:42 INFO  ClientCnxn:1653 - zookeeper.request.timeout value is 0. feature enabled=
2021-10-13 06:15:42 DEBUG ZkClient:1230 - Awaiting connection to Zookeeper server
2021-10-13 06:15:42 INFO  ZkClient:936 - Waiting for keeper state SyncConnected
2021-10-13 06:15:42 DEBUG SaslServerPrincipal:80 - Canonicalized address to dc3-prod-zk01.mydomain.com
2021-10-13 06:15:42 INFO  ClientCnxn:1112 - Opening socket connection to server dc3-prod-zk01.mydomain.com/10.15.14.90:2181. Will not attempt to authenticate using SASL (unknown error)
2021-10-13 06:15:42 INFO  ClientCnxn:959 - Socket connection established, initiating session, client: /10.244.0.34:50264, server: dc3-prod-zk01.mydomain.com/10.15.14.90:2181
2021-10-13 06:15:42 DEBUG ClientCnxn:1027 - Session establishment request sent on dc3-prod-zk01.mydomain.com/10.15.14.90:2181
2021-10-13 06:15:42 INFO  ClientCnxn:1394 - Session establishment complete on server dc3-prod-zk01.mydomain.com/10.15.14.90:2181, sessionid = 0x20002d6c468008a, negotiated timeout = 18000
2021-10-13 06:15:42 DEBUG ZkClient:615 - Received event: WatchedEvent state:SyncConnected type:None path:null
2021-10-13 06:15:42 INFO  ZkClient:713 - zookeeper state changed (SyncConnected)
2021-10-13 06:15:42 DEBUG ZkClient:659 - Leaving process event
2021-10-13 06:15:42 DEBUG ZkClient:950 - State is SyncConnected
2021-10-13 06:15:42 DEBUG Session:183 - Using ZooKeeper io.strimzi.operator.topic.zk.ZkImpl@78fdb660
2021-10-13 06:15:42 DEBUG ClientCnxn:923 - Reading reply sessionid:0x20002d6c468008a, packet:: clientPath:null serverPath:null finished:false header:: 1,3  replyHeader:: 1,219059315295,-101  request:: '/poc-kafka/strimzi/topics,F  response::  
2021-10-13 06:15:42 INFO  AdminClientConfig:372 - AdminClientConfig values: 
	bootstrap.servers = [dc1-kafka01.mydomain.com:9093, dc2-kafka01.mydomain.com:9093, dc3-kafka01.mydomain.com:9093]
	client.dns.lookup = use_all_dns_ips
	client.id = 
	connections.max.idle.ms = 300000
	default.api.timeout.ms = 60000
	metadata.max.age.ms = 300000
	metric.reporters = []
	metrics.num.samples = 2
	metrics.recording.level = INFO
	metrics.sample.window.ms = 30000
	receive.buffer.bytes = 65536
	reconnect.backoff.max.ms = 1000
	reconnect.backoff.ms = 50
	request.timeout.ms = 30000
	retries = 2147483647
	retry.backoff.ms = 100
	sasl.client.callback.handler.class = null
	sasl.jaas.config = null
	sasl.kerberos.kinit.cmd = /usr/bin/kinit
	sasl.kerberos.min.time.before.relogin = 60000
	sasl.kerberos.service.name = null
	sasl.kerberos.ticket.renew.jitter = 0.05
	sasl.kerberos.ticket.renew.window.factor = 0.8
	sasl.login.callback.handler.class = null
	sasl.login.class = null
	sasl.login.refresh.buffer.seconds = 300
	sasl.login.refresh.min.period.seconds = 60
	sasl.login.refresh.window.factor = 0.8
	sasl.login.refresh.window.jitter = 0.05
	sasl.mechanism = GSSAPI
	security.protocol = PLAINTEXT
	security.providers = null
	send.buffer.bytes = 131072
	socket.connection.setup.timeout.max.ms = 30000
	socket.connection.setup.timeout.ms = 10000
	ssl.cipher.suites = null
	ssl.enabled.protocols = [TLSv1.2, TLSv1.3]
	ssl.endpoint.identification.algorithm = https
	ssl.engine.factory.class = null
	ssl.key.password = null
	ssl.keymanager.algorithm = SunX509
	ssl.keystore.certificate.chain = null
	ssl.keystore.key = null
	ssl.keystore.location = null
	ssl.keystore.password = null
	ssl.keystore.type = JKS
	ssl.protocol = TLSv1.3
	ssl.provider = null
	ssl.secure.random.implementation = null
	ssl.trustmanager.algorithm = PKIX
	ssl.truststore.certificates = null
	ssl.truststore.location = null
	ssl.truststore.password = null
	ssl.truststore.type = JKS

2021-10-13 06:15:42 DEBUG AdminMetadataManager:245 - [AdminClient clientId=adminclient-2] Setting bootstrap cluster metadata Cluster(id = null, nodes = [dc3-kafka01.mydomain.com:9093 (id: -3 rack: null), dc1-kafka01.mydomain.com:9093 (id: -1 rack: null), dc2-kafka01.mydomain.com:9093 (id: -2 rack: null)], partitions = [], controller = null).
2021-10-13 06:15:42 WARN  AdminClientConfig:380 - The configuration 'application.id' was supplied but isn't a known config.
2021-10-13 06:15:42 INFO  AppInfoParser:119 - Kafka version: 2.8.0
2021-10-13 06:15:42 INFO  AppInfoParser:120 - Kafka commitId: ebb1d6e21cc92130
2021-10-13 06:15:42 INFO  AppInfoParser:121 - Kafka startTimeMs: 1634105742906
2021-10-13 06:15:42 DEBUG KafkaAdminClient:591 - [AdminClient clientId=adminclient-2] Kafka admin client initialized
2021-10-13 06:15:42 INFO  KafkaStreamsTopicStoreService:52 - Starting ...
2021-10-13 06:15:42 DEBUG NetworkClient:986 - [AdminClient clientId=adminclient-2] Initiating connection to node dc1-kafka01.mydomain.com:9093 (id: -1 rack: null) using address dc1-kafka01.mydomain.com/10.18.14.70
2021-10-13 06:15:42 DEBUG KafkaAdminClient:1362 - [AdminClient clientId=adminclient-2] Queueing Call(callName=listNodes, deadlineMs=1634105802984, tries=0, nextAllowedTryMs=0) with a timeout 60000 ms from now.
2021-10-13 06:15:43 DEBUG Selector:531 - [AdminClient clientId=adminclient-2] Created socket with SO_RCVBUF = 65536, SO_SNDBUF = 131072, SO_TIMEOUT = 0 to node -1
2021-10-13 06:15:43 DEBUG NetworkClient:953 - [AdminClient clientId=adminclient-1] Completed connection to node -3. Fetching API versions.
2021-10-13 06:15:43 DEBUG NetworkClient:953 - [AdminClient clientId=adminclient-2] Completed connection to node -1. Fetching API versions.
2021-10-13 06:15:43 DEBUG NetworkClient:967 - [AdminClient clientId=adminclient-1] Initiating API versions fetch from node -3.
2021-10-13 06:15:43 DEBUG NetworkClient:967 - [AdminClient clientId=adminclient-2] Initiating API versions fetch from node -1.
2021-10-13 06:15:43 DEBUG NetworkClient:522 - [AdminClient clientId=adminclient-2] Sending API_VERSIONS request with header RequestHeader(apiKey=API_VERSIONS, apiVersion=3, clientId=adminclient-2, correlationId=0) and timeout 3600000 to node -1: ApiVersionsRequestData(clientSoftwareName='apache-kafka-java', clientSoftwareVersion='2.8.0')
2021-10-13 06:15:43 DEBUG NetworkClient:522 - [AdminClient clientId=adminclient-1] Sending API_VERSIONS request with header RequestHeader(apiKey=API_VERSIONS, apiVersion=3, clientId=adminclient-1, correlationId=0) and timeout 3600000 to node -3: ApiVersionsRequestData(clientSoftwareName='apache-kafka-java', clientSoftwareVersion='2.8.0')
2021-10-13 06:15:44 DEBUG Selector:606 - [AdminClient clientId=adminclient-2] Connection with dc1-kafka01.mydomain.com/10.18.14.70 disconnected
java.io.EOFException: null
	at org.apache.kafka.common.network.NetworkReceive.readFrom(NetworkReceive.java:120) ~[org.apache.kafka.kafka-clients-2.8.0.jar:?]
	at org.apache.kafka.common.network.KafkaChannel.receive(KafkaChannel.java:452) ~[org.apache.kafka.kafka-clients-2.8.0.jar:?]
	at org.apache.kafka.common.network.KafkaChannel.read(KafkaChannel.java:402) ~[org.apache.kafka.kafka-clients-2.8.0.jar:?]
	at org.apache.kafka.common.network.Selector.attemptRead(Selector.java:674) ~[org.apache.kafka.kafka-clients-2.8.0.jar:?]
	at org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:576) [org.apache.kafka.kafka-clients-2.8.0.jar:?]
	at org.apache.kafka.common.network.Selector.poll(Selector.java:481) [org.apache.kafka.kafka-clients-2.8.0.jar:?]
	at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:561) [org.apache.kafka.kafka-clients-2.8.0.jar:?]
	at org.apache.kafka.clients.admin.KafkaAdminClient$AdminClientRunnable.processRequests(KafkaAdminClient.java:1333) [org.apache.kafka.kafka-clients-2.8.0.jar:?]
	at org.apache.kafka.clients.admin.KafkaAdminClient$AdminClientRunnable.run(KafkaAdminClient.java:1264) [org.apache.kafka.kafka-clients-2.8.0.jar:?]
	at java.lang.Thread.run(Thread.java:829) [?:?]
2021-10-13 06:15:44 DEBUG NetworkClient:936 - [AdminClient clientId=adminclient-2] Node -1 disconnected.
2021-10-13 06:15:44 DEBUG NetworkClient:986 - [AdminClient clientId=adminclient-2] Initiating connection to node dc2-kafka01.mydomain.com:9093 (id: -2 rack: null) using address dc2-kafka01.mydomain.com/10.16.14.70
2021-10-13 06:15:44 DEBUG Selector:531 - [AdminClient clientId=adminclient-2] Created socket with SO_RCVBUF = 65536, SO_SNDBUF = 131072, SO_TIMEOUT = 0 to node -2
2021-10-13 06:15:44 DEBUG NetworkClient:953 - [AdminClient clientId=adminclient-2] Completed connection to node -2. Fetching API versions.
2021-10-13 06:15:44 DEBUG NetworkClient:967 - [AdminClient clientId=adminclient-2] Initiating API versions fetch from node -2.
2021-10-13 06:15:44 DEBUG NetworkClient:522 - [AdminClient clientId=adminclient-2] Sending API_VERSIONS request with header RequestHeader(apiKey=API_VERSIONS, apiVersion=3, clientId=adminclient-2, correlationId=1) and timeout 3600000 to node -2: ApiVersionsRequestData(clientSoftwareName='apache-kafka-java', clientSoftwareVersion='2.8.0')
2021-10-13 06:15:44 DEBUG Selector:606 - [AdminClient clientId=adminclient-1] Connection with dc3-kafka01.mydomain.com/10.15.14.70 disconnected
java.io.EOFException: null
	at org.apache.kafka.common.network.NetworkReceive.readFrom(NetworkReceive.java:120) ~[org.apache.kafka.kafka-clients-2.8.0.jar:?]
	at org.apache.kafka.common.network.KafkaChannel.receive(KafkaChannel.java:452) ~[org.apache.kafka.kafka-clients-2.8.0.jar:?]
	at org.apache.kafka.common.network.KafkaChannel.read(KafkaChannel.java:402) ~[org.apache.kafka.kafka-clients-2.8.0.jar:?]
	at org.apache.kafka.common.network.Selector.attemptRead(Selector.java:674) ~[org.apache.kafka.kafka-clients-2.8.0.jar:?]
	at org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:576) [org.apache.kafka.kafka-clients-2.8.0.jar:?]
	at org.apache.kafka.common.network.Selector.poll(Selector.java:481) [org.apache.kafka.kafka-clients-2.8.0.jar:?]
	at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:561) [org.apache.kafka.kafka-clients-2.8.0.jar:?]
	at org.apache.kafka.clients.admin.KafkaAdminClient$AdminClientRunnable.processRequests(KafkaAdminClient.java:1333) [org.apache.kafka.kafka-clients-2.8.0.jar:?]
	at org.apache.kafka.clients.admin.KafkaAdminClient$AdminClientRunnable.run(KafkaAdminClient.java:1264) [org.apache.kafka.kafka-clients-2.8.0.jar:?]
	at java.lang.Thread.run(Thread.java:829) [?:?]
2021-10-13 06:15:44 DEBUG NetworkClient:936 - [AdminClient clientId=adminclient-1] Node -3 disconnected.
2021-10-13 06:15:44 DEBUG NetworkClient:986 - [AdminClient clientId=adminclient-1] Initiating connection to node dc1-kafka01.mydomain.com:9093 (id: -1 rack: null) using address dc1-kafka01.mydomain.com/10.18.14.70
2021-10-13 06:15:44 DEBUG Selector:531 - [AdminClient clientId=adminclient-1] Created socket with SO_RCVBUF = 65536, SO_SNDBUF = 131072, SO_TIMEOUT = 0 to node -1
2021-10-13 06:15:44 DEBUG NetworkClient:953 - [AdminClient clientId=adminclient-1] Completed connection to node -1. Fetching API versions.
2021-10-13 06:15:44 DEBUG NetworkClient:967 - [AdminClient clientId=adminclient-1] Initiating API versions fetch from node -1.
2021-10-13 06:15:44 DEBUG NetworkClient:522 - [AdminClient clientId=adminclient-1] Sending API_VERSIONS request with header RequestHeader(apiKey=API_VERSIONS, apiVersion=3, clientId=adminclient-1, correlationId=1) and timeout 3600000 to node -1: ApiVersionsRequestData(clientSoftwareName='apache-kafka-java', clientSoftwareVersion='2.8.0')
2021-10-13 06:15:44 DEBUG Selector:606 - [AdminClient clientId=adminclient-2] Connection with dc2-kafka01.mydomain.com/10.16.14.70 disconnected
java.io.EOFException: null
	at org.apache.kafka.common.network.NetworkReceive.readFrom(NetworkReceive.java:120) ~[org.apache.kafka.kafka-clients-2.8.0.jar:?]
	at org.apache.kafka.common.network.KafkaChannel.receive(KafkaChannel.java:452) ~[org.apache.kafka.kafka-clients-2.8.0.jar:?]
	at org.apache.kafka.common.network.KafkaChannel.read(KafkaChannel.java:402) ~[org.apache.kafka.kafka-clients-2.8.0.jar:?]
	at org.apache.kafka.common.network.Selector.attemptRead(Selector.java:674) ~[org.apache.kafka.kafka-clients-2.8.0.jar:?]
	at org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:576) [org.apache.kafka.kafka-clients-2.8.0.jar:?]
	at org.apache.kafka.common.network.Selector.poll(Selector.java:481) [org.apache.kafka.kafka-clients-2.8.0.jar:?]
	at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:561) [org.apache.kafka.kafka-clients-2.8.0.jar:?]
	at org.apache.kafka.clients.admin.KafkaAdminClient$AdminClientRunnable.processRequests(KafkaAdminClient.java:1333) [org.apache.kafka.kafka-clients-2.8.0.jar:?]
	at org.apache.kafka.clients.admin.KafkaAdminClient$AdminClientRunnable.run(KafkaAdminClient.java:1264) [org.apache.kafka.kafka-clients-2.8.0.jar:?]
	at java.lang.Thread.run(Thread.java:829) [?:?]
2021-10-13 06:15:44 DEBUG NetworkClient:936 - [AdminClient clientId=adminclient-2] Node -2 disconnected.
2021-10-13 06:15:44 DEBUG Selector:606 - [AdminClient clientId=adminclient-1] Connection with dc1-kafka01.mydomain.com/10.18.14.70 disconnected
java.io.EOFException: null
	at org.apache.kafka.common.network.NetworkReceive.readFrom(NetworkReceive.java:120) ~[org.apache.kafka.kafka-clients-2.8.0.jar:?]
	at org.apache.kafka.common.network.KafkaChannel.receive(KafkaChannel.java:452) ~[org.apache.kafka.kafka-clients-2.8.0.jar:?]
	at org.apache.kafka.common.network.KafkaChannel.read(KafkaChannel.java:402) ~[org.apache.kafka.kafka-clients-2.8.0.jar:?]
	at org.apache.kafka.common.network.Selector.attemptRead(Selector.java:674) ~[org.apache.kafka.kafka-clients-2.8.0.jar:?]
	at org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:576) [org.apache.kafka.kafka-clients-2.8.0.jar:?]
	at org.apache.kafka.common.network.Selector.poll(Selector.java:481) [org.apache.kafka.kafka-clients-2.8.0.jar:?]
	at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:561) [org.apache.kafka.kafka-clients-2.8.0.jar:?]
	at org.apache.kafka.clients.admin.KafkaAdminClient$AdminClientRunnable.processRequests(KafkaAdminClient.java:1333) [org.apache.kafka.kafka-clients-2.8.0.jar:?]
	at org.apache.kafka.clients.admin.KafkaAdminClient$AdminClientRunnable.run(KafkaAdminClient.java:1264) [org.apache.kafka.kafka-clients-2.8.0.jar:?]
	at java.lang.Thread.run(Thread.java:829) [?:?]
2021-10-13 06:15:44 DEBUG NetworkClient:986 - [AdminClient clientId=adminclient-2] Initiating connection to node dc3-kafka01.mydomain.com:9093 (id: -3 rack: null) using address dc3-kafka01.mydomain.com/10.15.14.70
2021-10-13 06:15:44 DEBUG NetworkClient:936 - [AdminClient clientId=adminclient-1] Node -1 disconnected.
2021-10-13 06:15:44 DEBUG NetworkClient:986 - [AdminClient clientId=adminclient-1] Initiating connection to node dc2-kafka01.mydomain.com:9093 (id: -2 rack: null) using address dc2-kafka01.mydomain.com/10.16.14.70
2021-10-13 06:15:44 DEBUG Selector:531 - [AdminClient clientId=adminclient-2] Created socket with SO_RCVBUF = 65536, SO_SNDBUF = 131072, SO_TIMEOUT = 0 to node -3
2021-10-13 06:15:44 DEBUG NetworkClient:953 - [AdminClient clientId=adminclient-2] Completed connection to node -3. Fetching API versions.
2021-10-13 06:15:44 DEBUG NetworkClient:967 - [AdminClient clientId=adminclient-2] Initiating API versions fetch from node -3.
2021-10-13 06:15:44 DEBUG NetworkClient:522 - [AdminClient clientId=adminclient-2] Sending API_VERSIONS request with header RequestHeader(apiKey=API_VERSIONS, apiVersion=3, clientId=adminclient-2, correlationId=2) and timeout 3600000 to node -3: ApiVersionsRequestData(clientSoftwareName='apache-kafka-java', clientSoftwareVersion='2.8.0')
2021-10-13 06:15:44 DEBUG Selector:531 - [AdminClient clientId=adminclient-1] Created socket with SO_RCVBUF = 65536, SO_SNDBUF = 131072, SO_TIMEOUT = 0 to node -2
2021-10-13 06:15:44 DEBUG NetworkClient:953 - [AdminClient clientId=adminclient-1] Completed connection to node -2. Fetching API versions.
2021-10-13 06:15:44 DEBUG NetworkClient:967 - [AdminClient clientId=adminclient-1] Initiating API versions fetch from node -2.
2021-10-13 06:15:44 DEBUG NetworkClient:522 - [AdminClient clientId=adminclient-1] Sending API_VERSIONS request with header RequestHeader(apiKey=API_VERSIONS, apiVersion=3, clientId=adminclient-1, correlationId=2) and timeout 3600000 to node -2: ApiVersionsRequestData(clientSoftwareName='apache-kafka-java', clientSoftwareVersion='2.8.0')
2021-10-13 06:15:45 WARN  BlockedThreadChecker: - Thread Thread[vert.x-eventloop-thread-0,5,main] has been blocked for 2517 ms, time limit is 2000 ms
2021-10-13 06:15:45 DEBUG Selector:606 - [AdminClient clientId=adminclient-2] Connection with dc3-kafka01.mydomain.com/10.15.14.70 disconnected
java.io.EOFException: null
	at org.apache.kafka.common.network.NetworkReceive.readFrom(NetworkReceive.java:120) ~[org.apache.kafka.kafka-clients-2.8.0.jar:?]
	at org.apache.kafka.common.network.KafkaChannel.receive(KafkaChannel.java:452) ~[org.apache.kafka.kafka-clients-2.8.0.jar:?]
	at org.apache.kafka.common.network.KafkaChannel.read(KafkaChannel.java:402) ~[org.apache.kafka.kafka-clients-2.8.0.jar:?]
	at org.apache.kafka.common.network.Selector.attemptRead(Selector.java:674) ~[org.apache.kafka.kafka-clients-2.8.0.jar:?]
	at org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:576) [org.apache.kafka.kafka-clients-2.8.0.jar:?]
	at org.apache.kafka.common.network.Selector.poll(Selector.java:481) [org.apache.kafka.kafka-clients-2.8.0.jar:?]
	at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:561) [org.apache.kafka.kafka-clients-2.8.0.jar:?]
	at org.apache.kafka.clients.admin.KafkaAdminClient$AdminClientRunnable.processRequests(KafkaAdminClient.java:1333) [org.apache.kafka.kafka-clients-2.8.0.jar:?]
	at org.apache.kafka.clients.admin.KafkaAdminClient$AdminClientRunnable.run(KafkaAdminClient.java:1264) [org.apache.kafka.kafka-clients-2.8.0.jar:?]
	at java.lang.Thread.run(Thread.java:829) [?:?]
2021-10-13 06:15:45 DEBUG NetworkClient:936 - [AdminClient clientId=adminclient-2] Node -3 disconnected.
2021-10-13 06:15:45 DEBUG NetworkClient:986 - [AdminClient clientId=adminclient-2] Initiating connection to node dc1-kafka01.mydomain.com:9093 (id: -1 rack: null) using address dc1-kafka01.mydomain.com/10.18.14.70
2021-10-13 06:15:45 DEBUG Selector:606 - [AdminClient clientId=adminclient-1] Connection with dc2-kafka01.mydomain.com/10.16.14.70 disconnected
java.io.EOFException: null
	at org.apache.kafka.common.network.NetworkReceive.readFrom(NetworkReceive.java:120) ~[org.apache.kafka.kafka-clients-2.8.0.jar:?]
	at org.apache.kafka.common.network.KafkaChannel.receive(KafkaChannel.java:452) ~[org.apache.kafka.kafka-clients-2.8.0.jar:?]
	at org.apache.kafka.common.network.KafkaChannel.read(KafkaChannel.java:402) ~[org.apache.kafka.kafka-clients-2.8.0.jar:?]
	at org.apache.kafka.common.network.Selector.attemptRead(Selector.java:674) ~[org.apache.kafka.kafka-clients-2.8.0.jar:?]
	at org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:576) [org.apache.kafka.kafka-clients-2.8.0.jar:?]
	at org.apache.kafka.common.network.Selector.poll(Selector.java:481) [org.apache.kafka.kafka-clients-2.8.0.jar:?]
	at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:561) [org.apache.kafka.kafka-clients-2.8.0.jar:?]
	at org.apache.kafka.clients.admin.KafkaAdminClient$AdminClientRunnable.processRequests(KafkaAdminClient.java:1333) [org.apache.kafka.kafka-clients-2.8.0.jar:?]
	at org.apache.kafka.clients.admin.KafkaAdminClient$AdminClientRunnable.run(KafkaAdminClient.java:1264) [org.apache.kafka.kafka-clients-2.8.0.jar:?]
	at java.lang.Thread.run(Thread.java:829) [?:?]
2021-10-13 06:15:45 DEBUG NetworkClient:936 - [AdminClient clientId=adminclient-1] Node -2 disconnected.
2021-10-13 06:15:45 DEBUG NetworkClient:986 - [AdminClient clientId=adminclient-1] Initiating connection to node dc3-kafka01.mydomain.com:9093 (id: -3 rack: null) using address dc3-kafka01.mydomain.com/10.15.14.70
2021-10-13 06:15:45 DEBUG Selector:531 - [AdminClient clientId=adminclient-2] Created socket with SO_RCVBUF = 65536, SO_SNDBUF = 131072, SO_TIMEOUT = 0 to node -1
2021-10-13 06:15:45 DEBUG NetworkClient:953 - [AdminClient clientId=adminclient-2] Completed connection to node -1. Fetching API versions.
2021-10-13 06:15:45 DEBUG NetworkClient:967 - [AdminClient clientId=adminclient-2] Initiating API versions fetch from node -1.
2021-10-13 06:15:45 DEBUG NetworkClient:522 - [AdminClient clientId=adminclient-2] Sending API_VERSIONS request with header RequestHeader(apiKey=API_VERSIONS, apiVersion=3, clientId=adminclient-2, correlationId=3) and timeout 3600000 to node -1: ApiVersionsRequestData(clientSoftwareName='apache-kafka-java', clientSoftwareVersion='2.8.0')
2021-10-13 06:15:45 DEBUG Selector:531 - [AdminClient clientId=adminclient-1] Created socket with SO_RCVBUF = 65536, SO_SNDBUF = 131072, SO_TIMEOUT = 0 to node -3
2021-10-13 06:15:45 DEBUG NetworkClient:953 - [AdminClient clientId=adminclient-1] Completed connection to node -3. Fetching API versions.
2021-10-13 06:15:45 DEBUG NetworkClient:967 - [AdminClient clientId=adminclient-1] Initiating API versions fetch from node -3.
2021-10-13 06:15:45 DEBUG NetworkClient:522 - [AdminClient clientId=adminclient-1] Sending API_VERSIONS request with header RequestHeader(apiKey=API_VERSIONS, apiVersion=3, clientId=adminclient-1, correlationId=3) and timeout 3600000 to node -3: ApiVersionsRequestData(clientSoftwareName='apache-kafka-java', clientSoftwareVersion='2.8.0')
2021-10-13 06:15:45 DEBUG Selector:606 - [AdminClient clientId=adminclient-2] Connection with dc1-kafka01.mydomain.com/10.18.14.70 disconnected
java.io.EOFException: null
	at org.apache.kafka.common.network.NetworkReceive.readFrom(NetworkReceive.java:120) ~[org.apache.kafka.kafka-clients-2.8.0.jar:?]
	at org.apache.kafka.common.network.KafkaChannel.receive(KafkaChannel.java:452) ~[org.apache.kafka.kafka-clients-2.8.0.jar:?]
	at org.apache.kafka.common.network.KafkaChannel.read(KafkaChannel.java:402) ~[org.apache.kafka.kafka-clients-2.8.0.jar:?]
	at org.apache.kafka.common.network.Selector.attemptRead(Selector.java:674) ~[org.apache.kafka.kafka-clients-2.8.0.jar:?]
	at org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:576) [org.apache.kafka.kafka-clients-2.8.0.jar:?]
	at org.apache.kafka.common.network.Selector.poll(Selector.java:481) [org.apache.kafka.kafka-clients-2.8.0.jar:?]
	at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:561) [org.apache.kafka.kafka-clients-2.8.0.jar:?]
	at org.apache.kafka.clients.admin.KafkaAdminClient$AdminClientRunnable.processRequests(KafkaAdminClient.java:1333) [org.apache.kafka.kafka-clients-2.8.0.jar:?]
	at org.apache.kafka.clients.admin.KafkaAdminClient$AdminClientRunnable.run(KafkaAdminClient.java:1264) [org.apache.kafka.kafka-clients-2.8.0.jar:?]
	at java.lang.Thread.run(Thread.java:829) [?:?]
2021-10-13 06:15:45 DEBUG NetworkClient:936 - [AdminClient clientId=adminclient-2] Node -1 disconnected.
2021-10-13 06:15:45 DEBUG NetworkClient:986 - [AdminClient clientId=adminclient-2] Initiating connection to node dc2-kafka01.mydomain.com:9093 (id: -2 rack: null) using address dc2-kafka01.mydomain.com/10.16.14.70
2021-10-13 06:15:45 DEBUG Selector:606 - [AdminClient clientId=adminclient-1] Connection with dc3-kafka01.mydomain.com/10.15.14.70 disconnected
java.io.EOFException: null
	at org.apache.kafka.common.network.NetworkReceive.readFrom(NetworkReceive.java:120) ~[org.apache.kafka.kafka-clients-2.8.0.jar:?]
	at org.apache.kafka.common.network.KafkaChannel.receive(KafkaChannel.java:452) ~[org.apache.kafka.kafka-clients-2.8.0.jar:?]
	at org.apache.kafka.common.network.KafkaChannel.read(KafkaChannel.java:402) ~[org.apache.kafka.kafka-clients-2.8.0.jar:?]
	at org.apache.kafka.common.network.Selector.attemptRead(Selector.java:674) ~[org.apache.kafka.kafka-clients-2.8.0.jar:?]
	at org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:576) [org.apache.kafka.kafka-clients-2.8.0.jar:?]
	at org.apache.kafka.common.network.Selector.poll(Selector.java:481) [org.apache.kafka.kafka-clients-2.8.0.jar:?]
	at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:561) [org.apache.kafka.kafka-clients-2.8.0.jar:?]
	at org.apache.kafka.clients.admin.KafkaAdminClient$AdminClientRunnable.processRequests(KafkaAdminClient.java:1333) [org.apache.kafka.kafka-clients-2.8.0.jar:?]
	at org.apache.kafka.clients.admin.KafkaAdminClient$AdminClientRunnable.run(KafkaAdminClient.java:1264) [org.apache.kafka.kafka-clients-2.8.0.jar:?]
	at java.lang.Thread.run(Thread.java:829) [?:?]
2021-10-13 06:15:45 DEBUG NetworkClient:936 - [AdminClient clientId=adminclient-1] Node -3 disconnected.
2021-10-13 06:15:45 DEBUG NetworkClient:986 - [AdminClient clientId=adminclient-1] Initiating connection to node dc1-kafka01.mydomain.com:9093 (id: -1 rack: null) using address dc1-kafka01.mydomain.com/10.18.14.70
2021-10-13 06:15:45 DEBUG Selector:531 - [AdminClient clientId=adminclient-2] Created socket with SO_RCVBUF = 65536, SO_SNDBUF = 131072, SO_TIMEOUT = 0 to node -2
2021-10-13 06:15:45 DEBUG NetworkClient:953 - [AdminClient clientId=adminclient-2] Completed connection to node -2. Fetching API versions.
2021-10-13 06:15:45 DEBUG NetworkClient:967 - [AdminClient clientId=adminclient-2] Initiating API versions fetch from node -2.
2021-10-13 06:15:45 DEBUG NetworkClient:522 - [AdminClient clientId=adminclient-2] Sending API_VERSIONS request with header RequestHeader(apiKey=API_VERSIONS, apiVersion=3, clientId=adminclient-2, correlationId=4) and timeout 3600000 to node -2: ApiVersionsRequestData(clientSoftwareName='apache-kafka-java', clientSoftwareVersion='2.8.0')
2021-10-13 06:15:45 DEBUG Selector:531 - [AdminClient clientId=adminclient-1] Created socket with SO_RCVBUF = 65536, SO_SNDBUF = 131072, SO_TIMEOUT = 0 to node -1
2021-10-13 06:15:45 DEBUG NetworkClient:953 - [AdminClient clientId=adminclient-1] Completed connection to node -1. Fetching API versions.
2021-10-13 06:15:45 DEBUG NetworkClient:967 - [AdminClient clientId=adminclient-1] Initiating API versions fetch from node -1.
2021-10-13 06:15:45 DEBUG NetworkClient:522 - [AdminClient clientId=adminclient-1] Sending API_VERSIONS request with header RequestHeader(apiKey=API_VERSIONS, apiVersion=3, clientId=adminclient-1, correlationId=4) and timeout 3600000 to node -1: ApiVersionsRequestData(clientSoftwareName='apache-kafka-java', clientSoftwareVersion='2.8.0')
2021-10-13 06:15:45 DEBUG Selector:606 - [AdminClient clientId=adminclient-1] Connection with dc1-kafka01.mydomain.com/10.18.14.70 disconnected
java.io.EOFException: null
	at org.apache.kafka.common.network.NetworkReceive.readFrom(NetworkReceive.java:120) ~[org.apache.kafka.kafka-clients-2.8.0.jar:?]
	at org.apache.kafka.common.network.KafkaChannel.receive(KafkaChannel.java:452) ~[org.apache.kafka.kafka-clients-2.8.0.jar:?]
	at org.apache.kafka.common.network.KafkaChannel.read(KafkaChannel.java:402) ~[org.apache.kafka.kafka-clients-2.8.0.jar:?]
	at org.apache.kafka.common.network.Selector.attemptRead(Selector.java:674) ~[org.apache.kafka.kafka-clients-2.8.0.jar:?]
	at org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:576) [org.apache.kafka.kafka-clients-2.8.0.jar:?]
	at org.apache.kafka.common.network.Selector.poll(Selector.java:481) [org.apache.kafka.kafka-clients-2.8.0.jar:?]
	at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:561) [org.apache.kafka.kafka-clients-2.8.0.jar:?]
	at org.apache.kafka.clients.admin.KafkaAdminClient$AdminClientRunnable.processRequests(KafkaAdminClient.java:1333) [org.apache.kafka.kafka-clients-2.8.0.jar:?]
	at org.apache.kafka.clients.admin.KafkaAdminClient$AdminClientRunnable.run(KafkaAdminClient.java:1264) [org.apache.kafka.kafka-clients-2.8.0.jar:?]
	at java.lang.Thread.run(Thread.java:829) [?:?]
2021-10-13 06:15:45 DEBUG NetworkClient:936 - [AdminClient clientId=adminclient-1] Node -1 disconnected.
2021-10-13 06:15:45 DEBUG NetworkClient:986 - [AdminClient clientId=adminclient-1] Initiating connection to node dc2-kafka01.mydomain.com:9093 (id: -2 rack: null) using address dc2-kafka01.mydomain.com/10.16.14.70
2021-10-13 06:15:45 DEBUG Selector:606 - [AdminClient clientId=adminclient-2] Connection with dc2-kafka01.mydomain.com/10.16.14.70 disconnected
java.io.EOFException: null
	at org.apache.kafka.common.network.NetworkReceive.readFrom(NetworkReceive.java:120) ~[org.apache.kafka.kafka-clients-2.8.0.jar:?]
	at org.apache.kafka.common.network.KafkaChannel.receive(KafkaChannel.java:452) ~[org.apache.kafka.kafka-clients-2.8.0.jar:?]
	at org.apache.kafka.common.network.KafkaChannel.read(KafkaChannel.java:402) ~[org.apache.kafka.kafka-clients-2.8.0.jar:?]
	at org.apache.kafka.common.network.Selector.attemptRead(Selector.java:674) ~[org.apache.kafka.kafka-clients-2.8.0.jar:?]
	at org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:576) [org.apache.kafka.kafka-clients-2.8.0.jar:?]
	at org.apache.kafka.common.network.Selector.poll(Selector.java:481) [org.apache.kafka.kafka-clients-2.8.0.jar:?]
	at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:561) [org.apache.kafka.kafka-clients-2.8.0.jar:?]
	at org.apache.kafka.clients.admin.KafkaAdminClient$AdminClientRunnable.processRequests(KafkaAdminClient.java:1333) [org.apache.kafka.kafka-clients-2.8.0.jar:?]
	at org.apache.kafka.clients.admin.KafkaAdminClient$AdminClientRunnable.run(KafkaAdminClient.java:1264) [org.apache.kafka.kafka-clients-2.8.0.jar:?]
	at java.lang.Thread.run(Thread.java:829) [?:?]
2021-10-13 06:15:45 DEBUG NetworkClient:936 - [AdminClient clientId=adminclient-2] Node -2 disconnected.
2021-10-13 06:15:45 DEBUG NetworkClient:986 - [AdminClient clientId=adminclient-2] Initiating connection to node dc3-kafka01.mydomain.com:9093 (id: -3 rack: null) using address dc3-kafka01.mydomain.com/10.15.14.70
2021-10-13 06:15:45 DEBUG Selector:531 - [AdminClient clientId=adminclient-1] Created socket with SO_RCVBUF = 65536, SO_SNDBUF = 131072, SO_TIMEOUT = 0 to node -2
2021-10-13 06:15:45 DEBUG NetworkClient:953 - [AdminClient clientId=adminclient-1] Completed connection to node -2. Fetching API versions.
2021-10-13 06:15:45 DEBUG NetworkClient:967 - [AdminClient clientId=adminclient-1] Initiating API versions fetch from node -2.
2021-10-13 06:15:45 DEBUG NetworkClient:522 - [AdminClient clientId=adminclient-1] Sending API_VERSIONS request with header RequestHeader(apiKey=API_VERSIONS, apiVersion=3, clientId=adminclient-1, correlationId=5) and timeout 3600000 to node -2: ApiVersionsRequestData(clientSoftwareName='apache-kafka-java', clientSoftwareVersion='2.8.0')
2021-10-13 06:15:46 DEBUG Selector:531 - [AdminClient clientId=adminclient-2] Created socket with SO_RCVBUF = 65536, SO_SNDBUF = 131072, SO_TIMEOUT = 0 to node -3
2021-10-13 06:15:46 DEBUG NetworkClient:953 - [AdminClient clientId=adminclient-2] Completed connection to node -3. Fetching API versions.
2021-10-13 06:15:46 DEBUG NetworkClient:967 - [AdminClient clientId=adminclient-2] Initiating API versions fetch from node -3.
2021-10-13 06:15:46 DEBUG NetworkClient:522 - [AdminClient clientId=adminclient-2] Sending API_VERSIONS request with header RequestHeader(apiKey=API_VERSIONS, apiVersion=3, clientId=adminclient-2, correlationId=5) and timeout 3600000 to node -3: ApiVersionsRequestData(clientSoftwareName='apache-kafka-java', clientSoftwareVersion='2.8.0')

Is there anything im missing here?

I am able to connect using a client (conducktor)

Thanks in advance

scholzj · 2021-10-13T06:33:21Z

scholzj
Oct 13, 2021
Maintainer

I don't know if that is the only issue, but having STRIMZI_TLS_ENABLED set to false sounds wrong if you are passing TLS certificates to it.

0 replies

itaydj · 2021-10-13T06:37:28Z

itaydj
Oct 13, 2021
Author

Thanks Jakub,

I was playing with this (and other) settings and found that when I enable STRIMZI_TLS_ENABLED the log shows this:

 kubectl logs -f strimzi-topic-operator-6767b596d4-2mrwj  
Preparing trust store certificates for internal communication
Adding /etc/tls-sidecar/cluster-ca-certs/*.crt to truststore /tmp/topic-operator/replication.truststore.p12 with alias *
keytool error: java.io.FileNotFoundException: /etc/tls-sidecar/cluster-ca-certs/*.crt (No such file or directory)

And thats the only thing in the log

2 replies

scholzj Oct 13, 2021
Maintainer

Well, that suggests that you are passing the certificates in a wrong format? Where did you found the keystore and truststore options? Maybe you can deploy a full Strimzi cluster and check what configuration does the Topic Operator use there => that is using TLS client authentication as well, so you should be able to copy the options and just change the values.

itaydj Oct 13, 2021
Author

Got it,
Thats actually a good idea
I will try it and update soon

itaydj · 2021-10-13T10:54:01Z

itaydj
Oct 13, 2021
Author

Ok, I took your advise and created a Strimzi TLS enabled cluster and checked the topic operator settings.
it seems that the topic operator uses P12 files and not JKS

I mounted the joined client certificate and key as a p12 file (instead of JKS)

I than saw in the manual that for the I need to use both the ca.crt and ca.key

Any reason we need the ca.key for?
I only want to connect to the remote Kafka cluster , not to sign certificates on its behalf.

11 replies

itaydj Oct 14, 2021
Author

Im back

Changed the keystore and trustore to contain the folowing:

keystore.p12 - contains the client certificate and key needed to connect to the external Kafka cluster

truststore.p12 - contains the ca certificate for the kafka cluster.

tried to launch the standalone operator and got this:

$ kubectl logs -f strimzi-topic-operator-66f78b4fc8-7xfbh 
Preparing trust store certificates for internal communication
Adding /etc/tls-sidecar/cluster-ca-certs/*.crt to truststore /tmp/topic-operator/replication.truststore.p12 with alias *
keytool error: java.io.FileNotFoundException: /etc/tls-sidecar/cluster-ca-certs/*.crt (No such file or directory)

What am I (still) missing?

scholzj Oct 14, 2021
Maintainer

Wait ... are you using the TLS sidecar for something?

itaydj Oct 14, 2021
Author

Im not, I dont even know what is triggering it

itaydj Oct 14, 2021
Author

just to clarify I only trying to use the standalone topic operator, not the whole strimzi suite
(we are using the whole suite in a different cluster)

scholzj Oct 14, 2021
Maintainer

Yeah, that is why I'm confused.

itaydj · 2021-10-14T21:29:45Z

itaydj
Oct 14, 2021
Author

UPDATE

I did some reverse engineering on the
https://github.com/strimzi/strimzi-kafka-operator/blob/main/topic-operator/scripts/topic_operator_run.sh

and found out I need to add the following flags to prevent the tls side car from being used:

            - name: STRIMZI_SECURITY_PROTOCOL
              value: "SSL"
            - name: STRIMZI_PUBLIC_CA
              value: "true"
            - name: STRIMZI_TLS_AUTH_ENABLED
              value: "false"

when I did that it seems that the topic operator finally connects to my cluster!

Now im getting the following error:

.
2021-10-14 21:20:42 DEBUG KafkaAdminClient:1099 - [AdminClient clientId=adminclient-2] Sending CreateTopicsRequestData(topics=[CreatableTopic(name='__strimzi_store_topic', numPartitions=1, replicationFactor=3, assignments=[], configs=[CreateableTopicConfig(name='min.insync.replicas', value='2')])], timeoutMs=30000, validateOnly=false) to esh-poc-kafka01.mydomain.com:9093 (id: 31 rack: ESH). correlationId=5
2021-10-14 21:20:42 DEBUG NetworkClient:522 - [AdminClient clientId=adminclient-2] Sending CREATE_TOPICS request with header RequestHeader(apiKey=CREATE_TOPICS, apiVersion=7, clientId=adminclient-2, correlationId=5) and timeout 30000 to node 31: CreateTopicsRequestData(topics=[CreatableTopic(name='__strimzi_store_topic', numPartitions=1, replicationFactor=3, assignments=[], configs=[CreateableTopicConfig(name='min.insync.replicas', value='2')])], timeoutMs=30000, validateOnly=false)
2021-10-14 21:20:42 DEBUG NetworkClient:880 - [AdminClient clientId=adminclient-2] Received CREATE_TOPICS response from node 31 for request with header RequestHeader(apiKey=CREATE_TOPICS, apiVersion=7, clientId=adminclient-2, correlationId=5): CreateTopicsResponseData(throttleTimeMs=0, topics=[CreatableTopicResult(name='__strimzi_store_topic', topicId=AAAAAAAAAAAAAAAAAAAAAA, errorCode=36, errorMessage='Topic '__strimzi_store_topic' already exists.', topicConfigErrorCode=0, numPartitions=-1, replicationFactor=-1, configs=[])])
2021-10-14 21:20:42 WARN  KafkaStreamsTopicStoreService:69 - Failed to start.
java.util.concurrent.CompletionException: org.apache.kafka.common.errors.TopicExistsException: Topic '__strimzi_store_topic' already exists.
	at java.util.concurrent.CompletableFuture.encodeRelay(CompletableFuture.java:367) ~[?:?]
	at java.util.concurrent.CompletableFuture.completeRelay(CompletableFuture.java:376) ~[?:?]
	at java.util.concurrent.CompletableFuture$UniRelay.tryFire(CompletableFuture.java:1019) ~[?:?]
	at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506) ~[?:?]
	at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2088) ~[?:?]
	at io.strimzi.operator.topic.KafkaStreamsTopicStoreService.lambda$toCS$13(KafkaStreamsTopicStoreService.java:178) ~[io.strimzi.topic-operator-0.25.0.jar:0.25.0]
	at org.apache.kafka.common.internals.KafkaFutureImpl$WhenCompleteBiConsumer.accept(KafkaFutureImpl.java:175) ~[org.apache.kafka.kafka-clients-2.8.0.jar:?]
	at org.apache.kafka.common.internals.KafkaFutureImpl$WhenCompleteBiConsumer.accept(KafkaFutureImpl.java:162) ~[org.apache.kafka.kafka-clients-2.8.0.jar:?]
	at org.apache.kafka.common.internals.KafkaFutureImpl.completeExceptionally(KafkaFutureImpl.java:238) ~[org.apache.kafka.kafka-clients-2.8.0.jar:?]
	at org.apache.kafka.common.KafkaFuture$AllOfAdapter.accept(KafkaFuture.java:73) ~[org.apache.kafka.kafka-clients-2.8.0.jar:?]
	at org.apache.kafka.common.KafkaFuture$AllOfAdapter.accept(KafkaFuture.java:57) ~[org.apache.kafka.kafka-clients-2.8.0.jar:?]
	at org.apache.kafka.common.internals.KafkaFutureImpl.completeExceptionally(KafkaFutureImpl.java:238) ~[org.apache.kafka.kafka-clients-2.8.0.jar:?]
	at org.apache.kafka.clients.admin.KafkaAdminClient$1.handleResponse(KafkaAdminClient.java:1545) ~[org.apache.kafka.kafka-clients-2.8.0.jar:?]
	at org.apache.kafka.clients.admin.KafkaAdminClient$AdminClientRunnable.handleResponses(KafkaAdminClient.java:1189) ~[org.apache.kafka.kafka-clients-2.8.0.jar:?]
	at org.apache.kafka.clients.admin.KafkaAdminClient$AdminClientRunnable.processRequests(KafkaAdminClient.java:1341) ~[org.apache.kafka.kafka-clients-2.8.0.jar:?]
	at org.apache.kafka.clients.admin.KafkaAdminClient$AdminClientRunnable.run(KafkaAdminClient.java:1264) ~[org.apache.kafka.kafka-clients-2.8.0.jar:?]
	at java.lang.Thread.run(Thread.java:829) [?:?]
Caused by: org.apache.kafka.common.errors.TopicExistsException: Topic '__strimzi_store_topic' already exists.
2021-10-14 21:20:42 INFO  KafkaStreamsTopicStoreService:161 - Stopping services ...
2021-10-14 21:20:42 DEBUG KafkaAdminClient:632 - [AdminClient clientId=adminclient-2] Initiating close operation.
2021-10-14 21:20:42 DEBUG KafkaAdminClient:648 - [AdminClient clientId=adminclient-2] Waiting for the I/O thread to exit. Hard shutdown in 31535999999 ms.
2021-10-14 21:20:42 INFO  AppInfoParser:83 - App info kafka.admin.client for adminclient-2 unregistered
2021-10-14 21:20:42 INFO  Metrics:659 - Metrics scheduler closed
2021-10-14 21:20:42 INFO  Metrics:663 - Closing reporter org.apache.kafka.common.metrics.JmxReporter
2021-10-14 21:20:42 INFO  Metrics:669 - Metrics reporters closed
2021-10-14 21:20:42 DEBUG KafkaAdminClient:1283 - [AdminClient clientId=adminclient-2] Exiting AdminClientRunnable thread.
2021-10-14 21:20:42 DEBUG KafkaAdminClient:657 - [AdminClient clientId=adminclient-2] Kafka admin client closed.
2021-10-14 21:20:42 ERROR Session:201 - Failed to create topic store.
java.util.concurrent.CompletionException: org.apache.kafka.common.errors.TopicExistsException: Topic '__strimzi_store_topic' already exists.
	at java.util.concurrent.CompletableFuture.encodeRelay(CompletableFuture.java:367) ~[?:?]
	at java.util.concurrent.CompletableFuture.completeRelay(CompletableFuture.java:376) ~[?:?]
	at java.util.concurrent.CompletableFuture$UniRelay.tryFire(CompletableFuture.java:1019) ~[?:?]
	at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506) ~[?:?]
	at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2088) ~[?:?]
	at io.strimzi.operator.topic.KafkaStreamsTopicStoreService.lambda$toCS$13(KafkaStreamsTopicStoreService.java:178) ~[io.strimzi.topic-operator-0.25.0.jar:0.25.0]
	at org.apache.kafka.common.internals.KafkaFutureImpl$WhenCompleteBiConsumer.accept(KafkaFutureImpl.java:175) ~[org.apache.kafka.kafka-clients-2.8.0.jar:?]
	at org.apache.kafka.common.internals.KafkaFutureImpl$WhenCompleteBiConsumer.accept(KafkaFutureImpl.java:162) ~[org.apache.kafka.kafka-clients-2.8.0.jar:?]
	at org.apache.kafka.common.internals.KafkaFutureImpl.completeExceptionally(KafkaFutureImpl.java:238) ~[org.apache.kafka.kafka-clients-2.8.0.jar:?]
	at org.apache.kafka.common.KafkaFuture$AllOfAdapter.accept(KafkaFuture.java:73) ~[org.apache.kafka.kafka-clients-2.8.0.jar:?]
	at org.apache.kafka.common.KafkaFuture$AllOfAdapter.accept(KafkaFuture.java:57) ~[org.apache.kafka.kafka-clients-2.8.0.jar:?]
	at org.apache.kafka.common.internals.KafkaFutureImpl.completeExceptionally(KafkaFutureImpl.java:238) ~[org.apache.kafka.kafka-clients-2.8.0.jar:?]
	at org.apache.kafka.clients.admin.KafkaAdminClient$1.handleResponse(KafkaAdminClient.java:1545) ~[org.apache.kafka.kafka-clients-2.8.0.jar:?]
	at org.apache.kafka.clients.admin.KafkaAdminClient$AdminClientRunnable.handleResponses(KafkaAdminClient.java:1189) ~[org.apache.kafka.kafka-clients-2.8.0.jar:?]
	at org.apache.kafka.clients.admin.KafkaAdminClient$AdminClientRunnable.processRequests(KafkaAdminClient.java:1341) ~[org.apache.kafka.kafka-clients-2.8.0.jar:?]
	at org.apache.kafka.clients.admin.KafkaAdminClient$AdminClientRunnable.run(KafkaAdminClient.java:1264) ~[org.apache.kafka.kafka-clients-2.8.0.jar:?]
	at java.lang.Thread.run(Thread.java:829) [?:?]
Caused by: org.apache.kafka.common.errors.TopicExistsException: Topic '__strimzi_store_topic' already exists.
2021-10-14 21:20:42 ERROR Main:55 - Error deploying Session
java.util.concurrent.CompletionException: org.apache.kafka.common.errors.TopicExistsException: Topic '__strimzi_store_topic' already exists.
	at java.util.concurrent.CompletableFuture.encodeRelay(CompletableFuture.java:367) ~[?:?]
	at java.util.concurrent.CompletableFuture.completeRelay(CompletableFuture.java:376) ~[?:?]
	at java.util.concurrent.CompletableFuture$UniRelay.tryFire(CompletableFuture.java:1019) ~[?:?]
	at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506) ~[?:?]
	at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2088) ~[?:?]
	at io.strimzi.operator.topic.KafkaStreamsTopicStoreService.lambda$toCS$13(KafkaStreamsTopicStoreService.java:178) ~[io.strimzi.topic-operator-0.25.0.jar:0.25.0]
	at org.apache.kafka.common.internals.KafkaFutureImpl$WhenCompleteBiConsumer.accept(KafkaFutureImpl.java:175) ~[org.apache.kafka.kafka-clients-2.8.0.jar:?]
	at org.apache.kafka.common.internals.KafkaFutureImpl$WhenCompleteBiConsumer.accept(KafkaFutureImpl.java:162) ~[org.apache.kafka.kafka-clients-2.8.0.jar:?]
	at org.apache.kafka.common.internals.KafkaFutureImpl.completeExceptionally(KafkaFutureImpl.java:238) ~[org.apache.kafka.kafka-clients-2.8.0.jar:?]
	at org.apache.kafka.common.KafkaFuture$AllOfAdapter.accept(KafkaFuture.java:73) ~[org.apache.kafka.kafka-clients-2.8.0.jar:?]
	at org.apache.kafka.common.KafkaFuture$AllOfAdapter.accept(KafkaFuture.java:57) ~[org.apache.kafka.kafka-clients-2.8.0.jar:?]
	at org.apache.kafka.common.internals.KafkaFutureImpl.completeExceptionally(KafkaFutureImpl.java:238) ~[org.apache.kafka.kafka-clients-2.8.0.jar:?]
	at org.apache.kafka.clients.admin.KafkaAdminClient$1.handleResponse(KafkaAdminClient.java:1545) ~[org.apache.kafka.kafka-clients-2.8.0.jar:?]
	at org.apache.kafka.clients.admin.KafkaAdminClient$AdminClientRunnable.handleResponses(KafkaAdminClient.java:1189) ~[org.apache.kafka.kafka-clients-2.8.0.jar:?]
	at org.apache.kafka.clients.admin.KafkaAdminClient$AdminClientRunnable.processRequests(KafkaAdminClient.java:1341) ~[org.apache.kafka.kafka-clients-2.8.0.jar:?]
	at org.apache.kafka.clients.admin.KafkaAdminClient$AdminClientRunnable.run(KafkaAdminClient.java:1264) ~[org.apache.kafka.kafka-clients-2.8.0.jar:?]
	at java.lang.Thread.run(Thread.java:829) [?:?]
Caused by: org.apache.kafka.common.errors.TopicExistsException: Topic '__strimzi_store_topic' already exists.
2021-10-14 21:20:44 DEBUG ClientCnxn:821 - Got ping response for sessionid: 0x4016282a08a11ed after 63ms
2021-10-14 21:20:50 DEBUG ClientCnxn:821 - Got ping response for sessionid: 0x4016282a08a11ed after 63ms
2021-10-14 21:20:56 DEBUG ClientCnxn:821 - Got ping response for sessionid: 0x4016282a08a11ed after 62ms
2021-10-14 21:21:02 DEBUG ClientCnxn:821 - Got ping response for sessionid: 0x4016282a08a11ed after 62ms
2021-10-14 21:21:08 DEBUG ClientCnxn:821 - Got ping response for sessionid: 0x4016282a08a11ed after 62ms
2021-10-14 21:21:14 DEBUG ClientCnxn:821 - Got ping response for sessionid: 0x4016282a08a11ed after 62ms
2021-10-14 21:21:20 DEBUG ClientCnxn:821 - Got ping response for sessionid: 0x4016282a08a11ed after 62ms
2021-10-14 21:21:26 DEBUG ClientCnxn:821 - Got ping response for sessionid: 0x4016282a08a11ed after 62ms
2021-10-14 21:21:32 DEBUG ClientCnxn:821 - Got ping response for sessionid: 0x4016282a08a11ed after 62ms
2021-10-14 21:21:38 DEBUG ClientCnxn:821 - Got ping response for sessionid: 0x4016282a08a11ed after 63ms
2021-10-14 21:21:44 DEBUG ClientCnxn:821 - Got ping response for sessionid: 0x4016282a08a11ed after 62ms
2021-10-14 21:21:50 DEBUG ClientCnxn:821 - Got ping response for sessionid: 0x4016282a08a11ed after 63ms

Any tips? feels like im at the finish line :-)

1 reply

scholzj Oct 14, 2021
Maintainer

Sorry, since you started in the first post with truststore and keystore, I thought you are using self-signed certs with TLS authentication ... I guess you are not given the final configuration.

I don't know what the error means. Could it be similar to #5674? @sknot-rh and @tombentley are the experts, they might be able to help.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Strimzi

Standalone Topic Operator - failed to connect to secured cluster #5712

{{title}}

Replies: 4 comments 14 replies

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

Select a reply

Strimzi

Standalone Topic Operator - failed to connect to secured cluster #5712

itaydj Oct 13, 2021

Replies: 4 comments · 14 replies

scholzj Oct 13, 2021 Maintainer

itaydj Oct 13, 2021 Author

scholzj Oct 13, 2021 Maintainer

itaydj Oct 13, 2021 Author

itaydj Oct 13, 2021 Author

itaydj Oct 14, 2021 Author

scholzj Oct 14, 2021 Maintainer

itaydj Oct 14, 2021 Author

itaydj Oct 14, 2021 Author

scholzj Oct 14, 2021 Maintainer

itaydj Oct 14, 2021 Author

scholzj Oct 14, 2021 Maintainer

itaydj
Oct 13, 2021

Replies: 4 comments 14 replies

scholzj
Oct 13, 2021
Maintainer

itaydj
Oct 13, 2021
Author

scholzj Oct 13, 2021
Maintainer

itaydj Oct 13, 2021
Author

itaydj
Oct 13, 2021
Author

itaydj Oct 14, 2021
Author

scholzj Oct 14, 2021
Maintainer

itaydj Oct 14, 2021
Author

itaydj Oct 14, 2021
Author

scholzj Oct 14, 2021
Maintainer

itaydj
Oct 14, 2021
Author

scholzj Oct 14, 2021
Maintainer