WARN Cannot open channel to 1 at election address

[ChiragR7]

Why I am getting this warn on start of 3 node kafka cluster.

2024-10-01 09:28:30,215 WARN Cannot open channel to 1 at election address kafka-cluster-zookeeper-0.kafka-cluster-zookeeper-nodes.kafka.svc/10.xx.x.xxx:3888 (org.apache.zookeeper.server.quorum.QuorumCnxManager) [QuorumConnectionThread-[myid=2]-1]
java.net.ConnectException: Connection refused
	at java.base/sun.nio.ch.Net.pollConnect(Native Method)
	at java.base/sun.nio.ch.Net.pollConnectNow(Net.java:672)
	at java.base/sun.nio.ch.NioSocketImpl.timedFinishConnect(NioSocketImpl.java:554)
	at java.base/sun.nio.ch.NioSocketImpl.connect(NioSocketImpl.java:602)
	at java.base/java.net.SocksSocketImpl.connect(SocksSocketImpl.java:327)
	at java.base/java.net.Socket.connect(Socket.java:633)
	at java.base/sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:304)
	at org.apache.zookeeper.server.quorum.QuorumCnxManager.initiateConnection(QuorumCnxManager.java:384)
	at org.apache.zookeeper.server.quorum.QuorumCnxManager$QuorumConnectionReqThread.run(QuorumCnxManager.java:458)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
	at java.base/java.lang.Thread.run(Thread.java:840)
2024-10-01 09:28:30,354 INFO Notification time out: 400 ms (org.apache.zookeeper.server.quorum.FastLeaderElection) [QuorumPeer[myid=2](plain=127.0.0.1:12181)(secure=[0:0:0:0:0:0:0:0]:2181)]
2024-10-01 09:28:30,355 WARN Cannot open channel to 1 at election address kafka-cluster-zookeeper-0.kafka-cluster-zookeeper-nodes.kafka.svc/10.xx.x.xxx:3888 (org.apache.zookeeper.server.quorum.QuorumCnxManager) [QuorumConnectionThread-[myid=2]-2]
java.net.ConnectException: Connection refused
	at java.base/sun.nio.ch.Net.pollConnect(Native Method)
	at java.base/sun.nio.ch.Net.pollConnectNow(Net.java:672)
	at java.base/sun.nio.ch.NioSocketImpl.timedFinishConnect(NioSocketImpl.java:554)
	at java.base/sun.nio.ch.NioSocketImpl.connect(NioSocketImpl.java:602)
	at java.base/java.net.SocksSocketImpl.connect(SocksSocketImpl.java:327)
	at java.base/java.net.Socket.connect(Socket.java:633)
	at java.base/sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:304)
	at org.apache.zookeeper.server.quorum.QuorumCnxManager.initiateConnection(QuorumCnxManager.java:384)
	at org.apache.zookeeper.server.quorum.QuorumCnxManager$QuorumConnectionReqThread.run(QuorumCnxManager.java:458)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
	at java.base/java.lang.Thread.run(Thread.java:840)

cluster.yaml:-

apiVersion: kafka.strimzi.io/v1beta2
kind: Kafka
metadata:
  name: kafka-cluster
spec:
  kafka:
    version: 3.8.0
    replicas: 3
    listeners:
      - name: plain
        port: 9092
        type: internal
        tls: false
    config:
      offsets.topic.replication.factor: 3
      transaction.state.log.replication.factor: 3
      transaction.state.log.min.isr: 2
      default.replication.factor: 3
      min.insync.replicas: 2
      inter.broker.protocol.version: "3.8"
      auto.create.topics.enable: "false"
    storage:
      type: persistent-claim
      size: 100Gi
      deleteClaim: false
      kraftMetadata: shared
      class: longhorn
  zookeeper:
    replicas: 3
    storage:
      type: persistent-claim
      size: 100Gi
      deleteClaim: false
      class: longhorn
      kraftMetadata: shared
    jvmOptions:
      javaSystemProperties:
        - name: zookeeper.ssl.hostnameVerification
          value: "false"
        - name: zookeeper.ssl.quorum.hostnameVerification
          value: "false"
  entityOperator:
    topicOperator:
      watchedNamespace: kafka
      reconciliationIntervalSeconds: 2
      topicMetadataMaxAttempts: 5
    userOperator: {}

[scholzj]

ZooKeeper has all kind of errors like this, especially at startup when the nodes are looking for each other. Without the context (e.g. full logs from all components to properly understand what is going on), it is hard to say what exactly it means. If everything seems to work fine, you can likely ignore it

[ChiragR7]

I just followed steps :-

    1.  Download the release wget https://github.com/strimzi/strimzi-kafka-operator/releases/download/0.43.0/strimzi-0.43.0.tar.gz
    2.  tar -xvzf strimzi-0.43.0.tar.gz
    3.  cd strimzi-0.43.0
    4.  kubectl create namespace kafka
    5.  sed -i 's/namespace: .*/namespace: kafka/‘ install/cluster-operator/*RoleBinding*.yaml
    6.  kubectl apply -f install/cluster-operator -n kafka
    7.  kubectl apply -f cluster.yaml -n kafka
   

and I see those warn on startup of zookeepers. They eventually come down to running state, but shouldn't we have some way to relax/ put a delay before zookeeper pods starts discovering each other ?

[scholzj]

That still doesn't provide any of the missing the context. But in general, the pods start asynchronously, likely run on different nodes, each might need to wait different time for storage provisioning, node assignment and capacity, pulling of the container images etc. DNS also takes some time to recognize all the services which again is asynchronous. So I don't think you can synchronize this easily.