Strimzi
Kafka cluster not created after zookeeper pods started #6350
Answered
by
scholzj
gabriel-farache
asked this question in
Q&A
-
|
I followed the instructions to install strimzi v0.27.1 from YAML file: the cluster operator is UP and running but when I create a Kafka resource, only the zookeeper resources are created (stateful sets, services configmaps and secrets). A bit of context: I am using a private repository to host the images. The private repository has proxy to download images from quay.io. I want to set replicas to 3 for the zookeeper part but when I do, the zookeeper pods are not working and are stuck in a restart loop with the error Now back to the current issue: the zookeeper pods are running fine, here are the logs: I do not understand the ruok message and i do not see any other container in the pod. Any TLS sidecar pod sould be present? Here are the logs of the cluster operator: I see the timeout in it but I do not understand from where they come nor what I should do to solve them. Maybe that is the source of the issue? Here are the services created: I am cluess from where to start looking for the issue, has anyone has any idea where I could start looking and/or what I did wrong? Thank you! |
Beta Was this translation helpful? Give feedback.
Answered by
scholzj
Feb 11, 2022
Replies: 1 comment 10 replies
-
|
Beta Was this translation helpful? Give feedback.
10 replies
-
|
Quick update: I deployed busybox images on these nodes and I was able to ping the busybox pods but I was not able to ping the zookeeper pod.
FYI: I used RKE to deploy my cluster and my cluster has 23 nodes. I use Flannel as CNI. |
Beta Was this translation helpful? Give feedback.
-
Ping is not a connection. But there are network policies => so you should not be able to connect to the ZooKeeper pods from a random pod. But if the connectivity works between the pods, you should be able to connect from one ZooKeeper replica to another one in the same cluster. But that seems to timeout. You can also try to disable the network policy management (env var |
Beta Was this translation helpful? Give feedback.
-
|
So, I finally took the time to test and that does not solve the connection issue when using a single cluster (r-cluster) with a replicas of 3 zookeepers (and brokers): What is weird is that I get a SSL error on some IP address and then it succeed: So I do not have any network policy on the namespace, there is no sidecar on the zookeeper pod but the zookeeper cluster does not "want" to form. Any idea where I could take a look? |
Beta Was this translation helpful? Give feedback.
-
|
Well, this now looks better. It is not timing out anymore. ZooKeeper has a weird TLS hostname verification which depends a lot on your DNS configuration. That seems to be failing right now. You can disable the hostname verification with this: apiVersion: kafka.strimzi.io/v1beta2
kind: Kafka
metadata:
name: my-cluster
spec:
...
zookeeper:
replicas: 3
jvmOptions:
javaSystemProperties:
- name: zookeeper.ssl.hostnameVerification
value: "false"
- name: zookeeper.ssl.quorum.hostnameVerification
value: "false"
...There are also multiple issues describing how some fixed it in their Kube clusters in the DNS configuration => so you can search through the issues and see if some of that helps. IIRC, RKE was mentioned there few times in the past in relation to this. |
Beta Was this translation helpful? Give feedback.
-
Yep, that was it, I misunderstood the last error and it was indeed caused by the SSL hostname verification error. Thank you very much for the time, it is much appreciated! |
Beta Was this translation helpful? Give feedback.
Answer selected by
gabriel-farache
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
bin/zookeeepr-shell.shscript to connect to localhost:12181 and try what theconfigcommand returns to you.