Can I use Strimzi without minikube?

[networkingana]

I have vanilla Kubernetes installation, and I want to use Kafka with Strimzi for production. Currently, I am organizing the whole environment with all of the applications of the system.

I encountered issues regarding the persistent volumes and storage. For now, I want to use local storage for the Kafka and zookeeper data.

My cluster has one master node and two worker nodes

I created a storage class for kubernetes with the following YAML file:

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: local-storage
provisioner: kubernetes.io/no-provisioner
volumeBindingMode: WaitForFirstConsumer

I also created persistent volume with the following YAML file:

apiVersion: v1
kind: PersistentVolume
metadata:
  name: kafka-cluster-pv
spec:
  capacity:
    storage: 10Gi
  accessModes:
  - ReadWriteMany
  persistentVolumeReclaimPolicy: Retain
  local:
    path: /data
  storageClassName: local-storage
  nodeAffinity:
    required:
      nodeSelectorTerms:
      - matchExpressions:
        - key: kubernetes.io/hostname
          operator: In
          values:
          - master-node

And finally, this is my Kafka YAML used from the strimzi-.0.24.0.zip example folder and modified with class

apiVersion: kafka.strimzi.io/v1beta2
kind: Kafka
metadata:
  name: my-cluster
spec:
  kafka:
    version: 2.8.0
    replicas: 3
    listeners:
      - name: plain
        port: 9092
        type: internal
        tls: false
      - name: tls
        port: 9093
        type: internal
        tls: true
      - name: external
        port: 9094
        type: nodeport
        tls: false
    config:
      offsets.topic.replication.factor: 3
      transaction.state.log.replication.factor: 3
      transaction.state.log.min.isr: 2
      log.message.format.version: "2.8"
      inter.broker.protocol.version: "2.8"
    storage:
      type: persistent-claim
      size: 10Gi
      deleteClaim: false
      class: "local-storage"
  zookeeper:
    replicas: 3
    storage:
      type: persistent-claim
      size: 10Gi
      deleteClaim: false
      class: "local-storage"
  entityOperator:
    topicOperator: {}
    userOperator: {}

My zookeepers fail to bind to my persistent volume can you please help me with this?

kubect_describe_all.txt
kubect_describe_pvc.txt

[scholzj]

Well, first of all, you need persistent volume per node. So it is not enough to create just one PV ... for 3 ZooKeepers and 3 Kafkas you will need 6 of them -> each for a different path. Not sure why at least the one them didn't bound, but this seems to be the complaint from Kubernetes:

Warning  ProvisioningFailed   15m (x12 over 18m)    persistentvolume-controller  no volume plugin matched name: kubernetes.io/no-provisioner

[scholzj]

Could be if there are new errors.

[networkingana]

firewall-cmd --permanent --add-port=30000-32767/tcp on my cluster node and it seemed that there are changes in the cluster so I created new report archive please take a look
report-30-07-2021_14-08-55.zip

[networkingana]

2021-07-30 22:06:21,161 INFO Client environment:java.library.path=/usr/java/packages/lib:/usr/lib64:/lib64:/lib:/usr/lib (org.apache.zookeeper.ZooKeeper) [main]
2021-07-30 22:06:21,161 INFO Client environment:java.io.tmpdir=/tmp (org.apache.zookeeper.ZooKeeper) [main]
2021-07-30 22:06:21,161 INFO Client environment:java.compiler=<NA> (org.apache.zookeeper.ZooKeeper) [main]
2021-07-30 22:06:21,161 INFO Client environment:os.name=Linux (org.apache.zookeeper.ZooKeeper) [main]
2021-07-30 22:06:21,161 INFO Client environment:os.arch=amd64 (org.apache.zookeeper.ZooKeeper) [main]
2021-07-30 22:06:21,161 INFO Client environment:os.version=3.10.0-1160.el7.x86_64 (org.apache.zookeeper.ZooKeeper) [main]
2021-07-30 22:06:21,161 INFO Client environment:user.name=kafka (org.apache.zookeeper.ZooKeeper) [main]
2021-07-30 22:06:21,162 INFO Client environment:user.home=/home/kafka (org.apache.zookeeper.ZooKeeper) [main]
2021-07-30 22:06:21,162 INFO Client environment:user.dir=/opt/kafka (org.apache.zookeeper.ZooKeeper) [main]
2021-07-30 22:06:21,162 INFO Client environment:os.memory.free=117MB (org.apache.zookeeper.ZooKeeper) [main]
2021-07-30 22:06:21,162 INFO Client environment:os.memory.max=456MB (org.apache.zookeeper.ZooKeeper) [main]
2021-07-30 22:06:21,162 INFO Client environment:os.memory.total=130MB (org.apache.zookeeper.ZooKeeper) [main]
2021-07-30 22:06:21,166 INFO Initiating client connection, connectString=my-cluster-zookeeper-client:2181 sessionTimeout=18000 watcher=kafka.zookeeper.ZooKeeperClient$ZooKeeperClientWatcher$@78a773fd (org.apache.zookeeper.ZooKeeper) [main]
2021-07-30 22:06:21,279 INFO jute.maxbuffer value is 4194304 Bytes (org.apache.zookeeper.ClientCnxnSocket) [main]
2021-07-30 22:06:21,288 INFO zookeeper.request.timeout value is 0. feature enabled= (org.apache.zookeeper.ClientCnxn) [main]
2021-07-30 22:06:21,290 INFO Starting poller (io.strimzi.kafka.agent.KafkaAgent) [main]
2021-07-30 22:06:21,300 INFO [ZooKeeperClient Kafka server] Waiting until connected. (kafka.zookeeper.ZooKeeperClient) [main]
2021-07-30 22:06:21,315 INFO Opening socket connection to server my-cluster-zookeeper-client/10.99.117.223:2181. Will not attempt to authenticate using SASL (unknown error) (org.apache.zookeeper.ClientCnxn) [main-SendThread(my-cluster-zookeeper-client:2181)]
2021-07-30 22:06:22,168 INFO SSL handler added for channel: [id: 0x05c4f4aa] (org.apache.zookeeper.ClientCnxnSocketNetty) [nioEventLoopGroup-2-1]
2021-07-30 22:06:39,302 INFO [ZooKeeperClient Kafka server] Closing. (kafka.zookeeper.ZooKeeperClient) [main]
2021-07-30 22:06:39,590 WARN Client session timed out, have not heard from server in 18297ms for sessionid 0x0 (org.apache.zookeeper.ClientCnxn) [main-SendThread(my-cluster-zookeeper-client:2181)]
2021-07-30 22:06:39,593 INFO channel is told closing (org.apache.zookeeper.ClientCnxnSocketNetty) [main]
2021-07-30 22:06:39,595 INFO Session: 0x0 closed (org.apache.zookeeper.ZooKeeper) [main]
2021-07-30 22:06:39,596 INFO EventThread shut down for session: 0x0 (org.apache.zookeeper.ClientCnxn) [main-EventThread]
2021-07-30 22:06:39,597 INFO future isn't success, cause: (org.apache.zookeeper.ClientCnxnSocketNetty) [nioEventLoopGroup-2-1]
java.nio.channels.ClosedChannelException
        at io.netty.channel.nio.AbstractNioChannel.doClose(AbstractNioChannel.java:502)
        at io.netty.channel.socket.nio.NioSocketChannel.doClose(NioSocketChannel.java:342)
        at io.netty.channel.AbstractChannel$AbstractUnsafe.doClose0(AbstractChannel.java:772)
        at io.netty.channel.AbstractChannel$AbstractUnsafe.close(AbstractChannel.java:749)
        at io.netty.channel.AbstractChannel$AbstractUnsafe.close(AbstractChannel.java:620)
        at io.netty.channel.nio.NioEventLoop.closeAll(NioEventLoop.java:769)
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526)
        at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
        at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
        at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
        at java.base/java.lang.Thread.run(Thread.java:829)
2021-07-30 22:06:39,597 INFO [ZooKeeperClient Kafka server] Closed. (kafka.zookeeper.ZooKeeperClient) [main]
2021-07-30 22:06:39,609 ERROR Fatal error during KafkaServer startup. Prepare to shutdown (kafka.server.KafkaServer) [main]
kafka.zookeeper.ZooKeeperClientTimeoutException: Timed out waiting for connection while in state: CONNECTING
        at kafka.zookeeper.ZooKeeperClient.$anonfun$waitUntilConnected$3(ZooKeeperClient.scala:271)
        at kafka.zookeeper.ZooKeeperClient.waitUntilConnected(ZooKeeperClient.scala:267)
        at kafka.zookeeper.ZooKeeperClient.<init>(ZooKeeperClient.scala:125)
        at kafka.zk.KafkaZkClient$.apply(KafkaZkClient.scala:1948)
        at kafka.server.KafkaServer.createZkClient$1(KafkaServer.scala:431)
        at kafka.server.KafkaServer.initZkClient(KafkaServer.scala:456)
        at kafka.server.KafkaServer.startup(KafkaServer.scala:191)
        at kafka.Kafka$.main(Kafka.scala:109)
        at kafka.Kafka.main(Kafka.scala)
2021-07-30 22:06:39,615 INFO shutting down (kafka.server.KafkaServer) [main]
2021-07-30 22:06:39,629 INFO App info kafka.server for 0 unregistered (org.apache.kafka.common.utils.AppInfoParser) [main]
2021-07-30 22:06:39,630 INFO shut down completed (kafka.server.KafkaServer) [main]
2021-07-30 22:06:39,630 ERROR Exiting Kafka. (kafka.Kafka$) [main]
2021-07-30 22:06:39,632 INFO shutting down (kafka.server.KafkaServer) [kafka-shutdown-hook]

[networkingana]

[root@master strimzi-0.24.0]# kubectl describe pod/my-cluster-kafka-0 -n kafka
Name:         my-cluster-kafka-0
Namespace:    kafka
Priority:     0
Node:         node1/192.168.100.27
Start Time:   Fri, 30 Jul 2021 18:05:42 -0400
Labels:       app.kubernetes.io/instance=my-cluster
              app.kubernetes.io/managed-by=strimzi-cluster-operator
              app.kubernetes.io/name=kafka
              app.kubernetes.io/part-of=strimzi-my-cluster
              controller-revision-hash=my-cluster-kafka-69d78d5b56
              statefulset.kubernetes.io/pod-name=my-cluster-kafka-0
              strimzi.io/cluster=my-cluster
              strimzi.io/kind=Kafka
              strimzi.io/name=my-cluster-kafka
Annotations:  cni.projectcalico.org/podIP: 10.0.166.150/32
              cni.projectcalico.org/podIPs: 10.0.166.150/32
              strimzi.io/broker-configuration-hash: 5822b2bb37416ead07cedf790f8bb210f8c60035e10dda7f262722113c43a515d133c3868e48132d
              strimzi.io/clients-ca-cert-generation: 0
              strimzi.io/cluster-ca-cert-generation: 0
              strimzi.io/generation: 0
              strimzi.io/inter-broker-protocol-version: 2.8
              strimzi.io/kafka-version: 2.8.0
              strimzi.io/log-message-format-version: 2.8
              strimzi.io/logging-appenders-hash: e893ac9f32f7958042d1c82477e3484b0f8ebad5
              strimzi.io/storage: {"type":"persistent-claim","size":"1Gi","class":"glusterfs-storage"}
Status:       Running
IP:           10.0.166.150
IPs:
  IP:           10.0.166.150
Controlled By:  StatefulSet/my-cluster-kafka
Init Containers:
  kafka-init:
    Container ID:  docker://d9a8497943b17937996d05558b3bc6434530ddbcdb890a1c67574a1fd7c5fe2a
    Image:         quay.io/strimzi/operator:0.24.0
    Image ID:      docker-pullable://quay.io/strimzi/operator@sha256:786ea69c77e0651f009a73a8b3ce0b3d41b3cd71b917dc0d9bdc61c725f494a6
    Port:          <none>
    Host Port:     <none>
    Args:
      /opt/strimzi/bin/kafka_init_run.sh
    State:          Terminated
      Reason:       Completed
      Exit Code:    0
      Started:      Fri, 30 Jul 2021 18:05:45 -0400
      Finished:     Fri, 30 Jul 2021 18:05:50 -0400
    Ready:          True
    Restart Count:  0
    Limits:
      cpu:     1
      memory:  256Mi
    Requests:
      cpu:     100m
      memory:  128Mi
    Environment:
      NODE_NAME:          (v1:spec.nodeName)
      EXTERNAL_ADDRESS:  TRUE
    Mounts:
      /opt/kafka/init from rack-volume (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-thnpq (ro)
Containers:
  kafka:
    Container ID:  docker://17e09c77ac76740fbd5dcd520840b707a61ad3278fb717155b12445b3141de78
    Image:         quay.io/strimzi/kafka:0.24.0-kafka-2.8.0
    Image ID:      docker-pullable://quay.io/strimzi/kafka@sha256:fbb08410d9595029bc4a02ed859971264e6ce2dc85dd6a9855eaa7bb58b52a25
    Ports:         9090/TCP, 9091/TCP, 9092/TCP, 9093/TCP, 9094/TCP
    Host Ports:    0/TCP, 0/TCP, 0/TCP, 0/TCP, 0/TCP
    Command:
      /opt/kafka/kafka_run.sh
    State:          Waiting
      Reason:       CrashLoopBackOff
    Last State:     Terminated
      Reason:       Error
      Exit Code:    1
      Started:      Fri, 30 Jul 2021 18:07:13 -0400
      Finished:     Fri, 30 Jul 2021 18:07:43 -0400
    Ready:          False
    Restart Count:  3
    Liveness:       exec [/opt/kafka/kafka_liveness.sh] delay=15s timeout=5s period=10s #success=1 #failure=3
    Readiness:      exec [test -f /var/opt/kafka/kafka-ready] delay=15s timeout=5s period=10s #success=1 #failure=3
    Environment:
      KAFKA_METRICS_ENABLED:         false
      STRIMZI_KAFKA_GC_LOG_ENABLED:  false
      KAFKA_HEAP_OPTS:               -Xms128M
    Mounts:
      /opt/kafka/broker-certs from broker-certs (rw)
      /opt/kafka/client-ca-certs from client-ca-cert (rw)
      /opt/kafka/cluster-ca-certs from cluster-ca (rw)
      /opt/kafka/custom-config/ from kafka-metrics-and-logging (rw)
      /opt/kafka/init from rack-volume (rw)
      /tmp from strimzi-tmp (rw)
      /var/lib/kafka/data from data (rw)
      /var/opt/kafka from ready-files (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-thnpq (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             False
  ContainersReady   False
  PodScheduled      True
Volumes:
  data:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  data-my-cluster-kafka-0
    ReadOnly:   false
  rack-volume:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:
    SizeLimit:  <unset>
  strimzi-tmp:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:     Memory
    SizeLimit:  <unset>
  cluster-ca:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  my-cluster-cluster-ca-cert
    Optional:    false
  broker-certs:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  my-cluster-kafka-brokers
    Optional:    false
  client-ca-cert:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  my-cluster-clients-ca-cert
    Optional:    false
  kafka-metrics-and-logging:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      my-cluster-kafka-config
    Optional:  false
  ready-files:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:     Memory
    SizeLimit:  <unset>
  kube-api-access-thnpq:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   Burstable
Node-Selectors:              <none>
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason       Age                  From               Message
  ----     ------       ----                 ----               -------
  Normal   Scheduled    2m42s                default-scheduler  Successfully assigned kafka/my-cluster-kafka-0 to node1
  Warning  FailedMount  2m41s                kubelet            MountVolume.SetUp failed for volume "kafka-metrics-and-logging" : failed to sync configmap cache: timed out waiting for the condition
  Normal   Pulled       2m40s                kubelet            Container image "quay.io/strimzi/operator:0.24.0" already present on machine
  Normal   Created      2m39s                kubelet            Created container kafka-init
  Normal   Started      2m39s                kubelet            Started container kafka-init
  Warning  BackOff      102s (x2 over 104s)  kubelet            Back-off restarting failed container
  Normal   Killing      72s                  kubelet            Container kafka failed liveness probe, will be restarted
  Warning  Unhealthy    72s (x3 over 2m12s)  kubelet            Liveness probe failed:
  Warning  Unhealthy    72s (x3 over 2m12s)  kubelet            Readiness probe failed:
  Normal   Created      71s (x4 over 2m33s)  kubelet            Created container kafka
  Normal   Started      71s (x4 over 2m33s)  kubelet            Started container kafka
  Normal   Pulled       71s (x4 over 2m33s)  kubelet            Container image "quay.io/strimzi/kafka:0.24.0-kafka-2.8.0" already present on machine

[scholzj]

Having to do things such as enable firewalls suggests the cluster is probably not correctly set up? But I'm not sure I would expect that to have some impact - I'm not sure any of this goes over 30000-32767 ports which are normally node ports.

The log suggests that the ZooKeeper cluster did not formed and the ZooKeeper nodes cannot properly talk with each other. But I do not know why.

[networkingana]

report-07-10-2021_22-58-33.zip

Still struggling with this. Can you please take a look?

Thank you

[scholzj]

Isn't that the same one which you shared a month ago and which didn't seem to have anything pointing to some specific issue?

[networkingana]

Sorry, you are right, I attached the wrong archive
report-31-10-2021_09-49-47.zip

[scholzj]

I think it looks still the same. The ZooKeeper do not seem to be able to communicate with each other so they fail to form the cluster. The operator log also seems to have a lot of issues which seem to be network related. But I'm not really a networking expert, so I don' really know what would be the cause. It can be probably all kind of things - firewalls, SDN issue etc. Sorry, I don't think I can provide much help for this.

[networkingana]

I scheduled all zookeepers pods on the same node, and the zookeepers cluster is fine. Not sure if calico is the issue or I'm missing some configuration in calico, or eventually there is some network configuration in our data center that prevents this communication.

I did a network trace and as I can see, the one zookeeper/node sends syn to 3888 port but from the tunnel interface IP address not the actual IP address of the pod.

[scholzj]

I scheduled all zookeepers pods on the same node, and the zookeepers cluster is fine. Not sure if calico is the issue or I'm missing some configuration in calico, or eventually there is some network configuration in our data center that prevents this communication.

I use Calico on my cluster and do had any problems - so I do not think Calico in general is an issue. But it has lot of options, so there might be some issues with your config / cluster maybe? It could be also that Calico works fine, but some firewall on the worker node level doesn't allow the communication through to other nodes? I had this kind of problem in the past while playing with some stuff (since it was not really a production-grade environment, I just disabled the firewall which might not be the option for you).