updating operator vsersion from 0.26.0 to 0.27.0 did not update kafkacluster

[sorushsaghari]

Describe the bug
after updating kafka operator in one my clusters all kafka cluster were updated except one of them ant its entity operator version is 0.26.0, just its zookeepers were rolled out.

Expected behavior
expected all clusters to get rolled out and updated (specially entity opearator to 0.27.0)

Environment (please complete the following information):

• Strimzi version: 0.27.0
• Installation method: YAML files
• Kubernetes cluster: Kubernetes 1.21
• Infrastructure: self hosted

YAML files and logs
operator log

2022-01-03 09:17:58 INFO  AbstractOperator:373 - Reconciliation #1(watch) Kafka(name-prod/name-kafka-cluster): Reconciliation is in progress
2022-01-03 09:18:58 INFO  AbstractOperator:373 - Reconciliation #1(watch) Kafka(name-prod/name-kafka-cluster): Reconciliation is in progress
2022-01-03 09:18:58 INFO  ClusterOperator:121 - Triggering periodic reconciliation for namespace name-prod
2022-01-03 09:18:58 INFO  AbstractOperator:226 - Reconciliation #99(timer) KafkaRebalance(name-prod/name-kafka-rebalance): KafkaRebalance name-kafka-rebalance will be checked for creation or modification
2022-01-03 09:18:58 INFO  KafkaRebalanceAssemblyOperator:359 - Reconciliation #99(timer) KafkaRebalance(name-prod/name-kafka-rebalance): Rebalance action from state [Ready]
2022-01-03 09:18:58 WARN  KafkaRebalanceAssemblyOperator:1021 - Reconciliation #99(timer) KafkaRebalance(name-prod/name-kafka-rebalance): strimzi.io/rebalance: Ignore annotation none={}
2022-01-03 09:18:58 INFO  KafkaRebalanceAssemblyOperator:386 - Reconciliation #99(timer) KafkaRebalance(name-prod/name-kafka-rebalance): State updated to [Ready] with annotation strimzi.io/rebalance=null 
2022-01-03 09:18:58 INFO  AbstractOperator:517 - Reconciliation #99(timer) KafkaRebalance(name-prod/name-kafka-rebalance): reconciled

kafka yaml

apiVersion: kafka.strimzi.io/v1beta2
kind: Kafka
spec:
  kafka:
    version: 3.0.0
    replicas: 7
    listeners:
      - name: plain
        port: 9092
        type: internal
        tls: false
      - name: external
        port: 9094
        type: loadbalancer
        tls: false
        configuration:
          brokers:
            - broker: 0
              advertisedHost: ip
              advertisedPort: port
            - broker: 1
              advertisedHost: ip
              advertisedPort: port
            - broker: 2
              advertisedHost: ip
              advertisedPort: port
            - broker: 3
              advertisedHost: ip
              advertisedPort: port
            - broker: 4
              advertisedHost: ip
              advertisedPort: port
            - broker: 5
              advertisedHost: ip 
              advertisedPort: port
            - broker: 6
              advertisedHost: ip
              advertisedPort: port
    config:
      auto.create.topics.enable: false
      inter.broker.protocol.version: "3.0"
      log.message.format.version: "3.0"
      offsets.topic.replication.factor: 3
      transaction.state.log.min.isr: 2
      transaction.state.log.replication.factor: 3
    storage:
      class: storage-classname
      deleteClaim: true
      size: 4.1Ti
      type: persistent-claim
    jvmOptions:
      -Xms: 8192m
      -Xmx: 8192m
    resources:
      limits:
        cpu: "6"
        ephemeral-storage: 4Gi
        memory: 32Gi
      requests:
        cpu: "6"
        ephemeral-storage: 4Gi
        memory: 32Gi
    metricsConfig:
      type: jmxPrometheusExporter
      valueFrom:
        configMapKeyRef:
          key: kafka-metrics-config.yml
          name: name-kafka-config
    template:
      pod:
        affinity:
          nodeAffinity:
            requiredDuringSchedulingIgnoredDuringExecution:
              nodeSelectorTerms:
                - matchExpressions:
                    - key: nodepool
                      operator: In
                      values:
                        - rdm
          podAntiAffinity:
            preferredDuringSchedulingIgnoredDuringExecution:
              - podAffinityTerm:
                  labelSelector:
                    matchLabels:
                      strimzi.io/cluster: name-kafka-cluster
                      strimzi.io/name: name-kafka-cluster-kafka
                  topologyKey: kubernetes.io/hostname
                weight: 100
        tolerations:
          - effect: NoSchedule
            key: nodepool
            operator: Equal
            value: rdm
  zookeeper:
    replicas: 3
    storage:
      class: rawfile-1
      deleteClaim: true
      size: 2Gi
      type: persistent-claim
    metricsConfig:
      type: jmxPrometheusExporter
      valueFrom:
        configMapKeyRef:
          key: zookeeper-metrics-config.yml
          name: name-kafka-config
    template:
      pod:
        affinity:
          podAntiAffinity:
            preferredDuringSchedulingIgnoredDuringExecution:
              - podAffinityTerm:
                  labelSelector:
                    matchLabels:
                      strimzi.io/cluster: name-kafka-cluster
                      strimzi.io/name: name-kafka-cluster-zookeeper
                  topologyKey: kubernetes.io/hostname
                weight: 100
  entityOperator:
    topicOperator: {}
    userOperator: {}
  cruiseControl:
    brokerCapacity:
      disk: 4.1Ti
      cpuUtilization: 70
      inboundNetwork: 1GB/s
    config:
      cpu.balance.threshold: 2
      cpu.capacity.threshold: 0.9
      cruise.control.metrics.topic.min.insync.replicas: 2
      cruise.control.metrics.topic.num.partitions: 2
      cruise.control.metrics.topic.replication.factor: 3
      default.goals: com.linkedin.kafka.cruisecontrol.analyzer.goals.MinTopicLeadersPerBrokerGoal,com.linkedin.kafka.cruisecontrol.analyzer.goals.ReplicaCapacityGoal,com.linkedin.kafka.cruisecontrol.analyzer.goals.DiskCapacityGoal,com.linkedin.kafka.cruisecontrol.analyzer.goals.NetworkInboundCapacityGoal,com.linkedin.kafka.cruisecontrol.analyzer.goals.NetworkOutboundCapacityGoal,com.linkedin.kafka.cruisecontrol.analyzer.goals.CpuCapacityGoal,com.linkedin.kafka.cruisecontrol.analyzer.goals.ReplicaDistributionGoal,com.linkedin.kafka.cruisecontrol.analyzer.goals.PotentialNwOutGoal,com.linkedin.kafka.cruisecontrol.analyzer.goals.DiskUsageDistributionGoal,com.linkedin.kafka.cruisecontrol.analyzer.goals.NetworkInboundUsageDistributionGoal,com.linkedin.kafka.cruisecontrol.analyzer.goals.NetworkOutboundUsageDistributionGoal,com.linkedin.kafka.cruisecontrol.analyzer.goals.CpuUsageDistributionGoal,com.linkedin.kafka.cruisecontrol.analyzer.goals.TopicReplicaDistributionGoal,com.linkedin.kafka.cruisecontrol.analyzer.goals.LeaderReplicaDistributionGoal,com.linkedin.kafka.cruisecontrol.analyzer.goals.LeaderBytesInDistributionGoal
      hard.goals: com.linkedin.kafka.cruisecontrol.analyzer.goals.NetworkInboundCapacityGoal,com.linkedin.kafka.cruisecontrol.analyzer.goals.ReplicaDistributionGoal,com.linkedin.kafka.cruisecontrol.analyzer.goals.MinTopicLeadersPerBrokerGoal,com.linkedin.kafka.cruisecontrol.analyzer.goals.ReplicaCapacityGoal,com.linkedin.kafka.cruisecontrol.analyzer.goals.DiskCapacityGoal,com.linkedin.kafka.cruisecontrol.analyzer.goals.CpuCapacityGoal
      leader.replica.count.balance.threshold: 1.1
      replica.count.balance.threshold: 1.1
      sampling.allow.cpu.capacity.estimation: true
      topic.replica.count.balance.max.gap: 10
      topic.replica.count.balance.threshold: 1.8
      webserver.http.cors.enabled: false
  kafkaExporter:
    groupRegex: .*
    topicRegex: .*
metadata:
  name: name-kafka-cluster
  namespace: name-prod

[scholzj]

Can you share the full log (ideally at DEBUG level) and not just few lines from it?

[sorushsaghari]

scaling up kafka by adding a broker doesnt work when storageclass is configured to WaitForFirstConsumer, it creates the pvc but the pvc status changes to pending because it waits for pod to be scheduled, and it doesnt scale up the statefulSet because it waits for the pvc

[sorushsaghari]

is there any other usefull information that i can share with you ??

[scholzj]

I don't think there is anything what waits for the PVCs to be bound in the operator. And there is also nothing in the StatefulSet what would wait for it. I use WaitForFirstConsumer normally in my clusters without any problems. So I do not think this is the issue here.

Are you scaling the Kafka cluster during the upgrade?

[sorushsaghari]

about the cluster1, its zookeepers rolled out but kafka brokers didn't roll out.
and about scaling up i have tested in on another cluster which was upgraded successfully

[scholzj]

Sorry, I'm now completely lost how these things relate to each other.