Strimzi Kafka brokers rolled due to [Pod has old revision]

[jslusher]

We're using Strimzi version 0.42.0 with Kafka version 3.7.1

Seemingly out of the blue, the brokers on our Kafka cluster got rolled. I see these in the operator logs:

2024-09-27 20:46:19 INFO  KafkaRoller:481 - Reconciliation #20846(timer) Kafka(strimzi-kafka-cluster/kafka-prod): Rolling Pod kafka-prod-kafka-0/0 due to [Pod has old revision]

We haven't made any changes to the StrimziPodSet configuration today. We did increase the size of the PVCs earlier in the week and not all of the pods restarted at that time, but 5 days later seems like a long time for the operator to finally discover something like that and take action.

Is there any other reason this might have happened? Is there any concrete way for me to find out? I'm not seeing much else that's helpful in the operator logs.

[scholzj]

I don't think the PVC size change would cause this as the Pod has no reference to the PVC size normally. But it is not always simple to say what caused it. The Pod revision is generated based on the Pod definition. The includes the different options of the Pod itself (all the things that the Kubernetes Pod might have). But it also includes some special annotations to indicate that some Kafka configuration that cannot by dynamically configured changed, some special logging changes such as logger configuration or for example new certificates. Some of these things would be obvious from the operator log. Some might be less.

If you would have this problem more often, the easiest thing to debug this is to just watch the Pod or StrimziPodSet YAMLs and dump them into a file so that you can compare what changes between the restarts.

[jslusher]

Based on what I can see in the logs this hasn't happened before or since recently, so without knowing what might have changed when it did actually happened, I'm not sure it's possible to say what might have caused this. I'll keep an eye out for it if it happens again, but it seems like a blip so far.