Is there a way to implement kafka-broker autoscaling with Strimzi Operator.

[shiva-appani-hash]

Hello Team,

Could anyone assist with implementing Kafka broker autoscaling using the Strimzi Operator? Specifically, I'm looking for ways to enable autoscaling for a Kafka cluster managed by Strimzi, where the number of replicas can dynamically adjust between 5 and 10 pods based on consumption.

I am using Strimzi within an AKS environment and am familiar with autoscaling mechanisms like Cluster Autoscaler, HPA, and VPA. However, I couldn't find any specific information on autoscaling Kafka brokers with Strimzi.

Is there a way to achieve this with the Strimzi Operator? If not, is there any plan to include this feature in the roadmap, or is it considered impractical for Kafka?

Thank you !

[scholzj]

If you use KafkaNodePools, you can scale tham as any other Kubernetes resource. Then you need to do rebalancing to move that data - that will be automated one day, but currently you have to do it yourself. But Kafka will never scale-up and down for few minutes - it will never have that kind of elasticity (or at least not anytime soon). So there migth be situations where it adds some value, but also many where it would not.

[shiva-appani-hash]

Thank you @scholzj !
Do you mean after scaling Kafka cluster by adding new brokers, we need to rebalance the cluster using Cruise Control to ensure the existing topic partitions are redistributed to the new brokers, optimizing the performance and resource utilization of your Kafka cluster.

[ppatierno]

Yes exactly. After scaling up the cluster, the newly added brokers will be used for new topics' partitions only not for the existing ones. So you would like to run a rebalancing for using the newly added brokers by redistributing the already existing partitions. Right now it's a manual process you need to create a KafkaRebalance resource in add-brokers mode (see more on the documentation).
We are working on an auto-rebalancing mechanism (optional) where this manual process is not needed anymore.

[shiva-appani-hash]

Thank you @ppatierno !
Is KafkaNodePool feature still in preview.
As per below documentation, I see it as an Alpha feature, could you confirm, please.

https://strimzi.io/blog/2023/08/14/kafka-node-pools-introduction/
https://strimzi.io/docs/operators/0.36.1/full/deploying#proc-migrating-clusters-node-pools-str

[ppatierno]

It's not in preview, it's enabled by default since 0.41.0. You are poiting to a very old blog post and very old Strimzi version (0.36.0)