Apache Pekko based Streaming application is idle and not consuming from kafka topic post rebalancing

[dahiyahimanshu]

Issue: Apache Pekko-based Streaming Application Idle After Kafka Rebalancing

Description: Our application, which runs on Kubernetes with KEDA scaling based on consumer group lag, has encountered issues post-rebalancing. During testing, the application scales up and down as expected. However, after a rebalance, certain Kafka topic partitions are no longer being consumed.

We observed that while the consumer is assigned multiple partitions, some partitions are paused and never resumed, resulting in unprocessed messages. The application only resumes consuming these pending messages when there is another surge of input messages, which triggers pod scaling and, subsequently, another rebalance.

[mdedetrich]

Can you isolate whether this is a pekko-connectors-kafka issue or whether it's due to the underlying kafka client?

[pjfanning]

also, we need to know which version of pekko-connectors-kafka that you are using - it is recommended to use pekko-connectors-kafka 1.1.0 if you can

[dahiyahimanshu]

it is exactly the same issue as described here. akka/alpakka-kafka#194

it is not underlying client issue but something to do with rebalance which pauses some partition but never resume them again, i checked KafkaConsumerActor file which has poll method where for some partitions we are not calling resume method. i checked this behaviour in debug logs.

i am using pekko-connectors-kafka 1.0.0 version. actually i am running with scala 2.12. Can i use 1.1.0 with scala 2.12?