Is there an option to slow down the pulling of messages from kafka topic when the consumer queue is full.

[ddaeyongee]

hi.
My name is Taeyong Kang. I am working in the software architecture organization of LG CNS. I have a question while doing this MSA project, so I'm posting an issue.

I am looking at the potential issue of back-pressure i.e. fast producer and slow consumer. In this case, producer will be the component that will read messages from kafka topic.
Is there an option to slow down the pulling of messages from kafka topic when the consumer queue is full.
If yes, how is it effective to apply?

To implement this, I am using TPL data flow BufferBlock class. BufferBlock has a bounded capacity. If the component is not able to post messages to the buffer block, I want to slow down the kafka component that is pulling messages from topic. Please advice!

[cer]

See

eventuate-messaging-kafka/eventuate-messaging-kafka-spring-basic-consumer/src/main/java/io/eventuate/messaging/kafka/spring/basic/consumer/EventuateKafkaConsumerSpringConfigurationProperties.java

Line 13 in 2e95af2

private BackPressureConfig backPressure = new BackPressureConfig();

and

eventuate-messaging-kafka/eventuate-messaging-kafka-integration-test/src/main/resources/application.properties

Lines 12 to 13 in 2e95af2

	eventuate.local.kafka.consumer.backPressure.low=5
	eventuate.local.kafka.consumer.backPressure.high=100

The goal of the BackPressureConfig mechanism to 'slow down' the consumer so that it doesn't read (into memory) a large number of messages from a topic.