Distributed Approach: 3% of messages lost

[nelsou]

Hello guys,

When we have less than 5.000 messages / minutes, it works fine ...
But as soon as the traffic increase, I got some message lost (~3%)

Description

1. I pushed 40.000 messages in the queue.
2. The RabbitMQ receiver (Distributed Approach) received the 40.000 messages verified by summing the 9 log lines corresponding to
   https://github.com/Stratio/Spark-RabbitMQ/blob/8d485890e7e0ea6ee03faddfd1ad80e10840af14/src/main/scala/org/apache/spark/streaming/rabbitmq/distributed/RabbitMQRDD.scala#L217
3. but than, if I count the number of message in my RDD, rabbitMQStream.count() I got only 39.843 messages ...
   No error in the logs ! I have no idea where my missing lines are :/

I tried to play with all the options (ackType, fairDispatch, prefetchCount, maxReceiveTime, maxRate, levelParallelism, maxMessagesPerPartition & backpressure). None fixed the problem.

Has anyone encountered the same error and how did you fix it ?

BTW, the old Receiver-based Approach works fine !!!
Duplicate of #57 ?

[nelsou]

• actType = basic
• fairDispatch = true
• prefetchCount = 100

fixes the problem