[Question] The best way to efficiently produce messages while ensuring their order

[lcy0321]

Hi Experts,

We are developing a service that requires the Kafka producer to support the following features:

• At least once delivery, no message loss is acceptable
• No re-ordering of messages
• Retry as much as possible

I've read the INTRODUCTION.md of librdkafka and realize that the Idempotent Producer is close to our needs. However, there seems to be a very small chance that one or more messages in a queue could be lost due to a timeout. Is my understanding correct?

Currently, we are meeting these requirements by sending the messages synchronously, like this:

for {
    // Some retry logic here
    ...

    err := sendMessage(p, message)
    if err != nil {
        continue
    }
    break
}

func sendMessage(p *kafka.Producer, message *kafka.Message) error {
    deliveryChan := make(chan kafka.Event)
    err := p.Produce(message, deliveryChan)
    if err != nil {
        return err
    }
    e := <-deliveryChan
    // Check if the message was delivered; if not, return an error
    ...
    return nil
}

But producing messages one by one synchronously is heavily affecting the throughput.

Could you please provide recommendations for more efficient ways to meet these requirements?