Support log compaction for streams

[nvx]

Is your feature request related to a problem? Please describe.

Kafka has a handy feature called log compaction where the most recent message with a given "key" is retained when compaction of old log segments occurs. This allows for infinite duration retention without needing infinite disk (as long as there is enough disk to hold the latest message for each key, plus any non-compacted segments).

This would really close the gap in terms of RabbitMQ vs Kafka featuresets.

Describe the solution you'd like

Similar functionality to Kafka

Describe alternatives you've considered

No response

Additional context

No response

[michaelklishin]

This feature is known as "retained messages" in the messaging world.

This is a very brief problem description, so belongs to a discussion.

[michaelklishin]

There's ongoing work on stream filtering, most likely to ship in RabbitMQ 4.0 in 2024. Message retention will very likely require that to be finished first.

[michaelklishin]

Apparently there are two complications even on the message filtering branch:

• The original key is not stored in segment files, only a "filtering tag"
• Individual messages are fairly opaque to the storage format

[adefilippo83]

Looks like stream filtering is available now.
Any chance to get this feature available?

[kjnilsson]

There are no immediate plans to implement log compaction for streams. We know it is a potentially useful feature but it's a big change and non trivial to do efficiently and well.