Question about dictionary option

[ethanresnick]

Hi, thanks for this awesome package!

My understanding of zstd is that messages compressed with a dictionary can only be decompressed if that dictionary is available at decompression time, and that the dictionary is not embedded in the compressed data, so it must be provided out-of-band (with the options this package supports). Is that right?

If so, I'm very confused how a dictionary would work with Kafka, because my understanding is that the broker needs to decompress the batch of compressed messages it receives to (e.g.) find the boundaries between messages, verify the offsets, and (in the case of Confluent Schema Registry) validate the messages against the topic's schema.

Am I misunderstanding the extent to which the broker decompresses/processes/validates the incoming messages? If not, how does this all work with dictionaries + zstd?

Thanks!