Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Document consumption modes #21

Closed
bartelink opened this issue Aug 19, 2019 · 1 comment
Closed

Document consumption modes #21

bartelink opened this issue Aug 19, 2019 · 1 comment

Comments

@bartelink
Copy link
Collaborator

bartelink commented Aug 19, 2019

When consuming via Kafka, the high level modes include:

Representation Description Implemented In Notes
Serial Messages Messages are individual events tagged with a key. No parallelism is or can be employed. Some current usage uses this pattern but should be deprecated
Batched Consume, processing batches for a given partition together. Checkpoint when batch completed Jet.ConfluentKafka.FSharp
Stream Event Spans Messages are spans of events from a specified Stream Name (also the Key), with a specified index Propulsion.Kafka Implemented in StreamedConsumer and proProjector + proConsumer template. Allows production in parallel without max.in.flight=1. Consumer deduplicates
Stream Summary Events Messages contain a rendition of a summary as at a particular version. Version is monotonic and consumer is free to / should only use the most recent one TODO build a dotnet-templates example and provide default wiring in StreamedConsumer. Consumer can deduplicate by taking highest version (probably tracking last-seen per stream to deduplicate). Ref http://verraes.net/2019/05/patterns-for-decoupling-distsys-summary-event/
Ordered Summaries Messages represent state at a point in time. The topic's ordering within a key defines the 'version'. Consumer needs to group by key, dropping all but the newest. There's no way to de-duplicate double sends. Stream Summary Events are preferred, but if the producer can guarantee the ordering when producing, it's a perfectly reasonable representation.

Surely there's a glossary for this sort of thing somewhere? please comment!

@bartelink
Copy link
Collaborator Author

Broader information in #200

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant