Skip to content

Latest commit

 

History

History
161 lines (118 loc) · 6.53 KB

README.md

File metadata and controls

161 lines (118 loc) · 6.53 KB
Raw

kafka-scheduler

A scalable, distributed Kafka message scheduler with only a Kafka cluster as a dependency.

Why?

Some applications or microservices run without any database or external dependency only sending and receiving messages through Kafka.

If they have to schedule some tasks for example

  • Send out a reminder email every day if a user has not verified their email
  • Generate an export every 30 minutes
  • ...

the application has to make sure that the task is executed exactly once. Even if the application moves to another node on a Kubernetes cluster.

The usual way to handle persistent jobs would be to use Quartz, Hangfire or Jobrunr. But all of them depend on a database.

Why should I have to introduce a database into my stack if I can "just" stay with Kafka?

How?

Kafka-scheduler lets you schedule a Kafka message with a desired key, value and headers and when you want to receive them on which topic. This way your application receives the message as a trigger and can execute the business logic on the desired schedule.

OneTimeCommand

A fire-and-forget command will be triggered at a certain point in the future.

sequenceDiagram
    autonumber
    note right of Application: Topic: one-time-commands<br>Headers<br>kafka-scheduler-key: myKey<br>kafka-scheduler-topic: mytopic<br>kafka-scheduler-when:2011-12-03T10:15:30+01:00
    Application->>Kafka-Scheduler: key:value
    Kafka-Scheduler->>Kafka-Scheduler: Schedule
    note right of Application: Topic: mytopic
    Kafka-Scheduler->>Application: mykey:value
Loading

Headers

Header Description
kafka-scheduler-key The kafka message key used as a unique id so it can be cancelled
kafka-scheduler-value The Kafka message value you want to receive back
kafka-scheduler-when A ISO-6801 configured Timestamp

Key and Value

Field Description
Key The kafka message key used as a unique id so it can be cancelled
Value The Kafka message value you want to receive back

RecurringCommand

A command which will be triggered on a CRON schedule

sequenceDiagram
    autonumber
    note right of Application: Topic: recurring-commands<br>Headers<br>kafka-scheduler-key: myKey<br>kafka-scheduler-topic: mytopic<br>kafka-scheduler-cron: */5 * * * * *
    Application->>Kafka-Scheduler: key:value
    Kafka-Scheduler->>Kafka-Scheduler: Schedule
    note right of Application: Topic: mytopic
    Kafka-Scheduler->>Application: mykey:value
    note right of Application: Topic: mytopic
    Kafka-Scheduler->>Application: mykey:value
    note right of Application: Topic: mytopic
    Kafka-Scheduler->>Application: mykey:value
Loading

Headers

Header Description
kafka-scheduler-key The kafka message key used as a unique id so it can be cancelled
kafka-scheduler-value The Kafka message value you want to receive back
kafka-scheduler-cron A cron expression with extended syntax support]

Key and Value

Field Description
Key The kafka message key used as a unique id so it can be cancelled
Value The Kafka message value you want to receive back

FixedRateCommand

A command which a defined duration between runs

sequenceDiagram
    autonumber
    note right of Application: Topic: fixed-rate-commands<br>Headers<br>kafka-scheduler-key: myKey<br>kafka-scheduler-topic: mytopic<br>kafka-scheduler-period: PT5S
    Application->>Kafka-Scheduler: key:value
    Kafka-Scheduler->>Kafka-Scheduler: Schedule
    note right of Application: Topic: mytopic
    Kafka-Scheduler->>Application: mykey:value
    note right of Application: Topic: mytopic
    Kafka-Scheduler->>Application: mykey:value
    note right of Application: Topic: mytopic
    Kafka-Scheduler->>Application: mykey:value
Loading

Headers

Header Description
kafka-scheduler-key The kafka message key used as a unique id so it can be cancelled
kafka-scheduler-value The Kafka message value you want to receive back
kafka-scheduler-period A ISO-8601 Duration

Key and Value

Field Description
Key The kafka message key used as a unique id so it can be cancelled
Value The Kafka message value you want to receive back

Cancelling Commands

All commands can be cancelled by sending a Kafka tombstone message with the same key which was used to schedule the command.

sequenceDiagram
    autonumber
    note right of Application: Topic: one-time-commands<br>Headers<br>kafka-scheduler-key: myKey<br>kafka-scheduler-topic: mytopic<br>kafka-scheduler-when:2011-12-03T10:15:30+01:00
    Application->>Kafka-Scheduler: key:value
    Kafka-Scheduler->>Kafka-Scheduler: Schedule
    note right of Application: Topic: one-time-commands<br>Headers
    Application->>Kafka-Scheduler: key:null
    Kafka-Scheduler->>Kafka-Scheduler: Cancel scheduled command
Loading

Client SDK

TBD

Detailed Documentation

TBD

Other Kafka scheduler implementations

kafka-message-scheduler - Awesome implementation by @etf in Go. Only supports OneTimeCommands but has memory optimizations and an Admin UI.

high-available-task-scheduling - Implementation by @cbenaveen using Kafka Streams.

Acknowledgements

@rdehuyss for creating JobRunr the easiest to use scheduling library in the JVM space.