Awesome Open-Source Contribs for Apache Kafka

A curated list of awesome open-source frameworks, libraries, tools and examples for the Apache Kafka project.

Contributing to this project is easy!Just send a pull request and make the list growth!

Table of Contents

• Change Data Capture
• Clients
• CLI / Tools
• Configuration
• Examples
• Infrastructure
• Kafka Streams
• Performance
• Query/SQL
• Replication
• REST API
• Schema / Data
• Security / ACL
• Site Reliability Tools
• Test
• UI/Cluster Management
• Other Awesome Lists about Apache Kafka & Data Engineering
• License

Change Data Capture

• Debezium: Debezium is an open source project that provides a low latency data streaming platform for change data capture (CDC). Apache License 2.0

• Maxwell: Maxwell’s daemon, a mysql-to-json kafka producer. Apache License 2.0

Clients

• kafkajs: A modern Apache Kafka client for node.js MIT License

• kafka-clients-kotlin: This projects gives Kotlin bindings and several extensions for Apache Kafka Clients.

• Kafka Rust: Rust client for Apache Kafka. MIT License

• librdkafka: The Apache Kafka C/C++ library. 2-clause BSD license

• Sarama: Sarama is a Go library for Apache Kafka 0.8, and up. MIT License

CLI / Tools

• Kafkacat: Generic command line non-JVM Apache Kafka producer and consumer. librdkafka - Apache Kafka C driver library

• kafkacli: CLI and Go Clients to manage Kafka components (Kafka Connect & SchemaRegistry). Apache License 2.0

• Kafka Shell: ⚡A supercharged, interactive Kafka shell built on top of the existing Kafka CLI tools. Apache License 2.0

• kukulcan: A REPL for Apache Kafka. Apache License 2.0

• Zoe: The Kafka CLI for humans. MIT License

Configuration

• kafka-gitops: Manage Apache Kafka topics and generate ACLs through a desired state file. Apache License 2.0

• Kafka Specs: Tool to ease and automate Apache Kafka cluster configuration management. Apache License 2.0

• kattlo: Kattlo gives a way to manage topics, ACLs, users, schemas, ksqlDB, employing a evolutionary configuration. Easy to understand, easy to write and easy to apply. Apache License 2.0

• julie: An operational manager for Apache Kafka (Automation, GitOps, SelfService) MIT License

Examples

• Confluent demo-scene: 👾 Scripts and samples to support Confluent Demos and Talks.

• Confluent Kafka Streams Examples: Demo applications and code examples for Apache Kafka’s Streams API. Apache License 2.0

• Confluent Examples: Apache Kafka and Confluent examples and demos. Apache License 2.0

• kafka-examples: Snippets and small examples demonstrating kafka features and configs. Apache License 2.0

Infrastructure

• Confluent Docker Images: Docker images for Confluent Platform. [Apache License 2.0](Apache License 2.0)

• Confluent Platform Ansible: Ansible playbooks for the Confluent Platform. Apache License 2.0

• Strimzi: Strimzi provides a way to run an Apache Kafka cluster on Kubernetes in various deployment configurations. Apache License 2.0

Kafka Streams

• Azkarra Streams: Azkarra Streams is a lightweight Java framework which makes easy to develop and operate Kafka Streams applications. Apache License 2.0

• Faust: A library for building streaming applications in Python. BSD License

• GoKa: Goka is a compact yet powerful distributed stream processing library for Apache Kafka written in Go. BSD 3-Clause "New" or "Revised" License

• Kafka Streams CEP: Complex Event Processing on top of Kafka Streams (Apache License 2.0

• Kafka Streams .NET: .NET Stream Processing Library for Apache Kafka

• Kafka Streams Viz: A tool helps visualize stream topologies by generating nice looking diagrams from a kafka stream topology descriptions.

Performance

• kloadgen: KLoadGen is kafka load generator plugin for jmeter designed to work with AVRO schema Registries. MPL 2.0

Query/SQL

• ksqlDB: The event streaming database purpose-built for stream processing applications. Confluent Community License Agreement Version 1.0

• ksql-jdbc-driver: JDBC driver for Apache Kafka. Apache License 2.0

Replication

• Mirror Maker 2: MM2 leverages the Connect framework to replicate topics between Kafka clusters. Apache License 2.0

• Uber uReplicator: Improvement of Apache Kafka Mirrormaker. Apache License 2.0

REST API

• Kafka REST Proxy: Confluent REST Proxy for Kafka. Confluent Community License Agreement Version 1.0

Schema / Data

• Javro: Javro is an interface allowing you to write Avro schemas with ease, to validate it, and to preview it into JSON format with samples. Gpl 3.0

• Confluent Schema Registry: Confluent Schema Registry for Kafka. Confluent Community License Agreement Version 1.0

Security / ACL

• kafka-security-manager: Manage your Kafka ACL at scale. MIT License

• kafka-encryption: About Kafka End to End Encryption. Apache License 2.0

Site Reliability Tools

• Burrow(Linkedin): Kafka Consumer Lag Checking. Apache License 2.0

• Chaperone( Uber): As Kafka audit system, Chaperone monitors the completeness and latency of data stream.The audit metrics are persisted in database for Kafka users to quantify the loss of their topics if any. Apache License 2.0

• Cruise Control(Linkedin): Cruise-control is the first of its kind to fully automate the dynamic workload rebalance and self-healing of a kafka cluster.It provides great value to Kafka users by simplifying the operation of Kafka clusters. BSD 2-Clause "Simplified" License

• Doctor Kafka(Pinterest): DoctorKafka is a service for Kafka cluster auto healing and workload balancing. Apache License 2.0

• kafka-kit (Datadog): Kafka data mapping and recovery tools. Apache License 2.0

• Kafka Monitor(Linkedin): Xinfra Monitor monitors the availability of Kafka clusters by producing synthetic workloads using end-to-end pipelines to obtain derived vital statistics - E2E latency, service availability & message loss rate.It reassigns partition & trigger preferred leader election to ensure each broker acts as leader of at least 1 partition of monitor topic. Apache License 2.0

• kafka-monitoring-stack-docker-compose: Demonstration on how to monitor Kafka using Prometheus and Grafana. Apache License 2.0

Test

• Kafka Junit: This library wraps Kafka’s embedded test cluster, allowing you to more easily create and run integration tests using JUnit against a "real" kafka server running within the context of your tests.No need to stand up an external kafka cluster! BSD 3-Clause "New" or "Revised" License

• kafka-streams-test-utils: Testing Kafka Streams.Apache License 2.0

• Mocked Streams: Scala DSL for Unit-Testing Processing Topologies in Kafka Streams. Apache License 2.0

• Test Container for Apache Kafka: Testcontainers can be used to automatically instantiate and manage Apache Kafka containers.More precisely Testcontainers uses the official Docker images for Confluent OSS Platform. MIT License

UI/Cluster Management

• AKHQ: Kafka GUI for Apache Kafka to manage topics, topics data, consumers group, schema registry, connect and more. Apache License 2.0

• CMAK: CMAK (previously known as Kafka Manager) is a tool for managing Apache Kafka clusters.See below for details about the name change. Apache License 2.0

• Kafdrop – Kafka Web UI: Kafdrop is a web UI for viewing Kafka topics and browsing consumer groups.The tool displays information such as brokers, topics, partitions, consumers, and lets you view messages. Apache License 2.0

• Kafka Eagle: A easy and high-performance monitoring system, as well as offsets or metadata and other kafka information. Apache License 2.0

• Kafkawize: Kafkawize : A Self service Apache Kafka Topic Management tool/portal.A Web application which automates the process of creating and browsing Kafka topics, acls, schemas by introducing roles/authorizations to users of various teams of an org. Apache License 2.0

• klustr: Monitoring tool and graphic visualizer for Apache Kafka that helps you track key metrics about your cluster in real-time. MIT License

• Remora: Kafka consumer lag-checking application for monitoring, written in Scala and Akka HTTP; a wrap around the Kafka consumer group command.Integrations with Cloudwatch and Datadog.Authentication recently added. MIT License

Other Awesome Lists about Apache Kafka & Data Engineering

• Awesome Open-Source Data Engineering

• Awesome Data Engineering

• infoslack/awesome-kafka: This list is for anyone wishing to learn about Apache Kafka, but do not have a starting point.

• dharmeshkakadia/awesome-kafka: Everything about Apache Kafka

• Apache Kafka Ecosystem

License

This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.