librdkafka - Apache Kafka C/C++ client library

Copyright (c) 2012-2016, Magnus Edenhill.

https://github.com/edenhill/librdkafka

librdkafka is a C library implementation of the Apache Kafka protocol, containing both Producer and Consumer support. It was designed with message delivery reliability and high performance in mind, current figures exceed 1 million msgs/second for the producer and 3 million msgs/second for the consumer.

librdkafka is licensed under the 2-clause BSD license.

For an introduction to the performance and usage of librdkafka, see INTRODUCTION.md

See the wiki for a FAQ.

NOTE: The master branch is actively developed, use latest release for production use.

Apache Kafka 0.8 & 0.9 & 0.10 support:

• Branch: master
• Producer: supported
• High-level balanced KafkaConsumer: supported (requires broker >= 0.9)
• Simple Consumer: supported
• Compression: snappy, gzip, lz4
• Broker version support: >=0.8
• Debian package: librdkafka1 and librdkafka-dev in Debian and Ubuntu
• ZooKeeper: not supported
• C API: Stable, ABI safe
• C++ API: Stable
• Tests: Regression tests in tests/ directory.
• Statistics: JSON formatted, see rd_kafka_conf_set_stats_cb in rdkafka.h.
• Status: Testing

Apache Kafka 0.7 support:

• Branch: 0.7
• Producer: supported
• High-level Consumer: not supported
• Simple Consumer: supported
• Compression: not supported
• Broker version support: 0.7.x
• ZooKeeper: not supported
• Status: Stable, Deprecated, Unsupported

#Language bindings#

• Haskell: haskakafka
• Haskell: haskell-kafka
• Node.js: node-kafka
• Node.js: kafka-native
• Lua: luardkafka
• OCaml: ocaml-kafka
• PHP: phpkafka
• PHP: php-rdkafka
• Python: confluent-kafka-python
• Python: PyKafka
• Ruby: Hermann
• Tcl: KafkaTcl
• C#/.NET: rdkafka-dotnet

#Users of librdkafka#

• kafkacat - Apache Kafka swiss army knife
• Wikimedia's varnishkafka - Varnish cache web log producer
• Wikimedia's kafkatee - Kafka multi consumer with filtering and fanout
• rsyslog
• syslog-ng
• collectd
• logkafka - Collect logs and send to Kafka
• redBorder
• Headweb
• Produban's log2kafka - Web log producer
• fuse_kafka - FUSE file system layer
• node-kafkacat
• OVH - AntiDDOS
• otto.de's trackdrd - Varnish log reader
• Microwish has a range of Kafka utilites for log aggregation, HDFS integration, etc.
• large unnamed financial institution
• Let me know if you are using librdkafka

Usage

Requirements

The GNU toolchain
GNU make
pthreads
zlib (optional, for gzip compression support)
libssl-dev (optional, for SSL support)
libsasl2-dev (optional, for SASL support)
liblz4-dev (optional, for LZ4 compression support)

Instructions

Building

  ./configure
  make
  sudo make install

NOTE: See README.win32 for instructions how to build on Windows with Microsoft Visual Studio.

Usage in code

See examples/rdkafka_example.c for an example producer and consumer.

Link your program with -lrdkafka -lz -lpthread -lrt.

Documentation

The public APIs are documented in their respective header files:

• The C API is documented in src/rdkafka.h
• The C++ API is documented in src-cpp/rdkafkacpp.h

To generate Doxygen documents for the API, type:

make docs

Configuration properties are documented in CONFIGURATION.md

For a librdkafka introduction, see INTRODUCTION.md

Examples

See the examples/sub-directory.

Tests

See the tests/sub-directory.

Support

File bug reports, feature requests and questions using GitHub Issues

Questions and discussions are also welcome on irc.freenode.org, #apache-kafka, nickname Snaps.

Commercial support

Commercial support is available from Edenhill services