v1.0.1

librdkafka v1.0.1 release

v1.0.1 is a maintenance release with the following fixes:

• Fix consumer stall when broker connection goes down (issue #2266 introduced in v1.0.0)
• Fix AdminAPI memory leak when broker does not support request (@souradeep100, #2314)
• Update/fix protocol error response codes (@benesch)
• Treat ECONNRESET as standard Disconnects (#2291)

v1.0.0

librdkafka v1.0.0 release

v1.0.0 is a major feature release:

• Idempotent producer - guaranteed ordering, exactly-once producing.
• Sparse/on-demand connections - connections are no longer maintained to all brokers in the cluster.
• KIP-62 - max.poll.interval.ms for high-level consumers

This release also changes configuration defaults and deprecates a set
of configuration properties, make sure to read the Upgrade considerations
section below.

Upgrade considerations (IMPORTANT)

librdkafka v1.0.0 is API (C & C++) and ABI (C) compatible with older versions of librdkafka, but there are changes to configuration properties that may require changes to existing applications.

Configuration default changes

The following configuration properties have changed default values, which
may require application changes:

• acks (alias request.required.acks) default is now all (wait for ack from all in-sync replica brokers), the previous default was 1 (only wait for ack from partition leader) which could cause data loss if the leader broker goes down.
• enable.partition.eof is now false by default. Applications that rely on ERR__PARTITION_EOF to be emitted must now explicitly set this property to true. This change was made to simplify the common case consumer application consume loop.
• broker.version.fallback was changed from 0.9 to 0.10 and broker.version.fallback.ms was changed to 0.
  Users on Apache Kafka <0.10 must set api.version.request=false and broker.version.fallback=.. to their broker version. For users >=0.10 there is no longer any need to specify any of these properties. See https://github.com/edenhill/librdkafka/wiki/Broker-version-compatibility for more information.

Deprecated configuration properties

• topic.metadata.refresh.fast.cnt is no longer used.
• socket.blocking.max.ms is no longer used.
• reconnect.backoff.jitter.ms is no longer used, see reconnect.backoff.ms and reconnect.backoff.max.ms.
• offset.store.method=file is deprecated.
• offset.store.path is deprecated.
• offset.store.sync.interval.ms is deprecated.
• queuing.strategy was an experimental property that is now deprecated.
• msg_order_cmp was an experimental property that is now deprecated.
• produce.offset.report is no longer used. Offsets are always reported.
• auto.commit.enable (topic level) for the simple (legacy) consumer is
  now deprecated.

Use of any deprecated configuration property will result in a warning when the client instance is created.
The deprecated configuration properties will be removed in a future version of librdkafka.
See issue #2020 for more information.

Configuration checking

The checks for incompatible configuration has been improved, the client
instantiation (rd_kafka_new()) will now fail if incompatible configuration
is detected.

max.poll.interval.ms is enforced

This release adds support for max.poll.interval.ms (KIP-62), which requires
the application to call rd_kafka_consumer_poll()/rd_kafka_poll() at least every max.poll.interval.ms.
Failure to do so will make the consumer automatically leave the group, causing a group rebalance,
and not rejoin the group until the application has called ..poll() again, triggering yet another group rebalance.
max.poll.interval.ms is set to 5 minutes by default.

Idempotent Producer

This release adds support for Idempotent Producer, providing exactly-once
producing and guaranteed ordering of messages.

Enabling idempotence is as simple as setting the enable.idempotence
configuration property to true.

There are no required application changes, but it is recommended to add
support for the newly introduced fatal errors that will be triggered when the idempotent producer encounters an unrecoverable error that would break the ordering or duplication guarantees.

See Idempotent Producer in the manual and the Exactly once semantics blog post for more information.

Sparse connections

In previous releases librdkafka would maintain open connections to all
brokers in the cluster and the bootstrap servers.

With this release librdkafka now connects to a single bootstrap server
to retrieve the full broker list, and then connects to the brokers
it needs to communicate with: partition leaders, group coordinators, etc.

For large scale deployments this greatly reduces the number of connections
between clients and brokers, and avoids the repeated idle connection closes
for unused connections.

See Sparse connections in the manual for more information.

Original issue #825.

Features

• Add support for ZSTD compression (KIP-110, @mvavrusa. Caveat: will not currently work with topics configured with compression.type=zstd, instead use compression.type=producer, see #2183)
• Added max.poll.interval.ms (KIP-62, #1039) to allow long processing times.
• Message Header support for C++ API (@davidtrihy)

Enhancements

• Added rd_kafka_purge() API to purge messages from producer queues (#990)
• Added fatal errors (see ERR__FATAL and rd_kafka_fatal_error()) to
  raise unrecoverable errors to the application. Currently only triggered
  by the Idempotent Producer.
• Added rd_kafka_message_status() producer API that may be used from
  the delivery report callback to know if the message was persisted to brokers
  or not. This is useful for applications that want to perform manual retries
  of messages, to know if a retry could lead to duplication.
• Backoff reconnects exponentially (See reconnect.backoff.ms and reconnect.backoff.max.ms).
• Add broker[..].req["reqType"] per-request-type metrics to statistics.
• CONFIGURATION.md: Added Importance column.
• ./configure --install-deps (and also --source-deps-only) will automatically install dependencies through the native package manager and/or from source.

Fixes

General

• rd_kafka_version() was not thread safe
• Round up microsecond->millisecond timeouts to 1ms in internal scheduler
  to avoid CPU-intensive busy-loop.
• Send connection handshake requests before lower prio requests.
• Fix timespec conversion to avoid infinite loop (#2108, @boatfish)
• Fix busy-loop: Don't set POLLOUT (due to requests queued) in CONNECT state (#2118)
• Broker hostname max size increased from 127 to 255 bytes (#2171, @Vijendra07Kulhade)

Consumer

• C++: Fix crash when Consumer ctor fails
• Make sure LeaveGroup is sent on unsubscribe and consumer close (#2010, #2040)
• Remember assign()/seek():ed offset when pause()ing (#2105)
• Fix handling of mixed MsgVersions in same FetchResponse (#2090)

Producer

• Added delivery.timeout.ms -> message.timeout.ms alias
• Prevent int overflow while computing abs_timeout for producer request… (#2050, @KseniyaYakil).
• Producer: fix re-ordering corner-case on retry.

Windows

• win32: cnd_timedwait*() could leave the cond signalled, resulting in high CPU usage.

Build/installation/tooling

• Makefile: fix install rule (#2049, @pacovn)
• Fixing Counting error in rdkafka_performance #1542 (#2028, @gnanasekarl)
• OpenSSL 1.1.0 compatibility (#2000, @nouzun, @wiml)
• Set OpenSSL locking callbacks as required, dont call CRYPTO_cleanup_all_ex_data (#1984, @ameihm0912)
• Fix 64-bit IBM i build error (#2017, @ThePrez)
• CMake: Generate pkg-config files (@Oxymoron79, #2075)
• mklove: suggest brew packages to install on osx
• rdkafka_performance: Add an option to dump the configuration (@AKohn)
• Check for libcrypto explicitly (OSX Mojave, #2089)

v0.11.6

v0.11.6 is a maintenance release.

Critical fixes

• The internal timer could wrap in under 8 days on Windows, causing stalls and hangs. Bug introduced in v0.11.5. #1980
• Purge and retry buffers in outbuf queue on connection fail (#1913). Messages could get stuck in internal queues on retry and broker down.

Enhancements

• Enable low latency mode on Windows by using TCP "pipe". Users no longer need to set socket.blocking.max.ms to improve latency. (#1930, @LavaSpider)
• Added rd_kafka_destroy_flags() to control destroy behaviour. Can be used to force an consumer to terminate without leaving the group or committing final offsets.

Fixes

• Producer: Serve UA queue when transitioning topic from UNKNOWN. Messages could get stuck in UA partition queue on metadata timeout (#1985).
• Fix partial read issue on unix platforms without recvmsg()
• Improve disconnect detection on Windows (#1937)
• Use atomics for refcounts on all platforms with atomics support (#1873)
• Message err was not set for on_ack interceptors on broker reply (#1892)
• Fix consumer_lag to -1 when neither app_offset or commmitted_offset is available (#1911)
• Fix crash: Insert retriable messages on partition queue, not xmit queue (#1965)
• Fix crash: don't enqueue messages for retry when handle is terminating.
• Disconnect regardless of socket.max.fails when partial request times out (#1955)
• Now builds with libressl (#1896, #1901, @secretmike)
• Call poll when flush() is called with a timeout of 0 (#1950)
• Proper locking of set_fetch_state() when OffsetFetch response is outdated
• Destroy unknown and no longer desired partitions (fixes destroy hang)
• Handle FetchResponse for partitions that were removed during fetch (#1948)
• rdkafka_performance: Default the message size to the length of the pattern unless given explicitly (#1899, @ankon)
• Fix timeout reuse in queue serving/polling functions (#1863)
• Fix rd_atomic*_set() to use __atomic or __sync when available
• NuGet: change runtime from win7-.. to more generic win-.. (CLIENTS-1188)
• Fix crash: failed ops enq could be handled by original dest queue callbacks
• Update STATISTICS.md to match code (@ankon, #1936)
• rdkafka_performance: Don't sleep while waiting for delivery reports (#1918, @ankon)
• Remove unnecessary 100ms sleep when broker goes from UP -> DOWN (#1895)
• rdhdrhistogram: Fix incorrect float -> int cast causing havoc on MIPS (@andoma)
• "Message size too large": The receive buffer would grow by x2 (up to the limit) each time a EOS control message was seen. (#1472)
• Added support for system-provided C11 threads, e.g. alpine/musl. This fixes weird behaviour on Alpine (#1998)
• Fix high CPU usage after disconnect from broker while waiting for SASL response (#2032, @stefanseufert)
• Fix dont-destroy-from-rdkafka-thread detection logic to avoid assert when using plugins.
• Fix LTO warnings with gcc 8 (#2038, @Romain-Geissler-1A)

v0.11.5

v0.11.5 is a feature release that adds support for the Kafka Admin API (KIP-4).

Admin API

This release adds support for the Admin API, enabling applications and users to perform administrative Kafka tasks programmatically:

• Create topics - specifying partition count, replication factor and topic configuration.
• Delete topics - delete topics in cluster.
• Create partitions - extend a topic with additional partitions.
• Alter configuration - set, modify or delete configuration for any Kafka resource (topic, broker, ..).
• Describe configuration - view configuration for any Kafka resource.

The API closely follows the Java Admin API:

https://github.com/edenhill/librdkafka/blob/master/src/rdkafka.h#L4495

New and updated configuration

• Added compresion.level configuration option, which allows fine-tuning of gzip and LZ4 comression level (@erkoln)
• Implement ssl.curves.list and ssl.sigalgs.list configuration settings (@jvgutierrez)
• Changed queue.buffering.backpressure.threshold default (#1848)

Enhancements

• Callback based event notifications (@fede1024)
• Event callbacks may now optionally be triggered from a dedicated librdkafka background thread, see rd_kafka_conf_set_background_event_cb.
• Log the value that couldn't be found for flag configuration options (@ankon)
• Add support for rd_kafka_conf_set_events(conf, ..EVENT_ERROR) to allow generic errors to be retrieved as events.
• Avoid allocating BIOs and copying for base64 processing (@agl)
• Don't log connection close for idle connections (regardless of log.connection.close)
• Improve latency by using high-precision QPC clock on Windows
• Added make uninstall
• Minor documentation updates from replies to #1794 (@briot)
• Added rd_kafka_controllerid() to return the current controller.
• INTRODUCTION.md: add chapter on latency measurement.
• Add relative hyperlinks to table of contents in INTRODUCTION.md (#1791, @stanislavkozlovski)
• Improved statistics:
  • Added Hdr Histograms for all windowed stats (rtt, int_latency, throttle) #1798
  • Added top-level totals for broker receive and transmit metrics.
  • Added batchcnt, batchsize histograms to consumer.
  • Added outbuf_latency histograms.
  • STATISTICS.md moved from wiki to source tree.

Fixes

• Fixed murmur2 partitioner to make it compatible with java version (#1816, @lins05)
• Fix pause/resume: next_offset was not properly initialized
• Fix a segment fault error in rdkafka_buf with zero-length string (@sunny1988)
• Set error string length in rkmessage.len on error (#1851)
• Don't let metadata ERR_UNKNOWN set topic state to non-existent.
• The app_offset metric is now reset to INVALID when the fetcher is stopped.
• consumer_lag is now calculated as consumer_lag = hi_wmark_offset - MAX(app_offset, committed_offset), which makes it correct after a reassignment but before new messages have been consumed (#1878)
• socket.nagle.disable=true was never applied on non-Windows platforms (#1838)
• Update interface compile definitions for Windows using CMake (#1800, @raulbocanegra)
• Fix queue hang when queue_destroy() is called on rdkafka-owned queue (#1792)
• Fix hang on unclean termination when there are outstanding requests.
• Metadata: fix crash when topic is in transitionary state
• Metadata: sort topic partition list
• rdkafka_example emitted bogus produce errors
• Increase BROKERS_MAX to 10K and PARTITIONS_MAX to 100K
• Proper log message on SSL connection close
• Missing return on error causes use-after-free in SASL code (@sidhpurwala-huzaifa)
• Fix configure --pkg-config-path=... (#1797, @xbolshe)
• Fix -fsanitize=undefined warning for overflowed OP switches (#1789)

v0.11.4

Maintenance release

Default changes

• socket.max.fails changed to 1 to provide same functionality (fail request immediately on error) now when retries are working properly again.
• fetch.max.bytes (new config property) is automatically adjusted to be >= message.max.bytes, and receive.message.max.bytes is automatically adjusted to be > fetch.max.bytes. (#1616)

New features

• Message Headers support (with help from @johnistan)
• Java-compatible Murmur2 partitioners (#1468, @barrotsteindev)
• Add PKCS#12 Keystore support - ssl.keystore.location (#1494, @AMHIT)

Noteworthy fixes

• Formalise and fix Producer retries and retry-ordering (#623, #1092, #1432, #1476, #1421)
  • Ordering is now retained despite retries if max.in.flight=1.
  • Behaviour is now documented
• Fix timeouts for retried requests and properly handle retries for all request types (#1497)
• Add and use fetch.max.bytes to limit total Fetch response size (KIP-74, #1616). Fixes "Invalid response size" issues.

Enhancements

• Added sasl.mechanism and compression.type configuration property aliases for conformance with Java client.
• Improved Producer performance
• C++: add c_ptr() to Handle,Topic,Message classes to expose underlying librdkafka object
• Honour per-message partition in produce_batch() if MSG_F_PARTITION set (@barrotsteindev, closes #1604)
• Added on_request_sent() interceptor
• Added experimental flexible producer queuing.strategy=fifo|lifo
• Broker address DNS record round-robin: try to maintain round-robin position across resolve calls.
• Set system thread name for internal librdkafka threads (@tbsaunde)
• Added more concise and user-friendly 'consumer' debug context
• Add partitioner (string) topic configuration property to set the builtin partitioners
• Generate rdkafka-static.pc (pkg-config) for static linking

Fixes

• Fix producer memory leak on <0.11 brokers when compressed messageset is below copy threshold (closes #1534)
• CRC32C - fix unaligned access on ARM (@Soundman32)
• Fix read after free in buf_write_seek
• Fix broker wake up (#1667, @gduranceau)
• Fix consumer hang when rebalancing during commit (closes #1605, @ChenyuanHu)
• CMake fixes for Windows (@raulbocanegra)
• LeaveGroup was not sent on close when doing final offset commits
• Fix for consumer slowdown/stall on compacted topics where actual last offset < MsgSet.LastOffset (KAFKA-5443)
• Fix global->topic conf fallthru in C++ API
• Fix infinite loop on LeaveGroup failure
• Fix possible crash on OffsetFetch retry
• Incorporate compressed message count when deciding on fetch backoff (#1623)
• Fix debug-only crash on Solaris (%s NULL) (closes #1423)
• Drain broker ops queue on termination to avoid hang (closes #1596)
• cmake: Allow build static library (#1602, @proller)
• Don't store invalid offset as next one when pausing (#1453, @mfontanini)
• use #if instead of #ifdef / defined() for atomics (#1592, @vavrusa)
• fixed .lib paths in nuget packaging (#1587)
• Fixes strerror_r crash on alpine (#1580, @skarlsson)
• Allow arbitrary lengthed (>255) SASL PLAIN user/pass (#1691, #1692)
• Trigger ApiVersionRequest on reconnect if broker.version.fallback supports it (closes #1694)
• Read Fetch MsgAttributes as int8 (discovered by @tvoinarovskyi, closes #1689)
• Portability: stop using typeof in rdkafka_transport.c (#1708, @tbsaunde)
• Portability: replace use of #pragma once with header guards (#1688, @tbsaunde)
• mklove: add LIBS in reverse order to maintain dependency order
• Fix build when python is not available #1358

v0.11.3

Maintenance release

Default changes

• Change default queue.buffering.max.kbytes and queued.max.message.kbytes to 1GB (#1304)
• win32: Use sasl.kerberos.service.name for broker principal, not sasl.kerberos.principal (#1502)

Enhancements

• Default producer message offsets to OFFSET_INVALID rather than 0
• new nuget package layout + debian9 librdkafka build (#1513, @mhowlett)
• Allow for calling rd_kafka_queue_io_event_enable() from the C++ world (#1483, @akhi3030)
• rdkafka_performance: allow testing latency with different size messages (#1482, @tbsaunde)

Fixes

• Improved stability on termination (internal queues, ERR__DESTROY event)
• offsets_for_times() return ERR__TIMED_OUT if brokers did not respond in time
• Let list_groups() return ERR__PARTIAL with a partial group list (#1508)
• Properly handle infinite (-1) rd_timeout:s throughout the code (#1539)
• Fix offsets_store() return value when at least one valid partition
• portability: rdendian: add le64toh() alias for older glibc (#1463)
• Add MIPS build and fix CRC32 to work on big endian CPUs (@andoma, closes #1498)
• osx: fix endian checking for software crc32c
• Fix comparison in rd_list_remove_cmp (closes #1493)
• stop calling cnd_timedwait() with a timeout of 0h (#1481, @tbsaunde)
• Fix DNS cache logic broker.address.ttl (#1491, @dacjames)
• Fix broker thread "hang" in CONNECT state (#1397)
• Reset rkb_blocking_max_ms on broker DOWN to avoid busy-loop during CONNECT (#1397)
• Fix memory leak when producev() fails (#1478)
• Raise cmake minimum version to 3.2 (#1460)
• Do not assume LZ4 worst (best?) case 255x compression (#1446 by @tudor)
• Fix ALL_BROKERS_DOWN re-generation (fix by @ciprianpascu, #1101)
• rdkafka-performance: busy wait to wait short periods of time

v0.11.1

Maintenance release

NOTE: If you are experiencing lousy producer performance, try setting the linger.ms configuration property to 100 (ms).

Noteworthy critical bug fixes

• Fix OpenSSL instability on Windows (fix thread id callback) - the bug resulted in SSL connections being torn down for no apparent reason.
• Fetch response fix: read all MessageSets, not just the first one. (#1384) - Huge performance degradation (compared to v0.9.5) when fetching small batches.

Enhancements

• Add api.version.request.timeout.ms (#1277, thanks to @vinipuh5)
• Point users to documentation when attempting to use Java security properties (#1412)

Fixes

• Adjust log level for partial message reads when debug is enabled (#1433)
• Allow app metadata() requests regardless of outstanding metadata requests (#1430)
• Proper size calculation from flags2str to include nul (#1334)
• Thread-safe rd_strerror() (#1410)
• Proper re-sends of ProduceRequests in-transit on connection down (#1390)
• Producer: invalid offsets reported back when produce.offset.report=false
• Consumer: Message Null Key/Value were messed up (regression in v0.11.0, #1386)
• Fix metadata querying for topics with LEADER_UNAVAIL set (#1313)
• Treat request __TIMED_OUT as a retryable error
• Let timed out ProduceRequests result in MSG_TIMED_OUT error code for messages
• Fix crash on leader rejoin when outstanding assignor metadata (#1371)
• sasl_cyrus: Fix dangling stack ptr to sasl_callback_t (#1329, thanks to @gnpdt )

v0.11.0

Feature release

v0.11.0 is a new feature release of librdkafka with support for the new Kafka message format (MsgVersion 2) which makes librdkafka (and any librdkafka-based clients) transparently compatible for use with the EOS (Exactly-Once-Semantics) supporting Java client released with Apache Kafka v0.11.0.

This release also includes enhancements and fixes as listed below.

NOTE: While librdkafka implements the new Message version and features, it does not yet implement the EOS (Exactly-Once-Semantics) functionality itself.

NOTE: The librdkafka C++ API is unfortunately not ABI safe (the API stability is guaranteed though): C++ users will need to recompile their applications when upgrading librdkafka.

Upgrade notes

api.version.request:
The api.version.request property (see https://github.com/edenhill/librdkafka/wiki/Broker-version-compatibility) default value has changed from false to true, meaning that librdkafka will make use of the latest protocol features of the broker without the need to set the property to true explicitly on the client.

WARNING: Due to a bug in Apache Kafka 0.9.0.x, the ApiVersionRequest (as sent by the client when connecting to the broker) will be silently ignored by the broker causing the request to time out after 10 seconds. This causes client-broker connections to stall for 10 seconds during connection-setup before librdkafka falls back on the broker.version.fallback protocol features. The workaround is to explicitly configure api.version.request to false on clients communicating with &lt=0.9.0.x brokers.

Producer:
The default value of queue.buffering.max.ms was changed from 1000ms to 0ms (no delay). This property denotes the internal buffering time (and latency) for messages produced.

Features

• Added support for MsgVersion v2 (message format of KIP-98) - message format compatible with EOS clients
• Added support for client interceptors
• Added support for dynamically loaded plugins (plugin.library.paths, for use with interceptors)
• Added SASL SCRAM support (KIP-84)
• Added builtin SASL PLAIN provider (for Win32, #982)

Enhancements

• Deprecate errno usage, use rd_kafka_last_error() instead.
• Deprecate rd_kafka_wait_destroyed().
• Implemented per-partition Fetch backoffs, previously all partitions for the given broker were backed off.
• Added updated Kafka protocol and error enums
• Added rd_kafka_message_latency()
• Added rd_kafka_clusterid() and rd_kafka_type()
• SSL: set default CA verify locations if ssl.ca.location is not specified
• C++: add yield() method
• Added support for stats as events (#1171)
• Build with system liblz4 if available, else fall back on built-in lz4, for improved portability.
• Use SNI when connecting through SSL (@vincentbernat)
• Improve broker thread responsiveness, decreasing internal latency
• Improve OpenSSL config error propagation (#1119)
• Prioritize all relevant user-facing ops (callbacks) over messages on poll queue (#1088)
• Added global->topic config fallthru: default topic config properties
  can now be set effortlessly on global config object.
• Log offset commit failures when there is no offset_commit_cb (closes #1043)
• Add CRC checking support to consumer (#1056)
• C++: Added seek() support to KafkaConsumer
• Added rd_kafka_conf_dup_filter() to selectively copy a config object.

Fixes:

• Avoid _ALIGN re-definition on BSD (#1225)
• rdkafka_performance: exit with code 1 if not all messages were delivered
• Fix endianism issues that were causing snappy to compress incorrectly (#1219, @rthalley)
• Fix stability on AIX (#1211)
• Document that rd_kafka_message_errstr() must not be used on producer
• Add support for re-queuing half-processed ops to honour yield()
• Handle null Protocol in JoinGroupResponse (#1193)
• Consumer: Proper relative offset handling (#1192, @rthalley)
• OSX: silence libsasl deprecated warnings
• partition count should be per topic in offset request buffer (closes #1199, @matthew-d-jones)
• fix build on SmartOS (#1186 by @misterdjules)
• ERR_remove_thread_state OpenSSL version checking
• Don't emit TIMED_OUT_QUEUE for timed out messages (revert)
• producev() default partition should UA, not 0 (#1153)
• Fix SaslHandshakeRequest timeout to 10s
• SASL: fix memory leak: received SASL auth frames were not freed
• Clean up OpenSSL per-thread memory on broker thread exit
• Properly auto-set metadata.max.age.ms when metadata.refresh.interval.ms is disabled (closes #1149)
• Fix memory alignment issues (#1150)
• configure: auto add brew openssl pkg-config path
• Fix consumer_lag calculation (don't use cached hi_offset)
• rdkafka_example: fix message_errstr usage, not allowed on producer
• Avoid use of partially destroyed topic object (#1125)
• Improve reconnect delay handling (#1089)
• C++: fix conf->get() allocation (closes #1118)
• Use app_offset to calculate consumer_lag (closes #1112)
• Fix retrybuf memory leak on termination when broker is down
• Fix small memory leak in metadata_leader_query
• Fix use-after-free when log.queue and debug was used
• consumer_example: fix crash on -X dump (closes #841)
• Added rd_kafka_offsets_store() (KafkaConsumer::offsets_store) (closes #826)
• Optimize broker id lookups (closes #523)
• Don't log broker failures when an error_cb is registered (closes #1055)
• Properly log SSL connection close (closes #1081)
• Win32 SASL GSSAPI: protection level and message size were not sent
• C++: improved error reporting from Conf::set()
• Flush partition fetch buffer on seek (from decide())
• Fix recursive locking on periodic refresh of leader-less partition (closes #1311)

v0.9.5

Maintenance release

Critical fixes

• q_concat: don't wakeup listeners if srcq is empty (fix by @orthrus in #1121): Fixes idle consumer CPU usage on Windows
• Prioritize commit_cb over messages on poll queue (closes #1088)
• Prioritize all relevant user-facing ops (callbacks) (#1088)
• Fix SaslHandshakeRequest timeout

Fixes

• Properly log SSL connection close (closes #1081)
• Don't log broker failures when an error_cb is registered (closes #1055)
• Fix use-after-free when log.queue and debug was used
• Log offset commit failures when there is no offset_commit_cb (closes #1043)
• Fix retrybuf memory leak on termination when broker is down
• Fix small memory leak in metadata_leader_query
• Use app_offset to calculate consumer_lag (closes #1112)
• Fix consumer_lag calculation (dont use cached hi_offset)
• C++: fix conf->get() allocation (closes #1118)
• Avoid use of partially destroyed topic object (#1125, fixed by @benli123)
• sasl win32: protection level and message size not sent (fixed by @zyzil )
• Properly auto-set metadata.max.age.ms when metadata.refresh.interval.ms is disabled (closes #1149)
• Fix memory alignment issues (#1150)
• consumer_example: fix crash on -X dump (closes #841)
• configure: auto add brew openssl pkg-config path
• producev() default partition should be UA, not 0 (#1153)

Enhancements

• Added global->topic config fallthru: Topic-level configuration properties can now be set on the global configuration object. The property will be applied on the default_topic_conf object (if no such object exists one is created automatically).
• Use SNI when connecting through SSL (by @vincentbernat )
• Windows performance improvements (use atomics instead of locks, avoid locking in some cases)
• Add lz4/lib sources for in-tree building when external lz4 is not available
• Add CRC checking support to consumer (#1056)
• Added op priority to queues (for #1088)
• Fail known unsupported requests locally (closes #1091)
• Improve reconnect delay handling (#1089)
• Improve OpenSSL config error propagation (#1119)
• C++: improved error reporting from Conf::set()
• Don't emit TIMED_OUT_QUEUE (revert)

v0.9.4

librdkafka v0.9.4 is a small feature release with a horde of enhancements and fixes.

NOTE: The librdkafka C++ API is unfortunately not ABI safe (the API stability is guaranteed though): C++ users will need to recompile their applications when upgrading librdkafka.

New features

• SASL: Windows SASL GSSAPI support using native Windows SSPI (#888)
• Added rd_kafka_offsets_for_times (KIP-79, #842)
• Added rd_kafka_producev() using va-args to construct message fields, including timestamps (#858, #707, #908, #345)
• C++: added produce(..., timestamp, ..)
• Added support for partition-specific queues (@lambdaknight)
  • allows redirecting partitions to separate queues and separates application threads.
• True low-latency (on linux and osx) by fd-based queue wakeups

Critical fixes

• leader failover error detection and recovery for both producer and consumer
• cgrp: failover robustness improvements
• cgrp: certain failed Heartbeats could mask out future heartbeat requests
• buffer handle callbacks could previously be triggered on wrong thread

Enhancements

• Events: expose EVENT_OFFSET_COMMIT
• Experimental cmake support (thanks to @ruslo)
• Validate topics in subscribe() (mainly regex validation)
• Added set_log_queue() - allows application thread log handling (#355)
• Add support for Metadata v1..2 (part of KIP-4)
• New metadata request and caching framework:
  • much fewer metadata requests
  • use cached metadata when deemed possible
  • KIP-4 support for lean metadata requests in clusters with many topics
  • improved (faster, more reliable) leader lookups
• C++: added TopicPartition::create(, ..offset)
• C++: added const attribute to TopicPartition getters
• C++: added TopicPartition::destroy(vector) helper
• C++: added commitSync(.., OffsetCommitCb) variant
• C++: Enrichment of Conf getter for callbacks. (#883, @vin-d)
• Expose rd_kafka_topic_partition_list_sort()
• Added max.in.flight -> max.in.flight.requests.per.connection alias
• Win32: add lz4 support (using NuGet lz4 package)
• Add cgrp stats: rebalance_age, rebalance_cnt, assignment_size

Fixes

• increase maximum supported topic limit to 1M
• fix static linking on osx
• Fix infinte wait in .._commit_queue() if callback on temp queue was used
• Improved handling of assign() and outstanding rebalance_cb on termination
• commit_queue: avoid global offset_commit_cb if local specified
• Imrpoved Heartbeat error handling
• Improve OffsetCommit failure handling
• Query topics with missing or down leaders from topics_scan()
• Handle rebalance from consumer_close (#1002)
• rd_kafka_producev() would send 0 timestamp by default instead of current time
• queue.buffering.max.ms and fetch.error.backoff.ms now takes precedence over socket.blocking.max.ms (#966)
• flush() fix: msg_cnt not initialized
• rdkafka_performance: fix multi -p partitions allocation error and memory leak
• Refactor query_watermark_offsets to use new leader lookup code
• Proper error propagation on invalid topic names (#977)
• callbacks will no longer cut consumer_poll() timeouts short
• improved queue handling and serving
• rd_kafka_producev(): fix topic/rkt referencing
• Fix ApiVersionRequest timeout (10s)
• rd_kafka_q_concat: fix erroneous unlock() on enq failure
• Speed up termination by emitting TERMINATE ops
• cgrp: enforce metadata update check if matched subscribed topics is empty
• Proper suppression of connect failure logs (#847)
• Proper ATOMIC_OPn (#708)
• fix: use out of scope variable (#974, @HeChuanXUPT)
• C++: Fix undefined references to static const ints (#961, #962, @bbfgelman1)
• mklove: lib_check now attempts compilation after pkg-config passed (for false positive of SSL on OSX)
• mklove: fix ./configure --reconfigure
• stats: fix brokers.toppars key duplication