Skip to content

Kafka Toolkit v3.0.0
Compare
Choose a tag to compare
released this 26 Nov 12:20
· 104 commits to develop since this release
v3.0.0
3db53ad

What's new in this release and what has changed

  • The included Kafka client has been upgraded from version 2.2.1 to 2.3.1.
  • The schema of the output port of the KafkaProducer operator supports optional types for the error description.
  • When in autonomous region, the optional input port of the KafkaConsumer operator can be used to change the topic subscription, not only the partition assignment. To create JSON for changing topic subscriptions, the toolkit contains new SPL functions. For consistent region, this function is not available.
  • The guaranteeOrdering parameter now enables the idempotent producer when set to true, which allows a higher throughput by allowing more in-flight requests per connection (requires Kafka server version 0.11 or higher).
  • The KafkaConsumer operator now enables and benefits from group management when the user does not specify a group identifier, i.e. when the operator generates a group identifier.
  • Checkpoint reset of the KafkaConsumer is optimized in consistent region when the consumer operator is the only group member.
  • The KafkaConsumer operator can be configured as a static consumer group member (requires Kafka server version 2.3 or higher). See also the Static Consumer Group Membership chapter in the KafkaConsumer's documentation.
  • The KafkaConsumer operator now uses read_committed as the default isolation.level configuration unless the user has specified a different value. In read_committed mode, the consumer will read only those transactional messages which have been successfully committed. Messages of aborted transactions are now skipped. The consumer will continue to read non-transactional messages as before. This new default setting is incompatible with Kafka 0.10.2.

Deprecated features

The use of the input control port has been deprecated when the KafkaConsumer is used in a consistent region.

Changes in behaviour

KafkaConsumer operator

When the user does not specify a group identifier, and does not specify partitions to consume (via the partition parameter), the KafkaConsumer now subscribes to the given topics and benefits from group management, for example it automatically gets assigned new partitions when the number of partitions of the subscribed topics change. In previous versions, the KafkaConsumer operator used to self-assign all available topic partitions of the topics at startup, and the partition assignment of the operator never changed during the runtime.

Incompatible changes

  • The toolkit requires at minimum Streams version 4.3.
  • The guaranteeOrdering parameter of the KafkaProducer operator is incompatible with Kafka version 0.10.x when set to true. The work-around for Kafka 0.10.x is given in the guaranteeOrdering parameter description.
  • When the KafkaConsumer operator is configured with input port, the topic, pattern, partition, and startPosition parameters used to be ignored in previous versions. Now an SPL compiler error is raised when one of these parameters is used together with the input port.
  • The default isolation.level configuration of the KafkaConsumer operator is incompatible with Kafka broker version 0.10.x. When connecting with Kafka 0.10.x, isolation.level=read_uncommitted must be used for the consumer configuration.

The SPL documentation can be found at https://ibmstreams.github.io/streamsx.kafka/docs/user/SPLDoc/

Assets 4
Loading