Kafka Toolkit v2.0.0

What's new in this toolkit release

• The included Kafka client has been upgraded from version 2.1.1 to 2.2.1, #160
• Support for Kafka broker 2.2 has been added, #161
• The toolkit has enhancements for the KafkaConsumer when it is used in an autonomous region (i.e. not part of a consistent region):
  • The KafkaConsumer operator can now participate in a consumer group with startPosition parameter values Beginning, 'End, and Time`, #94
  • After re-launch of the PE, the KafkaConsumer operator does not overwrite the initial fetch offset to what the startPosition parameter is, i.e. after PE re-launch the consumer starts consuming at last committed offset, #107

The new startPosition handling requires that the application always includes a JobControlPlane operator when startPosition is different from Default.

Incompatible changes, which may break existing applications

The behavior of the KafkaConsumer operator changes when

1. the operator is not used in a consistent region, and
2. the startPosition parameter is used with Beginning, End, Time, or Offset.

In all other cases the behavior of the KafkaConsumer is unchanged. Details of the changed behavior, including sample code that breaks, can be found in the Toolkit documentation on Github.

You find the SPL documentation online at https://ibmstreams.github.io/streamsx.kafka/docs/user/SPLDoc/

Kafka Toolkit v1.9.4

What's new in this toolkit release

This toolkit release fixes following issues:

• #158 Kafka consumer: Failed offset commit makes the PE restart when in consistent region
• support StateHandler function resetToInitialState when the operator is reset before checkpoint creation in an autonomous region

You find the SPL documentation online at https://ibmstreams.github.io/streamsx.kafka/docs/user/SPLDoc/

Kafka Toolkit v1.9.3

What's new in this toolkit release

This toolkit release fixes following issues:

• #155 (found in toolkit releases v1.9.0, v1.9.1, and v1.9.2 after upgrade of Kafka client to 2.1.1)

You find the SPL documentation online at https://ibmstreams.github.io/streamsx.kafka/docs/user/SPLDoc/

Kafka Toolkit v1.9.2

What's new in this toolkit release

Following fixes have been implemented:

• #153 You can now use the {applicationDir} token in Kafka properties when you place files within the application directory, bundle them with the SAB, and reference them in Kafka properties. As an example, you can place an SSL keystore file in the etc directory of your SPL application and reference it in the ssl.keystore.location property by

ssl.keystore.location={applicationDir}/etc/keystore.jks

You find the SPL documentation online at https://ibmstreams.github.io/streamsx.kafka/docs/user/SPLDoc/

Known issues

• When the KafkaConsumer is used in consistent region and group management is in effect, the KafkaConsumer sometimes does not seek to fetch position to the previous state on consistent region reset. This behaviour is by now only observed when run on RHEL6. #155

Kafka Toolkit v1.9.1

What's new in this toolkit release

• improved low memory detection for the consumer when records are queued in the consumer, #91
• No adjustment of retries producer configuration to 10, as long as consistent region policy is not Transactional. The version 2.1.1 producer's default retries config (2,147,483,647) is used instead when retries is not specified. #151

You find the SPL documentation online at https://ibmstreams.github.io/streamsx.kafka/docs/user/SPLDoc/

Known issues

• When the KafkaConsumer is used in consistent region and group management is in effect, the KafkaConsumer sometimes does not seek to fetch position to the previous state on consistent region reset. This behaviour is by now only observed when run on RHEL6. #155

Kafka Toolkit v1.9.0

What's new in this toolkit release

This release has following changes and new features:

• The Kafka client has been upgraded to version 2.1.1. For the consumer and producer configuration, please consult the Consumer Configs or Producer Configs section of the Apache Kafka 2.1 documentation.

• The KafkaProducer can now optionally be flushed after a fixed tuple count. Use the optional flush parameter.

Bug fixes:

• #148

You find the SPL documentation online at https://ibmstreams.github.io/streamsx.kafka/docs/user/SPLDoc/

Known issues

• When the KafkaConsumer is used in consistent region and group management is in effect, the KafkaConsumer sometimes does not seek to fetch position to the previous state on consistent region reset. This behaviour is by now only observed when run on RHEL6. #155

Kafka Toolkit v1.8.0

What's new in this toolkit release

This release has following changes and new features:

• The KafkaConsumer can subscribe dynamically to multiple topics by specifying a regular expression using the new pattern parameter. Assignment of partitions happens for matching topics at the time of periodic check. When someone creates a new topic with a name that matches, a rebalance will happen almost immediately and the consumers will start consuming from the new topic. #144

• The control port of the KafkaConsumer allows assignment with default fetch position. New SPL functions and SPL types to generate the JSON string for the control port have been added. #87

• For the KafkaConsumer, you can now specify a time period for committing offsets. Use the new commitPeriod parameter. In previous versions, you could specify only a tuple count, after which offsets are committed.

• The time policy for offset commit is now the default policy of the KafkaConsumer when not in a consistent region. The time policy avoids too high commit request rates, which can occur with count based policy and high tuple rates. The default commit interval is 5 seconds.

• The operators can now be configured with a config checkpoint: clause when used in an autonomous region. The KafkaProducer operator simply ignores the config instead of throwing an error at compile time. The KafkaConsumer operator can be configured with operator driven and periodic checkpointing. Checkpointing is in effect when the operator is configured with the optional input port. Then, the operator checkpoints the assigned partitions. On reset, the assigned partitions are restored, and the consumer resumes fetching at last committed offset. #138

Bug fixes:

The following fixes affect the SPL documentation:

• #145
• #146

You find the SPL documentation online at https://ibmstreams.github.io/streamsx.kafka/docs/user/SPLDoc/

Kafka Toolkit v1.7.3

This bugfix release contains following fixes

• #142 New compression.type setting potentially breaks existing non-Streams consumers

The producer property compression.type is not set by default any more. Release 1.6.0 to 1.7.2 had lz4 as the default value.

Kafka Toolkit v1.7.2

This bugfix release contains following fixes

• #139 Change trace level for metrics dump to debug or trace
• Improved exception handling for the consumer

Kafka Toolkit v1.7.1

This bugfix release fixes following issues:

• #110 - Offsets are committed during partition rebalance within a consumer group, more precisely before partition revocation
• #136 - KafkaConsumer shows old metric value for "<topic-partition>:records-lag". partition related custom metrics, which are not valid for an operator any more now show -1 instead of keeping their last value.