forked from ClickHouse/clickhouse-kafka-connect
-
Notifications
You must be signed in to change notification settings - Fork 0
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Managed by Terraform, updates managed through: https://github.com/Air…
- Loading branch information
1 parent
85f45f2
commit a76898c
Showing
1 changed file
with
52 additions
and
38 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,38 +1,52 @@ | ||
# ClickHouse Kafka Connect Sink | ||
|
||
## About | ||
clickhouse-kafka-connect is the official Kafka Connect sink connector for [ClickHouse](https://clickhouse.com/). | ||
|
||
The Kafka connector delivers data from a Kafka topic to a ClickHouse table. | ||
## Documentation | ||
|
||
See the [ClickHouse website](https://clickhouse.com/docs/en/integrations/kafka/clickhouse-kafka-connect-sink) for the full documentation entry. | ||
|
||
## Design | ||
For a full overview of the design and how exactly-once delivery semantics are achieved, see the [design document](./docs/DESIGN.md). | ||
|
||
## Help | ||
For additional help, please [file an issue in the repository](https://github.com/ClickHouse/clickhouse-kafka-connect/issues) or raise a question in [ClickHouse public Slack](https://clickhouse.com/slack). | ||
|
||
## KeyToValue Transformation | ||
We've created a transformation that allows you to convert a Kafka message key into a value. | ||
This is useful when you want to store the key in a separate column in ClickHouse - by default, the column is `_key` and the type is String. | ||
|
||
```sql | ||
CREATE TABLE your_table_name | ||
( | ||
`your_column_name` String, | ||
... | ||
... | ||
... | ||
`_key` String | ||
) ENGINE = MergeTree() | ||
``` | ||
|
||
Simply add the transformation to your connector configuration: | ||
|
||
```properties | ||
transforms=keyToValue | ||
transforms.keyToValue.type=com.clickhouse.kafka.connect.transforms.KeyToValue | ||
transforms.keyToValue.field=_key | ||
``` | ||
<!-- START doctoc generated TOC please keep comment here to allow auto update --> | ||
<!-- DON'T EDIT THIS SECTION, INSTEAD RE-RUN doctoc TO UPDATE --> | ||
|
||
<!-- END doctoc generated TOC please keep comment here to allow auto update --> | ||
|
||
[![Airthings][logo]](https://www.airthings.com) | ||
|
||
# clickhouse-kafka-connect | ||
|
||
This repository is used to maintain the code base for the kotlin library clickhouse-kafka-connect. | ||
This code base in this repo is maintained by the Airthings kotlin_core_maintainers team. So please reach out to them before making any breaking changes. | ||
|
||
[logo]: https://upload.wikimedia.org/wikipedia/commons/d/d1/Airthings_logo.svg | ||
|
||
# Contribute | ||
When a PR is made to this repository, make sure to add a label corresponding to the type of change you are making. The labels are: | ||
- `major` : For any breaking changes | ||
- `minor` : For any new features | ||
- `patch` : For any bug fixes | ||
|
||
## Installation | ||
|
||
Declare the dependency in your `build.gradle` file: | ||
|
||
- Add this helper function | ||
```kotlin | ||
fun RepositoryHandler.airthings(repository: String) = maven { | ||
name = "com.airthings.github.packages" | ||
url = uri("https://maven.pkg.github.com/Airthings/clickhouse-kafka-connect") | ||
credentials { | ||
username = env.fetchOrNull("GITHUB_USERNAME") | ||
?: System.getenv("GITHUB_USERNAME") | ||
?: System.getenv("GITHUB_ACTOR") | ||
password = env.fetchOrNull("GITHUB_ACCESS_TOKEN_READ_PACKAGES") | ||
?: System.getenv("GITHUB_ACCESS_TOKEN_READ_PACKAGES") | ||
} | ||
} | ||
``` | ||
|
||
- Add the repository | ||
```kotlin | ||
repositories { airthings(repository = "clickhouse-kafka-connect") } | ||
``` | ||
|
||
- Add the dependency using catalog | ||
```kotlin | ||
dependencies { implementation(libs.airthings.replace.me) } | ||
``` | ||
|
||
## Usage | ||
|
||
Use Example.client() to create a client instance. |