flink-connector-jdbc

Java library provides Apache Flink connector sink for JDBC database that can be used with Flink 1.8 runtime version. Connector code is backported from the latest Flink version (1.11) in order to be used in Amazon Kinesis Data Analytics applications.

At Klarna we use streaming applications extensively. Amazon Kinesis Data Analytics with Flink 1.8 is starting to be one of the choices for the development of new streaming analytics applications at Klarna. Unfortunately, some of the latest features developed in the Apache Flink project version after 1.8 are not available yet in Amazon Kinesis Data Analytics.

flink-connector-jdbc-1.8 is a Java library that contains code backported from the latest Flink version (1.11) flink-connector-jdbc library that can be used in Amazon Kinesis Data Analytics / Flink 1.8.

Usage example

import com.klarna.org.apache.flink.api.java.io.jdbc.JDBCOptions;
import com.klarna.org.apache.flink.api.java.io.jdbc.JDBCUpsertOutputFormat;
import com.klarna.org.apache.flink.api.java.io.jdbc.JDBCUpsertSinkFunction;

...
env.addSource(createConsumer())
   .addSink(new JDBCUpsertSinkFunction(JDBCUpsertOutputFormat.builder()
            .setFieldNames(new String[]{
                    "event_id",
                    "created_at"
            })
            .setFieldTypes(new int[]{
                    Types.VARCHAR,
                    Types.TIMESTAMP
            })
            .setFlushIntervalMills(10000)
            .setFlushMaxSize(5000)
            .setKeyFields(new String[]{ "event_id" })
            .setMaxRetryTimes(3)
            .setOptions(JDBCOptions.builder()
                    .setDBUrl(dbUrl)
                    .setDriverName(Driver.class.getName())
                    .setUsername(dbUsername)
                    .setPassword(dbPassword)
                    .setTableName(tableName)
                    .build())
            .build()));

Development setup

This project uses Maven to set up the development environment. The recommended workflow to build and install the library is the following.

mvn clean install

How to contribute

See our guide on contributing.

Release History

See our changelog.

License

Copyright © 2020 Klarna Bank AB

For license details, see the LICENSE file in the root of this project.