[FLINK-35055][cdc] Exclude connector dependencies with questionable l…

…icenses from fat jar packages (apache#3212)
wuzhenhua01 · Aug 4, 2024 · 48df714 · 48df714
1 parent 1878d66
commit 48df714
Show file tree

Hide file tree

Showing 19 changed files with 217 additions and 10 deletions.
diff --git a/docs/content.zh/docs/connectors/legacy-flink-cdc-sources/db2-cdc.md b/docs/content.zh/docs/connectors/legacy-flink-cdc-sources/db2-cdc.md
@@ -55,6 +55,26 @@ put it under `<FLINK_HOME>/lib/`.
 [flink-sql-connector-db2-cdc](https://mvnrepository.com/artifact/com.ververica/flink-sql-connector-db2-cdc),
 more released versions will be available in the Maven central warehouse.
 
+由于 Db2 Connector 采用的 IPLA 协议与 Flink CDC 项目不兼容，我们无法在 jar 包中提供 Db2 连接器。
+您可能需要手动配置以下依赖：
+
+<div class="wy-table-responsive">
+<table class="colwidths-auto docutils">
+    <thead>
+      <tr>
+        <th class="text-left">依赖名称</th>
+        <th class="text-left">说明</th>
+      </tr>
+    </thead>
+    <tbody>
+      <tr>
+        <td><a href="https://mvnrepository.com/artifact/com.ibm.db2.jcc/db2jcc/db2jcc4">com.ibm.db2.jcc:db2jcc:db2jcc4</a></td>
+        <td>用于连接到 Db2 数据库。</td>
+      </tr>
+    </tbody>
+</table>
+</div>
+
 Setup Db2 server
 ----------------
 

diff --git a/docs/content.zh/docs/connectors/legacy-flink-cdc-sources/mysql-cdc.md b/docs/content.zh/docs/connectors/legacy-flink-cdc-sources/mysql-cdc.md
@@ -58,6 +58,26 @@ MySQL CDC 连接器允许从 MySQL 数据库读取快照数据和增量数据。
 
 **注意:** 参考 [flink-sql-connector-mysql-cdc](https://mvnrepository.com/artifact/com.ververica/flink-sql-connector-mysql-cdc) 当前已发布的所有版本都可以在 Maven 中央仓库获取。
 
+由于 MySQL Connector 采用的 GPLv2 协议与 Flink CDC 项目不兼容，我们无法在 jar 包中提供 MySQL 连接器。
+您可能需要手动配置以下依赖：
+
+<div class="wy-table-responsive">
+<table class="colwidths-auto docutils">
+    <thead>
+      <tr>
+        <th class="text-left">依赖名称</th>
+        <th class="text-left">说明</th>
+      </tr>
+    </thead>
+    <tbody>
+      <tr>
+        <td><a href="https://mvnrepository.com/artifact/mysql/mysql-connector-java/8.0.27">mysql:mysql-connector-java:8.0.27</a></td>
+        <td>用于连接到 MySQL 数据库。</td>
+      </tr>
+    </tbody>
+</table>
+</div>
+
 配置 MySQL 服务器
 ----------------
 

diff --git a/docs/content.zh/docs/connectors/legacy-flink-cdc-sources/oceanbase-cdc.md b/docs/content.zh/docs/connectors/legacy-flink-cdc-sources/oceanbase-cdc.md
@@ -61,7 +61,7 @@ OceanBase CDC 连接器允许从 OceanBase 读取快照数据和增量数据。
 
 **注意:** 参考 [flink-sql-connector-oceanbase-cdc](https://mvnrepository.com/artifact/com.ververica/flink-sql-connector-oceanbase-cdc) 当前已发布的所有版本都可以在 Maven 中央仓库获取。
 
-对于 JDBC 驱动，上述的 cdc jar 文件中已经包含了我们推荐的 MySQL 驱动版本 5.1.47。由于开源许可证的原因，我们不能在上述 cdc jar 文件中包含 OceanBase 的官方 JDBC 驱动，如果您需要使用它，可以从[这里](https://repo1.maven.org/maven2/com/oceanbase/oceanbase-client/2.4.2/oceanbase-client-2.4.2.jar)下载，然后放到 `<FLINK_HOME>/lib/` 目录下，同时需要将配置项 `jdbc.driver` 设为 `com.oceanbase.jdbc.Driver`。
+由于开源许可证的原因，我们不能在上述 cdc jar 文件中包含 OceanBase 的官方 JDBC 驱动，如果您需要使用它，可以从[这里](https://repo1.maven.org/maven2/com/oceanbase/oceanbase-client/2.4.2/oceanbase-client-2.4.2.jar)下载，然后放到 `<FLINK_HOME>/lib/` 目录下，同时需要将配置项 `jdbc.driver` 设为 `com.oceanbase.jdbc.Driver`。
 
 ### 配置 OceanBase 数据库和 oblogproxy 服务
 

diff --git a/docs/content.zh/docs/connectors/legacy-flink-cdc-sources/oracle-cdc.md b/docs/content.zh/docs/connectors/legacy-flink-cdc-sources/oracle-cdc.md
@@ -45,6 +45,32 @@ Download [flink-sql-connector-oracle-cdc-3.0.1.jar](https://repo1.maven.org/mave
 
 **Note:** Refer to [flink-sql-connector-oracle-cdc](https://mvnrepository.com/artifact/com.ververica/flink-sql-connector-oracle-cdc), more released versions will be available in the Maven central warehouse.
 
+由于 Oracle Connector 采用的 FUTC 协议与 Flink CDC 项目不兼容，我们无法在 jar 包中提供 Oracle 连接器。
+您可能需要手动配置以下依赖：
+
+<div class="wy-table-responsive">
+<table class="colwidths-auto docutils">
+    <thead>
+      <tr>
+        <th class="text-left">依赖名称</th>
+        <th class="text-left">说明</th>
+      </tr>
+    </thead>
+    <tbody>
+      <tr>
+        <td><a href="https://mvnrepository.com/artifact/com.oracle.ojdbc/ojdbc8/19.3.0.0">com.oracle.ojdbc:ojdbc8:19.3.0.0</a></td>
+        <td>用于连接到 Oracle 数据库。</td>
+      </tr>
+    </tbody>
+    <tbody>
+      <tr>
+        <td><a href="https://mvnrepository.com/artifact/com.oracle.database.xml/xdb/19.3.0.0">com.oracle.database.xml:xdb:19.3.0.0</a></td>
+        <td>用于存储 XML 文件。</td>
+      </tr>
+    </tbody>
+</table>
+</div>
+
 Setup Oracle
 ----------------
 You have to enable log archiving for Oracle database and define an Oracle user with appropriate permissions on all databases that the Debezium Oracle connector monitors.

diff --git a/docs/content.zh/docs/connectors/mysql.md b/docs/content.zh/docs/connectors/mysql.md
@@ -28,6 +28,27 @@ under the License.
 
 MySQL CDC Pipeline 连接器允许从 MySQL 数据库读取快照数据和增量数据，并提供端到端的整库数据同步能力。 本文描述了如何设置 MySQL CDC Pipeline 连接器。
 
+## 依赖配置
+
+由于 MySQL Connector 采用的 GPLv2 协议与 Flink CDC 项目不兼容，我们无法在 jar 包中提供 MySQL 连接器。
+您可能需要手动配置以下依赖：
+
+<div class="wy-table-responsive">
+<table class="colwidths-auto docutils">
+    <thead>
+      <tr>
+        <th class="text-left">依赖名称</th>
+        <th class="text-left">说明</th>
+      </tr>
+    </thead>
+    <tbody>
+      <tr>
+        <td><a href="https://mvnrepository.com/artifact/mysql/mysql-connector-java/8.0.27">mysql:mysql-connector-java:8.0.27</a></td>
+        <td>用于连接到 MySQL 数据库。</td>
+      </tr>
+    </tbody>
+</table>
+</div>
 
 ## 示例
 

diff --git a/docs/content/docs/connectors/legacy-flink-cdc-sources/db2-cdc.md b/docs/content/docs/connectors/legacy-flink-cdc-sources/db2-cdc.md
@@ -55,6 +55,26 @@ put it under `<FLINK_HOME>/lib/`.
 [flink-sql-connector-db2-cdc](https://mvnrepository.com/artifact/com.ververica/flink-sql-connector-db2-cdc),
 more released versions will be available in the Maven central warehouse.
 
+Since Db2 Connector's IPLA license is incompatible with Flink CDC project, we can't provide Db2 connector in prebuilt connector jar packages. 
+You may need to configure the following dependencies manually.
+
+<div class="wy-table-responsive">
+<table class="colwidths-auto docutils">
+    <thead>
+      <tr>
+        <th class="text-left">Dependency Item</th>
+        <th class="text-left">Description</th>
+      </tr>
+    </thead>
+    <tbody>
+      <tr>
+        <td><a href="https://mvnrepository.com/artifact/com.ibm.db2.jcc/db2jcc/db2jcc4">com.ibm.db2.jcc:db2jcc:db2jcc4</a></td>
+        <td>Used for connecting to Db2 database.</td>
+      </tr>
+    </tbody>
+</table>
+</div>
+
 Setup Db2 server
 ----------------
 

diff --git a/docs/content/docs/connectors/legacy-flink-cdc-sources/mysql-cdc.md b/docs/content/docs/connectors/legacy-flink-cdc-sources/mysql-cdc.md
@@ -52,6 +52,26 @@ Download [flink-sql-connector-mysql-cdc-3.0.1.jar](https://repo1.maven.org/maven
 
 **Note:** Refer to [flink-sql-connector-mysql-cdc](https://mvnrepository.com/artifact/com.ververica/flink-sql-connector-mysql-cdc), more released versions will be available in the Maven central warehouse.
 
+Since MySQL Connector's GPLv2 license is incompatible with Flink CDC project, we can't provide MySQL connector in prebuilt connector jar packages.
+You may need to configure the following dependencies manually.
+
+<div class="wy-table-responsive">
+<table class="colwidths-auto docutils">
+    <thead>
+      <tr>
+        <th class="text-left">Dependency Item</th>
+        <th class="text-left">Description</th>
+      </tr>
+    </thead>
+    <tbody>
+      <tr>
+        <td><a href="https://mvnrepository.com/artifact/mysql/mysql-connector-java/8.0.27">mysql:mysql-connector-java:8.0.27</a></td>
+        <td>Used for connecting to MySQL database.</td>
+      </tr>
+    </tbody>
+</table>
+</div>
+
 Setup MySQL server
 ----------------
 

diff --git a/docs/content/docs/connectors/legacy-flink-cdc-sources/oceanbase-cdc.md b/docs/content/docs/connectors/legacy-flink-cdc-sources/oceanbase-cdc.md
@@ -53,7 +53,7 @@ Download [flink-sql-connector-oceanbase-cdc-3.0.1.jar](https://repo1.maven.org/m
 
 **Note:** Refer to [flink-sql-connector-oceanbase-cdc](https://mvnrepository.com/artifact/com.ververica/flink-sql-connector-oceanbase-cdc), more released versions will be available in the Maven central warehouse.
 
-For JDBC driver, the cdc jar above already contains MySQL JDBC driver 5.1.47, which is our recommended version. Due to the license issue, we can not include the OceanBase JDBC driver in the cdc jar. If you need to use it, you can download it from [here](https://repo1.maven.org/maven2/com/oceanbase/oceanbase-client/2.4.2/oceanbase-client-2.4.2.jar) and put it under `<FLINK_HOME>/lib/`, you also need to set the start option `jdbc.driver` to `com.oceanbase.jdbc.Driver`.
+Due to the license issue, we can not include the OceanBase JDBC driver in the cdc jar. If you need to use it, you can download it from [here](https://repo1.maven.org/maven2/com/oceanbase/oceanbase-client/2.4.2/oceanbase-client-2.4.2.jar) and put it under `<FLINK_HOME>/lib/`, you also need to set the start option `jdbc.driver` to `com.oceanbase.jdbc.Driver`.
 
 Setup OceanBase and LogProxy Server
 ----------------------

diff --git a/docs/content/docs/connectors/legacy-flink-cdc-sources/oracle-cdc.md b/docs/content/docs/connectors/legacy-flink-cdc-sources/oracle-cdc.md
@@ -45,6 +45,33 @@ Download [flink-sql-connector-oracle-cdc-3.0.1.jar](https://repo1.maven.org/mave
 
 **Note:** Refer to [flink-sql-connector-oracle-cdc](https://mvnrepository.com/artifact/com.ververica/flink-sql-connector-oracle-cdc), more released versions will be available in the Maven central warehouse.
 
+
+Since Oracle Connector's FUTC license is incompatible with Flink CDC project, we can't provide Oracle connector in prebuilt connector jar packages.
+You may need to configure the following dependencies manually.
+
+<div class="wy-table-responsive">
+<table class="colwidths-auto docutils">
+    <thead>
+      <tr>
+        <th class="text-left">Dependency Item</th>
+        <th class="text-left">Description</th>
+      </tr>
+    </thead>
+    <tbody>
+      <tr>
+        <td><a href="https://mvnrepository.com/artifact/com.oracle.ojdbc/ojdbc8/19.3.0.0">com.oracle.ojdbc:ojdbc8:19.3.0.0</a></td>
+        <td>Used for connecting to Oracle database.</td>
+      </tr>
+    </tbody>
+    <tbody>
+      <tr>
+        <td><a href="https://mvnrepository.com/artifact/com.oracle.database.xml/xdb/19.3.0.0">com.oracle.database.xml:xdb:19.3.0.0</a></td>
+        <td>Used for storaging XML files.</td>
+      </tr>
+    </tbody>
+</table>
+</div>
+
 Setup Oracle
 ----------------
 You have to enable log archiving for Oracle database and define an Oracle user with appropriate permissions on all databases that the Debezium Oracle connector monitors.

diff --git a/docs/content/docs/connectors/mysql.md b/docs/content/docs/connectors/mysql.md
@@ -29,6 +29,27 @@ under the License.
 MySQL connector allows reading snapshot data and incremental data from MySQL database and provides end-to-end full-database data synchronization capabilities.
 This document describes how to setup the MySQL connector.
 
+## Dependencies
+
+Since MySQL Connector's GPLv2 license is incompatible with Flink CDC project, we can't provide MySQL connector in prebuilt connector jar packages.
+You may need to configure the following dependencies manually.
+
+<div class="wy-table-responsive">
+<table class="colwidths-auto docutils">
+    <thead>
+      <tr>
+        <th class="text-left">Dependency Item</th>
+        <th class="text-left">Description</th>
+      </tr>
+    </thead>
+    <tbody>
+      <tr>
+        <td><a href="https://mvnrepository.com/artifact/mysql/mysql-connector-java/8.0.27">mysql:mysql-connector-java:8.0.27</a></td>
+        <td>Used for connecting to MySQL database.</td>
+      </tr>
+    </tbody>
+</table>
+</div>
 
 ## Example
 

diff --git a/flink-cdc-connect/flink-cdc-pipeline-connectors/flink-cdc-pipeline-connector-mysql/pom.xml b/flink-cdc-connect/flink-cdc-pipeline-connectors/flink-cdc-pipeline-connector-mysql/pom.xml
@@ -165,7 +165,6 @@ limitations under the License.
                                     <include>org.apache.flink:flink-connector-mysql-cdc</include>
                                     <include>org.antlr:antlr4-runtime</include>
                                     <include>org.apache.kafka:*</include>
-                                    <include>mysql:mysql-connector-java</include>
                                     <include>com.zendesk:mysql-binlog-connector-java</include>
                                     <include>com.fasterxml.*:*</include>
                                     <include>com.google.guava:*</include>

diff --git a/flink-cdc-connect/flink-cdc-source-connectors/flink-sql-connector-db2-cdc/pom.xml b/flink-cdc-connect/flink-cdc-source-connectors/flink-sql-connector-db2-cdc/pom.xml
@@ -63,6 +63,7 @@ limitations under the License.
                                 <excludes>
                                     <exclude>org.codehaus.janino:*</exclude>
                                     <exclude>org.apache.calcite:*</exclude>
+                                    <exclude>com.ibm.db2.jcc:*</exclude>
                                 </excludes>
                             </artifactSet>
                             <filters>

diff --git a/flink-cdc-connect/flink-cdc-source-connectors/flink-sql-connector-mysql-cdc/pom.xml b/flink-cdc-connect/flink-cdc-source-connectors/flink-sql-connector-mysql-cdc/pom.xml
@@ -62,7 +62,6 @@ limitations under the License.
                                     <include>org.apache.flink:flink-connector-mysql-cdc</include>
                                     <include>org.antlr:antlr4-runtime</include>
                                     <include>org.apache.kafka:*</include>
-                                    <include>mysql:mysql-connector-java</include>
                                     <include>com.zendesk:mysql-binlog-connector-java</include>
                                     <include>com.fasterxml.*:*</include>
                                     <include>com.google.guava:*</include>

diff --git a/flink-cdc-connect/flink-cdc-source-connectors/flink-sql-connector-oceanbase-cdc/pom.xml b/flink-cdc-connect/flink-cdc-source-connectors/flink-sql-connector-oceanbase-cdc/pom.xml
@@ -57,7 +57,6 @@ limitations under the License.
                                     <include>io.debezium:debezium-core</include>
                                     <include>org.apache.flink:flink-connector-debezium</include>
                                     <include>org.apache.flink:flink-connector-oceanbase-cdc</include>
-                                    <include>mysql:mysql-connector-java</include>
                                     <include>com.oceanbase:*</include>
                                     <include>io.netty:*</include>
                                     <include>com.google.protobuf:protobuf-java</include>

diff --git a/flink-cdc-connect/flink-cdc-source-connectors/flink-sql-connector-oracle-cdc/pom.xml b/flink-cdc-connect/flink-cdc-source-connectors/flink-sql-connector-oracle-cdc/pom.xml
@@ -62,8 +62,6 @@ limitations under the License.
                                     <include>org.apache.flink:flink-connector-oracle-cdc</include>
                                     <include>org.antlr:antlr4-runtime</include>
                                     <include>com.github.jsqlparser:jsqlparser</include>
-                                    <include>com.oracle.ojdbc:*</include>
-                                    <include>com.oracle.database.xml:*</include>
                                     <include>org.apache.kafka:*</include>
                                     <include>com.fasterxml.*:*</include>
                                     <include>com.google.guava:*</include>

diff --git a/flink-cdc-e2e-tests/flink-cdc-source-e2e-tests/pom.xml b/flink-cdc-e2e-tests/flink-cdc-source-e2e-tests/pom.xml
@@ -368,6 +368,37 @@ limitations under the License.
                             <outputDirectory>${project.build.directory}/dependencies
                             </outputDirectory>
                         </artifactItem>
+
+                        <artifactItem>
+                            <groupId>com.ibm.db2.jcc</groupId>
+                            <artifactId>db2jcc</artifactId>
+                            <version>db2jcc4</version>
+                            <destFileName>db2-jcc.jar</destFileName>
+                            <type>jar</type>
+                            <outputDirectory>${project.build.directory}/dependencies
+                            </outputDirectory>
+                        </artifactItem>
+
+                        <artifactItem>
+                            <groupId>com.oracle.ojdbc</groupId>
+                            <artifactId>ojdbc8</artifactId>
+                            <version>19.3.0.0</version>
+                            <destFileName>oracle-ojdbc.jar</destFileName>
+                            <type>jar</type>
+                            <outputDirectory>${project.build.directory}/dependencies
+                            </outputDirectory>
+                        </artifactItem>
+
+                        <artifactItem>
+                            <groupId>com.oracle.database.xml</groupId>
+                            <artifactId>xdb</artifactId>
+                            <version>19.3.0.0</version>
+                            <destFileName>oracle-xdb.jar</destFileName>
+                            <type>jar</type>
+                            <outputDirectory>${project.build.directory}/dependencies
+                            </outputDirectory>
+                        </artifactItem>
+
                     </artifactItems>
                 </configuration>
             </plugin>

diff --git a/...dc-source-e2e-tests/src/test/java/org/apache/flink/cdc/connectors/tests/Db2E2eITCase.java b/...dc-source-e2e-tests/src/test/java/org/apache/flink/cdc/connectors/tests/Db2E2eITCase.java
@@ -56,6 +56,7 @@ public class Db2E2eITCase extends FlinkContainerTestEnvironment {
 
     private static final Path db2CdcJar = TestUtils.getResource("db2-cdc-connector.jar");
     private static final Path mysqlDriverJar = TestUtils.getResource("mysql-driver.jar");
+    private static final Path db2JccJar = TestUtils.getResource("db2-jcc.jar");
 
     public static final String DB2_IMAGE = "ibmcom/db2";
     public static final String DB2_CUSTOM_IMAGE = "custom/db2-cdc:1.4";
@@ -161,7 +162,7 @@ public void testDb2CDC() throws Exception {
                         "INSERT INTO products_sink",
                         "SELECT * FROM products_source;");
 
-        submitSQLJob(sqlLines, db2CdcJar, jdbcJar, mysqlDriverJar);
+        submitSQLJob(sqlLines, db2CdcJar, jdbcJar, mysqlDriverJar, db2JccJar);
         waitUntilJobRunning(Duration.ofSeconds(30));
 
         try (Connection conn = getDb2Connection();

diff --git a/...-source-e2e-tests/src/test/java/org/apache/flink/cdc/connectors/tests/MySqlE2eITCase.java b/...-source-e2e-tests/src/test/java/org/apache/flink/cdc/connectors/tests/MySqlE2eITCase.java
@@ -43,6 +43,8 @@ public class MySqlE2eITCase extends FlinkContainerTestEnvironment {
 
     private static final Path mysqlCdcJar = TestUtils.getResource("mysql-cdc-connector.jar");
 
+    private static final Path mySqlConnectorJar = TestUtils.getResource("mysql-driver.jar");
+
     @Test
     public void testMySqlCDC() throws Exception {
         List<String> sqlLines =
@@ -95,7 +97,7 @@ public void testMySqlCDC() throws Exception {
                         "INSERT INTO products_sink",
                         "SELECT * FROM products_source;");
 
-        submitSQLJob(sqlLines, mysqlCdcJar, jdbcJar);
+        submitSQLJob(sqlLines, mysqlCdcJar, jdbcJar, mySqlConnectorJar);
         waitUntilJobRunning(Duration.ofSeconds(30));
 
         // generate binlogs

diff --git a/...source-e2e-tests/src/test/java/org/apache/flink/cdc/connectors/tests/OracleE2eITCase.java b/...source-e2e-tests/src/test/java/org/apache/flink/cdc/connectors/tests/OracleE2eITCase.java
@@ -55,6 +55,8 @@ public class OracleE2eITCase extends FlinkContainerTestEnvironment {
     private static final String INTER_CONTAINER_ORACLE_ALIAS = "oracle";
     private static final Path oracleCdcJar = TestUtils.getResource("oracle-cdc-connector.jar");
     private static final Path mysqlDriverJar = TestUtils.getResource("mysql-driver.jar");
+    private static final Path oracleOjdbcJar = TestUtils.getResource("oracle-ojdbc.jar");
+    private static final Path oracleXdbJar = TestUtils.getResource("oracle-xdb.jar");
     public static final String ORACLE_IMAGE = "goodboy008/oracle-19.3.0-ee";
     private static OracleContainer oracle;
 
@@ -128,7 +130,7 @@ public void testOracleCDC() throws Exception {
                         "INSERT INTO products_sink",
                         "SELECT * FROM products_source;");
 
-        submitSQLJob(sqlLines, oracleCdcJar, jdbcJar, mysqlDriverJar);
+        submitSQLJob(sqlLines, oracleCdcJar, jdbcJar, mysqlDriverJar, oracleOjdbcJar, oracleXdbJar);
         waitUntilJobRunning(Duration.ofSeconds(30));
 
         // generate redo log