Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[FLINK-35055][cdc] Exclude connector dependencies with questionable licenses from fat jar packages #3212

Merged
merged 4 commits into from
Apr 23, 2024

Conversation

yuxiqian
Copy link
Contributor

@yuxiqian yuxiqian commented Apr 9, 2024

This resolves FLINK-35055.

Currently, Flink CDC connector releases both slim and fat jars. Apart from CDC itself, all of its dependencies are packaged into fat jars, including some with incompatible licenses:

  • Db2 connector: com.ibm.db2.jcc:db2jcc:db2jcc4 licensed under IBM's non-FOSS license (International Program License Agreement).
  • MySQL connector: mysql:mysql-connector-java licensed under GPLv2 license, which is incompatible with Apache 2.0.
  • OceanBase connector: mysql:mysql-connector-java licensed under GPLv2 license, which is incompatible with Apache 2.0.
  • Oracle connector: com.oracle.ojdbc licensed under Oracle's non-FOSS license (Oracle Free Use Terms and Conditions).

This PR:

  1. Excludes questionable dependencies from released fat jar packages;
  2. Adds explanation in docs to guide user providing dependencies manually.

Copy link
Contributor

@PatrickRen PatrickRen left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@yuxiqian Thanks for the patch! Just a minor comment for the documentation.

@PatrickRen
Copy link
Contributor

It reminds me that there's an automatic license check in CI of Flink repo. Maybe we can also do it in Flink CDC:

https://github.com/apache/flink/blob/dec5e9e659dd09346781c97c940a20a6cbc63678/tools/ci/flink-ci-tools/src/main/java/org/apache/flink/tools/ci/licensecheck/JarFileChecker.java#L141-L181

@yuxiqian
Copy link
Contributor Author

@PatrickRen Sure! Shall I add the license checking script in this PR or open another one?

@PatrickRen
Copy link
Contributor

@yuxiqian I won't take it as a blocker. We can have a try to see if it is complex.

@yuxiqian
Copy link
Contributor Author

It reminds me that there's an automatic license check in CI of Flink repo. Maybe we can also do it in Flink CDC:

https://github.com/apache/flink/blob/dec5e9e659dd09346781c97c940a20a6cbc63678/tools/ci/flink-ci-tools/src/main/java/org/apache/flink/tools/ci/licensecheck/JarFileChecker.java#L141-L181

Addressed in #3218.

@yuxiqian
Copy link
Contributor Author

yuxiqian commented Apr 19, 2024

Seems removing connector jar dependencies breaks SQL Job E2e tests. Need to fix this first.

Copy link
Contributor

@PatrickRen PatrickRen left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@yuxiqian Thanks for the update! LGTM

@PatrickRen PatrickRen merged commit 01ec7da into apache:master Apr 23, 2024
15 checks passed
wuzhenhua01 pushed a commit to wuzhenhua01/flink-cdc-connectors that referenced this pull request Aug 4, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants