Build: Add plugin to generate license and notice files #11977

bryanck · 2025-01-15T20:13:16Z

This PR adds a Gradle plugin for generating the LICENSE and NOTICE files for runtimes and bundles that include third party libraries. The PR includes a custom renderer for generating the files, so we can adjust that as needed.

The following Gradle tasks were added:

generateLicenseReport - generates new LICENSE and NOTICE files in the build directory
compareLicenseReport - compares the generated files with the existing ones and fails if they don't match
updateLicenseReport - replaces the existing files with the newly generated ones

The compareLicenseReport will run automatically as part of the build, when the check task runs. If the files don't match and the build fails, then the files will need to be updated by running the updateLicenseReport task manually.

I enabled the plugin for the following projects: aws-bundle, azure-bundle, gcp-bundle, and kafka-connect-runtime. These projects all used a variation of the plugin in the past to generate the files, so are the safest to include. I also regenerated the files for these projects using this plugin so the build will pass.

This addresses #11559

bryanck · 2025-01-15T20:14:30Z

buildSrc/build.gradle

+}
+
+dependencies {
+  implementation 'com.github.jk1:gradle-license-report:2.9'


The dependency needed to be declared here so that the classes were available outside of build.gradle. We could also move the other buildscript dependencies in here, but I left them as-is.

RussellSpitzer · 2025-01-15T22:34:47Z

I liked the idea that Ryan suggested this morning, where our plugin would always generate and check the generated files against a set of committed existing files and report an error if there was a change from that state. Basically a RevAPI for our license files

bryanck · 2025-01-16T03:54:41Z

I liked the idea that Ryan suggested this morning, where our plugin would always generate and check the generated files against a set of committed existing files and report an error if there was a change from that state. Basically a RevAPI for our license files

I made this change.

Fokko · 2025-01-16T20:42:20Z

This is looking great @bryanck, thanks for working on this 👍

How about documenting the process of checking if the NOTICE files are up to date as part of the how-to-release?

What Ryan also mentioned is that not everything in the NOTICE is required. The official ASF docs state: Do not add anything to NOTICE which is not legally required. For example, if we copy the NOTICE from Parquet-Java, the general NOTICE contains a section that's only relevant for parquet-protobuf that we don't use in the project.

I liked the idea that Ryan suggested this morning, where our plugin would always generate and check the generated files against a set of committed existing files and report an error if there was a change from that state. Basically a RevAPI for our license files

I like that as well, we could run the compareLicenseReport task as part of the CI.

bryanck · 2025-01-16T20:52:52Z

This is looking great @bryanck, thanks for working on this 👍

How about documenting the process of checking if the NOTICE files are up to date as part of the how-to-release?

Sure will do.

What Ryan also mentioned is that not everything in the NOTICE is required. The official ASF docs state: Do not add anything to NOTICE which is not legally required. For example, if we copy the NOTICE from Parquet-Java, the general NOTICE contains a section that's only relevant for parquet-protobuf that we don't use in the project.

I had some discussion w/ Dan around this, I'll investigate this some more, there was some conflicting information.

I like that as well, we could run the compareLicenseReport task as part of the CI.

Yes, I added this, the check task depends on compareLicenseReport.

RussellSpitzer · 2025-01-16T21:00:06Z

aws-bundle/LICENSE

@@ -203,320 +203,468 @@

 --------------------------------------------------------------------------------

-This binary artifact contains code from the following projects:
+This product includes a gradle wrapper.


There are a lot of licenses here which seem like they shouldn't be included? Why does the AWS bundle have a gradle wrapper, avro, parquet ... etc Looks like it got a lot of license notices from something else?

Am I missing something from this?
https://github.com/apache/iceberg/blob/main/aws-bundle/build.gradle#L26-L39

I'm appending to the root LICENSE and NOTICE files, I think some new items were added to those recently. I could remove that if we want.

Shouldn't they be in the other LICENSE Bundles then?

For example I see in this file

This product includes code from Apache Spark. * dev/check-license script * vectorized reading of definition levels in BaseVectorizedParquetValuesReader.java * portions of the extensions parser * casting logic in AssignmentAlignmentSupport * implementation of SetAccumulator. * Connector expressions. Copyright: 2011-2018 The Apache Software Foundation Home page: https://spark.apache.org/ License: https://www.apache.org/licenses/LICENSE-2.0

But I don't see that in azure-bundle or gcp-bundle

I do see it in kafka-connect but that one makes sense to me

Hm let me double check what's going on.

Ah yeah got it, I must have not expanded my diffs.

@bryanck But are they transitive dependencies of this particular bundle? That's I think where I am a bit lost

They may not be, so I'm thinking I should remove the appending to the root files. The root files used to be pretty minimal but some new things were added recently.

I'll go ahead and make that change, I don't think it makes sense to do this anymore (appending to the root files).

I changed this so it strips off the extra deps after the initial Apache license in the root files, so those extra deps don't get added.

snazy

Thanks @bryanck for the PR, it's great to see the work that has been originally added to Nessie and then to Apache Polaris is getting reused in other projects as well.

But instead of copying the work, I think it would be much better if all the efforts around the NOTICE and LICENSE files is bundled and shared so that not every Apache project has to redo all that work, but is instead reusable for all projects. It can easily become a Gradle plugin on its own.

I'm requesting changes to this PR, because as it is based on code of Apache Polaris (I think @jbonofre mentioned it on the last Iceberg community sync) it requires a mention - and I suspect with this also projectnessie as the original source of the work.

bryanck · 2025-01-16T21:32:48Z

I'm requesting changes to this PR, because as it is based on code of Apache Polaris (I think @jbonofre mentioned it on the last Iceberg community sync) it requires a mention - and I suspect with this also projectnessie as the original source of the work.

This was not copied from another project, we have used this in Iceberg before Polaris existed, but simply didn't contribute the code. #8261 (comment)

snazy · 2025-01-17T06:50:12Z

This was not copied from another project, we have used this in Iceberg before Polaris existed, but simply didn't contribute the code. #8261 (comment)

Then I retract my comment.

RussellSpitzer · 2025-01-17T21:49:31Z

I would like to be clear that it is extremely important that we assume all other engineers are contributing in good faith and the interaction above is failing to meet that standard. It is not appropriate to make public accusations of copying code in pull request comments.

In situations where a community member believes there has been some sort of infringement, I would recommend that the community member talk directly with the code contributor and try to resolve the issue privately first. In this community it is appropriate to help others if you believe they have inadvertently misattributed code or have doubts about provenance, but this must not be done in an aggressive or insulting manner.

If talking one on one fails to resolve the issue, immediately notify the PMC. Knowingly copying code into the project in disregard to copyright or without attribution is a serious offense and damages the entire Apache community. Because of how serious these accusations are, it's crucial that we do not make them lightly.

I'd like to again thank @bryanck for his contributions to the project and reiterate that we should strive to have positive interactions with each other and we should apologize when we fall short.

ajantha-bhat

Thanks for working on it.

I hope we do a follow up PR for other runtime jars like Spark, Flink, REST-testFixture etc.

ajantha-bhat · 2025-01-20T12:04:21Z

license.gradle

+    File licenseFile = new File("${projectDir}/build/reports/dependency-license/LICENSE")
+    File currentLicenseFile = new File("${projectDir}/LICENSE")
+    if (licenseFile.text != currentLicenseFile.text) {
+      throw new RuntimeException("License file has changed");


Maybe mention about what to do next, like Run updateLicenseReport with a complete gradle command.

Similar to how RevAPI or spotless prompts when fails about the next step.

Thanks, I added this.

Fokko

Thanks

It isn't curated as we like, but I do think this is more complete and will also notify us when new dependencies are added.

WDYT @danielcweeks @rdblue @jbonofre

Fokko · 2025-01-23T06:29:27Z

kafka-connect/kafka-connect-runtime/NOTICE

@@ -5,111 +5,21 @@ Copyright 2017-2024 The Apache Software Foundation
 This product includes software developed at
 The Apache Software Foundation (http://www.apache.org/).

--------------------------------------------------------------------------------


I do think we need to keep this one. This is not the result of a dependency, but because we copied code from the Kite project. This comes from the parent NOTICE: https://github.com/apache/iceberg/blob/main/NOTICE

Agree, it should not be there imho.

jbonofre · 2025-01-23T09:20:29Z

@bryanck thanks a lot for working on that ! I will take a look as I'm doing a complete pass on artifacts LICENSE/NOTICE.

Generally speaking, by experience on Apache projects, it's pretty hard to have automate tools to generate LICENSE/NOTICE (because the inner shading is hard to check, some "deps" LICENSE and NOTICE may have other names like COPYING, etc). However, I'm completely in favor on a tool to "assist" the contributors.

So, big thanks again to @bryanck and let me take a look on this PR 😄

jbonofre

I found a couple of "not fully correct" content in LICENSE (about slf4j-api).

I gonna take a new deeper look but I like this as "assistance tool".

I also found another issue in our distributed jar files (unrelated to this change), I will create another PR.

Thanks again @bryanck !

jbonofre · 2025-01-23T09:23:41Z

aws-bundle/NOTICE

@@ -5,92 +5,22 @@ Copyright 2017-2024 The Apache Software Foundation
 This product includes software developed at
 The Apache Software Foundation (http://www.apache.org/).

--------------------------------------------------------------------------------
-
-NOTICE for Group: commons-codec  Name: commons-codec  Version: 1.17.1


It is not necessary to duplicate the line "This product includes software developed at the Apache Software Foundation...", though the ASF copyright line and any other portions of NOTICE must be considered for propagation. I saw it's still included, which is fine, just wanted to provide some "background".

jbonofre · 2025-01-23T09:31:16Z

azure-bundle/LICENSE

-Group: org.reactivestreams  Name: reactive-streams  Version: 1.0.4
-Project URL: http://www.reactive-streams.org/
-License: MIT-0 - https://spdx.org/licenses/MIT-0.html
+Group: org.slf4j  Name: slf4j-api  Version: 2.0.16


Imho, that's not correct here: slf4j-api is excluded (https://github.com/apache/iceberg/blob/main/azure-bundle/build.gradle#L43), and not in the distributed jar file, so it should not be in LICENSE.

See this PR: #12052

jbonofre · 2025-01-23T09:33:19Z

gcp-bundle/LICENSE

+
+--------------------------------------------------------------------------------
+
+Group: org.slf4j  Name: slf4j-api  Version: 2.0.16


Same here, imho, it's not correct because slf4j-api is excluded (https://github.com/apache/iceberg/blob/main/gcp-bundle/build.gradle#L42) and it's not part of the distributed jar file.

See this PR: #12052

jbonofre · 2025-01-23T09:44:58Z

kafka-connect/kafka-connect-runtime/NOTICE

@@ -5,111 +5,21 @@ Copyright 2017-2024 The Apache Software Foundation
 This product includes software developed at
 The Apache Software Foundation (http://www.apache.org/).

--------------------------------------------------------------------------------


Agree, it should not be there imho.

github-actions bot added build AWS GCP AZURE KAFKACONNECT labels Jan 15, 2025

bryanck commented Jan 15, 2025

View reviewed changes

bryanck force-pushed the license-report branch 2 times, most recently from c5cb209 to c289050 Compare January 15, 2025 20:42

bryanck requested a review from ajantha-bhat January 15, 2025 20:50

bryanck force-pushed the license-report branch from 723708a to 29b1b3d Compare January 16, 2025 03:52

bryanck force-pushed the license-report branch from 06578fe to 0194508 Compare January 16, 2025 17:19

Fokko requested a review from jbonofre January 16, 2025 20:42

RussellSpitzer reviewed Jan 16, 2025

View reviewed changes

snazy suggested changes Jan 16, 2025

View reviewed changes

ajantha-bhat reviewed Jan 20, 2025

View reviewed changes

bryanck added 5 commits January 21, 2025 08:58

Build: Add plugin to generate license and notice files

b4ede61

Fail build if license or notice file has changed

87c939b

add new tasks

2c28811

always generate license

998663c

strip off extra deps from root files

274f531

bryanck force-pushed the license-report branch from df6ee18 to 274f531 Compare January 21, 2025 16:59

bryanck added 2 commits January 21, 2025 09:01

add note on how to update when compare fails

93386ed

update license report

b61614f

Fokko reviewed Jan 23, 2025

View reviewed changes

jbonofre reviewed Jan 23, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Build: Add plugin to generate license and notice files #11977

Build: Add plugin to generate license and notice files #11977

bryanck commented Jan 15, 2025 •

edited

Loading

bryanck Jan 15, 2025

RussellSpitzer commented Jan 15, 2025

bryanck commented Jan 16, 2025

Fokko commented Jan 16, 2025

bryanck commented Jan 16, 2025

RussellSpitzer Jan 16, 2025

RussellSpitzer Jan 16, 2025

bryanck Jan 16, 2025

RussellSpitzer Jan 16, 2025

bryanck Jan 16, 2025

RussellSpitzer Jan 16, 2025

RussellSpitzer Jan 16, 2025

bryanck Jan 16, 2025

bryanck Jan 16, 2025 •

edited

Loading

bryanck Jan 16, 2025

snazy left a comment

bryanck commented Jan 16, 2025 •

edited

Loading

snazy commented Jan 17, 2025

RussellSpitzer commented Jan 17, 2025

ajantha-bhat left a comment

ajantha-bhat Jan 20, 2025

bryanck Jan 21, 2025

Fokko left a comment

Fokko Jan 23, 2025

jbonofre Jan 23, 2025

jbonofre commented Jan 23, 2025

jbonofre left a comment

jbonofre Jan 23, 2025

jbonofre Jan 23, 2025

jbonofre Jan 23, 2025

jbonofre Jan 23, 2025


		--------------------------------------------------------------------------------

		Group: org.slf4j Name: slf4j-api Version: 2.0.16

Build: Add plugin to generate license and notice files #11977

Are you sure you want to change the base?

Build: Add plugin to generate license and notice files #11977

Conversation

bryanck commented Jan 15, 2025 • edited Loading

Choose a reason for hiding this comment

RussellSpitzer commented Jan 15, 2025

bryanck commented Jan 16, 2025

Fokko commented Jan 16, 2025

bryanck commented Jan 16, 2025

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bryanck Jan 16, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

snazy left a comment

Choose a reason for hiding this comment

bryanck commented Jan 16, 2025 • edited Loading

snazy commented Jan 17, 2025

RussellSpitzer commented Jan 17, 2025

ajantha-bhat left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Fokko left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jbonofre commented Jan 23, 2025

jbonofre left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bryanck commented Jan 15, 2025 •

edited

Loading

bryanck Jan 16, 2025 •

edited

Loading

bryanck commented Jan 16, 2025 •

edited

Loading