generate-copyright: Now generates a library file too.

[jonathanpallant]

We only run reuse once, so the output has to be filtered to find only the files that are relevant to the library tree.

Outputs COPYRIGHT.html and COPYRIGHT-library.html.

The license-metadata.json file is also now in the tree. We need a CI tool to check that it's correct.

r? kobzol

Remaining steps:

• Teach CI to double-check the license-metadata.json file is correct
• Add the COPYRIGHT.html and COPYRIGHT-license.html to the releases

[Kobzol]

It's a bit surprising to me that we commit the JSON file. My expectation was that the only reason we need to commit anything at all is to have a "pre-built" copyright file (in HTML format?) in the repo, which would be easily checkable by people without the need to run bootstrap. So in that case, I'd expect that on CI, we render the HTML file, and check if it is updated in git.

Is the JSON actually needed to be in git?

[jonathanpallant]

My expectation was that the only reason we need to commit anything at all is to have a "pre-built" copyright file (in HTML format?) in the repo, which would be easily checkable by people without the need to run bootstrap. So in that case, I'd expect that on CI, we render the HTML file, and check if it is updated in git.

As I understood it, building the COPYRIGHT-*.html files is trivial using bootstrap, because it's written in Rust. So we can let bootstrap build those.

But the license-metadata.json file requires reuse, which is a Python package, and we didn't want to force people to install it before they could run x.py dist or whatever. So that's the file we commit to git and check in CI.

Edit: Also Copyright-*.html contains the results of scraping the cargo dependency tree for the workspace(s), so it would very quickly get out of sync with Cargo.lock, and that could be annoying. license-metadata.json only contains information that is already in git.

[jonathanpallant]

Ah ha ha ha:

The following files have no copyright and licensing information:
* ../license-metadata.json

[Kobzol]

But the license-metadata.json file requires reuse, which is a Python package, and we didn't want to force people to install it before they could run x.py dist or whatever. So that's the file we commit to git and check in CI.

Bootstrap also requires Python (as of right now), plus a bunch of other dependencies, so that doesn't seem like a gigantic win to me, tbh.

Edit: Also Copyright-*.html contains the results of scraping the cargo dependency tree for the workspace(s), so it would very quickly get out of sync with Cargo.lock, and that could be annoying. license-metadata.json only contains information that is already in git.

It wouldn't ever get out of sync for the same reason the JSON won't get out of sync - we'd check in CI that it is in fact in sync :) Our Cargo.locks are also in git.

Maybe to take a step back - what does this PR try to achieve? My understanding was that we want to get rid of LICENSE-MIT, LICENSE-APACHE and the COPYRIGHT files (and maybe also a bunch of others), and automatically generate a copyright file instead. This is also what Pietro has originally described on Zulip:

Hook collect-license-metadata and generate-copyright into CI, to ensure COPYRIGHT.md reflects the state of things.

So from this point of view, it makes more sense to me to commit the human-readable (and partially machine readable) COPYRIGHT.md/html file, rather than the JSON file, which is mostly a temporary build artifact anyway.

The question is: do we want to generate and commit copyright metadata that is readable by humans, or metadata that is readable by machines, or both? CC @pietroalbini

[jonathanpallant]

From the Zulip:

• Created the src/tools/collect-license-metadata tool, which takes the REUSE output and condenses it in a compact JSON file. By the time the effort is complete, the goal is to commit that JSON file in the repository, and enforce in CI that the file is up to date. That way, contributors don't need REUSE installed unless they change the licensing of the repo.
• Created the src/tools/generate-copyright tool, which takes the JSON generated by the previous tool and renders a COPYRIGHT.md file. An example of how that looks is in this gist.

The main goal of this PR is to generate two COPYRIGHT files, one for the standard library and one for the toolchain as a whole. Tomorrow I will back out the changes in this PR to where the JSON lives so we can have that discussion separately.

[Kobzol]

Sorry, I completely missed the sentence about committing the JSON file 🤦‍♂️ I'm still interested to discuss whether we really need to commit the JSON file, vs just committing something human readable..

[jonathanpallant]

So my view is you shouldn't commit anything you can reasonably regenerate in CI. So to me the choice is, should we commit nothing, or should we commit the JSON to avoid people needing to have reuse installed.

[Kobzol]

I never needed to handle JSON copyright data, but I find it very useful to have a human readable COPYRIGHT file in the root of a repo - it is a standard thing that many repositories have and people kind of expect it will be there, IMO. So it makes sense to me to automatically pregenerate this file.

[jonathanpallant]

The plan is to deploy it with every release, so it's in /usr/local/share/rust/whatever, or wherever your toolchain is installed to. Because it's the product of a relatively expensive complication process, like rustc itself, or any of the other binaries we ship.

It's also only really useful to people who don't have the source code checked out. If you have the source code, you have much more fine-grained copyright information - in the source files themselves. This is just a summary of the source tree.

[Kobzol]

It's also only really useful to people who don't have the source code checked out. If you have the source code, you have much more fine-grained copyright information - in the source files themselves. This is just a summary of the source tree.

Sure, but it's still common to have a COPYRIGHT file in the root of a repo anyway. People don't want to go through the source tree to figure out copyright, not to mention that most of our Rust files don't actually have any. And you have created the awesome HTML copyright page, so let's use it and commit it so that it is easily accessible!

Anyway, I didn't want to derail this PR, sorry. IMO, we should definitely remove the previous COPYRIGHT file(s) and replace them with automatically generated COPYRIGHT files, commit these, and ensure that they stay fresh in CI. As long as we do that, I don't mind if we also commit the JSON file, even though I find it a bit redundant.

[jonathanpallant]

I dropped the changes regarding committing the JSON file.

I can do another PR where I add ./x test tools/check-copyright which will run ./x run tools/generate-copyright and then do a diff to tell you if ./COPYRIGHT.html and ./COPYRIGHT-library.html are OK. But I need a bit of guidance on the difference between ./x run and ./x test.

[Kobzol]

Thank you for reducing the scope of this PR. The code looks good. When I tried to generate the libstd copyright, I got these weird empty entries:

But I suppose it's still the same case as this.

[Kobzol]

I only noticed now that the definition of these (non-trivial) JSON structures is duplicated here and in collect-license-metadata. Eventually it would be nice to share these e.g. through the build_helper crate, but let's not complicate this PR with that.

[jonathanpallant]

But I suppose it's still the same case as this.

I'll poke at generate-license-metadata and see if we can make that better. But in general, reuse really doesn't like files without copyright information (which is kind of the point).

[Kobzol]

I haven't found an easy way to tell reuse to ignore untracked files, sadly. Anyway, that's not relevant to this PR, that's a separate issue.

[jonathanpallant]

If only CI runs the program, and CI doesn't have untracked files, it'll be moot.

[Kobzol]

Indeed. Thank you!

@bors r+

[bors]

📌 Commit 2932833 has been approved by Kobzol

It is now in the queue for this repository.