Look into ways to decouple some crate metadata from Cargo.toml

[jtgeibel]

An outline of our current scheme for crate and version level metadata (with quoted fields coming from Cargo.toml:

• Crate-wide metadata
  • description
  • homepage
  • documentation
  • repository
  • categories
  • keywords
  • badges (largely deprecated)
  • readme (we cache the most recent readme for search)
• Version specific metadata
  • features
  • license
  • crate_size
  • published_by
  • authors

With the exception of authors, all of the version specific metadata is clearly immutably tied to the specific crate that was published under that name and version. For the crate-wide metadata, it can be annoying to be required to publish a new version to update these fields.

It would be nice if the crate-wide metadata was managed through the web interface (Edit: and eventually a stable API to support cargo integration and other tooling), instead of via Cargo.toml.

Suggestions

For optional fields, we can probably do something like:

• If the latest stable doesn't specify the field, then it can be added/edited in the frontend UI. If the field is defined, the frontend can direct the user to first publish a new version without the field.
• If a new stable release overwrites the field by adding it to Cargo.toml, then we overwrite the value and return the previous contents in a warning to the user. (If this was a mistake, then the user has to publish a new crate that is otherwise a no-op. This is a bit unfortunate, but seems better than forcing a new publish to change any piece of metadata.)

The only required crate-wide field is description. I don't see changing this as high-priority, so we can focus on the optional fields.

A few additional notes

We've had bugs in the past where publishing 0.2.1 and backporting to 0.1.12 would overwrite metadata unexpectedly if they were published in that order. Off the top of my head, I don't know exactly how we handle pre-releases and metadata updates. My point is, saving the metadata from the "right" release isn't entirely straightforward and might cause unexpected results. It would be nice if we could reduce the user impact of this complexity.

The documentation field is a bit interesting, because there is no way for a crate version to link directly to version specific information. We fixup any docs.rs links automatically in the frontend so that they automatically point to the correct version specific documentation (matching the behavior for when a value is not provided). This means that user can either have automatic version-specific links to docs.rs, or one link to their most recent 3rd party documentation for all their releases. (Edit: I think we could improve this field by supporting some type of {$version} placeholder where crates.io would automatically customize the URL for each release but it would only need to be set once globally for the crate and could be easily modified if a project's 3rd party documentation moves.)

[Diggsey]

I would still like the non-versioned information to be managed through cargo, not through a crates.io-specific interface. (Although it's fine if crates.io also has a way to manage that as well). This would allow that information to be easily updated by CI/CD pipelines, and could be managed in a way that is not tied specifically to crates.io.

This is why I suggested putting that information in a separate file, which is also read by cargo publish.

[Ekleog]

Random thought: the information could also be written in Cargo.toml, and just stripped out of it by cargo publish before uploading to the repo?

Not sure whether this idea holds its weight, though, but it'd avoid most if not all the issues around deprecating the authors field I saw discussed in the RFC there.

[SOF3]

Currently crates.io cleans Cargo.toml with Cargo.toml.orig in a separate file. What about changing them to Cargo-version.toml and Cargo-package.toml in two separate files?

[jtgeibel]

Currently crates.io cleans Cargo.toml with Cargo.toml.orig in a separate file.

This is done by cargo actually.

I would still like the non-versioned information to be managed through cargo, not through a crates.io-specific interface.

I would also like to provide a stable API and maybe a cargo package-meta command or something. I certainly see the advantage of being able to modify this metadata in a stable way that can be accessed via tooling, but I think building this into the website first would satisfy most initial use-cases. Over the years there has been some discussion of a /api/v2/ namespace where we could provide better a better long-term API and eventually fix some warts in our existing APIs. I'd love to see something similar to the alternative registries RFC to stabilize some APIs around managing package-level metadata.

[kornelski]

Please keep alternative registries in mind. If the solution is crates-io-specific, then other registries will struggle to manage this data.

This data could theoretically be added to the git-based registry index, but the amount of extra data could be painful. However, the planned HTTP-based registry protocol could accommodate extra data. rust-lang/rfcs#2789

[VulpineAmethyst]

Regarding the author field: my opinion is that it should be injected in to the crate when it is fetched. This would allow the field to change as needed, retroactively, without requiring updating and resigning every affected crate.

[wizzwizz4]

It might help to think of this abstractly. We've got some data that is:

• Constant (e.g. crate names)
• Historical (e.g. the code that was published)
• Mutable (e.g. the latest unyanked version)

I think this is roughly how we're thinking of it at the moment. However, this isn't expressive enough to effectively represent all the information we want. There are other properties to consider:

• Mutable? (Does it change?)
• Historical? (Will we keep the historical record?)
• Certain? (Are we free from needing to correct the record?)
• Revocable? (Might we need to remove access to it? Includes yanking.)
• Indirect? (Does it refer to the thing we actually want to keep track of?)

(This list might not be exhaustive.) I've had a go at placing some things into this framework:

	M	H	C	R	I
crate name			✔️
current version	✔️		✔️
a release		✔️	✔️	✔️
a dependency		✔️	✔️		✔️: CR
an version's author		✔️		✔️
an author's current name	✔️		✔️	✔️	✔️: HR
contact details	✔️			✔️
download statistics	✔️	✔️		✔️

I haven't described indirection well enough here, because it has three levels of potential mutability:

• A change in the reference (e.g. changing mycrate:3.0 to mycrate:3.1)
• A change in the referencee (I have no examples, so *x = 5 is my example)
• A change in how the reference is looked up (e.g. mycrate version 3.1.2 is released, so mycrate:3.1's meaning has changed)

Are there any glaring holes in this framework?

[wizzwizz4]

Hmm… Really, we only need to keep things that affect builds immutable:

• The public API;
• The code's behaviour (including interaction with undocumented aspects of rustc); and
• Anything the build script reads. (For most scripts, it should be possible to find this statically, but it's not solvable in general. Build scripts already cause problems for immutable builds, though, so this probably isn't too big of a concern.)

We'd probably need guarantees from rustc that adding or removing comments (anywhere) or doc attributes, lint attributes etc. (outside of macro invocations) wouldn't change the build, if we're to do anything useful with this. (Plus, we'd need a new hash function.)

Though comments are most likely to contain metadata that should logically be mutable, so maybe we don't even need to go that far.

[Diggsey]

Everything downloaded from crates.io as part of a build must be immutable, even if it doesn't affect the final result. I would not use a registry that does anything else.

It's fine for there to be additional files/metadata that are "mutable" (although mutability is a lie anyway...) but that should not be downloaded when I run cargo build on a downstream crate.

When you publish a crate you are publishing a snapshot for people to depend on. You don't get to later change that snapshot, at all, that's what "publishing" means. If you publish a book, and make a mistake, you can't just locate all the copies and update them (and legally you would have no right to). All you can do is publish a new revised copy.

[wizzwizz4]

Does Cargo necessarily have to download the original source code for dependencies? I can imagine a cargo vendor --no-doc that doesn't download the info required for documentation, which would save on bandwidth for bandwidth-restricted users, build servers etc.. (Of course, we'd need some way to verify that we were actually getting an unmodified copy of the program, necessitating an additional stripped AST hash (or something) that would vary as the AST-stripping algorithm changed, so perhaps this isn't worth it.)

The main reason I raised this is that many licenses recommend adding a snippet of license text at the top of the source file. That, and other practices of embedding metadata in source files, puts a limit on how much we can do this.

Is the documentation logically part of the program? I could make arguments either way – consistency with README.md versus the argument-from-literate-programming.

[VulpineAmethyst]

Everything downloaded from crates.io as part of a build must be immutable, even if it doesn't affect the final result. I would not use a registry that does anything else.

It's fine for there to be additional files/metadata that are "mutable" (although mutability is a lie anyway...) but that should not be downloaded when I run cargo build on a downstream crate.

When you publish a crate you are publishing a snapshot for people to depend on. You don't get to later change that snapshot, at all, that's what "publishing" means. If you publish a book, and make a mistake, you can't just locate all the copies and update them (and legally you would have no right to). All you can do is publish a new revised copy.

Digital distribution is different from physical distribution in some pretty fundamental ways, in that you can make emendations at any time so that all future distributions of the product reflect those changes. And it's absurd to assert that changing ancillary metadata necessitates a new publication when there are no substantial changes to the work. What I'm proposing, in effect, is that crate-wide metadata be stapled to the package when it is downloaded, rather than requiring packages to contain a fixed version of that metadata that is then immutable.

[Diggsey]

What I'm proposing, in effect, is that crate-wide metadata be stapled to the package when it is downloaded, rather than requiring packages to contain a fixed version of that metadata that is then immutable.

I understand what you're proposing. I'm saying that, as a consumer of a crate, I did not agree to download your new ancillary metadata. When I add a dependency to my lockfile, I'm choosing to add it as it was. You don't get to decide in future that actually I wanted something else, even if the change should not affect the output of the build.

crates.io serves both crate publishers and crate consumers, it doesn't just serve the publishers.

[VulpineAmethyst]

What I'm proposing, in effect, is that crate-wide metadata be stapled to the package when it is downloaded, rather than requiring packages to contain a fixed version of that metadata that is then immutable.

I understand what you're proposing. I'm saying that, as a consumer of a crate, I did not agree to download your new ancillary metadata. When I add a dependency to my lockfile, I'm choosing to add it as it was. You don't get to decide in future that actually I wanted something else, even if the change should not affect the output of the build.

That ancillary metadata is already present in existing crates; moving it to a location where it can be more readily modified does not actually change what you're downloading, nor should it require additional network requests.

[Diggsey]

It's not about additional network requests, it's about the content of what I'm downloading. At the moment, if I add a crate to a lockfile, then I know what will be downloaded for that crate. It's never going to change unless I update the lockfile. That's the one thing I require as a crate consumer from a registry.

[wizzwizz4]

You don't get to decide in future that actually I wanted something else, even if the change should not affect the output of the build.

Surely the Cargo.lock is only for specifying a consistent build? If you want exactly the same thing, you use something like cargo vendor – “you don't get to decide” that all registries must serve those files to you in perpetuity. If it's a choice between the files changing (without affecting the build) and the crate version being completely removed, which do you pick?

“What you're downloading” is actually somewhat nebulous already. It's not the TLS or HTTPS, it's not the file attributes – it's already some subset of what you end up with. Everything's a trade-off:

• What guarantees do we currently have?
• What guarantees do people currently rely on?
• What guarantees would be useful?
• What are the natural guarantees to have?

Focusing on only the first of these questions is, imo, a mistake – especially if you're assuming guarantees that aren't actually there, just because you haven't encountered a violation of those guarantees yet. (I don't actually know where these guarantees are currently documented, if they even are.)

[Diggsey]

“you don't get to decide” that all registries must serve those files to you in perpetuity.

No, it's something the registry gets to decide. I'm saying I would not use crates from a registry that does not guarantee this.

If it's a choice between the files changing (without affecting the build) and the crate version being completely removed, which do you pick?

My personal preference would be for crate contents to be completely immutable unless compelled by law to be removed, in which case they are deleted. If the crates.io team want to permit crate deletion in more circumstances, then that's their prerogative: as long as it continues to be exceedingly rare then I have no real issue with that, although I think it won't scale, and it will be difficult to enforce rules consistently.

Changing crate contents is a different thing entirely: I would rather it never happen, but if it has to happen then it should be an extreme circumstance, and the change should be one made by the crates.io team, it shouldn't be something the crate author can do.

(By "crate contents" I am specifically referring to everything that is downloaded from the registry for that crate version when you run cargo build on a downstream crate.)

“What you're downloading” is actually somewhat nebulous already. It's not the TLS or HTTPS, it's not the file attributes – it's already some subset of what you end up with. Everything's a trade-off:

Not really: the part that the crate author has control over is very well defined. In using crates.io I'm trusting the crates.io team and involved protocols. That doesn't mean I trust every crate author, and even if I trusted them I wouldn't trust that their API tokens will never be compromised in the future.

[pietroalbini]

I don't think mutable crate metadata needs to be downloaded by cargo build at all: that metadata is not needed for builds, and Cargo must not expose it to the compiler to avoid losing reproducibility. The users that could benefit from that metadata (third-party tools and alternate crates.io frontends like lib.rs) can download it themselves from the registry, and ideally there should be a standard format so the same tools can be used for alternate registries too.

Still, I'd love to better understand @Diggsey's concerns around cargo build. What would the problem be if cargo build were to download such metadata but not use nor expose it to the build?

[wizzwizz4]

I'm saying I would not use crates from a registry that does not guarantee [serving files in perpetuity].

Well, crates.io already doesn't. Realistically, only a registry you control can guarantee this, where “in perpetuity” means “for as long as you need it”.

My personal preference would be for crate contents to be completely immutable unless compelled by law to be removed, in which case they are deleted.

“Always X, unless Y” isn't an elegant system. How much will break when the first fairly popular crate has illegal numbers snuck into the comments by some developer protesting something, and versions 0.2.3 to 1.4.5 get DMCA'd? Or, more realistically, if somebody copied sections of an ISO standard's text to describe the algorithm they were implementing, or if a chunk of text was grabbed from Wikipedia but the author forgot to add attribution…

I completely understand wanting some guarantee as to when a version is identical (for which there's already a checksum in the Cargo.lock, though I'm not sure how that's used), and for this to be a rare event (i.e., not something crate authors can do themselves under most circumstances) – and I'm completely behind the “identical builds” guarantee (otherwise what's the point of having version numbers?) –, but I can think of plenty of places where a change to the comments, or perhaps a variable name or two, could be a viable alternative to taking a version down entirely.

I think I can justify treating comments as somewhat metadata-like, though I'm not sure whether exposing a second “TokenTree hash” (to allow end-users to verify that the “identical builds” invariant is upheld when a version changes, without needing to trust the registry) would be a good idea.

[Diggsey]

“Always X, unless Y” isn't an elegant system. How much will break when the first fairly popular crate has illegal numbers snuck into the comments by some developer protesting something, and versions 0.2.3 to 1.4.5 get DMCA'd

Not sure where you're going with this: unless we build crates.io on a blockchain, then legally crates.io can be compelled to remove crates and there's nothing we can do about that. The fact that we still have to remove crates in some cases is not an argument to remove them in more cases, or an argument to allow them to be modified.

Are you suggesting that, by modifying a crate, we can remove the offending content without breaking builds? Perhaps, but having modifiable metadata does not help unless the offending content happens to be in that metadata, so are you now suggesting that the whole crate should be modifiable?

Still, I'd love to better understand @Diggsey's concerns around cargo build. What would the problem be if cargo build were to download such metadata but not use nor expose it to the build?

There are a ton of issues here, to list a few:

• What if I'm building a docker container? Then everything affects the build, even if you define it not to. Even if it is later deleted off the file-system, it still exists in that layer.
• What if the metadata has been modified to contain illegal content? Even if I vetted my dependencies at the time of adding them, I could now be unknowningly downloading illegal content on any number of machines.
• What if the build script for the crate contains a back-door which is only triggered after a change to the metadata? Now tools like cargo-audit can no longer reliably flag security issues.
• What if the metadata is updated to contain a malicious payload which is later executed by another security hole that would otherwise not be exploitable?
• What if the metadata is updated to contain innocent content which happens to falsely trigger a security alert?
• What if we get our definition of "changes that do not affect the build" wrong?

And all of this to solve a non-existent problem.

1. Anything that does not affect cargo build also does not need to be downloaded by cargo build.
2. If you need to make minor changes that might affect the build, then release a new minor version. That's the whole point of having minor and patch versions of a crate.

[wizzwizz4]

Perhaps, but having modifiable metadata does not help unless the offending content happens to be in that metadata, so are you now suggesting that the whole crate should be modifiable?

I'm suggesting that there are cases where it's useful to view everything that doesn't affect the build as modifiable metadata. (Even crate names, though the cases where that's something you can actually change are very, very rare.) Some are more convoluted than others, but I can make a case for all the ones I've thought of, and the chances of none of this coming up in practice is pretty slim. (Zero, if you count historical precedent, though I did use the future tense.)

I'm not saying it's a good idea for all cases (and hence, I'm not saying that all metadata should be modifiable by the crate author) but I think that something that actually happens – and that might be legally mandated even if everyone agrees that not doing it is more important than the (very important) reasons it currently happens – shouldn't be a tacked-on “kinda breaks stuff but we ignore it” aspect of the registry system.

• What if I'm building a docker container? Then everything affects the build, even if you define it not to. Even if it is later deleted off the file-system, it still exists in that layer.

The timestamps also affect the build, so no new problem is introduced. (And a crate version going missing entirely – which mutable metadata is in part a solution to – would affect the build a lot more.)

• What if the metadata has been modified to contain illegal content?
• What if the build script for the crate contains a back-door which is only triggered after a change to the metadata?

What if the build script for the crate contains a back-door which is only triggered after a change to the system clock? My advice: don't put malware on your computer. Anyone with access to the registry can already do these. (What if the registry gets cracked?) Again, no new problem is introduced.

• What if the metadata is updated to contain a malicious payload which is later executed by another security hole that would otherwise not be exploitable?
• What if the metadata is updated to contain innocent content which happens to falsely trigger a security alert?
• What if we get our definition of "changes that do not affect the build" wrong?

Good points!

The biggest difficulty, as you (and others) have pointed out, is build scripts. No thanks to those nasty mathematicians (Gödel, Tarski, Church, Turing, etc.), it's not actually possible, in the general case, to know what a build script is doing. Fortunately for us, though, we don't need to consider the general case: most build scripts will call std::env functions with a literal argument directly, and call functions that call std::fs with constant-knowable file paths, and verifiably not interact with the environment in any other way. Ensuring the determinism of arbitrary build scripts is probably out of scope of this issue, though.

For crates without build scripts, I'm guessing that non-doc comments and Cargo.toml metadata don't affect the build – unless an env!() macro is involved, but we can tell which metadata is being inspected that way. In general, environment-dependent builds are not very reproduceable, but I think we should make an effort for the CARGO_ environment variables, for crates without a build script.

Getting the definition of “changes that do not affect the build” right is already really important – there are many places in the toolchain where non-determinism can crop up as-is, so the changes I'm proposing wouldn't be that special in that regard.

It's just occurred to me that debug symbols are part of the build, so line numbers and internal variable names are probably not up for grabs – but I see no reason that modifying comments shouldn't be okay, unless rustc uses file hashes for name mangling or something. It definitely won't in RFC 2603 (rust-lang/rust#60705).

And all of this to solve a non-existent problem.

If it were non-existent, we wouldn't be talking about it.

[wizzwizz4]

rustc_symbol_mangling::legacy::get_symbol_hash doesn't look like it's got anything that would be affected by modifying comments, so it doesn't look like my harebrained “comments are kind of metadata too!” proposal has any barriers in rustc; the only thing it seems to affect is the crate checksum. It's probably not a good thing to focus on right now, though, thinking about it.

[Diggsey]

What if the build script for the crate contains a back-door which is only triggered after a change to the system clock? My advice: don't put malware on your computer. Anyone with access to the registry can already do these. (What if the registry gets cracked?) Again, no new problem is introduced.

I would like to avoid putting malware on my computer, but your proposal is making that difficult by breaking the link between crate versions and their contents. This is something which cargo audit relies on in order to work...

The biggest difficulty, as you (and others) have pointed out, is build scripts.

The biggest difficulty is the shear amount of unnecessary complexity that this would add.

If it were non-existent, we wouldn't be talking about it.

Well nobody's actually given a reason to do this, other than an emotional aversion to publishing new crate versions. Maybe you can explain why information that is mutable needs to be downloaded by cargo build?

Pros:

• You can change comments in some situations provided you don't change line numbers... As long as they are not doc comments.

Cons:

• All reproducibility guarantees are out the window unless you can exactly match rust's parsing rules to determine if anything changes. Everything which detects changes based on the filesystem is broken. Crate hashes are broken.

TBH, even if none of the cons existed, being able to change comments under these specific rules is still an anti-feature, because having a feature that only works in such narrow circumstances is a footgun by itself.

[VulpineAmethyst]

If it were non-existent, we wouldn't be talking about it.

Well nobody's actually given a reason to do this, other than an emotional aversion to publishing new crate versions. Maybe you can explain why information that is mutable needs to be downloaded by cargo build?

Up to now, author information has been attached to crate releases in a way which makes changes to that field require a new release. The problem with this approach is that name changes (for various cultural reasons) or address changes (for other reasons) means that ensuring that all releases have correct author information requires making a new release and does nothing about existing releases. I want to decouple this in a way that allows cargo to (continue to?) be able to provide such information.

I fully recognize that there are other approaches (e.g. fetching such information on demand), but on e.g. a metered connection I don't want tools making network requests without my knowledge.

[wizzwizz4]

This is something which cargo audit relies on in order to work...

Not really. A change in metadata isn't going to change whether non-malicious code is a security vulnerability or not. Hmm… I don't think a change in metadata is going to change whether non-malicious code is a security vulnerability any more than a change to the build environment is likely to, but that won't affect cargo audit anyway, because cargo audit is a list of known-bad versions, not a list of known-good versions. A crate with such a metadata-triggered vulnerability:

• doesn't have a deterministic-under-metadata-change build, making it ineligible for a metadata change; and
• has vulnerabilities regardless.

Maybe you can explain why information that is mutable needs to be downloaded by cargo build?

Requirements of the licenses, I think. Though it doesn't have to be associated with the crate contents (if you ignore my “comments” proposal), and so wouldn't have to be “part of the version”; it could be part of the registry instead (or something else entirely).

Apart from that, mere backwards-compatibility is the only reason for it. It's information that was available via the env! macro. (I don't know whether what cargo does to the environment has the same backwards-compatibility guarantees as std, but decoupling them entirely would take at least a new edition.)

All reproducibility guarantees are out the window unless you can exactly match rust's parsing rules to determine if anything changes. Everything which detects changes based on the filesystem is broken.

1. Rust throws out comments very early on in parsing. They're not even exposed by proc_macro.
2. Unless you delete and re-download from the registry, this won't affect anything.

Crate hashes are broken.

Yeah. I have no idea how much relies on those, so that's probably enough to ignore my comment proposal. (Though decoupling other metadata from Cargo.toml might well break hashes, too.)

TBH, even if none of the cons existed, being able to change comments under these specific rules is still an anti-feature, because having a feature that only works in such narrow circumstances is a footgun by itself.

But it's a feature that just so happens to line up with nearly all the reasons for deleting (rather than yanking) a crate version. Deleting crate versions is also a footgun.

[wizzwizz4]

Side-stepping this issue: can anyone think of a way to completely separate metadata from the crate versions? (Preferably in a way that lets us have mutable crate metadata and correctable version metadata.)

[pietroalbini]

There are a ton of issues here, to list a few: [...]

Thanks for the examples @Diggsey, I now agree that cargo build should not download that metadata!

In general I think that changing parts of the source code are out of scope for this proposal, and in my personal opinion that's not a feature I'd like to see implemented. Can we focus on how to provide mutable access to the metadata mentioned by Justin in the issue body instead?

[ssokolow]

I'd be OK with some kind of "what Crates.io displays may be newer information, similar to staging stuff in git, and it will reject publish if the crate hasn't yet been updated to match", but I want to preserve that a crate version (including metadata) is an immutable snapshot of the state of its world at the time of publish.

Compromising that feels like setting yourself on the road to people maintaining their own Crates.io proxy-caches to restore that property and possibly sabotaging efforts to assure enterprise users that the purpose of their internal processes relates to availability/uptime and auditing procedures/processes/workflow, not stability of contents.

[heaths]

Could this also include deprecation information a la nuget.org and npmjs.org (and related tools)? That is, a package in its entirety could be marked deprecated (with an optional array of packages that replace it, since a package may be refactored into multiple) or a range of versions e.g., a 1.x is deprecated and only 2.x is supported.

We are looking to deprecate a bunch of packages in the not-so-distant future and many will have supported replacements. In most other languages we support, we have either direct support from package managers (the site and tools, though I agree with the statement above that it's not critical for crates.io to have UI for this just yet) or a work around like adding a deprecation error message in Python's setup.py.

More context may be found in https://rust-lang.zulipchat.com/#narrow/stream/246057-t-cargo/topic/Deprecating.20crates/near/434808268.