Policy for non-functional packages

[emillon]

This is the summary of a discussion between @kit-ty-kate and I regarding how non-functional packages should be handled in the repository, more precisely in the context of #23649.

The current policy for opam-repository is to only add new packages, and not remove any. However, the maintainers fix package metadata so that installing a package should not result in a build failure.

However, there are some circumstances where there are benefits to making some releases unavailable. For example:

• when a package release has a known security issue
• when a package relies on an external service that does not exist anymore, or through an API that does not exist anymore

In the latter case, the package can be installed, but is not useful at runtime.

In addition, these non-functional packages cost a non-negligible amount of resources in CI: because they are included in the revdeps run of some "core" packages (dune, lwt,uri, the compilers...), they consume time and energy.

The goal of this issue is to list the possible solutions and discuss what should be implemented.

(In addition, having a growing number of packages has some negative consequences in itself: it is more work for the opam maintainers, it takes up disk space (and inodes) and slows down opam update. But this is a separate issue.)

One thing we want to avoid is to make solutions suddenly unavailable: if opam update triggers a reinstall of the project but it has now no solutions, this is a bad experience with no escape hatch. This also breaks the concept of lock file since an explicit version changes in a major way.

Solutions

1: git rm

This is the most invasive solution - just remove the package.

2: available: false

This ensures that the package is not installable, but keeps the metadata history. It makes it possible to copy the opam file in a repository with just this line removed and have it to work.

3: avoid-version

This flag puts a version at the end of the list of candidates, without explicitly forbidding it. (a variant of this opam feature is to only select a version only if explicitly requested but it has not been documented). We could use that as a signal not to trigger revdeps run in CI.

4: x-skip-ci

An x-field, ignored by all users, but implemented in CI to filter packages in the revdeps stage.

5: a configuration option in opam-repo-ci

This would be a way to configure opam-repo-ci to ignore some packages, especially in revdeps runs.

6: using a flag package installed in CI

The idea would be to add depends: "allow-broken-package" or conflicts: "forbid-broken-packages" to these packages. Normal users would allow installation, while CI would reject the installation (by putting forbid-broken-package in the switch invariant, for example).

7: encoding CI-ness in available:

Similar to the above, except it's encoded in available instead of package dependencies.
The goal is to find something that evaluates to true for normal users, and false in CI.

Analysis

Options 1 and 2 fail the requirement of not breaking existing switches.

Option 3 (avoid-version) and 4 (x-skip-ci) are a bit difficult to implement in the revdeps run, which uses opam list to determine which packages to rebuild. In addition, it does not fit with the semantics of avoid-version (with explicit-version on the other hand, it could work).

Option 5 (configuration of opam-repo-ci) is not in line with how opam-repo-ci is configured (usually through opam files themselves) and could cause confusion since changes in that configuration are not visible in the repository.

As for options 6 and 7: the important part is to make sure that the configuration step (being part of the CI group or not) should be an explicit step, and the error messages should make sense, but otherwise they can be suitable.

Recommendation

We recommend the option 7 - encoding the CI-ness in available:. An initial idea was to use available: opam-version < 2.2 since end-users are unlikely to use unreleased versions (while CI uses it), but this requires changing the repository often.

It is possible to leverage's opam support for undefined variable and use the following:

available: !(?ci & ci)

This evaluates to true if ci is unset or false, and to false if ci is set to false

[mseri]

However, there are some circumstances where there are benefits to making some releases unavailable. For example:

* when a package release has a known security issue

* when a package relies on an external service that does not exist anymore, or through an API that does not exist anymore

In the latter case, the package can be installed, but is not useful at runtime.

In those cases, we do allow packages to be marked as unavailable.

I do agree with your analysis that adding available:false but never removing packages does end up growing works for maintainers, solvers and wastes disk space.

Note that for the time being git rm is a problem since the way opam does updates would break on some systems (e.g. macos). So it would constantly break opam switches and updates.

I don't like any of the solutions so far, with a slight preference for a mix of 2 and 3. Where 2 is for broken/unusable packages and 3 for alpha releases.

I do think we should have some sort of "opam-repository releases" where we release a pruned index or were opam uses a pruned index (e.g. removing all the unavailable opam files and only preserve the latest of each mmajor releases) but I don't have a well formed idea for that (so far). It is also made harder by the fact that many packages do not follow semver so minor version bumps may relate to different/incompatible packages.

[mseri]

I would also add this type of removals as part of this discussion: #23649

[shonfeder]

I believe this is subsumed by the archiving process and policy under discussion in #23789 . Accordingly, I will close this. But please re-open or LMK if I am in error!