Skip to content

Package Archiving: Plan

Jump to bottom
Shon Feder edited this page Nov 7, 2024 · 3 revisions

Philosophy

The opam-repository is steadily growing, using a substantial amount of space and inodes. Yet a lot of packages have become stale or uninstallable. We could slim it significantly and reduce power and time waste in the CI and in local solves, by periodically pruning the uninstallable and unmaintained packages.

Plan

Phase 0

Phase 1

Ready to implement

  • Create the ocaml/opam-repository-archive github repository
    • Create a repo file
    • Create a README.md
      • explaining the role of the repo
      • explaining the uses of the repo (opam repository add …)
      • explaining the additional x- fields in the files
      • explaining that files in here may be modified to improve installability and such
  • Make a list of known unavailable or broken packages using opam-repo-ci, opam-health-check and opam admin check --installable
    • make a PR removing them from opam-repository
    • make a PR adding them to opam-repository-archive
  • write a Discuss post listing all the packages that are going to be removed and wait a week for comments
    • in the post, explain the problem with (g)patch (see Phase 0)
    • Push a new before-opam-repository-archive tag on opam-repository before any of the removal
    • Merge the PRs if the discussion above is ok

Preliminaries for Phase 2 from infra team #CI

  • Create a CI for this new repository
    • it should simply lint the files
      • checks that every dependency constraint is of the form "dep" {... & <= "<latest version of dep available in opam-repository at the time of the PR or lower>"} except for "ocaml" (needed for phase 2)
      • checks if the x-reason-for-archival is set (needed for phase 3)
      • checks if the x-opam-repository-commit-hash-at-time-of-archival is set (needed for phase 3)
  • Add the archive repo to the health-check but run it more infrequently (~1/month)

Phase 2 (OCaml < 4.08 -- the least aggressive pruning) - aim: 3 months from Phase 1

  • Call opam admin check --installability, make sure the output is correct and move all the packages it lists to opam-repository-archive #opam
  • Make a PR to move all ocaml packages < 4.08 (oldest version still in use in the wild according to https://repology.org/project/ocaml/versions) to the opam-repository-archive
  • Add a check to the CI lint of the opam-repository: the supported OCaml version is >= 4.08 #ci
  • Announce it in a Discuss post and wait a week for comments then merge

The above steps will be repeated each time we go on with this point with a more recent version of the compiler as bound.

Phase 3 (spring cleaning) - aim: 3-6 months from phase 2

  • Write a Discuss post to advertise the extension fields for maintainers to declare intention of maintainshing for their packages
    • for usability, it is sufficient to add the field to the latest version of a package and then the tooling will resolve this
    • in case of multiple values for this field in multiple versions of a package, the latest version is taken into account
    • The idea is that only the packages with the x-maintenance-intent will be considered for removal when the times come, the others will stay
    • Syntax for the new x-maintenance-intent field:
      • the value would be a list of strings matching on version numbers, with special keywords (latest), (any) and (none), or specific version numbers
      • For example:
        • ["(latest)"] the maintainer will only maintain the latest version
        • ["(latest)" "(latest-1)"] the maintainer will only maintain the latest X.Y.Z version and (X-1).Y.Z
        • ["(latest)" "(latest).(latest-1)"] the maintainer will only maintain the latest X.Y.Z version and X.(Y-1).Z
        • ["(any).(latest)"] the maintainer will maintain every major version X for each X.Y.Z
        • ["(latest).(any).(latest)"] the maintainer will maintain every Y for each X.Y.Z (where X is the latest)
        • ["(any)"] the maintainer will maintain every single versions
        • ["(none)"] the maintainer will not maintain any version
        • ["1.3"] the maintainer will maintain the latest version of "1.3.Z"
        • ["2.(latest)"] the maintainer will maintain the latest minor version specifically of version "2" of the package
    • Phase 3 is planned to happen on TODO: Fix a Date. Make sure that this proposal does allow to write any type of pattern that may apply to your packages and let us know if not, this is our chance to change it beforehand. (TODO: make this point boldface when doing the announcement)
    • TODO: do we need a mechanism to register interest in specific packages (a way to indicate "I depend on this package even though my work is not published in opam-repository")

To repeat each time

  • Use a script that reads every opam-repository package, takes the latest version of each package that has the x-maintenance-intent field and lists all versions that are not maintained anymore #tools #ci
  • use the installability check to ensure that we remove only packages that do not prevent installation of maintained packages #ci
  • Ping maintainers of large sets of packages individually so they can add this field to the latest version of each of their packages
  • Create the PR to remove the packages from the opam-repository and adds it to the opam-repository-archive
  • Announce the removal on discuss
    • once everything is ok merge

Tooling (basically to do once)

  • Support for intent field on dune release, opam publish, opam repo ci linter, ... to ensure it is present #ci #tools
  • Create a script that reads every opam-repository package, takes the latest version of each package that has the x-maintenance-intent field and lists all versions that are not maintained anymore #tools #ci
  • use the installability check to ensure that we remove only packages that do not prevent installation of maintained packages #ci

Phase 4 (coasting)

  • Repeat Phase 3 every 6 or so months
  • Repeat Phase 2 with updated OCaml compiler version when major distributions migrate
Clone this wiki locally