Skip to content

Latest commit

 

History

History
479 lines (343 loc) · 23.1 KB

README.md

File metadata and controls

479 lines (343 loc) · 23.1 KB
Raw

Cardano Haskell package repository ("CHaP")

Top How-Tos

This is a Cabal package repository ("CHaP") whose purpose is to contain all the Haskell packages used by the Cardano open-source project which are not on Hackage.

The package repository itself is available here. It is built from a git repository which contains the metadata specifying all the package versions. The package repository is built using foliage.

Help!

If you have trouble, open an issue, or contact the trustees: @intersectmbo/cardano-haskell-packages-trustees

Background

This section explains some concepts that are useful for understanding and working with CHaP.

What is a Cabal package repository?

A package repository is essentially a mapping from package name and version to the source distribution for the package. Most Haskell programmers will be familiar with the package repository hosted on Hackage, which is enabled by default in Cabal.

However, Cabal supports the use of additional package repositories. This is convenient for users who can't or don't want to put their packages on Hackage.

Cabal package repositories and source-repository-package

Using source-repository-package stanzas is another common way of getting dependencies that are not on Hackage. Both have their place: CHaP gives us proper versioning and simpler setup, source-repository-packages are useful for ad-hoc use of patched or pre-release versions.

Crucially, additional Cabal package repositories like CHaP and source-repository-package stanzas are compatible and source-repository-packages always win. That is, they interact in the same way as Hackage and source-repository-packages do. This gives us behaviour that we want: ad-hoc source-repository-package stanzas will override packages from Hackage or CHaP.

Using CHaP

To use CHaP with cabal, add the following lines to your cabal.project file:

repository cardano-haskell-packages
  url: https://chap.intersectmbo.org/
  secure: True
  root-keys:
    3e0cce471cf09815f930210f7827266fd09045445d65923e6d0238a6cd15126f
    443abb7fb497a134c343faf52f0b659bd7999bc06b7f63fa76dc99d631f9bea1
    a86a1f6ce86c449c46666bda44268677abf29b5b2d2eb5ec7af903ec2f117a82
    bcec67e8e99cabfa7764d75ad9b158d72bfacf70ca1d0ec8bc6b4406d1bf8413
    c00aae8461a256275598500ea0e187588c35a5d5d7454fb57eac18d9edb86a56
    d4a35cd3121aa00d18544bb0ac01c3e1691d618f462c46129271bccf39f7e8ee

The package repository will be understood by cabal, and can be updated with cabal update. You must run cabal update at least once so cabal can download the package index!

The index-state for the package repository can also be pinned as usual. You can either just use a single index-state for both Hackage and CHaP:

index-state: 2022-08-25T00:00:00Z

or you can specify a different index-state for each repository:

index-state:
  , hackage.haskell.org      2022-12-31T00:00:00Z
  , cardano-haskell-packages 2022-08-25T00:00:00Z

Note that a second index-state stanza completely ovverides the first, so

index-state: 2022-12-31T00:00:00Z
index-state: cardano-haskell-packages 2022-08-25T00:00:00Z

would override the index-state for Hackage to HEAD (since HEAD is the default).

... with haskell.nix

To use CHaP with haskell.nix, do the following:

  1. Add the package repository to your cabal.project as above.
  2. Setup a fetcher for the package repository. The easiest way is to use a flake input, such as:
inputs.CHaP = {
  url = "github:intersectmbo/cardano-haskell-packages?ref=repo";
  flake = false;
};
  1. Tell haskell-nix to map the CHaP url to the appropriate nix store path using the inputMap argument of one of haskell.nix project functions. Using cabalProject this would look like the following:
cabalProject {
  ...
  inputMap = { "https://chap.intersectmbo.org/" = CHaP; };
}

To build some of the pacakges from CHaP, some C libraries will need to be installed. If you don't have those libraries installed cabal may pick an older library version that might not compile. An easy way to add those libraries is using the overlays provided by iohk-nix. For example plutus-core==1.21.0.0 depends on libblst, haskell-nix-crypto and crypto are the two overlays that would make it available during the build.

When you want to update the state of CHaP, you can simply update the flake input (in the example above you would run nix flake lock --update-input CHaP).

If you have CHaP configured correctly, then when you run cabal build from inside a haskell.nix shell, you should not see any of the packages in CHaP being built by cabal. The exception is if you have a source-repository-package stanza which overrides a dependency of one of the packages in CHaP. Then cabal will rebuild them both. If this becomes a problem, you can consider adding the patched package to CHaP itself, see below.

Updating dependencies from CHaP

If you have a Haskell project which has bounds on CHaP packages, we provide a script which will update all of those bounds to pin the latest major version of each that is released to CHaP.

You can run it like so: nix run --update-input CHaP --no-write-lock-file "github:intersectmbo/cardano-haskell-packages#update-chap-deps" pkgA pkgB. This will update the bounds for everything from CHaP, except for those on pkgA or pkgB. This lets you blacklist packages that you don't want to update (e.g. because you know that the update will break you, or it's your own package!).

Creating a repository like CHaP

If you just want to test changes to CHaP, you should make a fork. If you want to replicate the setup from scratch you can clone this template.

Contributing packages and revisions

This section explains how to contribute to the main content of CHaP: packages and revisions. The contribution itself should be made in a PR.

Requirements for including a package

Monotonically increasing timestamps

When adding a package, it is important to use a timestamp (see below) that is greater than any other timestamp in the index. Indeed, cabal users rely on the changes to the repository index to be append-only. A non append-only change to the package index would change the repository index state as pinned by index-state, breaking reproducibility.

This condition is enforced by the CI, and we only allow FF-merges in order to ensure that we are always checking a linear history.

Tips for working with timestamps:

  • Most of the scripts will insert timestamps for you, e.g. ./scripts/add-from-github.sh
  • If you have a PR and the timestamps are now too old (e.g. because someone else made a PR in the meantime), then see below for tips
  • If you want to get a suitable timestamp for some other reason, ./scripts/current-timestamp.sh will produce one

No extra build configuration beyond what is given in the cabal file

When downstream users pull a package from CHaP, cabal will build it based only on the information in the cabal file. This means that if your package needs any additional configuration to build, then it will simply be broken for downstream users unless they replicate that configuration.

Typical examples of this are anything that you add in cabal.project:

  • constraints
  • allow-newer
  • source-repository-package

This is enforced by the CI, which will build newly added packages in PRs.

How to add a new package version

Package versions are defined using metadata files _sources/$pkg_name/$pkg_version/meta.toml, which you can create directly. The metadata files have the following format:

# REQUIRED timestamp at which the package appears in the index
timestamp = 2022-03-29T06:19:50+00:00
# REQUIRED URL pointing to the source code tarball (not necessarily a sdist)
url = 'https://github.com/intersectmbo/ouroboros-network/tarball/fa10cb4eef1e7d3e095cec3c2bb1210774b7e5fa'
# OPTIONAL subdirectory inside the tarball where the package is located
subdir = 'typed-protocols'

... from GitHub

There is a convenience script ./scripts/add-from-github.sh to simplify adding a package from a GitHub repository.

$ ./scripts/add-from-github.sh
Usage add-from-github.sh [-f OVERWRITE_VERSION] REPO_URL COMMIT-SHA [SUBDIRS...]

        -f OVERWRITE_VERSION      (DANGEROUS) uses OVERWRITE_VERSION as the package version instead of the one from the tarball
        REPO_URL                  the repository's Github URL
        COMMIT_SHA                the commit SHA for the package source
        SUBDIRS                   the list of relevant sub-directories

For example, to add a new version from plutus's plutus-core and plutus-ledger-api, etc from commit 75267027f157f1312964e7126280920d1245c52d, run

./scripts/add-from-github.sh "https://github.com/intersectmbo/plutus" 75267027f157f1312964e7126280920d1245c52d plutus-core plutus-ledger-api plutus-tx plutus-tx-plugin prettyprinter-configurable

The script will:

  1. Find the cabal files in the repo (either at the root or in the specified subdirectories)
  2. Obtain package names and versions from the cabal files
  3. Create the corresponding meta.toml files
  4. Commit the changes to the git repository

You can tell the script to override the package version either by passing the version explicitly or by adding a "revision number" (see below).

How to add a new package metadata revision

CHaP supports package metadata revisions just like Hackage. These allow you to provide an edited cabal file for a package version. The primary use of this is to tweak the dependency bounds of a package. In principle you can change other things too, but this is generally frowned upon.

This repository contains a convenience script for adding a revision to CHaP:

$ ./scripts/add-revision.sh _repo PACKAGE_NAME PACKAGE_VERSION

_repo needs to point to a built package repository. It will add a new revision and copy the current cabal file in as the revised cabal file. You can then edit that file and commit the result.

How to deprecate a package

CHaP supports package version deprecations just like Hackage. These allow you to make a package "not-preferred" by the cabal solver (note that the solver will still pick a deprecated package version if it cannot pick a non-deprecated one).

There is not currently a script for adding a deprecation, but you can find examples by serarching for "deprecations" in the repository. Deprecations must include a timestamp (like all events) and indicate the new deprecation state (so package versions can also be un-deprecated).

How to add a patched versions of a Hackage package

CHaP should mostly contain versions of packages which are not on Hackage.

If you need to patch a version of a package on Hackage, then there are two options:

  1. For short lived forks, use a source-repository-package stanza by preference.
  2. For long-lived forks (because e.g. the maintainer is unresponsive or the patch is large and will take time to upstream), then we can consider releasing a patched version in CHaP.

The main constraint when adding a patched version to CHaP is to be sure that we use a version number that won't ever conflict with a release made by upstream on Hackage. There are two approaches to doing this:

  1. Release the package in CHaP under a different name (for the fork). This is very safe, but may not be possible if the dependency is incurred via a package we don't control, as then we can't force it to depend on the renamed package.
  2. Release the package under a version that is very unlikely to be used by upstream. The scheme that we typically use is to take the existing version number, add four zero components and then a patch version, e.g. 1.2.3.4.0.0.0.0.1.

IMPORTANT: if you release a patched package to CHaP, make sure to open an issue about it so we can keep track of which patched packages we have. Ideally, include the conditions under which we can deprecate it, e.g. "can deprecate either when it's fixed upstream or when package X removes their dependency on it".

How to update Hackage index used by CHaP

If one of your packages requires a newer version of a package published on Hackage, you will need to run: nix flake lock --update-input hackage-nix. hackage.nix is automatically updated from Hackage once per day. If things still don't work because the version of the package is not available you'll need either wait for the automatic update or make a PR to hackage.nix first and then rerun the above command.

Releasing CHaP packages to Hackage

It's totally fine to release a package in CHaP to Hackage. The thing to avoid is to have the same package version in both repositories. The simplest solution is to just make sure to use a higher major version number when you start releasing to Hackage, even if this looks a bit odd. For example, if CHaP contains X-1.0 and X-1.1, then the first Hackage release should be X-1.2 or X-2.0.

Building and testing CHaP

For most contributors this section is not going to be necessary, and you can rely on the CI.

However if you are making a large number of changes (e.g. many revisions), it can be useful to test your work before making a PR.

How to get the built Cabal package repository

The Cabal package repository itself is built using the tool foliage. You can either fetch the latest version which is stored in git; or build it yourself locally, which can be convenient or necessary if you have local changes.

... by downloading it from Github

The built repository is stored in the repo branch of CHaP itself. You can get the contents of the repo branch from Github at https://github.com/intersectmbo/cardano-haskell-packages/archive/refs/heads/repo.zip .

Or you can check out that branch and copy the contents, e.g.

git checkout repo
git merge --ff-only
cp -aR . _repo
git checkout -

You can also check it out as a worktree:

git worktree add _repo repo

When using this later, remember to pull before creating a revision.

... by building it locally

foliage is available in the Nix dev shell, which you can get into using nix develop.

To build the repository, run foliage build -j 0 --write-metadata. This will build the repository and put it in _repo.

How to test changes

Sometimes it is useful to test in advance how a new package or a cabal file revision affects things.

First of all, build the repository. For the rest of this section we will assume the built repository is in /home/user/cardano-haskell-packages/_repo.

... by building packages with cabal

You can test a locally built CHaP with a small test project consisting of just a cabal.project file:

-- Give it a different name to avoid cabal confusing it with the
-- real CHaP
repository cardano-haskell-packages-local
  -- Point this to the *built* repository
  url: file:/home/user/cardano-haskell-packages/_repo
  secure: True
  -- You can skip the root-keys field

-- Add all the packages you want to try building
extra-packages:
  , cardano-prelude-0.1.0.0

You need to tell cabal about the new repository with cabal update (you might need to clear out ~/.cabal/packages/cardano-haskell-packages-local if you've been editing your repository destructively).

Then you can build whatever package version you want with cabal:

$ cabal build cardano-prelude

You can troubleshoot a failed build plan using the cabal flags --constraint, --allow-newer and --allow-older. Once you have obtained a working build plan, you should revise you cabal file with appropriate constraints.

... by building packages with Nix

You can build packages from CHaP using Nix like this:

nix build
  --override-input CHaP path:/home/user/cardano-haskell-packages/_repo
  '.#"ghc92/plutus-core/1.1.0.0"'

(Note the added quotes around .#"ghc92/plutus-core/1.1.0.0". For other shells than bash this is required to make sure that the above command works. See: NixOS/nix#4686. For bash, the quotes can be omitted.)

This will build all the components of that package version that CHaP cares about, namely libraries and executables (test suites and benchmarks are not built).

We need to use --override-input because the CHaP flake relies on a built repository. By default it points to a built repository on the main CHaP repo branch. But if you have just produced your own built repository (see above) then you want to use that instead, and --override-input will let you do that.

... by testing against a haskell.nix project

If you want to test a locally built CHaP against a project that uses CHaP via haskell.nix, you can build the project while overriding CHaP with your local version.

$ nix build --override-input CHaP path:/home/user/cardano-haskell-packages/_repo

Note that you will need to change the index-state for cardano-haskell-packages to be newer than the repository you just built, otherwise cabal will ignore your new package versions!

Also, you you can examine the build plan without completing the build:

$ nix build .#project.plan-nix.json \
	--out-link plan.json \
	--override-input CHaP path:/home/user/cardano-haskell-packages/_repo

This is useful if you just want to see whether cabal is able to successfully resolve dependencies and see what versions it picked.

Making changes

Changes to CHaP should simply be made using PRs.

Access control

CHaP uses CODEOWNERS to determine whose approval is needed to release a package. The general rules are:

  • If a package is clearly owned by a particular team, then set that team as the CODEOWNER.
  • Prefer to use GitHub teams over individual accounts wherever possible.
  • In the case of patched packages, the owner should be whichever team owns the package that causes the dependency on the package that needs patching.

Generally, use your judgement about what's appropriate.

CI

The CI for CHaP does the following things:

  • Checks that the timestamps in the git repository are monotonically increasing through commits. Along with requiring linear history, this ensures that package repository that we build is always an extension of the previous one.
  • Builds the package repository from the metadata using foliage.
  • Builds a small set of packages using the newly built repository.
    • We build with all the major GHC versions we expect to be in use.
    • At the moment we don't build all the packages in the repository, only the latest versions of a fixed set.
    • This happens twice, without Haddock and with Haddock.
  • Builds any newly added packages using the newly built repository.
    • This happens twice, without Haddock and with Haddock.
  • If on the master branch, deploys the package repository to the repo branch, along with some static web content.

Troubleshooting CI / GitHub Actions

In case the build-packages or build-new-packages actions fail, you can retrieve the built-repo Artifact from the Actions Summary page for the failed action. Then, unpack it into the _repo directory and proceed with the remaining steps in the CI.yml GitHub Workflow file. Note that the nixbuild flags are not relevant for reproducing the issue locally and can be ignored.

Dealing with timestamp conflicts

Since we require monotonically increasing timestamps, there can be timestamp conflicts if someone else merges a PR with later timestamps than yours. That means that your PR (once updated from main) will now introduce "old" timestamps, which is not allowed.

There are some scripts for dealing with this:

  • ./scripts/update-timestamps-in-revision.sh REV will look at the given revision, find any timestamps that were added in that commit, and make changes to update them to a fresh timestamp.
  • ./scripts/update-timestamps-and-fixup.sh REV will do the same but also commit the changes as a fixup commit. You can either leave these in your PR or get rid of them with git rebase main --autosquash

An easy way to run update-timestamps-and-fixup on a multi-commit PR is to run git rebase main --exec "./scripts/update-timestamps-and-fixup.sh HEAD". This will run the script at every step of the rebase on HEAD (i.e. the commit you have reached).

Debugging solver issues

Sometimes you may hit issues that are related to cabal's constraint solver making strange choices. For example, you are making a PR to release foo-X, which depends on bar via baz. But when the CI builds the package, instead of using the newest bar-Y, cabal inexplicably decides to build a very old verison of bar that either a) leads to solver errors; or b) leads to compilation errors.

The root cause is usually that the solver can't pick the version of bar that you want because it conflicts with your dependencies in some way, but it is often not obvious why. It's tricky to diagnose, since you'll sometimes want to rebuild the repository to add revisions while you're working.

The easiest way is to build the package in question with nix, i.e. something like this:

# repeat this whenever you change the repository, e.g. by adding a revision
> foliage build -j 0 --write-metadata
> nix build
  --override-input CHaP path:/home/user/cardano-haskell-packages/_repo
  '.#"ghc92/foo/X"'

There are then two ways to make progress:

  1. Add a constraint to force the good case. If you are expecting to build with bar-Y, you can add a bar == Y constraint and the solver should tell you why it's impossible! You can either add this to the cabal.project specified in nix/builder.nix, or add .addConstraint "bar == Y" to the nix invocation. For example:
nix build $(nix eval --raw '.#"ghc8107/cardano-ledger-core/0.1.0.0"' \
    --apply 'd: (d.passthru.addConstraint "cardano-crypto-class <2.0.0.1, cardano-binary<1.5.0.1").drvPath')
  1. Make a revision to rule out the bad case. If your build fails because baz-Z can't build with bar-P, then baz-Z should have a constraint that excludes bar-P. You can add this constraint by making a revision to baz-Z and try again.

Both of these should get you to a different error. The advantage of adding constraints is that it tends to more directly reveal the problem as it focusses on what you want to happen. The advantage of the revision is that it permanently records the incompatibility information, which is useful for future people.