Test and record Bazel compatibility for BCR modules

[meteorcloudy]

Goals

Since the release of Bazel 7, we now have two Bazel LTS releases that officially support Bzlmod. As the community migrates to Bzlmod, it'll be beneficial to test BCR modules against all supported Bazel LTS releases and document their compatibility.

This will enable:

• Easy lookup of a module's compatibility with specific Bazel LTS versions.
• Use this compatibility data by tools for automatic updates of dependency versions based on the current Bazel version.
• Detect scenarios where users might need to simultaneously upgrade both the module's major version (assuming SemVer used) and Bazel's major version. See https://bazel.build/release/rule-compatibility
• Detect regressions in Bazel minor/patch release candidates by testing modules compatible with its previous versions.

Solutions

1. In the metadata.json of the module

The original proposal in this doc is to document the compatibility info in the metadata.json file of each module.

For example, in modules/rules_apple/bazel_compatibility.json:

...
"bazel_compatibility" : {
    # Bazel 5.0.0 is compatible with the module from version 0.20.0 to 0.29.2
    "5" : {"min_version": "0.20.0", "max_version": "0.28.2"},
    "6" : {"min_version": "0.30.0", "max_version": "0.31.2"},
}
...

But this approach has some disadvantages:

• Hard to maintain the version range, it might even not be a complete range. For example, what if 0.31.0 actually breaks with Bazel 6 and then later fixed in 0.31.2.
• It doesn't express that a module version requires a minor version Bazel. For example, it's possible that 0.31.0 breaks with Bazel 6.0 but works with Bazel 6.1 and later 6.x versions. Although theoretically users can always upgrade to the latest minor release of Bazel 6, we still lose some information here.

2. In a per-version bazel_compatibility.json file

Instead of solution 1, we can document the compatibility info in a per-version bazel_compatibility.json file.

For example, in modules/rules_apple/3.1.0/bazel_compatibility.json:

{
    "5": null,
    "6": "6.4.0",
    "7": "7.0.0",
    "8": null,
}

In this JSON file, each key represents the major version number of a Bazel LTS release, and its corresponding value indicates the minimum version within that LTS release that is compatible with the current module version. null means none of the versions of that LTS release is compatible.

Differences from the bazel_compatibility attribute

With the bazel_compatibility attribute of the module directive in MODULE.bazel, module authors can already specify Bazel compatibility info. So why do we need a separate bazel_compatibility.json file?

There are two key differences:

• MODULE.bazel files checked in the BCR cannot be modified. If a new Bazel version is released after it's checked in, then there is no way we can update the bazel compatibility info retrospectively.
• The bazel_compatibility attribute is a general Bzlmod feature that works not only for BCR, but also for other Bazel registries and overridden modules.

Therefore, one cannot replace the other.

BCR presubmit changes

BCR presubmit will be updated to check Bazel compatibility:

1. A bazel_compatibility.json file should exist under modules/<module name>/<version>/ for the tested module. If not, assuming all Bazel LTS versions are compatible by defaulting to "X": "X.0.0" for each LTS release.
2. Calculate which Bazel LTS versions should be tested. If the value for LTS X is null, then this LTS release is not tested. Otherwise, the recorded minimal Bazel version should be tested.
3. Run tests in presubmit.yml with the above Bazel versions.

If anything breaks, the PR author can create/update the bazel_compatibility.json file to allow the presubmit to pass by bumping the minimal version or changing it to null.

Question: should we always test the latest LTS release? For example, even if "7": "7.0.0" was recorded, should we still test 7.1.0?

Update modules retrospectively

During the ramp up phase or when a new Bazel LTS release is out, we need to update existing modules retrospectively.

To do that, create/update the bazel_compatibility.json files for existing module versions with a PR. After the BCR presubmit update, the PR should trigger builds for all modified versions. If anything breaks, we update the bazel_compatibility.json files to reflect the actual compatibility.

It might be worth implementing a new pipeline to bisect the minimal version of a LTS release that is compatible for a given module version.

BCR UI changes

https://registry.bazel.build should be updated to include Bazel compatibility info. The UI should make it easy to lookup 1) Given a module version, which Bazel LTS releases are compatible. 2) Given a Bazel version, which module versions are compatible

Limitations

Even with the above process, the recorded Bazel compatibility may not be accurate as it depends on what tests are specified in the presubmit.yml for each module.

[meteorcloudy]

/cc @Wyverald @fmeum @rickeylev @keith @alexeagle @comius @hobofan

[alexeagle]

I would just point out this is a "defense-in-depth" as the rulesets already ought to have been testing against the matrix of "supported Bazel versions" as they merged PRs. Many of them use a reusable workflow: https://github.com/bazel-contrib/.github/blob/master/.github/workflows/bazel.yaml#L67-L78

[meteorcloudy]

In this JSON file, each key represents the major version number of a Bazel LTS release, and its corresponding value indicates the minimum version within that LTS release that is compatible with the current module version.

Considering minor releases are backwards compatible so users can always upgrade to the latest version effortlessly, I find recording the minimal compatible version of a LTS release might not be necessary. Therefore, alternatively, we can just record a true or false value which indicates whether the latest version of the LTS release is compatible with the module version.

{
    "5": false,
    "6": true,
    "7": true,
    "8": false,
}

This should make it much easier to update bazel_compatibility.json, and users can still set [">6.1.0"] in the bazel compatibility value in MOUDLE.bazel when necessary.

In presubmit, we'll just test the module version with the latest versions of all supported LTS releases.

[meteorcloudy]

After some discussion in the rules authors meeting, we decided to just simplify this to specifying bazel version inside the presubmit.yml instead of having a separate bazel_compatibility.json file. This should make the whole process much simpler.

[meteorcloudy]

Done in #1387