Support Bzlmod and add rules_scala to bazel-central-registry

[sfc-gh-pbennes]

Greetings!

It looks like rules_scala isn't part of the Bzlmod effort or added to bazel-central-registry yet. I've opened an issue both here and in https://github.com/bazelbuild/bazel-central-registry to request it: bazelbuild/bazel-central-registry#522

[chrislovecnm]

Just a note to anyone that starts the work native.register_toolchains is not supported when using bzlmod.

[sluongng]

@chrislovecnm it's still available here no? https://bazel.build/rules/lib/globals/module#register_toolchains

[chrislovecnm]

@sluongng When I last tried it, it did not work when using it inside an extension. In rules_python we are skipping the native call:

https://github.com/bazelbuild/rules_python/blob/36082079b514529d4009a2b104475fde1cdd5b30/python/repositories.bzl#L569

And build the toolchains here:

https://github.com/bazelbuild/rules_python/blob/36082079b514529d4009a2b104475fde1cdd5b30/python/extensions/private/pythons_hub.bzl#L56

We build the toolchains and use the following:

https://github.com/bazelbuild/rules_python/blob/36082079b514529d4009a2b104475fde1cdd5b30/MODULE.bazel#L49

[sluongng]

Yeah you would only want to prepare the toolchain (as a new repository) inside the extension I think. Registering it must be called from the MODULE.bazel file.

So native.register_toolchains call from the extension's starlark does not work, but having the extension create @pythons_hub//:all and then MODULE.bazel calling register_toolchains("@pythons_hub//:all") worked. 👍

[mateuszkuta256]

I prepared a minimal bzlmod support here and gonna split it into a few PRs
TODOs after merging it:

• write documentation on how to use rules_scala with bzlmod (some properties like SCALA_VERSION will probably be configurable via repo-env)
• prepare release notes
• contribute repo to the Bazel Central Registry
• add tests that the project builds using both WORKSPACE and MODULE.bazel
• migrate test/external workspaces to blzmod too

[pcj]

@mateuszkuta256 thanks for getting this started! Any progress updates or blockers?

[mateuszkuta256]

@mateuszkuta256 thanks for getting this started! Any progress updates or blockers?

hi, unfortunately I'm working on other things now and won't continue with this PR in the foreseeable future
I remember this migration was on hold because interfered with 'cross compilation support'
It looks like great progress was done it this area, so maybe someone can already take over the PR

[mbland]

I'd like to take on the task of Bzlmodifying this repo through a series of pull requests.

I've already created a (mostly) working branch in my own fork. Though I saw a couple draft pull requests here, I ended up taking a different approach and got it mostly working. In fact, I did exactly what @sluongng suggested in #1482 (comment). (I didn't notice this comment before just now—I might've read it, but not understood it at the time—but I did study rules_python and rules_go, and ended up doing exactly that.)

There are still issues I need help to address (recorded in some of the commit messages), I didn't strictly maintain WORKSPACE compatibility (fixable), and I only tested against Scala 2.13. But I can start teasing chunks out of it in a methodical fashion, to ensure that I maintain WORKSPACE and cross-version compatibility.

For an example of what it looks like from a client perspective, here's what the MODULE.bazel stanza from my local EngFlow/example repo looks like (as opposed to the current stanza, explanatory comment notwithstanding):

bazel_dep(name = "rules_scala", repo_name = "io_bazel_rules_scala")
local_path_override(
    module_name = "rules_scala",
    path = "../../bazelbuild/rules_scala"
)   

scala_dev_deps = use_extension(
    "@io_bazel_rules_scala//scala/extensions:deps.bzl",
    "scala_deps",
    dev_dependency = True,
)   
scala_dev_deps.toolchains(
    scalatest = True,
)

So if folks are game for me to do this, I'll start carving off pieces as separate pull requests, and we can resolve any outstanding problems in the process.

cc: @BillyAutrey @jayconrod @benjaminp @TheGrizzlyDev

[mbland]

A quick update for visibility: I'm very close to having the Bzlmodified rules_scala passing 100% of the tests, down to the last couple of failures:

• I need to replace the bind() calls from twitter_scrooge with alias() or some such to get test_version.sh to pass.
• I need to fix the Scala 3.1.2 and Scala 3.3.3 failures for the Scalafmt targets from test/shell/test_cross_build.sh (corresponding to the test3 and library3 test cases in test_scalafmt()).

I also need to get test_lint.sh working again, and update the dt_patches repos for Bzlmod. But the following all pass:

• test_rules_scala.sh
• test_reproducibility.sh
• test_examples.sh
• test_coverage.sh

Hopefully I can get these fixed up today, and I'll start peeling off the next pull request or two. And thanks to @simuons and @liucijus for reviewing and merging #1619 and #1620 already.

[mbland]

Again, I deeply appreciate the appreciation, but these have been helpful contributions, not heroism. 🙂 I'm compelled to push back politely on such characterizations, however honored I am to receive them.

To answer your question, though, I'm hoping we can land the remaining Bzlmodification changes quite soon.

---

That said, it's up to the time and bandwidth @simuons and @liucijus have to review the pull requests, whether they have any concerns and/or suggestions, and when they're comfortable cutting a release after that. They've been incredibly helpful and responsive to date, and I'm not trying to rush them. Several of the changes have benefitted substantially from review feedback, including several receiving serendipitous feedback from @WojciechMazur. (Plus, I indirectly roped @fmeum into effectively reviewing #1621 via bazelbuild/bazel-skylib#548. I'm so glad I did, because the end result was so much better.)

As I've been fond of telling people recently, this is Open Source at its best, working as intended. 🙂

Once #1633 gets reviewed, possibly updated, approved, and merged, that will clear the way for the rest of the essential Bzlmodification/toolchainization changes. My comments in #1633 explain what I'm shooting for, and link to the working prototype in my Bzlmod working branch—which now also maintains WORKSPACE compatibility. So hopefully the complete Bzlmodification itself will land quickly.

---

While I'm at it, let me recap the other activity related to this effort.

Most of the changes so far have been in preparation for landing the essential Bzlmodification/toolchainification changes, the first of which being #1633. Some have been essential prerequisites, fixing things that outright break under Bzlmod, such as #1621 or #1626. But with those fixed, now we're talking about providing a proper Bzlmod API that doesn't put the burden on users to import dozens of repos required by rules_scala toolchains (such as I had to do in the current MODULE.bazel from EngFlow/example). Technically, #1635 falls in this API design category as well.

Other pull requests have addressed small compatibility issues I've encountered along the way, like Bazel 7 compatibility in #1619 and #1620; Bzlmod compatible testing in #1622; and Scala version compatibility in #1628, #1632, and #1634. I've included a couple of helpful utilities I've written as well in #1629 and #1638. All of these should make maintaining the implementation and switching to Bzlmod even smoother, by resolving existing problems before anyone else encounters them.

I've also been identifying and making changes to ensure users aren't version locked on any dependencies thanks to rules_scala, such as #1631 (which was borderline essential, because Scalafmt 3.0.0 threw java.util.NoSuchElementException: last of empty IndexedSeq under Scala 2.13). I'm now focusing on trying to ensure more recent Protobuf module versions work with the scala_proto toolchain and with aspects implemented using protobuf/gRPC/ScalaPB/protoc-bridge (including the Scalafmt aspect). This is where changes like #1623, #1624, #1630, and #1637 come from. Much of that effort involves coordinating version bumps in the jar dictionaries from the third_party/repository/scala_*.bzl files and adding API adapters for specific Scala and/or dependency versions as needed. (Updating those dictionaries manually is why I'm also investing in updating scripts/create_repository.py in #1639, because doing all that work by hand in #1631 was the most time consuming part of this effort to date.)

---

Anywho, there's a few more changes to come, for Bzlmodification itself, for Protobuf upgradability, and for internal test and tooling updates. But as I've told friends and colleagues, I'm beyond seeing the light at the end of the tunnel. I'm out of the tunnel, off of the highway, and in the neighborhood, around the corner from the destination, turning the radio down so I can focus on where to park. 😛

[sundbry]

For my $0.02, I've checked out your mbland/bzlmod scala branch, and used it to build some scala programs - it works great! Thank you very much for pushing this code upstream.

The only issue I had, and I don't think this was related to your changes in particular, was figuring out how to set it up to use the scala3 toolchain- I ended up using --repo_env=SCALA_VERSION=3.4.3 in the build command to set it, but couldn't figure out how to set it in the MODULE.bazel or BUILD.bazel with the new configuration.

[mbland]

For my $0.02, I've checked out your mbland/bzlmod scala branch, and used it to build some scala programs - it works great! Thank you very much for pushing this code upstream.

My pleasure!

The only issue I had, and I don't think this was related to your changes in particular, was figuring out how to set it up to use the scala3 toolchain- I ended up using --repo_env=SCALA_VERSION=3.4.3 in the build command to set it, but couldn't figure out how to set it in the MODULE.bazel or BUILD.bazel with the new configuration.

Ah—my MODULE.bazel from EngFlow/example has an example of what this will look like. The only difference is the extension path will be @rules_scala//scala/extensions:config.bzl:

# This constant matches the default Scala version from rules_scala for now.
SCALA_VERSION = "2.13.12"
SCALA_VERSIONS = [SCALA_VERSION]

scala_config = use_extension("//scala/extensions:config.bzl", "scala_config")
scala_config.settings(
    scala_version = SCALA_VERSION,
    scala_versions = SCALA_VERSIONS,
)

Using named constants is optional, of course. Also, in this case, the SCALA_VERSIONS definition isn't necessary, because rules_scala will do the right thing without it. But if you do use SCALA_VERSIONS, it must contain SCALA_VERSION. Check out the relevant stanza from scala_config.bzl.

There's a similar, but slightly more detailed example in the MODULE.bazel file from my bzlmod working branch. The key difference there is the dev_dependency = True argument to use_extension. That's there because that configuration is only for developing rules_scala, not for its consumers.

[zcking]

@mbland Totally get it! My comment was mostly intended to show support for this issue and vested interest outside of the normal contributors. This work is really important if Scala is to join many other languages already up-to-speed in the Bazel registry.

Also, although I'm not familiar with development for bazel rules such as this, I'd be happy to help contribute in my spare time if any additional help is needed!

[mbland]

Another quick update...

@zcking I appreciate the offer for help. I feel like the actual code changes are essentially complete, modulo adjustments based on review feedback. Feel free to look at the outstanding and upcoming PRs and offer any feedback or questions if you have any.

@simuons provided some feedback on #1633 that prompted me to make some adjustments and to ask for confirmation on whether to make certain other changes. This one will establish the direction for pulling other changes from my Bzlmod working branch. Though it will remain compatible with WORKSPACE, there will be small yet breaking changes from the existing WORKSPACE API, so it's worth taking the time to get it right.

I've reached some conclusions with regard to my Protobuf compatibility and upgradeability side quest, which I've written up in #1647. I found that rules_scala can either support up to Protobuf v25.5 when building with both Bazel 6 and 7 (with C++ compiler flags set for Bazel 6), or can leap to supporting only Protobuf v28.2 and above. My suggestion is to support v25.5, since the current v21.7 seems too old, but I'd fear switching straight to v28.2 might break too many users right now.

I've opened some new PRs to make a few improvements to the build and test scripts along the way:

• Refactor scripts/create_repository.py, improve efficiency and correctness, and bump Scalatest, protobuf-java, kind-projector #1642
• Make every .bazelrc import the top level .bazelrc #1644
• Clean test_{runner,scalafmt_helper}, add diagnostic env vars, and improve test_cross_build performance #1646

The first and last of these, in particular, turbocharged my Protobuf investigation:

• The create_repository.py script is now super fast, does more to ensure the entries in the generated third_party/repository/scala_*.bzl files are accurate, and updates more core artifacts (including ScalaPB in my bzlmod-update-protocbridge-and-grpc and bzlmod-update-abseil-cpp-and-protobuf working branches).
• The new RULES_SCALA_TEST_ONLY and RULES_SCALA_TEST_VERBOSE env vars made it easier to iterate on specific test cases or files from test_all.sh, test_version.sh, etc., and to see what was happening. (The default is to only emit output on failure, after the command finishes, which proves especially problematic if a test hangs.)

I also figured out how to put a nail in the coffin of the generators-hanging-on-Protobuf-upgrade issue by wrapping ScalaPbCodeGenerator to catch and report previously uncaught exceptions.

At this point, it's just a matter of taking the time to land things properly. No need to rush, and now that I've got all my tools and investigations and proofs of concept working, I plan to take it easy for a while. 🙂

[mbland]

OK, here's my latest weekly update. 🙂 The executive summary is that we're in the home stretch here.

#1633 is super close to going in. And now that I think about it, I think there aren't any WORKSPACE API-breaking changes after all. All the old macros should continue to work with existing builds with no changes (though scala_toolchains is nicer and can shrink existing WORKSPACE files substantially). So technically, the only "breaking" change so far (or bugfix, depending how you look at it) comes from removing external/$REPO)_NAME path prefixes from embedded resources, from #1621.

Speaking of #1621, I've superseded the apparent_repo_name utility from that pull request in the new #1650. Not critical, but it's the right thing to do; I've also closed bazelbuild/bazel-skylib#548 as a result. And an update as of this morning: The Bazel maintainers may be considering a new repository_ctx field to render this workaround unnecessary (eventually).

Integrating the changes from #1648 broke my bzlmod-rebuilding working branch, because instantiating scala_proto toolchain repos collided with instantiating ScalaPB repos for scalafmt and Guava repos for twitter_scrooge under Bzlmod. The WORKSPACE implementation doesn't care, but it's an error to instantiate the same repo multiple times from a module extension:

$ bazel build //src/...

ERROR: .../scala/scala_maven_import_external.bzl:299:24:
  Traceback (most recent call last):
    File ".../scala/extensions/deps.bzl",
      line 140, column 21, in _scala_deps_impl
        scala_toolchains(
    File ".../scala/private/macros/toolchains.bzl",
      line 140, column 17, in scala_toolchains
        _scrooge(
    File ".../twitter_scrooge/twitter_scrooge.bzl",
      line 96, column 17, in twitter_scrooge
        repositories(
    File ".../third_party/repositories/repositories.bzl",
      line 113, column 37, in repositories
        _scala_maven_import_external(
    File ".../scala/scala_maven_import_external.bzl",
      line 263, column 30, in scala_maven_import_external
        jvm_maven_import_external(
    File ".../scala/scala_maven_import_external.bzl",
      line 299, column 24, in jvm_maven_import_external
        jvm_import_external(jar_urls = jar_urls, srcjar_urls = srcjar_urls, coordinates = artifact, **kwargs)

Error in repository_rule: A repo named
  io_bazel_rules_scala_guava_2_13_15
  is already generated by this module extension at
  .../scala/scala_maven_import_external.bzl:299:24

ERROR: Analysis of target
  '//src/java/io/bazel/rulesscala/worker:worker_test'
  failed; build aborted:
  error evaluating module extension scala_deps in
  //scala/extensions:deps.bzl

This led me to develop a schema whereby the toolchains declare their Maven artifact dependencies in a separate macro from the one that calls repositories to instantiate them. (The Scalafmt toolchain had defined a similar macro before, to accommodate different artifact sets for different Scala versions, which I've since updated.) This allows the new scala_toolchains macro from #1633 to collect all the artifact IDs, de-dupe them, and call repositories itself. You can see the final result in scala/private/macros/toolchains.bzl from my bzlmod working branch.

I've prepared commits for the testing toolchains and the Scalafmt toolchain already, to land in that order after #1633. I've started one for the scala_proto toolchain, but want to wait for #1648 to land before going further with it, since that PR introduces scala_proto_artifact_ids, which I'd used to accommodate differences between Scala 2.11 and other versions.

After that, there'll be:

• either one or two pull requests to add the jmh and twitter_scrooge toolchains; I do have twitter_scrooge building under Bzlmod already, with all native.bind calls removed and all tests passing
• a pull request to prepare the dt_patches repos for Bzlmodification. as well as remove duplication
• maybe one pull request to remove all remaining uses of @io_bazel_rules_scala, so that Bzlmod users can usebazel_dep(name = "rules_scala", version = "7.0.0") without a repo_name = "io_bazel_rules_scala" parameter
• at least one more pull request to drop in MODULE.bazel files, module extensions, and remove --noenable_bzlmod from .bazelrc; this could be more than one, depending on the maintainers' appetite to migrate all internal repos at once, or one at a time, or in batches
• either in the above pull request or in a separate one, turn the Bazel last green head build back on using Bzlmod to close Bazel green build CI job now fails persistently due to rules_java versioning #1625

We could also bump the default .bazelversion to 7.4.1 (or whatever the latest is by that point) and/or bump Protobuf to v25.5, per #1647, but that's not strictly critical. (It would technically break Bazel 6 users, because they'd have to add C++ compiler flags to their .bazelrc to build Protobuf v22 and later.)

At some point I'll look into adding the Publish to the Bazel Central Registry tool, probably after the official 7.0.0 release, unless the maintainers would prefer to do it themselves. After that, and when https://registry.bazel.build/modules/rules_scala goes live, I think we could close this issue!