The future of defaulting to Bloop for build tools that can also be build servers.

[ckipp01]

Historically Bloop has played an absolutely major role in Metals. It paved the way in BSP and has allowed us to support multiple build tools out of the gate with Metals. This was before sbt had BSP support and before Mill had BSP support.

Over the last year looking at conversations on Discord and other places it seems more and more common for sbt users to switch to using sbt as their build server instead of Bloop. This can help out in more complex builds, custom configurations, etc. sbt as a build server just knows how to handle sbt builds better. While there are some limitations (probably the fact that an sbt session blocks), most of the time using sbt as a build server is just a better choice.

We also find ourselves in a position where this may start to cause a bit of confusion as BSP support in tools like sbt and Mill get more popular. We see this in the discussion that took place here. In some ways we're ignoring BSP discovery (which Bloop doesn't follow) and instead favoring Bloop in build tools that Bloop supports. This may not be what the user wants or expects when there is an existing .bsp/tool.json entry. Switching to a new build server also adds a bit of mental overhead about when to switch, how to switch, and whether the switch worked.

This all sort of leads to the question of at what point do we stop favoring Bloop for build tools that can also serve as build servers? I propose that we start moving in this direction with the goals being:

1. More seamless getting started experience (no choice between Bloop and sbt for example)
2. More clear BSP discovery (if there is a .bsp/mill.json, then Mill is automatically used as a BSP server
3. Avoid having peoples first action being ignoring the Bloop import and switching to sbt if they want to use it

I'd love to hear other's thoughts on this.

---

Where we're at

• You can fine the Comparison write-up here, which has the latest state of things

[tanishiking]

I agree that it would be time to switch from "Bloop as a build server by default" to "sbt/mill BSP by default", as I see both sbt and mill BSP is becoming mature (and ScalaCLI too because it's mostly proxy for underlying bloop BSP server).

One concern before moving forward is something about closing server when we close Metals (maybe mill also has the same kind of issue?), I remember there was an issue related to this stuff, but sorry I didn't catch up with this area well...

[ckipp01]

One concern before moving forward is something about closing server when we close Metals

Yea this is a good point, and one that people will for sure hit on as they start using sbt as a build server for example. They loose out on the "one server" per machine benefit. For example a bit ago I was curious and looked at my own machine to see what was running, and this is what I was greeted with:

So having a discussion about keep alive time is also a good idea.

The Bloop one is the trickiest I think and we've had some discussion about it in here and I assume that's what you're referring to.

As for sbt, the standard keepalive time for the sbt server is 7 days. For Mill, I just asked and learned that it is 5 minutes.

[lolgab]

A good first step might be to make this configurable.
Bloop might continue to be the default, but power Mill/Sbt users could add a setting in their VSCode settings.json (something like "metals.forceMillBspServer": true or "metals.forceSbtBspServer": true) and have metals behave like what happens today when you perform Switch build server and you choose mill-bsp or sbt-bsp ?
Could this be a good middle ground to allow people to use build-tools' BSP servers (which, as you said, aren't free from defects) if they want, without necessarily changing the current default?

[jackcviers]

Needs to be accessible for more than just vscode users - but any metals configuration file or argument flag or env variable editable by users is fine for this, I agree.

[ckipp01]

Needs to be accessible for more than just vscode users

For sure, the nice thing is that a solution like this would work for everyone. The settings that you can pass in to Metals like this work more or less the same for most editors. So for example nvim-metals would also just pass in this setting. I'm assuming it'd be the same for emacs and friends.

I wouldn't recommend introducing two new settings though. Since we'd then have to do it for X amount of tools that could be build servers. Maybe it would be possible to instead just change the behavior to better response the .bsp entries or if we need to have a single setting to ensure that it does.

[dos65]

I support the future action to provide a better experience for working with sbt/mill bsp.

More seamless getting started experience (no choice between Bloop and sbt for example)

However, I don't agree that we should do anything with the choice.
I think shouldn't be a problem to keep this choice and provide a link to the docs with describes the difference.
In most cases I still bloop because opening a lot of projects with native-build-bsp leads to a huge memory consumption

[adpi2]

And I also don't love that sbt does generate it automatically.

We could change this right? @adpi2 I feel like I remember seeing a discussion about this, but do you is creating the .bsp/sbt.json by default necessary? Could this be changed?

It is not necessary since both Metals and Intellij knows how to run bspConfig. But this kind of defeats the purpose of BSP discovery: build clients should not know anything about the build tools, not even how to generate the build configuration. If Metals knows sbt bspConfig, it could as well know sbt -bsp which makes the .bsp file useless. I feel like we are missing the point of BSP discovery and that we are using it in the wrong way.

In any case, we can stop generating the BSP confing automatically in sbt. It would be rather easy. One important thing though is for sbt to update the .bsp file if it already exists which it already does since it overrides the file at each startup.

[ckipp01]

It is not necessary since both Metals and Intellij knows how to run bspConfig. But this kind of defeats the purpose of BSP discovery: build clients should not know anything about the build tools, not even how to generate the build configuration. If Metals knows sbt bspConfig, it could as well know sbt -bsp which makes the .bsp file useless. I feel like we are missing the point of BSP discovery and that we are using it in the wrong way.

I agree with this. But it's also sort of the state of things since as you said, both Metals and IntelliJ seem to know how to create the bsp entry needed. I think part of the issue here is also that Bloop just completely throws BSP discovery out the window and doesn't use a BSP entry. I feel like the easiest thing is either everyone follows discovery and relies on a .bsp/ entry or the thing falls apart.

[adpi2]

Another problem with BSP discovery currently is that the .bsp config is absent in freshly cloned project and so the new user needs to know what to do to generate it. It would be better if maintainers could push the .bsp to their repo but this is only possible if the content of the .bsp is always machine-independent. This would be better because then the maintainers could decide what the best way to import their project based on their knowledge of the build. And users could open Metals right away after cloning with no trouble.

[julienrf]

We actually run benchmarks comparing both sbt and bloop hot compile times and Bloop ends up being a clear winner there.

Example in Akka:

Do you know where this difference comes from? Is this still the case now that Bloop does not implement pipelined compilation anymore?

Also, IMHO it would be better to explain the benchmarking protocol, and show all the results instead of exhibiting just one data point.

My experience with the efficiency of using sbt vs bloop is that anyway I need at least one sbt instance to run (so that I can invoke tasks from there), so in that regard, invoking bloop is always a waste of resources for me.

[tgodzik]

I actually merged a PR that makes is easier to run benchmarks locally and also updated it to use the most recent repos. It seems that overall the performance depends on the repo with Bloop being up to 50% faster and in some cases around the same.

I am not sure about why that is, but it might be the fact that Bloop doesn't need to worry about a lot things that sbt deals with.

Also, IMHO it would be better to explain the benchmarking protocol, and show all the results instead of exhibiting just one data point.

The benchmarks runs compilation 7 times and only then starts 5 runs to measure performance both for Bloop and sbt. For sbt it keeps the session open.

My experience with the efficiency of using sbt vs bloop is that anyway I need at least one sbt instance to run (so that I can invoke tasks from there), so in that regard, invoking bloop is always a waste of resources for me

That's true, it might be suboptimal especially if you only need a quick look at a project. Anyway, I see a strong sentiment towards just using sbt as the default. I think we can go that way and focus on improving the experience. We can start with using .bsp as default if it exists.

[jackcviers]

Has code navigation for scala 3.x.x projects improved for sbt build servers? I'm on latest metals and tried to switch to a sbt build server from bloop on a 3.1.2 project yesterday and was informed that code navigation would not work on this build server. On bloop bsp it didn't complain.

If major features are breaking on sbt, it may not be a good idea to move the default away from bloop for newcomers - they take the default and code navigation breaks, or documentation or lenses, etc.

[tanishiking]

Hmm, that shouldn't be the issue with sbt, is it possible to open an issue?

As you mentioned, we shouldn't make sbt/mill BSP to default until it looks stable.

[adpi2]

Theoretically, sbt server should support the exact same set of Scala versions as Bloop.

However sometimes the user needs to reload sbt so that its project can be imported properly. This is because Metals writes a metals.sbt file in the project folder and it needs to be loaded for code navigation to work.

@jackcviers You can try this:

1. After you got the Navigation will not work message
2. Force reload sbt server with sbtn reload or sbt -client reload
3. Reimport the build in Metals by running Metals: Import build

And if it does not work, please open an issue with a reproduction.

This manual process is cumbersome, we should force an automatic reload of sbt if necessary.

[jackcviers]

Worked @adpi2, thank you!

[adpi2]

Thinking about this, weighing the pros and cons of each option, I conviced myself that BSP is used for two very different things and that we should consider them separately.

1. Importing the build

If we only consider the build import, with requests such as workspace/buildTargets, buildTarget/sources, then I would argue that it is more reliable and faster to use the build tool directly. Running sbt bloopInstall to export the build definition to Bloop, then connect to Bloop to import the build is slower and more complex than just connecting to sbt server and importing the build.

In terms of maintenance, if for instance we need to export more information, implementing the change in the build tool directly is more simple and more stable than implementing it in the N plugins that are supposed to work on many versions of many build tools, with the risk that something is inaccessible or that it breaks in any new version of a build tool.

2. Compiling and running

In contrast, Bloop is better for compiling and running. It optimizes the resource usage, it is non-blocking and it has better performance. Also it is easier to evolve: we only need to implement each new feature once in Bloop to make it available to everyone, rather than N times, in all build tools. For instance if we add a feature to run a single test, we can implement it in Bloop and then it is available to all.

Consequence

As a consequence, I propose that Metals should always connect to sbt or Mill to import the build, but still use Bloop to compile and run.

This is how it could work:

• Metals generates the .bsp configuration if it is not there
• It connects to the build tool and import the build
• It writes the corresponding Bloop configurations (I hope here that we can easily translate BSP into Bloop configuration files, that there is no missing information)
• It connects to Bloop to compile and run

The main advantages of this are:

• Deprecate the sbt-bloop and Mill plugins
• No need to write the metals.sbt file anymore to install the sbt-bloop plugin or to inject semanticdb
• Faster import because no need to recompile the build definition after adding metals.sbt
• Metals can hide the Bloop files in its own folder: less noise for the user

In some complex build, it would still make sense to use the build tools as the compile and run server as well.

While this plan is no panacea, I feel like it could improve a few things that are annoying for Metals users and maintainers. It takes advantage of Bloop while hiding it more behind the scene: no more sbt-bloop, no more bloopInstall running message, no more visible .bloop folder. It also takes advantage of the BSP support in build tools by deprecating the sbt and Mill plugins for Bloop.

Less maintenance, more performance and better user experience :)

PS: This plan is not much different from the idea that build tools could offload compilation to Bloop. But, since Metals would benefit the most from it, I claim that it should be implemented in Metals.

[adpi2]

That does seem like a good trade off, but we will need to be running 2 build servers at the same time

Metals needs to connect to the build tool to import the build but then it can simply disconnect which would/should stop the server.

and also deal with potential issues stemming from both.

Yes sure, but you trade that against the issues in sbt bloopInstall.

Currently Metals maintainers need to fix the issues in bloopInstall, sbt connection and Bloop connection. Here I am suggesting to deprecate one source of issues. So in the end I think it is a winning situation.

I think offloadding to Bloop from sbt is a bit more promising approach

Scala CLI offloads to Bloop, and that is a good design decision but the design of sbt or Mill is quite different. I feel like it will be hard to convince sbt or Mill maintainers to go in that direction. They already have a good integration with Zinc and they are running on the JVM anyway. Integrating with Bloop would be quite an effort and it won't bring much in return. Probably not worth it.

On the other hand, it don't think it would be hard for Metals to translate the BSP responses into Bloop configuration files.

[dos65]

@adpi2 I think it's a fair notice but IMO this suggestion is making things much more complicated. I would choose to look if there is way to speedup sbt bloopInstall rather than introducing this double-bsp approach.

[tgodzik]

I think the biggest issue with bloopInstall is that we shouldn't need to run compilation. We do have to run it with source generators, but maybe we could improve that as the source generators have been added to Bloop config. This would make the import faster.

[matthughes]

This also has the benefit of being able to work with projects that aren't currently compiling.

[ckipp01]

I also think what is outlined here just makes things more complicated. While it's helpful to think of BSP in two groups like this, I don't think they should be split. I don't think it was ever meant to.

The thing I'm still stuck on is that either way we have some effort that I don't want to say is "wasted", as it's not. But no matter what either we are focusing on Bloop too heavily and then the other BSP integrations suffer due to a lack of focus, or go the other direction, but then have multiple BSP servers to worry about. At least for me, it helps to go back in time a bit and look at why BSP was added to sbt. You can see a big discussion about this here and you can see the advisory board proposal here. The entire goal was to make the process of using sbt build via BSP easier. My opinion is that we haven't done that. We've instead fragmented the ecosystem. I want to stress the importance of getting this right as it can affect both Metals users and IntelliJ users depending on how we change things and address them. I know many of us here are attending the Tooling meetings. Maybe this should be a topic of discussion there or we can set up some sort of a discussion to go deeper on this topic?

[ckipp01]

BSP Comparisons

This is a continuation of the discussion that we were having up above about what Build
Server should be defaulted to. In order to try and look at this a bit
objectively I wanted to chart out the differences between the build tools and
bloop to get a better understanding of the levels of support a user will
receive depending on the build server they choose, discuss limitations that are
noticed, and hopefully come up with a solution about what the ideal
getting-started experience will be. I think it's important to remember that the
ability to switch BSP servers is important. It's equally important to note that
the best choice for an individual user may not be the default server. However,
the experience the average developer getting started with Metals is what we
should be most focused on. I'll be making the chart with these things in mind.

Since sbt and Mill are really the only two that are being commonly used as build
servers apart from Bloop, we'll focus on those. I'll take a hello world project
for both build tools and look at / do all the same things with both Bloop and
the respective build tool.

BSP Discovery

Before we dive into build tool and the differences first here is an overview of
how BSP discovery is currently working in Metals.

• In a workspace that no .bsp/ entry exists and that is using a build tool
  that Bloop supports (sbt, Mill, Gradle, Maven) then upon entering the
  workspace the user will be prompted to "Import build". This build import is
  actually running bloopInstall for the build tool, which will create a
  .bloop/.
• In a workspace that has an existing .bsp/ entry for the build tool that the
  workspace uses but is also a buidl tool that Bloop supports, the .bsp/
  entry will be ignored and the user will still receive an "Import build"
  request to import the build to Bloop.
• Situations where Metals defaults to the .bsp/ entry and auto-connects is in
  situations where it's a build tool that Metals knows Bloop does not support
  and therefore the only option would be to start the build server.

Build Import / Reload Request

Just a note about the "Build Import" request since this term is a bit overloaded
and it carries with it some Bloop-specific assumptions. The "Build Import"
language refers to importing a build to Bloop. When using Bloop even when you
change a build file and are prompting to re-import the build, the same process
happens behind the scenes, you're importing your build into a Bloop definition.

In the situation where you're not using Bloop, the same "Build Import" verbiage
is used, but behind the scenes it's actually doing a BSP
workspace/reload
when a build file is changed.

A General note about weaknesses

In each section I have some weaknesses listed for using that tool. This isn't
meant to always mean that it's a weakness on the server side, but could very
well be a weakness in how Metals is integrated with it. So in this context it's
basically "areas of improvement when using with Metals".

Bloop

It's helpful to take a look at the initialize response from the server to get
an understanding of the support, so here is the result from Bloop:

[Trace - 10:44:49 am] Received response 'build/initialize - (1)' in 6ms
Result: {
  "displayName": "bloop",
  "version": "1.5.6",
  "bspVersion": "2.1.0-M3",
  "capabilities": {
    "compileProvider": {
      "languageIds": [
        "scala",
        "java"
      ]
    },
    "testProvider": {
      "languageIds": [
        "scala",
        "java"
      ]
    },
    "runProvider": {
      "languageIds": [
        "scala",
        "java"
      ]
    },
    "debugProvider": {
      "languageIds": [
        "scala",
        "java"
      ]
    },
    "inverseSourcesProvider": true,
    "dependencySourcesProvider": true,
    "resourcesProvider": true,
    "buildTargetChangedProvider": false,
    "jvmRunEnvironmentProvider": true,
    "jvmTestEnvironmentProvider": true,
    "canReload": false
  }
}

A couple notes here:

• "buildTargetChangedProvider": false Bloop knows nothing about what files are part of
  your build. So it's up the client (Metals) to watch those files and re-trigger
  a build import when it's necessary. One way that it gets around this however
  is by using inverseSourcesProvider. There is a good overview of this in
  Introduce uses of buildTarget/inverseSources #3752.
• "canReload": false Reloading is only for servers that hold the state of the
  build in memory. Since Bloop holds that state on disc, this isn't relevant for
  Bloop

Everything else that is relevant for the Metals use-case is there with Bloop
meaning that a user has full support. There is also build file support, but this
is limited to sbt when using Bloop.

Strengths

• Fast compile times. Some of the benchmarks that were ran show Bloop being
  almost 2x as fast on certain repos.
• Multiple connections allowed meaning that using the Bloop CLI and Metals on
  the same workspace is not an issue, there is no blocking

Weaknesses

• There is no shared compilation between other parts of your build and Bloop.
  Very often you're running other tasks via sbt or Mill that won't re-use the
  compilation from Bloop.
• No knowledge of the task graph in your build tool. For example if you have a
  compile task that generates something (for example sbt buildinfo), Bloop won't
  have knowledge of this process.
• When used with sbt custom configurations are not picked up without additional
  code in your build definition.
• When you make a mistake in your build file and the build import fails, you get
  no reason why unless you look in the logs.

sbt

Here is the initialize response you'll see from sbt.

[Trace - 10:46:32 am] Received response 'build/initialize - (1)' in 14234ms
Result: {
  "displayName": "sbt",
  "version": "1.8.0",
  "bspVersion": "2.1.0-M1",
  "capabilities": {
    "compileProvider": {
      "languageIds": [
        "scala"
      ]
    },
    "testProvider": {
      "languageIds": [
        "scala"
      ]
    },
    "runProvider": {
      "languageIds": [
        "scala"
      ]
    },
    "dependencySourcesProvider": true,
    "resourcesProvider": true,
    "outputPathsProvider": true,
    "jvmRunEnvironmentProvider": true,
    "jvmTestEnvironmentProvider": true,
    "canReload": true
  }
}

A couple notes here:

• languageIds here is a bit confusing. sbt doesn't really have a concept of a
  pure Java target, everything is Scala based, but can have Java source. It
  may only have Java sources, but it's still considered a Scala target. So not
  having Java here isn't really an indication that it's not supporting Java.

Strengths

• You have access to the full talk graph of sbt this way. Arguably you're going
  to be getting the absolute best support for your sbt build when using sbt as
  your build server as it knows the inside / outs of your build definition.

Current limitations

• No Diagnostics for Java sources - BSP Diagnostics not reported for Java files sbt/sbt#7099 (This
  is the reason for the NO currently in the chart for Full Java Support)
• By default it's not using --client or the sbt script, but rather the
  launcher. We could switch this. There is some conversation about it in the
  following places:
  • Enable smoother sbt bsp integration. #2154 (comment)
  • Enable smoother sbt bsp integration. #2154 (comment)
    (there are some potential solutions listed in here)
  • Consider using -client when running sbt #2160 (didn't make a ton of
    sense when defaulting to Bloop, but if that changes...)
• sbt is blocking. This is probably the biggest hurdle imo. What this
  essentially means for Metals is that let's say you open the repl in sbt shell,
  it'll block and you'll be unable to compile your code in Metals until it's
  released. There have been a handful of discussions and even a draft pr to
  address this in the past:
  • Non-blocking run sbt/sbt#6178
  • Non-blocking console (Scala REPL) sbt/sbt#6179
  • Don't block sbt in run and console tasks sbt/sbt#6074

Mill

Here is the initialize response you'll see from mill.

[Trace - 11:12:05 am] Received response 'build/initialize - (1)' in 611ms
Result: {
  "displayName": "mill-bsp",
  "version": "0.10.10",
  "bspVersion": "2.0.0",
  "capabilities": {
    "compileProvider": {
      "languageIds": [
        "java",
        "scala"
      ]
    },
    "testProvider": {
      "languageIds": [
        "java",
        "scala"
      ]
    },
    "runProvider": {
      "languageIds": [
        "java",
        "scala"
      ]
    },
    "debugProvider": {
      "languageIds": []
    },
    "inverseSourcesProvider": true,
    "dependencySourcesProvider": true,
    "dependencyModulesProvider": true,
    "resourcesProvider": true,
    "outputPathsProvider": true,
    "buildTargetChangedProvider": false,
    "jvmRunEnvironmentProvider": true,
    "jvmTestEnvironmentProvider": true,
    "canReload": true
  }
}

Strengths

• Similar to sbt, Mill knows much more about your Mill build than Bloop will.
  This is important here especially since there are some known issues with the
  Bloop contrib package (Classpath regression in Bloop export com-lihaoyi/mill#1918)
• You get language support in your build files. Navigation mostly works, you
  get completions, and hovers. There are some limitations here such as no super
  method code lenses, which would be incredibly useful in your build.sc file.
  (Super method code lenses in your build.sc file. metals-feature-requests#321)

Weaknesses

• Mill isn't a debug server (Make Mill BSP server a debugProvider. com-lihaoyi/mill#2225),
  meaning you currently won't see any code lenses to run your code in your
  editor, and you won't have any debug ability.
• Java modules do not currently produce semanticDB meaning that when you open a
  fresh hello-world project you'll get a warning to open the Doctor since Java
  is not supported with semanticDB (Produce semanticDB for Java com-lihaoyi/mill#2226).
• The current workspace reload isn't really a reload, but it currently shuts the
  server down and restarts it. Not a huge deal, but still worth listing here.
  • Metals/BSP: errors in build.sc result in looped workspace reload com-lihaoyi/mill#1560
  • Issues with BSP workspace/reload com-lihaoyi/mill#1596

Comparison chart

More or less if you were to take the core features and table them out they'd
looking something like this:

Build Server	Full Scala Support	Full Java Support	Run/Test Support	Debugging Support	None Blocking	Tied into the Task Graph	Build File Support
Bloop	YES	YES	YES	YES	YES	NO	ONLY SBT
sbt	YES	NO	YES	YES	NO	YES	YES
mill	YES	NO	YES	NO	YES	YES	PARTIAL

How to proceed

Hopefully this gives us a better picture of the current state of things. Again
in #4505 we discussed what the
default should be and from taking some time and digging through all this, I
think we do need to tackle a few things before we make that switch. The
experience using sbt vs Bloop is incredibly similar. Unless you're doing a lot
of tasks in the console or shell then you might not notice the blocking nature
too much. However if you do hit on that, it will hinder your flow. While compile
speed may be slower, you're not having to double-compile anything, so in the end
it might be faster. The experience with Mill vs Bloop is a bit of a different
story. There are still some large things like debugging and Java support that
are lacking in mill-bsp, so I'd caution defaulting to that before those are
tackled.

Here is what I propose as some concrete next steps:

• Get some more feedback from all those involved in this here
• Tackle some of the issues I created while making this and also create new ones
  where necessary
• When we've done that, again make the decision if we change the default (I
  think we should, but we need to ensure parity).
• Make the necessary changes in Metals to respect .bsp/<entry> first and then
  default to Bloop for those tools that can't be a build server. There will
  also still be the option to switch to Bloop as well.

[tgodzik]

Thanks Chris for the great summary! I finally had the time to look into this.

The biggest obstacle from what I see to making sbt the default is the Java support I think, once that is fixed we can recommend that as the default over Bloop.

Mill seems a different story, but we should for sure use Mill BSP if it's present in .bsp since it's not being created eagerly. Once all the other issues are fixed we could also recommend it over Bloop.

[Baccata]

languageIds here is a bit confusing. sbt doesn't really have a concept of a
pure Java target, everything is Scala based, but can have Java source. It
may only have Java sources, but it's still considered a Scala target. So not
having Java here isn't really an indication that it's not supporting Java.

It is possible to build pure Java targets with SBT, it is driven by the autoScalaLibrary setting : https://www.scala-sbt.org/1.x/docs/Configuring-Scala.html#Configuring+the+scala-library+dependency

[ckipp01]

It is possible to build pure Java targets with SBT, it is driven by the autoScalaLibrary setting

All that setting really does is not add the scala library to the classpath. I actually checked with the sbt gurus about this since I had the same question, but even when using this (I tested it) for BSP related purposes sbt still treats these as Scala build targets when this is being used.

[Baccata]

I see :/