Client Metals: connection to server is erroring. Shutting down server.

[huntario]

Describe the bug

Things were going awesome as I was step debugging through some tests using the inline test debugging option. Then it started to get slower and slower, until it locked up. Then I restarted the computer, uninstalled/reinstalled the Metals extension, and VSCode as well.

I will say I'm on Scala 2.13.5 but everything was working great fir my needs until this debug session seemed to trip me up and lock me out. Upgrading isn't an immediate option for me so mainly just posting in case others run into it.

I'm also seeing the same behavior as #1488 when this occurs. The buttons are unreactive.

Have "metals.javaHome": "/Users/hunter.leachman/.sdkman/candidates/java/current/bin/java" set in settings.json

Using coursier located at /usr/local/bin/coursier
/Users/hunter.leachman/.sdkman/candidates/java/current/bin/java -version:
openjdk version "17.0.11" 2024-04-16
/Users/hunter.leachman/.sdkman/candidates/java/current/bin/java -version:
OpenJDK Runtime Environment Temurin-17.0.11+9 (build 17.0.11+9)
OpenJDK 64-Bit Server VM Temurin-17.0.11+9 (build 17.0.11+9, mixed mode)
Using Java Home: /Users/hunter.leachman/.sdkman/candidates/java/current
Error: Could not find or load main class scala.meta.metals.Main
Caused by: java.lang.ClassNotFoundException: scala.meta.metals.Main
[Error - 4:11:10 AM] Server initialization failed.
  Message: Pending response rejected since connection got disposed
  Code: -32097 
[Info  - 4:11:10 AM] Connection to server got closed. Server will restart.
true
[Error - 4:11:10 AM] Metals client: couldn't create connection to server.
  Message: Pending response rejected since connection got disposed
  Code: -32097 
Error: Could not find or load main class scala.meta.metals.Main
Caused by: java.lang.ClassNotFoundException: scala.meta.metals.Main
[Error - 4:11:10 AM] Server initialization failed.
  Message: Pending response rejected since connection got disposed
  Code: -32097 
[Info  - 4:11:10 AM] Connection to server got closed. Server will restart.
true
[Error - 4:11:10 AM] Metals client: couldn't create connection to server.
  Message: Pending response rejected since connection got disposed
  Code: -32097 
Could not launch Metals Language Server:
Pending response rejected since connection got disposed
Release notes was not shown: not showing release notes since they've already been seen for your current version
Error: Could not find or load main class scala.meta.metals.Main
Caused by: java.lang.ClassNotFoundException: scala.meta.metals.Main
[Error - 4:11:10 AM] Server initialization failed.
  Message: Pending response rejected since connection got disposed
  Code: -32097 
[Info  - 4:11:10 AM] Connection to server got closed. Server will restart.
true
[Error - 4:11:10 AM] Metals client: couldn't create connection to server.
  Message: Pending response rejected since connection got disposed
  Code: -32097 
[Error - 4:11:10 AM] Restarting server failed
  Message: Pending response rejected since connection got disposed
  Code: -32097 
Error: Could not find or load main class scala.meta.metals.Main
Caused by: java.lang.ClassNotFoundException: scala.meta.metals.Main
[Error - 4:11:10 AM] Server initialization failed.
  Message: Pending response rejected since connection got disposed
  Code: -32097 
[Info  - 4:11:10 AM] Connection to server got closed. Server will restart.
true
[Error - 4:11:10 AM] Metals client: couldn't create connection to server.
  Message: Pending response rejected since connection got disposed
  Code: -32097 
[Error - 4:11:10 AM] Restarting server failed
  Message: Pending response rejected since connection got disposed
  Code: -32097 
Error: Could not find or load main class scala.meta.metals.Main
Caused by: java.lang.ClassNotFoundException: scala.meta.metals.Main
[Error - 4:11:10 AM] Server initialization failed.
  Message: Pending response rejected since connection got disposed
  Code: -32097 
[Error - 4:11:10 AM] The Metals server crashed 5 times in the last 3 minutes. The server will not be restarted. See the output for more information.
[Error - 4:11:10 AM] Metals client: couldn't create connection to server.
  Message: Pending response rejected since connection got disposed
  Code: -32097 
[Error - 4:11:10 AM] Restarting server failed
  Message: Pending response rejected since connection got disposed
  Code: -32097

I've tried the Pre-release version. I noticed Bloop had a new release a few hours before my issues began? - https://scalacenter.github.io/bloop/setup#sbt -

I'm also getting these messages -

Client Metals: connection to server is erroring. Shutting down server.
write EPIPE
Metals client: couldn't create connection to server.
Restarting server failed
The Metals server crashed 5 times in the last 3 minutes. The server will not be restarted. See the output for more information.

Everything was going so well, it's like these errors are cached somewhere and won't go away?

The project compiles with sbt

Nothing seems to make the errors go away/

To Reproduce Steps to reproduce the behavior:

1. Use the inline test debugging for a few semi-long sessions, setting a few break points in a few files
2. Repeat and maybe notice the debugger taking longer and longer to connect?
3. Eventually it locks up, and the errors above continue after restarting, reinstalling, switching to pre/release versions

Expected behavior
The errors at least go away after restarting computer, the plugin, or uinstalling/reinstalling

Screenshots

Installation:

• Operating system: macOS/Windows/Linux

- VSCode version:

• VSCode extension version:
• Metals version: (found in VSCode settings, under metals.serverVersion)

Additional context
Everything was working so perfectly until these errors started seemingly out of no where.

Really appreciate y'all and the work you are doing. This has come a long way in the last 2 years since the last time I tried out Metals. I'm thiiiiiiiiis close (finger to thumb) to replacing IntelliJ, almost there.

"Metals client: couldn't create connection to server."
"he Metals server crashed 5 times in the last 3 minutes. The server will not be restarted. See the output for more information."
"Restarting server failed"

[huntario]

The strange thing about this is that I've never seen anything so resistant to restarting, reinstalling, etc. I deleted VS code entirely, remove all the traces of it I could fine, re-cloned a new copy of the repo I'm working on, reinstalled everything, and it just continues to fail.

[tgodzik]

Is there any Java process running in the background? I can't figure out why this would be happening. Maybe we are picking up JAVA_OPTS and that is interfering somehow?

[huntario]

Thank you so much, commenting out my JAVA_OPTS and restarting seems to resolve the issue for me. It will also let me add export JAVA_OPTS="-Xmx4g" back fine.

Adding export JAVA_OPTS="-agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=localhost:5050" seems to bring the issue back. I also tried a couple different ports and it does the same. Don't see any difference in behavior between the pre-release and release versions either. I was using this to activate the debugger but I think I can find another way to do it for now.

Probably something local to me if no one else runs into it. Really appreciate your suggestion, got me out of a bad place.

[tgodzik]

Och, I think I know what happened. Metals reads JAVA_OPTS and adds it to java process, but it needs to have agentlib silent if you add -agentlib option, since it will print some stuff to stdout, which will not be understood by VS Code, it only expects jsons.

Would be great to detect that automatically and print to the user 🤔

[huntario]

Good call, If I add quiet=y ( https://stackoverflow.com/questions/48431989/suppress-listening-for-transport-dt-socket-message-when-remote-debugging-java ) it seems to be cool with that.

Here is the process that is working for me now to step through test code AND code under test using ScalaTest.

---

in build.sbt place fork in Test := false which prevents SBT from running the tests in a forkedJVM which doesn't seem to want to listen, expected from my perspective.

Then in ./zshrc

export JAVA_OPTS="-agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=localhost:5050,quiet=y -Xmx4g"
(don't forget to source ~/.zshrc)

sbt
run

Then I'm able to run/debug the tests from the side test panel list (beaker icon) or the Debug play button/green triangle icon with this .vscode/launch.json :

{
  "version": "0.2.0",
  "configurations": [
    {
      "type": "scala",
      "request": "launch",
      "name": "Scala Test",
      "testClass": "com.hunter.applicationname.thing.HuntersDebugSpec"
    }
}

Where is com.hunter.applicationname.thing is the namespace at the top of the test file and HuntersDebugSpec is the name of the test class.

---

I still don't completely understand when the icons that show up inline next to the tests are expected to appear. It seems to show right when I start VSCode and import or clean and restart, but if I lose it I can't figure out how to get them back. Even after clicking the beaker. Then sometimes they come back when I least expect it Not super worried about it because the above two options are great.

FWIW this started around 9-11pm central US time for me yesterday. This all seemed to be working without the quiet=y? I don't believe I changed any configuration, was just stepping through some tests and adding some mocks to my tests. The fans came on the laptop and all hell eventually broke loose! Probably my fault in some way I'm sure but maybe this will help other people.

[soronpo]

I'm experiencing this as well. Started now out of the blue.

[soronpo]

Oh, it's because the VS-Code plugin was updated. I downgraded. Works now.

[tgodzik]

The only thing that could cause the problem is #1498 the rest added in the plugin are only fallbacks. Do you have any java options settings that could cause metals to stop starting?

[soronpo]

This is what I get when upgrading from 1.31.0 to 1.33.0

Metals version: 1.2.2
Using coursier located at C:\Users\oronpo\null\Coursier\data\bin\coursier.bat
C:\Users\oronpo\AppData\Local\Coursier\cache\arc\https\github.com\AdoptOpenJDK\openjdk16-binaries\releases\download\jdk-16%252B36\OpenJDK16-jdk_x64_windows_hotspot_16_36.zip\jdk-16+36\bin\java -version:
openjdk version "16" 2021-03-16
OpenJDK Runtime Environment AdoptOpenJDK (build 16+36)
C:\Users\oronpo\AppData\Local\Coursier\cache\arc\https\github.com\AdoptOpenJDK\openjdk16-binaries\releases\download\jdk-16%252B36\OpenJDK16-jdk_x64_windows_hotspot_16_36.zip\jdk-16+36\bin\java -version:
OpenJDK 64-Bit Server VM AdoptOpenJDK (build 16+36, mixed mode, sharing)
Found java executable under C:\Program Files\Eclipse Adoptium\jdk-17.0.10.7-hotspot\bin\java.exe that resolves to C:\Program Files\Eclipse Adoptium\jdk-17.0.10.7-hotspot\bin\java.exe
C:\Program Files\Eclipse Adoptium\jdk-17.0.10.7-hotspot\bin\java -version:
openjdk version "17.0.10" 2024-01-16
OpenJDK Runtime Environment Temurin-17.0.10+7 (build 17.0.10+7)
C:\Program Files\Eclipse Adoptium\jdk-17.0.10.7-hotspot\bin\java -version:
OpenJDK 64-Bit Server VM Temurin-17.0.10+7 (build 17.0.10+7, mixed mode, sharing)
Using Java Home: C:\Program Files\Eclipse Adoptium\jdk-17.0.10.7-hotspot
Error: Could not find or load main class scala.meta.metals.Main
Caused by: java.lang.ClassNotFoundException: scala.meta.metals.Main
[Error - 7:51:39 PM] Server initialization failed.
  Message: Pending response rejected since connection got disposed
  Code: -32097 
[Info  - 7:51:39 PM] Connection to server got closed. Server will restart.
true
[Error - 7:51:39 PM] Metals client: couldn't create connection to server.
  Message: Pending response rejected since connection got disposed
  Code: -32097 
Error: Could not find or load main class scala.meta.metals.Main
Caused by: java.lang.ClassNotFoundException: scala.meta.metals.Main
[Error - 7:51:39 PM] Server initialization failed.
  Message: Pending response rejected since connection got disposed
  Code: -32097 
[Info  - 7:51:39 PM] Connection to server got closed. Server will restart.
true
[Error - 7:51:39 PM] Metals client: couldn't create connection to server.
  Message: Pending response rejected since connection got disposed
  Code: -32097 
Could not launch Metals Language Server:
Pending response rejected since connection got disposed
Release notes was not shown: not showing release notes since they've already been seen for your current version
Error: Could not find or load main class scala.meta.metals.Main
Caused by: java.lang.ClassNotFoundException: scala.meta.metals.Main
[Error - 7:51:39 PM] Server initialization failed.
  Message: Pending response rejected since connection got disposed
  Code: -32097 
[Info  - 7:51:39 PM] Connection to server got closed. Server will restart.
true
[Error - 7:51:39 PM] Metals client: couldn't create connection to server.
  Message: Pending response rejected since connection got disposed
  Code: -32097 
[Error - 7:51:39 PM] Restarting server failed
  Message: Pending response rejected since connection got disposed
  Code: -32097 
Error: Could not find or load main class scala.meta.metals.Main
Caused by: java.lang.ClassNotFoundException: scala.meta.metals.Main
[Error - 7:51:39 PM] Server initialization failed.
  Message: Pending response rejected since connection got disposed
  Code: -32097 
[Info  - 7:51:39 PM] Connection to server got closed. Server will restart.
true
[Error - 7:51:39 PM] Metals client: couldn't create connection to server.
  Message: Pending response rejected since connection got disposed
  Code: -32097 
[Error - 7:51:39 PM] Restarting server failed
  Message: Pending response rejected since connection got disposed
  Code: -32097 
Error: Could not find or load main class scala.meta.metals.Main
Caused by: java.lang.ClassNotFoundException: scala.meta.metals.Main
[Error - 7:51:39 PM] Server initialization failed.
  Message: Pending response rejected since connection got disposed
  Code: -32097 
[Error - 7:51:39 PM] The Metals server crashed 5 times in the last 3 minutes. The server will not be restarted. See the output for more information.
[Error - 7:51:39 PM] Metals client: couldn't create connection to server.
  Message: Pending response rejected since connection got disposed
  Code: -32097 
[Error - 7:51:39 PM] Restarting server failed
  Message: Pending response rejected since connection got disposed
  Code: -32097 

[tgodzik]

Is the same Java used for 1.2.2 version also? What is your server properties? Anything in the environment that could interfere with Metals managing to start the JVM ?

[soronpo]

This is how it look when it's successful under 1.31.0.

Metals version: 1.2.2
Using coursier located at C:\Users\oronpo\null\Coursier\data\bin\coursier.bat
C:\Users\oronpo\AppData\Local\Coursier\cache\arc\https\github.com\AdoptOpenJDK\openjdk16-binaries\releases\download\jdk-16%252B36\OpenJDK16-jdk_x64_windows_hotspot_16_36.zip\jdk-16+36\bin\java -version:
openjdk version "16" 2021-03-16
C:\Users\oronpo\AppData\Local\Coursier\cache\arc\https\github.com\AdoptOpenJDK\openjdk16-binaries\releases\download\jdk-16%252B36\OpenJDK16-jdk_x64_windows_hotspot_16_36.zip\jdk-16+36\bin\java -version:
OpenJDK Runtime Environment AdoptOpenJDK (build 16+36)
OpenJDK 64-Bit Server VM AdoptOpenJDK (build 16+36, mixed mode, sharing)
Found java executable under C:\Program Files\Eclipse Adoptium\jdk-17.0.10.7-hotspot\bin\java.exe that resolves to C:\Program Files\Eclipse Adoptium\jdk-17.0.10.7-hotspot\bin\java.exe
C:\Program Files\Eclipse Adoptium\jdk-17.0.10.7-hotspot\bin\java -version:
openjdk version "17.0.10" 2024-01-16
OpenJDK Runtime Environment Temurin-17.0.10+7 (build 17.0.10+7)
OpenJDK 64-Bit Server VM Temurin-17.0.10+7 (build 17.0.10+7, mixed mode, sharing)
Using Java Home: C:\Program Files\Eclipse Adoptium\jdk-17.0.10.7-hotspot
2024.05.16 19:53:38 INFO  logging to files C:\Users\oronpo\IdeaProjects\dfhdl\.metals\metals.log
2024.05.16 19:53:38 INFO  Started: Metals version 1.2.2 in folders 'C:\Users\oronpo\IdeaProjects\dfhdl' for client Visual Studio Code 1.89.1.
Initializing Scala Debugger
Release notes was not shown: not showing release notes since they've already been seen for your current version
2024.05.16 19:53:40 INFO  Attempting to connect to the build server...
2024.05.16 19:53:40 INFO  skipping build import with status 'Installed'
2024.05.16 19:53:40 INFO  Bloop uses C:\Program Files\Eclipse Adoptium\jdk-17.0.10.7-hotspot defined at C:\Users\oronpo\.bloop\bloop.json
2024.05.16 19:53:40 INFO  tracing is disabled for protocol BSP, to enable tracing of incoming and outgoing JSON messages create an empty file at C:\Users\oronpo\IdeaProjects\dfhdl\.metals\bsp.trace.json or C:\Users\oronpo\AppData\Local\scalameta\metals\cache\bsp.trace.json
2024.05.16 19:53:40 INFO  Attempting to connect to the build server...
2024.05.16 19:53:40 INFO  Bloop uses C:\Program Files\Eclipse Adoptium\jdk-17.0.10.7-hotspot defined at C:\Users\oronpo\.bloop\bloop.json
2024.05.16 19:53:40 INFO  Attempting to connect to the build server...
2024.05.16 19:53:40 INFO  Bloop uses C:\Program Files\Eclipse Adoptium\jdk-17.0.10.7-hotspot defined at C:\Users\oronpo\.bloop\bloop.json
2024.05.16 19:53:40 INFO  tracing is disabled for protocol BSP, to enable tracing of incoming and outgoing JSON messages create an empty file at C:\Users\oronpo\IdeaProjects\dfhdl\project\project\.metals\bsp.trace.json or C:\Users\oronpo\AppData\Local\scalameta\metals\cache\bsp.trace.json
2024.05.16 19:53:41 INFO  tracing is disabled for protocol BSP, to enable tracing of incoming and outgoing JSON messages create an empty file at C:\Users\oronpo\IdeaProjects\dfhdl\project\.metals\bsp.trace.json or C:\Users\oronpo\AppData\Local\scalameta\metals\cache\bsp.trace.json
2024.05.16 19:53:41 INFO  time: Connected to build server in 0.54s
2024.05.16 19:53:41 INFO  Connected to Build server: Bloop v1.5.15
2024.05.16 19:53:41 INFO  time: Imported build in 0.15s

[tgodzik]

I need to maybe add some debug information. Do you have JVM_OPTS variable, .jvmopts files or specified metals.serverProperties?

[soronpo]

Do you have JVM_OPTS variable, .jvmopts files or specified metals.serverProperties?

None of these

[tgodzik]

Not sure what is wrong, but I think I reverted the offending change. 1.34.0 should have it fixed. Sorry for the trouble!

[soronpo]

1.34.0

I'm sorry to report that 1.34.0 still has the same issue. Would be happy to help anyway I can to provide more information.

[soronpo]

1.34.0

I'm sorry to report that 1.34.0 still has the same issue. Would be happy to help anyway I can to provide more information.

SORRY, MY MISTAKE. 1.34.0 WORKS