Eglot causes lock up when language server tries to watch files in large directory

[2xsaiko]

I'm using Eglot with nil, a language server for the Nix language. This language server wants to watch two files in the current directory over LSP protocol, which causes a find process to be spawned:

find -H . ( -path */SCCS/* -o -path */RCS/* -o [...] -o -path *.pyc -o -path *.pyo ) -prune -o -type f -print0

Since this recursively searches the whole current working directory, it can take a very long time to get the effect the language server wants (to watch files ./flake.nix and ./flake.lock), when running in a large directory such as the user home. (As a workaround, the find process can be killed.)

Here's the event buffer log (whole log):

[server-request] (id:1) Sat Jul 29 21:12:16 2023:
(:jsonrpc "2.0" :id 1 :method "client/registerCapability" :params
	  (:registrations
	   [(:id "workspace/didChangeWatchedFiles" :method "workspace/didChangeWatchedFiles" :registerOptions
		 (:watchers
		  [(:globPattern "/home/saiko/flake.lock")
		   (:globPattern "/home/saiko/flake.nix")]))]))
[client-reply] (id:1) ERROR Sat Jul 29 21:13:35 2023:
(:jsonrpc "2.0" :id 1 :error
	  (:code -32603 :message "Internal error"))

(the internal error is a result of me killing the find process)

See oxalica/nil#98. This also has a debugger backtrace (thanks @sauricat!)

Minimum reproducible example

1. Get Emacs, Eglot, nix-mode (for setting up eglot to run nil in .nix files) and the nil language server

• Using the Nix package manager: Run nix develop --impure --expr 'let pkgs = import (builtins.getFlake "nixpkgs/11cf5e1c74fe6892e860afeeaf3bfb84fdb7b1c3") {}; in pkgs.mkShell { buildInputs = [pkgs.nil ((pkgs.emacsPackagesFor pkgs.emacs-nox).emacsWithPackages (epkgs: [epkgs.nix-mode epkgs.eglot]))]; }' to open a shell with the necessary packages and Emacs configured to load plugins
• Manually install Emacs 28.2, eglot 1.15, nix-mode 1.5.0, nil 2023-05-09 (these are the same versions as you get with the Nix command)

2. Run emacs ~/test.nix (or a file in any other large directory). The file doesn't have to exist.
3. Enable eglot, M-x eglot
4. Emacs starts the language server, last status line being something like "[eglot] Connected! Server `nil' now managing `(nix-mode)' buffers in project `saiko'."
5. Emacs waits for the find process

[joaotavora]

2. Run emacs ~/test.nix (or a file in any other large directory).

Yes. More accurately, any "file in a non-project, non-Git controlled or any kind of version controlled extremely large directory".
It seems a bit contrived to expect Emacs/Eglot to be very fast in finding files in the user's HOME directory, especially if you don't have Git-versioned. Eglot is almost always used to manage files in a project, or at least a subdirectory, not at the very top of a hierarchy like a user's HOME.

Is this really a use case? I don't have nix or have any plans to try it, but do you develop with the Nix language directly in your $HOME?

Anyway, the slowness you experience almost certainly comes from project-files, which Eglot calls so that it can match what are usually glob expressions (file-name lookalikes with * and ** in them) instead of absolute file names.

However in this case where a glob isn't anywhere, I guess Eglot could skip calling project-files. But if the server does request globs, then project-files and the inherent problems of find are unavoidable (unless you use Git or manage to configure project.el to use some other more efficient file-listing method).

In this case, even if I do this optimization it won't work. I think the Nix server is non-compliant is that it is sending absolute file names as the glob pattern, when the Eglot client told it in the beginning of the conversation that the base path is /home/saiko. So it should ask the client to watch flake.lock and flake.nix instead.

[2xsaiko]

Is this really a use case? I don't have nix or have any plans to try it, but do you develop with the Nix language directly in your $HOME?

I pretty frequently write standalone Nix expression files for a shell or a package, like the Nix expression in quotes for the Emacs shell command above. In fact, that's how I found this out, since I was configuring emacs in ~/emacs.nix before adding it to my main configuration once it was working, and I configured it to automatically enable eglot in certain major modes including Nix. Had to disable that specifically for Nix because of this issue. I don't think it's unreasonable to not want your editor to lock up because you want to use LSP for loose files in your home directory, and this also has not been an issue in any other LSP editor I've used so far (i.e. Kate and Kakoune with kak-lsp).

(I do this for small C/C++ files too but there it's not an issue because clangd doesn't watch anything.)

IMO, outside of a project, when you'd otherwise be scanning an unknown number of files, it should only consider already open files in the editor unless specific files like here are given. That way you can ensure you never get into this situation.

I think the Nix server is non-compliant is that it is sending absolute file names as the glob pattern, when the Eglot client told it in the beginning of the conversation that the base path is /home/saiko. So it should ask the client to watch flake.lock and flake.nix instead.

I'm unfamiliar with the LSP protocol details, maybe @oxalica can say something about that.

[joaotavora]

In fact, that's how I found this out, since I was configuring emacs in ~/emacs.nix before adding it to my main configuration once it was working,

Why don't you do this in a directory which isn't parent to all your files? Like ~/Source or ~/Tmp. Eglot is for working in projects, it was designed for that. They can be small projects, of course, but Eglot uses the project.el infrastructure and -- by default -- it thinks that if you are editing a file A in X/Y/Z then Z and its subdirectories are the project, give or take.

So just start doing these one-offs in a subdirectory. I do them in ~/Tmp all the time. Never in ~/.

[mhanberg]

I'm experiencing this same error in my git project. we have about a 1000 files that would be file watched.

elixir-tools/next-ls#270