Taking JDT LS to next level

[gayanper]

Hi All,

Given that most of the stakeholders behind JDT.LS doesn't focus on Eclipse Java IDE from what I can see, I have the following thoughts which I would like to have a discussion around.

Today JDT.LS use JDT.Core underneath for implement LS support for Java. JDT.Core comes with a long history and therefore the design might be bit of complex to move forward faster. As result the movement of JDT.LS requires lot of effort.

If we think about stability, JDT has some issues around when it comes the Java Parser and AST resolution which brings issues time to time mostly in Completions but also in Compilation as well. One issues that I observe is the Parser implementation has become some what complex and only known by few who has worked from the beginning. It takes lot of time to investigate an issue and provide fix for a new developer in this area.

Now one suggestion I have is, What if we start to look at using a more modern parser which is actively maintain ? We could start to use it in LS for completions, then diagnostics which will provide us the opportunity add new features easily by designing it better from what we have learned so far.

For the compiler part, we could keep on using Eclipse JDT Compiler or look for Javac if we don't see any issues not using JDT Compiler. I know that JDT compiler is fast and more memory efficient, But considering effort we have to put to maintain it while Java team already provides a compiler, Should we spend that time on improving the LS for better Java Editing experience ?

We could still use the Eclipse JDT Indexer as it is if we could wire all the events to trigger indexing.

Lets discuss ...

[gayanper]

@jdneo @testforstephen @rgrunber FYI

[mickaelistria]

You may want to try and compare with https://github.com/georgewfraser/java-language-server , your feedback would be welcome.

Do you have any proposal of concrete parser generator/frameworks that could be used instead of jikespg in JDT Core? Is the parser output all internal and hidden behind an API, so one could try another one nd all changes would remain internal?

[gayanper]

Could we use something similar to Antlr ? At least for the code completions needs if javac parser is not capable of building a AST with compilation error java source.

[testforstephen]

Both NetBeans and georgewfraser/java-language-server are built on top of javac. Using javac as AST parser is one direction we want to explore.

[gayanper]

Good point, I think we should investigate this, Becasue specially for completions, we need a AST, and most of the time completions happens on a source code where there are parsing errors until the statement is completed.

[mickaelistria]

I'm still not capable of answering all that, but at least, here is a bit of code that can help to evaluate the capabilities of javac when it comes to getting the AST for a faulty source file.

package demo;

import java.io.IOException;
import java.net.URISyntaxException;

import javax.tools.JavaFileObject.Kind;
import javax.tools.SimpleJavaFileObject;

public class TestJavac {

	public static void main(String[] args) throws URISyntaxException {
		SimpleJavaFileObject javaFile = new SimpleJavaFileObject(TestJavac.class.getResource("parseError.java").toURI(), Kind.SOURCE) {
			@Override
			public CharSequence getCharContent(boolean ignoreEncodingErrors) throws IOException {
				return """
public class parseError {
	pub
}
						""";
			}
		};
		com.sun.tools.javac.main.JavaCompiler javac = com.sun.tools.javac.main.JavaCompiler.instance(new com.sun.tools.javac.util.Context());
		com.sun.tools.javac.tree.JCTree.JCCompilationUnit cu = javac.parse(javaFile);
		cu.accept(new com.sun.tools.javac.tree.TreeScanner() {
			 public void scan(com.sun.tools.javac.tree.JCTree that) {
				 if (that != null) {
					 System.err.println(that.getKind() + "=" + that.toString());
				 }
				 super.scan(that);
			 }
		});
	}
}

and this can be run with $ java --add-modules=ALL-SYSTEM --add-opens java.base/java.lang=ALL-UNNAMED --add-opens jdk.compiler/com.sun.tools.javac.main=ALL-UNNAMED --add-opens jdk.compiler/com.sun.tools.javac.util=ALL-UNNAMED --add-opens jdk.compiler/com.sun.tools.javac.tree=ALL-UNNAMED --add-opens jdk.compiler/com.sun.tools.javac.api=ALL-UNNAMED demo.TestJavac

[mickaelistria]

It looks like javac is at least partly fault-tolerant:

public class parseError {
	Stri
	@Override
	public void lol() {
		int n
	}
	public void rofl() {
		System.err.prin
	}
}

generates some not too bad AST,:

LASS=
public class parseError {
    (ERROR) <error>;
    public void lol() {
        int n;
    }
    public void rofl() {
        (ERROR);
    }
}
MODIFIERS=public 
VARIABLE=(ERROR) <error>
MODIFIERS=
ERRONEOUS=(ERROR)
METHOD=
public void lol() {
    int n;
}
MODIFIERS=public 
PRIMITIVE_TYPE=void
BLOCK={
    int n;
}
VARIABLE=int n
MODIFIERS=
PRIMITIVE_TYPE=int
METHOD=
public void rofl() {
    (ERROR);
}
MODIFIERS=public 
PRIMITIVE_TYPE=void
BLOCK={
    (ERROR);
}
EXPRESSION_STATEMENT=(ERROR);
ERRONEOUS=(ERROR)

But another aspect of the problem is illustrated in this picture: ECJ's Parser is the root of a relatively big class hierarchy; and it's not obvious to know which parsers are meant to do what, what specificity they provide that make an extra parser useful. For example, I don't understand some things such as indexing rely on a parser instead of on the higher-level AST.

[testforstephen]

Good findings.

Yes, Javac has been optimized a lot to support error recovery. And regarding the ECJ parser variants, I also had the question why we need so many variants in JDT.

For the assist parser, I can understand that it's optimized as a layered parser. It first performs dietParse to parse the high-level Java elements (package/import statement/type/field/method declarations), then performs parseBlockStatements to parse method bodies. It can be used to improve the parsing speed for use cases like code completion. For example, during code completion, it firstly diet parse to find the member declaration enclosing the cursor, then parse only the corresponding method body. This can get an incomplete parsing tree with enough completion context.

For other variants, looks like they are used to analyse the code structure, which I think can be replaced by Javac.

The featues that depend on these ECJ parser variants are the real challenging part if switching to Javac parser. We may need to resturcture the JDT code or even rewrite the feature to make it compatible with Javac.

[gayanper]

If I remember correctly, the indexing parser is used when indexing code which only parse definitions and imports only without the method blocks. But there is a special case where we parse methods containing lambdas if I remember correct

[akurtakov]

@mpalat This is probably interesting topic for you too.

[mickaelistria]

I'm drafting a "framework" to make progress on this topic, basically ensuring we can run unit tests that just try to parse a very minimal file with Javac so we can then "fill the holes" in the code with some new JavacASTConverter-like class.
My progress can be found at https://github.com/mickaelistria/eclipse.jdt.core/tree/use-javac-api , it can be imported and run from workspace, trying to run some tests from StandAloneASTParserWithJavacTest (that will fail because code is not there, but still managed to find the Javac compiler and so on!). To run those, or to even run JDT using Javac, you'll need to include org.eclipse.jdt.core.visibleJavac system fragment to your launch so OSGi can see the jdk.compiler internal modules, and the VM must be configured with -add-opens java.base/java.lang=ALL-UNNAMED --add-opens jdk.compiler/com.sun.tools.javac.main=ALL-UNNAMED --add-opens jdk.compiler/com.sun.tools.javac.util=ALL-UNNAMED --add-opens jdk.compiler/com.sun.tools.javac.tree=ALL-UNNAMED --add-opens jdk.compiler/com.sun.tools.javac.api=ALL-UNNAMED.

@testforstephen I believe you were also working, or planning to work, on this topic. Is there some possible collaboration at this stage already?

[testforstephen]

I think it will be a good move, that is starting with the build aspect. But even with that then we might have more issues with things such as quick fixes if we find that certain problems reported by javac will not be handled in the same way in ECJ. But those kind of scenarios should be really minimum.

But what if we change the build and code actions computations based on javac ? Do you think with javac we can also implement the ASTRewrites we have for things like quick fixes and refactoring ?

Because then down the line we can look into Completions, Selection parsers and see how we should replace them.

The quickfix is associated with problem-id. The problem-id from ECJ does not map exactly to javac error code. For example, some warnings in ECJ like unused variable are not reported by Javac. To avoid feature degradation, we can still use ECJ to report temp diagnostics for the open WorkingCopies and serve quick fixes. But the final project build will then use the build tool or Javac to generate class files and report diagnostics. We just need to distinguish the diagnostic of different tools well in the user interface, and give users transparency about them. This is similar to Intellij IDEA, which uses its own AST Parser to provide quick fixes for open files, but javac is used for the final project compilation and displays its diagnostic results in a different view.

Regarding ASTRewrite, Javac includes its ASTNode (e.g. JCTree) data structure in the API com.sun.tools.javac.tree.*. Technically, we can use the javac tree API to replicate the AST tree, modify the tree and compute the diff for similar purposes. But I will be more cautious to move in this direction. At least for me, the ROI is not clear so far.

[akaroml]

Echo here. We started with isolating the build process and delegating that to the actual build tools. We started with Gradle and released the Build Server for Gradle as the bridge between the Language Server and the Build Server. We are starting to see an increase in the success rate of importing Gradle projects and better compatibility when build plugins are involved.

[gayanper]

Yes its great improvement which i know use daily.

[guw]

@testforstephen Could ErrorProne with javac provide the missing problems?

[testforstephen]

@guw Have a quick look at the problem list of ErrorProne, and yes, it covers some of the missing problems, such as Unused Variable, unused Import, etc. This is a good addition to Javac compiler.

[mickaelistria]

I've tried a few more experiments:

• Reusing some Javac code (definitions mostly at the moment) in JDT, to make JDT and Javac more aligned, and hopefully JDT more capable to leveraging new features from Javac with less code (this later part doesn't have a valid example yet): [WIP] Use some javac classes eclipse-jdt/eclipse.jdt.core#1234
• Converting Javac's parser AST to JDT's compiler parser AST: some data is lost in translation, such as comment or some annotations. In some case, the lost data is important for further edition, so the loss is too important. This is IMO a dead-end 🛑

I'm going to investigate the following:

• convert JDT's compiler/parser AST to Javac's parser AST. If this is working decently enough, JDT could get rid of the "codegen" (AST to bytecode) duty, replacing it with JDT AST -> Javac AST which is probably easier to operate as new Java features get incorporated.

[mickaelistria]

Here is the output of my later experiment: https://github.com/mickaelistria/eclipse.jdt.core/tree/parse-with-jdt-codegen-with-javac . With this I manage to generate some (simple) .java files parsed with JDT's parser (which is better for several things, particularly to hook it inthe IDE), and to take the output of this parser, convert it to Javac compiler AST, and then run the next code generation steps on the Javac AST.
One benefit of this is that then JDT has less duties and can achieve the same outcome with less maintenance effort. However, one pitfall would then be that if JDT adds extra value on the codegen phase compared to Javac, then this extra value is lost.
So this is more a question: is JDT bringing extra value over Javac in the phase that come after parsing?

[mickaelistria]

I also mentioned it in eclipse-jdt/eclipse.jdt.core#1234 (comment) , but with upcoming java.lang.classfile package, it would become easily possible to replace a lot of the complex work done in CodeStream and other places where .class are read/written by the new standard Java API.
While it's not necessarily a trivial thing to change, it seems like a low-hanging fruit to me: a work we do once in JDT and then supporting newer Java constructs would be noticeably cheeper. and safer by doing it with a proper API instead of doing it with byte stream.

[laeubi]

@mickaelistria Oracle announced a Language-Server based on Javac here:
https://inside.java/2023/10/18/announcing-vscode-extension/

even though they mention VSCode I wonder if this can be used in eclipse or even other LS servers (like jdt-ls) to perform some work.

[gayanper]

I just checked few of what you have mentioned above:

• The editors provide diagnostics while typing
• When you try to run the code, it expect the whole project to compile, even thought the class which has the main method doesn't have any dependency to uncompilable code. The reason for this is, they use gradle compile command to compile the project before running. I also had to add gradle plugin application to run the project.
• Debugging doesn't support lambda breakpoints like vscode-java-debug does. But adding a line breakpoint will take you through all the lambda's as well in that line.

[mickaelistria]

Thanks for the details.

When you try to run the code, it expect the whole project to compile, even thought the class which has the main method doesn't have any dependency to uncompilable code. The reason for this is, they use gradle compile command to compile the project before running. I also had to add gradle plugin application to run the project.

OK, so that's IMO the main issue with this LS at the moment as it tends to make a longer feedback loop. It has often been a complaint against Netbeans as well that it requires to almost always rebuild when doing small edit/debug loops.

Debugging doesn't support lambda breakpoints like vscode-java-debug does

The Java debugger in VSCode is not provided by VSCode-Java, but by https://github.com/microsoft/java-debug/ .
But it's interesting to note that the scope of netbeans LS is also to provide a debug adapter (I just noticed it a couple of hours ago by editing the code and it is IMO a good surprise).

[gayanper]

It do provide diagnostic while typing without needing to build.

[gayanper]

Unfortunately https://github.com/microsoft/java-debug/ do depends on JDT.LS to work. It used JDTLS for finding resolving main class, classpath and also to performa evaluations and conditional breakpoints.

The oracle solution actually use the underlying build system to run the application, at least thats what I observed when I try to run a project without having a proper run target in the gradle build config.

[gayanper]

Also the debugger features far way behind when comparing features to vscode-java-debug and Eclipse Debugger. If we keep pushing changes to JDT project, getting more active contributors to help reviewing and getting things in, I think VSCodeJava gonna be the leading Java Extension for vscode for foreseeable future.

[snjeza]

The oracle solution actually use the underlying build system to run the application, at least thats what I observed when I try to run a project without having a proper run target in the gradle build config.

It requires that the whole project is built properly.

[gayanper]

Yes thats a big disadvantage. What I like about JDT is you can just run while you have tons of compiler errors, if the class you try to run doesn't have any errors which fails at runtime.

[mickaelistria]

Generating .class files at the right target location on save so a full build isn't always required is indeed the biggest strength of JDT. AFAIK, it doesn't come from the compiler per se, but more from the IJavaProject model which allows to keep track of the right options to generate the .class (eg classpath, build options...) for a given .java. So it should be possible to anyone interested to implement something similar invoking javac CLI or internal JDK compiler API instead of JDT compiler; as long as there is a good underlying build model that the IDE can read in order to orchestrate the compilation (what JDT has, and what most tools have partially at least to resolve deps and other settings in order to provide good contextual edition assistance).

Note that JDT also has a cool feature, but maybe less critical, that it can generate .class even for files with errors in those (replacing faulty subtrees in AST by thow new Error(...)) thanks to proceedOnError option; this feature is not available in any javac-based solution as far as I know. It would be nice if javac directly had such options.
However, I see it as way less critical; as it seems acceptable for users that there file must compile decently before trying to run it.

[mickaelistria]

Here is a followup issue on JDT Core about moving from ECJ to Javac: eclipse-jdt/eclipse.jdt.core#1827

[gayanper]

@mickaelistria I do see now JDT LS use a jdt.core incubating version, does that means usage of Javac is already used in jdt.ls ?

[rgrunber]

No,

eclipse.jdt.ls/org.eclipse.jdt.ls.target/org.eclipse.jdt.ls.tp.target

Lines 30 to 35 in 0acbde9

	<location includeAllPlatforms="false" includeConfigurePhase="false" includeMode="planner" includeSource="true" type="InstallableUnit">
	<unit id="org.eclipse.jdt.core.compiler.batch" version="0.0.0"/>
	<unit id="org.eclipse.jdt.core" version="0.0.0"/>
	<unit id="org.eclipse.jdt.apt.core" version="0.0.0"/>
	<repository location="https://download.eclipse.org/jdtls/jdt-core-incubator/snapshots/"/>
	</location>

was mainly about being able to have clients (eg. vscode-java and others) users trying out preview features / certain fixes at a faster pace, while we get them merged upstream. The dev branch of the incubator just tracks jdt.core and may have some extra patches. You want to look for the dom branches.

[gayanper]

@mickaelistria i just peak at the dom branch mentioned above and i see you and others have made lot of progress there. I also saw some DOM base completions stuff. Does that means you have started to add support dom base Completions Engine support? If so I was very much like to test this branch and start contribute to help this work. After spending lot of ours to figure out ECJ completion parser issues and fix them, i want to see if the javac approach will be much smoother.

[mickaelistria]

Yes, we're making good progress; using javac/dom-based strategies, we get the JDT model working, codeSelect, build... working (with some glitches) and the beginning of a completion engine that doesn't use ECJ.
Contributions to the DOMCompletionEngine are more than welcome! Do you need any further pointer to get started?

[gayanper]

I will have a look at the branch and post here if i need any more help the get started. I'm really excited about this work and progress 🥳

[mickaelistria]

OK, thanks. Please prefer starting new threads to https://github.com/eclipse-jdtls/eclipse-jdt-core-incubator/discussions now.

[gayanper]

@mickaelistria @rgrunber and others, Should we build a separate LS nightly version with the changes ongoing for integrating Javac changes?

The reason I'm asking is, I would very much like to explore the possibility of implementing completions based on some sort of javac based parser take a different approach than what is taken in JDT.Core completion parser. I'm assuming all the magic happens in Netbeans starting at this class https://github.com/apache/netbeans/blob/master/java/java.completion/src/org/netbeans/modules/java/completion/JavaCompletionTask.java, which I find very straight forward to maintain and anyone can add new functionality provided the Parser is updated. WDYT ?

[mickaelistria]

Should we build a separate LS nightly version with the changes ongoing for integrating Javac changes?

It seems a bit too early as Javac-based JDT is not considered as functional enough yet. IIRC, @fbricon has documented the process to build JDT-LS with the Javac work and enable it in vscode-java; this is currently good enough IMO as it allows complete testing while forcing testers to be at code-level (ie to become contributors), which seems still necessary right now as the project isn't mature enough to reach general usage and to require us to provide user-level support.

The reason I'm asking is, I would very much like to explore the possibility of implementing completions based on some sort of javac based parser take a different approach than what is taken in JDT.Core completion parser.

In general, we try to strongly avoid any "generic edition" feature to depend on Javac. We try to work with a DOM, and this DOM can be provided by either ECJ or Javac.
This works on the JDT's completion engine. We'd actually like to see a "DOMCompletionEngine" in JDT that would allow to work with a DOM directly instead of using ECJ's parser; but this is not trivial development; so help there is really welcome.
The best way to work on it is IMO to stay at JDT-Core level: writing tests in JDT-Core and so on. This will make things simpler as the LS is an extra level of indirection which can make development more complex.

[gayanper]

@fbricon can you drop the link to above documentation @mickaelistria mentioned. I checked wiki and couldn't find it.

[fbricon]

To try it out in vscode-java, you can manually install a recent vscode-java.vsix I regularly upload to https://github.com/fbricon/vscode-java/releases/

• make sure JDK 22 is enabled:
• Enable with "java.jdt.ls.javac.enabled":"on" in vscode settings

If you want to build it locally:

• First install jdt.core from the dom-with-javac branch with mvn clean install -DskipTests -Pp2-repo
• Then checkout (and probably rebase) jdt.ls PR#3123
• Then checkout (and probably rebase) vscode-java PR#3558
• Build the server jars with npm run build-server
• Generate the vsix with npx @vscode/vsce package

The completion is still delegated to ECJ, as we haven't started providing a replacement based on javac yet.

[testforstephen]

@fbricon I think we might need to update the branch vscode-java pod branch so that it uses the latest flag for enabling the javac compiler

'-DAbstractImageBuilder.compilerFactory=org.eclipse.jdt.internal.javac.JavacCompilerFactory'

@testforstephen please correct me If I got above wrong.

Correct. we should replace the old flag with the new one above.

[fbricon]

Done https://github.com/redhat-developer/vscode-java/pull/3558/files#diff-2d9ec606b265a83596a734df153c4ef01523700dc2576dcf1a80c9092c14d6d3R134

New vscode-java(c) build available at https://github.com/fbricon/vscode-java/releases/tag/javac-prototype

[gayanper]

@fbricon I'm working on DOMCompletionEngine adding support for more completions. it would be nice to have a flag to control this from the settings so that I don't need to alter the jdt.ls vmargs to add the following flag

-DCompilationUnit.codeComplete.DOM_BASED_OPERATIONS=true

[fbricon]

@gayanper try the latest build https://github.com/fbricon/vscode-java/releases/tag/javac-prototype and set "java.completion.engine": "dom":

[gayanper]

Nice 👌, thanks @fbricon