Java editor syntax highlighting via extension point

[BoykoAlex]

Language servers (LSP4E project) could provide semantic tokens with highlight information for Java files. These semantic tokens are primarily for languages embedded into Java. For example Spring Java code can have embedded:

• SPEL expressions
• JPQL
• HQL
• SQL
• Cron expressions

These embedded languages syntax fits nicely into LSP semantic tokens feature.

Bringing in semantic highlighting for these embedded languages into JDT Java editor is a challenge because syntax highlighting in Java editor does not allow for contributions.

(cc: @martinlippert)

[martinlippert]

We implemented the necessary logic for the highlighting via semantic tokens and LSP already inside of the language server that we are building for Spring and have this feature working nicely in VSCode (e.g. https://x.com/springtools4/status/1816014400379032005), now trying to bring this to Eclipse as well. It already works out-of-the-box when using the generic editor for Java source files, but since everybody is using the real JDT UI Java source editor in Eclipse, we would love to bring this to JDT UI as well - if possible.

@BoykoAlex submitted an initial PR for this (#1595), which is more a PoC than a ready-to-merge contribution, but would be awesome to hear from you whether the team would be interested in pushing this forward. Since it involves adding new public API to JDT UI, we would love to hear your thoughts about this.

[iloveeclipse]

Haven't looked in the code yet, but the idea sounds great.

[noopur2507]

As I understand, this is an enhancement that will add highlighting for embedded languages in the Java editor via an extension point. Sounds good!

[martinlippert]

Sounds like the is some consensus that this would be a good thing to implement. What are the next steps? I think we (specifically @BoykoAlex ) would need some advice on the next steps, since the PR is more meant as a draft to discuss than the final implementation (as mentioned on the PR). So would someone be willing to jump on the PR and provide overall feedback on the questions at hand? I am sure @BoykoAlex would also be around for a screenshare call to get some advice, provide some overview of the PR, etc.

[martinlippert]

(and thanks to @noopur2507 , @jukzi , @iloveeclipse for your initial reactions here, much appreciated)

[BoykoAlex]

Any feedback on the attached PR anyone?
I suppose my main questions are:

1. Any wished for the extension point schema?
2. Ideas for slow/async semantic tokens computation integration into the existing Java editor SemanticHighlightingReconciler?

Currently the PR computes contributed semantic tokens synchronously which is an obvious no go... Wonder if for example if can trigger a semantic highlight reconcile once async semantic tokens are computed? We'd need to to re-compute the sync tokens and add the already computed lazy ones on top, right? Not familiar with this code so would appreciate some guidance :-)

[BoykoAlex]

The async approach resulted in some ugly flickering. The sync solution worked much better. I'd go with the sync solution for now. Please review the PR #1683