Language Server guide

[kristianmandrup]

Would be great if you could include some kind of example or reference guide for how to write a Language Server (standard IDE protocol) for a language written in Chevrotain.

Even for something as simple as the calculator or tinyc. It seems like quite a jungle to get started. Does anyone have experience in this regard or know of some good resources?

[kristianmandrup]

I started an app-ml project here.
I will try to add VSCode extension and Language Server support with well documented instructions. You can then use it as reference.

[christianvoigt]

I wrote one here (but I am using my Chevrotain parser only indirectly there).

In my experience the official example was very helpful. It is not difficult to integrate your Chevrotain parser once you have learned how to set up a language server, so I am not sure if a Chevrotain-specific example is really necessary.

[bd82]

This is a topic that greatly interests me.
For example my Toml-Tools project

• https://github.com/bd82/toml-tools
  Could be expanded to include more editor tooling capabilities (not just a formatter)
  and wrap those tooling in the Language Server Protocol.

As always the limiting factor is free time... I doubt I will personally get around to it
but will be willing to accept contributions 😄

[bd82]

I wrote one here (but I am using my Chevrotain parser only indirectly there).

Thanks for that link @christianvoigt I may use it as a reference in my Toml-Tools project 👍

[kristianmandrup]

Great! Thanks. Well, it could just be a link of reference examples for "in the wild solutions" or perhaps an awesome-chevrotain page or repo... Cheers!

[kristianmandrup]

I'm pretty far now, with a well documented project, investigating how to do the "full shebang".

app-domain-ml

Still a WIP.

[jmrog]

FWIW, Stardog Union (where I work) uses chevrotain to build parsers for RDF-related languages, and these parsers back our stardog-language-servers. So, if you follow the trail from our language servers repo to our parsers repo, you'll see some "in the wild" uses of chevrotain with the LSP (we also have VSCode extensions that use the language servers, if that's of interest). Our language servers are all relatively new, and they'll continue to improve as time goes on; right now, the one for SPARQL has the most features, so you might look at it and at the AbstractLanguageServer class that our language servers extend.

[kristianmandrup]

Sweet :) Thanks! I'm trying to build a "best practices" how to guide with a walk through and examples.

Also trying to add better support for CST with location info for source maps, and how to best deliver auto-suggest etc

[bd82]

ArgDown also seems to contain a Language Server which uses a Chevrotain Parser.

- https://github.com/christianvoigt/argdown/tree/master/packages

Oops it was already linked in this thread 😢

[kristianmandrup]

Sweet. I'll get back to it shortly. Thanks for helping to make this happen!!

[kristianmandrup]

I've since then converted the official mini SQL language lexer, parser, actions and error recovery tutorial to TypeScript here and now working to add it as a VSC language extension with syntax highlighting and LSP.

Would love any help. I'm also planning to create some kind of infrastructure for chevrotain to make this process much easier.

[kristianmandrup]

@jmrog I'm now finally looking into your stardog LSPs to see what I can use. I also know that some new "token tracking capabilities" have since been added to chevrotain which should make it easier to add such features and also more powerful/useful.

Looks like your stardog-language-utils is the place to start?

[bd82]

On a related note, I am working on some editor services logic for XML.
Specifically content assist and validations.

• See: https://github.com/sap/xml-tools

This is the pure logic of implementing the editor services, not the language server protocol.

[kristianmandrup]

@bd82 Good stuff! I've been working on sql-lang the mini SQL chevrotain tutorial ported to TypeScript and now with utilities and documentation for writing a VS Code language extension complete with Language Server. Almost there...

[bd82]

@kristianmandrup is all the source code in that repo supposed to be there in the final version?
or are those left overs from copying stardog code?

[kristianmandrup]

From language-server-dot-visual-studio/

To add the completion provider (aka "content assist) for a VSC extension

connection.onInitialize((params): InitializeResult => {  
    return {        
        capabilities: {
           // ...
            completionProvider: {
                resolveProvider: true,
                "triggerCharacters": [ '=' ]
            },
            hoverProvider: true     
        }
    }
});

Sample onCompletion handler:

connection.onCompletion((textDocumentPosition: TextDocumentPositionParams): CompletionItem[] => {
    let text = documents.get(textDocumentPosition.textDocument.uri).getText();  
    let position = textDocumentPosition.position;       

   // use parsed model to lookup via position
    // return a list of auto complete suggestions (for = assignment)
    return results;  

In my case it should use the compiled model with scope information to present the varsAvailable array as the return value. How would I best do this I wonder? I need to somehow retain the position of each assignment, then perhaps have an index that links to each assignment object.

{
        "type": "SCOPE",
        "statements": [
          {
            "type": "ASSIGNMENT",
            "variableName": "d",
            "valueAssigned": "4",
            "position": 17,
            "varsAvailable": [
              "b",
              "d"
            ]
          }
        ]
      }

const assignmentIndex = {
  3: {varsAvailable: ['a'] },
  9: {varsAvailable: ['a', 'b'] },
  17: {varsAvailable: ['b', 'c'] },
}

Alternatively have a range for each, then find overlapping range?

Then for a given document char position, say 18, it should look up the closes assignment index less than that (or in range), in this case 17. Perhaps there is a better yay to do this with chevrotain?

I guess a primitive approach would be to simply iterate through this list until it finds first one with position greater than current pos (or at end of list) then use the one before that.

[kristianmandrup]

To add position info, I need to add this to the parser. Looks a little cumbersome IMO.

  assignment(ctx: any) {
    console.log("assignment = ", ctx);
    const variableName = ctx.Identifier[0].image;
    const {
      startOffset,
      endOffset,
      startLine,
      endLine,
      startColumn,
      endColumn
    } = ctx;
    const position = {
      startOffset,
      endOffset,
      startLine,
      endLine,
      startColumn,
      endColumn
    };
    // value assigned
    const valueAssigned = this.visit(ctx.reference);
    return {
      type: "ASSIGNMENT",
      variableName,
      valueAssigned,
      position
    };

Would have been more elegant to have

position: {
  offset: {
    start: 13
    end: 14
  },
  line: {
    start: 1,
    end: 1
  },
  column: {
    start: 7,
    end: 14
  }
}

Perhaps retain both position state models (nested and list), even if duplicate, so they are easy to work with either way (grouped or individually)?

[bd82]

position: {
  offset: {
    start: 13
    end: 14
  },
  line: {
    start: 1,
    end: 1
  },
  column: {
    start: 7,
    end: 14
  }
}

This is prettier, but also creates 4 times as many objects which would be slower and consume more memory. (By how much I do not know though...)

[kristianmandrup]

I tried extracting positioning node decoration into a separate visitor baseclass

export class PositionVisitor extends BaseVisitor {
  positioned = false;

  constructor(opts) {
    super();
    this.positioned = opts.positioned;
  }

  get isPositioned() {
    return Boolean(this.positioned);
  }

  decorate(node, ctx) {
    return this.decoratePosition(node, ctx);
  }

  decoratePosition(node, ctx) {
    if (this.isPositioned) {
      const {
        startOffset,
        endOffset,
        startLine,
        endLine,
        startColumn,
        endColumn
      } = ctx;
      const position = {
        startOffset,
        endOffset,
        startLine,
        endLine,
        startColumn,
        endColumn
      };
      node.position = position;
    }
    return node;
  }
}

So that I can simply decorate in assignment

  assignment(ctx: any) {
    // console.log("assignment = ", ctx);
    const variableName = ctx.Identifier[0].image;

    // value assigned
    const valueAssigned = this.visit(ctx.reference);
    const node = {
      type: "ASSIGNMENT",
      variableName,
      valueAssigned
    };
    return this.decorate(node, ctx);
  }

However it complains:

Errors Detected in CST Visitor <AstVisitor>:
        Redundant visitor method: <decorate> on AstVisitor CST Visitor
        There is no Grammar Rule corresponding to this method's name.
        For utility methods on visitor classes use methods names that do not match /^[a-zA-Z_]\w*$/.
    
        Redundant visitor method: <decoratePosition> on AstVisitor CST Visitor
        There is no Grammar Rule corresponding to this method's name.
        For utility methods on visitor classes use methods names that do not match /^[a-zA-Z_]\w*$/.

How do I circumvent this? using _ or $ prefix perhaps to indicate these are not to be treated as visitor handler methods?

Ah yes, that do not match /^[a-zA-Z_]\w*$/. so either would do :)

[bd82]

Then for a given document char position, say 18, it should look up the closes assignment index less than that (or in range), in this case 17. Perhaps there is a better yay to do this with chevrotain?

Are you trying to link the compiled Model (AST+Symbol Table) with the syntactic offset position in the text?

[bd82]

How do I circumvent this? using _ or $ prefix perhaps to indicate these are not to be treated as visitor handler methods?

Disable (delete) the Visitor validation...
I think the CST Visitor needs some upgrades/refactoring...

• Re-think CSTVisitor APIs #1039

[kristianmandrup]

Are you trying to link the compiled Model (AST+Symbol Table) with the syntactic offset position in the text?

Yes, exactly. I almost have it working already. Will likely finish it tmrw or at least by this week. See latest commit ;)

I kept the validation for now. Didn't realise the validate call in the visitor constructor did this method validation.

I think #1039 sounds like a good idea. Many other (better and more flexible) ways to go about it.

[bd82]

What I did was provide an AST in the content assist "scenario identification" CST Visitor. And manually traverse the AST while visiting the CST.

• Pass AST a top
• Manually traverse AST SubNodes as we go deeper in the CST

[kristianmandrup]

Now have a working content assist solution with nested scopes, displaying list of variables in scope as completion items for assignment :)

See chevrotain-nested-scope-lang-content-assist repo and Readme

Renamed my mini sql project chevrotain-mini-sql-lang

[kristianmandrup]

Regarding content assist, I found the following quick start

[bd82]

Now have a working content assist solution with nested scopes, displaying list of variables in scope as completion items for assignment :)

Great I found this repository much easier to understand as it uses has a small scope/grammar and fewer files.

I provided some feedback here:

• Feedback kristianmandrup/chevrotain-nested-scope-lang-content-assist#1

There still seem to be code parts that are unrelated, e.g:

• JSON grammar:
  • https://github.com/kristianmandrup/chevrotain-nested-scope-lang-content-assist/tree/master/src/error-recovery
• Large syntax file:
  https://github.com/kristianmandrup/chevrotain-nested-scope-lang-content-assist/blob/master/syntaxes/sqf.json

But I guess these are leftovers from the source it was copied from or place holders
for future features... In more productive code this would be a bit of source code history pollution, but in demo/example code I guess it does not matter as consumers are normally only interested in the latest master branch.

[kristianmandrup]

Exactly! The unrelated parts were for reference (source code co-location) until I add the full set of features to the library. WIP ;)

I’ll have a look at your comments shortly. Thanks again :)

[bd82]

While this is not a guide, it may still be of interest: https://github.com/langium/langium

[bd82]

Not sure there would ever be time to make a full language server guide. I will try to preserve this by moving it to a discussion

[kristianmandrup]

Would be great to add the link to https://langium.org/ on the main Chevrotain site documentation as an option to add LSP support

[bd82]

@kristianmandrup

Langium is linked on the "Where Used" section in the main readme.
However, its not quite an "add-on" to Chevrotain but a different project with different APIs and usage manner.
So I am not sure how to to reference it...

[msujew]

I agree Langium is difficult to place in this, given that Chevrotain only powers the parsing process. Developers are barely exposed (if at all) to Chevrotain.

It might make sense to add a "non-goals" section somewhere. It's quite some effort to build a language server on top of purely Chevrotain, so linking interested folks to Langium in that case could be a good idea. I'm biased though, given my affiliation with Langium ;)