Looking to build a typeahead like search with chevrotain, how to handle async backend calls in visitor?

[nair1abhishek]

I'm looking to build a search feature using chevrotain for my product and I'm was able setup the parser and visitor for the grammar we use in our search box. We use the same grammar for many products that then, will individual create there own visitors to transform the search grammar to there own DSL.
Next step for this is that we want to show typeahead results that load as the user is typing the query in the search box.

The basic idea I have now is that I use the recoveryMode of the parser, so that as the user is type I still reach the visitor, where I then reach the context to decide which fetch should be called based on the context stack. This fetch is outside the visitor and is invoked by a event creation.

Is this a good solution, or is there a implementation of this that I can reference where this scenario is already solved?

[bd82]

If I understand correctly you are trying to implement content assist / auto-complete during user input.

One option you can check is Langium which builds on-top of Chevrotain to implement language servers.

If you prefer to build the logic yourself

I am not sure I fully understood your description, So I will instead reference a use-case where I worked on content assist.
Perhaps you can expand on this for your use-case.

• This VSCode Extensions provides content assist for SAP UI5 XML views.
  • https://marketplace.visualstudio.com/items?itemName=SAPOSS.vscode-ui5-language-assistant (view the GIF).

Main logic parts:

1. Parsing the full document to CST

   • with slight modifications to error recovery to better recover in XML text.
   • Output may be a recovered CST, the input does not have to be 100% valid...
   • Source: https://github.com/SAP/xml-tools/blob/master/packages/parser/lib/parser.js

2. Logic to understand the syntactic context from a text position for a given CST.

   • e.g.: are we in elementContent or elementName or attributeName or attributeValue ?
   • Source: https://github.com/SAP/xml-tools/blob/master/packages/content-assist/lib/content-assist.js

3. Logic to apply the externally provided semantic content assist "providers".

   • e.g.: in SAP UI5 XML view there are different semantics than a Maven pom.xml file.
   • In this case these are synchronized not async.
   • Source: https://github.com/SAP/xml-tools/blob/master/packages/content-assist/lib/api.js

Sync vs Async semantic providers

I don't think this is a big issue, the parsing and figuring the syntactic parts are synchronized.
But this works on the whole document (in your case user search field input).
Once you know the type of suggestions which can be offered (syntactic context).
You can perform the semantic suggestions computation in an async manner.

e.g.: lets imagine the grammar in your search box is similar to SQL:

1. The user inputs: select * from p⇝ ( ⇝ marks the content assist position)
2. The parser you implemented should return a CST with enough data for the next step (using error recovery)
3. The syntactic content assist logic you implement on the CST structure should "figure out" the syntactic content assist context is TABLE NAME with prefix p
4. This syntactic context can be passed in a async manner to the semantic logic which would then filter by the
   list of tables the user is allowed to access which also start with p e.g:
   • persons | prices
   • but not prices_upcoming_2024 which only L1 managers are allowed to access.

Basically don't try to introduce async flows inside Chevrotain APIs (parsing/visitor).
Instead perform the async parts after the relevant context has been extracted from the sync parts.

[nair1abhishek]

Thanks for the detailed explanation. I'm planning to do something similar where I have a visitor just for content assist, so that it can provide enough hints to my system (including the prefix like in you example), such that it can make the appropriate server call to populate the hints. And I'm using the error recovery feature so that the parser doesn't error out and sends the context to visitor.

Thanks again.