feat: implement new PEG parser #154
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
The existing parser, based in the pest crate, has been good-enough so far, but lacks some features that makes it unsuitable for some tasks, like implementing a language server for YARA that can be integrated in text editors like VSCode. For implementing a language server we need a parser that is fault-tolerant, recovering from syntax errors as much as possible and parsing the code that comes after the problematic code. As an additional benefit the CLI tool can report multiple syntax errors at once, instead of a single one as it currently does. The new parser should be faster too.
Formatting error messages every time `unexpected_token_error` was called was a major bottleneck. Many of these error messages are not actually used at all, now the formatting of the error message is deferred to the moment that this error is actually needed.
Also rename `if_found` to `if_next`.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
The current parser, based on the pest crate, has been adequate as a starting point but lacks several features that make it unsuitable for specific tasks, such as implementing a language server for YARA that can be integrated into text editors like VSCode.
To implement a language server, we need an error-tolerant parser capable of recovering from syntax errors and continuing to parse the subsequent code. Additionally, this enhancement allows the CLI tool to report multiple syntax errors simultaneously, instead of just one as it currently does.
The new parser is a hand-written PEG parser.