remove scanner, rewrite grammar.js #16

justinmk · 2022-09-16T19:49:59Z

benefits:
- introduce url and block (group of lines)
- fixes most open issues
- no C scanner
- faster
- doesn't hang on filetype.txt , usr_24.txt
todo:
- keycodes more syntax features: keycodes, special, parameters #1
- [range] more syntax features: keycodes, special, parameters #1
- highlights
  - debug w/ vim.treesitter.get_captures_at_cursor()
- ~~code blocks~~
- ~~fixup remaining test failures~~

fix #7
fix #9
fix #10
fix #11
fix #14
fix #12 (non-nested)
fix #13 (non-nested)
fix #17

justinmk · 2022-09-17T13:40:19Z

corpus/headline.txt


 ================================================================================
-Headline with tag at the start
+headline with tag at the start


use lowercase descriptions so that tree-sitter test --filter headline is easy to use.

justinmk · 2022-09-18T23:22:44Z

Incompatible language version 14. Expected minimum 13, maximum 13

@vigoux any idea how I can fix this? Is CI installing an older version of tree-sitter CLI?

justinmk · 2022-09-19T00:47:46Z

code_block is kinda working but still needs work. Other than that, all tests are passing (plus several new tests).

vigoux · 2022-09-19T04:40:04Z

It is pretty surprising... I'll look into this problem today.

vigoux · 2022-09-19T05:27:59Z

I pushed a fix for that in latest master.

To ensure consistency of the parser version, I recommend you to:

Run npm install
Run make

This should handle all the version management for us now.

clason · 2022-09-19T06:39:05Z

Yes, looks like CI installs an old parser (0.20.0)?

As ABI 14 is now (0.20.7) the default, it would be better to update the CI.

vigoux · 2022-09-19T07:05:23Z

The CI uses the locked version of tree-sitter.

In latest master the locked version now supports ABI 14, and furthermore, the default ABI for this parser should now be 14.

clason · 2022-09-19T07:13:41Z

Any interest in including queries in this repo, too, so CI can catch breaking changes that require query updates on PRs?

(Does this PR require query updates?)

vigoux · 2022-09-19T07:24:01Z

Would be good too.
I'll have to look up the names of the captures for markup languages.

clason · 2022-09-19T07:27:40Z

You can start with https://github.com/nvim-treesitter/nvim-treesitter/blob/master/queries/help/highlights.scm
(sans the quote conceal).

We don't need to follow the strict corset of nvim-treesitter, but it would help; the allowed text captures are here: https://github.com/nvim-treesitter/nvim-treesitter/blob/master/CONTRIBUTING.md#text

justinmk · 2022-09-19T10:53:56Z

(Does this PR require query updates?)

yeah, mostly the introduction of:

block (which contains line).
url
keycode (TODO)
special (TODO)

I will probably alias (line_noeol) => (line) and (block_end) => (block) since there isn't much reason to expose those.

For "inner content" I have named the field either "name" or "text", to add some consistency.

If we're open to renaming things, I suggest:

hotlink => taglink
option => optionlink
headline with "===" => h1
headline with "---" => h2
UPPERCASE WORDS => h3
column_heading unchanged, it is special and not really a "heading"
backtick => codespan
code_block => codeblock

README.md

grammar.js

justinmk · 2022-09-19T13:43:10Z

code_block is working. Only 2 failing tests remaining for "option". Then will add keycode and keychord, then this is ready I think.

clason · 2022-09-19T14:44:10Z

If we're open to renaming things, I suggest:

We are. If you add the new queries here, I'll just plonk them into nvim-treesitter when bumping this parser. (Makes my life much easier.)

(The parser is currently marked as "experimental", so I have no qualms about breaking things. Could think about removing that label, though, if you deem things stable enough after this PR.)

vigoux · 2022-09-20T09:16:16Z

Without changing the "spec" for help files, I think that this cannot be handled in the parser.

The parser user would have to guess the filetype of the codeblock by looking at what it contains otherwise.

justinmk · 2022-09-20T11:56:22Z

any chance the new code_block can specify the language (either Lua or Vimscript

Isn't that something that can be added later? Could also extend the vimdoc spec as @vigoux mentioned, e.g. vimscript> and lua>.

vigoux · 2022-09-20T13:53:46Z

By the way, I am okay with renaming things. I think taking a fresh start is great.

vigoux · 2022-09-23T08:17:10Z

For the record, this seems to fail parsing some options.

justinmk · 2022-09-23T09:30:00Z

For the record, this seems to fail parsing some options.

Yes, I need to fix some stuff

justinmk · 2022-09-24T14:14:52Z

options are fixed (locally, WIP), now 2 other issues to fix :D getting close...

justinmk · 2022-09-24T16:22:30Z

Ok all tests are passing and I think the results are pretty good. I will note the compromises as PR comments.

I didn't rename anything yet, nor add highlights. Should that wait for a followup PR?

justinmk · 2022-09-24T16:23:33Z

corpus/arguments.txt

+    (line
+      (argument
+        (word)
+        (ERROR))


I gave up on this for now, ideally it should be:

(argument (word) (MISSING "}"))

corpus/backtick.txt

justinmk · 2022-09-24T16:25:42Z

corpus/backtick.txt

 ================================================================================

-Hello `world`, I am a markup language
+Hello `world`, I am `markup language`. But `this is


The grammar supports same-line whitespace in a codespan, but I'm not sure if that's actually supported by vimdoc format.

clason · 2022-09-25T08:40:24Z

I didn't rename anything yet, nor add highlights. Should that wait for a followup PR?

Best to rip the bandaid off in one go.

And, yes, please add queries; it's good practice to keep them in sync, otherwise I'll have to manually fix the ones we bundle or it will block all updates in nvim-treesitter. (Can be improved later; the main point is that they don't error out with the new parser.)

justinmk · 2022-09-26T11:24:36Z

Plan

~~fix column_heading to not parse special forms like links, tags, etc. It only supports plaintext.~~
- skipping this for now, consumers can just check if the parent is column_heading and then treat its children as plaintext.
✅ rename atoms
✅ add queries/highlights
future PR: try to eliminate block_end and line_noeol (just for elegance)?
future PR: introduce (keycode), (special) for more syntax features: keycodes, special, parameters #1

Problem: Hand-written C scanner is hard to maintain, slow, and hangs on files like `filetype.txt` and `usr_24.txt`. Solution: Delete hand-written C scanner, define grammar fully in `grammar.js` - introduce `url` - introduce `block`, a group of lines. (does not support nesting yet) - introduce `line_li` for listitems. (does not support nesting yet) - keycodes #1 - `[range]` #1 fix #1 fix #7 fix #9 fix #10 fix #11 fix #14 fix #12 (except nested) fix #13 (except nested)

neovim/tree-sitter-vimdoc#16

justinmk · 2022-09-26T13:55:14Z

Latest push includes renamed forms. See updated comment #16 (comment) for summary.

neovim/tree-sitter-vimdoc#16

vigoux · 2022-09-27T13:19:33Z

You should have commit access now, feel free to merge this PR and take the parser over from here

fixes for url, column_name, etc.

neovim/tree-sitter-vimdoc#16

- adapt to parser changes from neovim/tree-sitter-vimdoc#16 - numerous other generator improvements

justinmk mentioned this pull request Sep 16, 2022

single-char 'x' is not an option hotlink #14

Closed

justinmk commented Sep 17, 2022

View reviewed changes

justinmk marked this pull request as ready for review September 19, 2022 00:44

justinmk changed the title ~~WIP: remove scanner, rewrite parser in grammar.js~~ remove scanner, rewrite grammar.js Sep 19, 2022

vigoux reviewed Sep 19, 2022

View reviewed changes

README.md Outdated Show resolved Hide resolved

grammar.js Outdated Show resolved Hide resolved

This comment was marked as off-topic.

Sign in to view

clason mentioned this pull request Sep 20, 2022

embedded languages in codeblocks #2

Closed

justinmk commented Sep 24, 2022

View reviewed changes

corpus/backtick.txt Outdated Show resolved Hide resolved

justinmk commented Sep 24, 2022

View reviewed changes

justinmk marked this pull request as draft September 24, 2022 18:03

justinmk marked this pull request as ready for review September 24, 2022 18:04

justinmk added a commit to justinmk/neovim that referenced this pull request Sep 26, 2022

feat(gen_help_html.lua): adapt to new parser

c90cc7f

neovim/tree-sitter-vimdoc#16

justinmk mentioned this pull request Sep 26, 2022

feat(gen_help_html.lua): adapt to new parser neovim/neovim#20352

Merged

This was referenced Sep 26, 2022

Hot-links are not recognized inside parenthesis #18

Closed

Inline code highlighting has issues #17

Closed

justinmk added a commit to justinmk/neovim that referenced this pull request Sep 26, 2022

feat(gen_help_html.lua): adapt to new parser

dd10ebd

neovim/tree-sitter-vimdoc#16

justinmk added a commit to justinmk/neovim that referenced this pull request Sep 27, 2022

feat(gen_help_html.lua): adapt to new parser

6223d1d

neovim/tree-sitter-vimdoc#16

justinmk added 2 commits September 28, 2022 13:17

rename forms, introduce (h2), move (column_name) into (line)

2b81f6c

fixes for url, column_name, etc.

commit generated files

0f85e4d

justinmk merged commit 2ba61cf into neovim:master Sep 28, 2022

justinmk deleted the remove_scanner branch September 28, 2022 11:31

This was referenced Sep 28, 2022

nested blocks #20

Open

nested list items #21

Open

justinmk added a commit to justinmk/neovim that referenced this pull request Sep 28, 2022

feat(gen_help_html.lua): adapt to new parser

516f8ce

neovim/tree-sitter-vimdoc#16

justinmk added a commit to justinmk/neovim that referenced this pull request Sep 28, 2022

feat(gen_help_html.lua): adapt to new parser

83f7ed4

neovim/tree-sitter-vimdoc#16

justinmk added a commit to justinmk/neovim that referenced this pull request Sep 28, 2022

feat(gen_help_html.lua): adapt to new parser

5ebb282

- adapt to parser changes from neovim/tree-sitter-vimdoc#16 - numerous other generator improvements

justinmk added a commit to justinmk/neovim that referenced this pull request Sep 28, 2022

feat(gen_help_html.lua): adapt to new parser

408ce9e

- adapt to parser changes from neovim/tree-sitter-vimdoc#16 - numerous other generator improvements

justinmk added a commit to justinmk/neovim that referenced this pull request Sep 28, 2022

feat(gen_help_html.lua): adapt to new parser

16336c4

- adapt to parser changes from neovim/tree-sitter-vimdoc#16 - numerous other generator improvements

lvimuser pushed a commit to lvimuser/neovim that referenced this pull request Oct 6, 2022

feat(gen_help_html.lua): adapt to new parser

8a294f3

- adapt to parser changes from neovim/tree-sitter-vimdoc#16 - numerous other generator improvements

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

remove scanner, rewrite grammar.js #16

remove scanner, rewrite grammar.js #16

justinmk commented Sep 16, 2022 •

edited

Loading

justinmk Sep 17, 2022

justinmk commented Sep 18, 2022

justinmk commented Sep 19, 2022

vigoux commented Sep 19, 2022

vigoux commented Sep 19, 2022

clason commented Sep 19, 2022 •

edited

Loading

vigoux commented Sep 19, 2022

clason commented Sep 19, 2022

vigoux commented Sep 19, 2022

clason commented Sep 19, 2022 •

edited

Loading

justinmk commented Sep 19, 2022 •

edited

Loading

justinmk commented Sep 19, 2022

clason commented Sep 19, 2022 •

edited

Loading

This comment was marked as off-topic.

vigoux commented Sep 20, 2022

justinmk commented Sep 20, 2022

This comment was marked as off-topic.

vigoux commented Sep 20, 2022

This comment was marked as off-topic.

vigoux commented Sep 23, 2022

justinmk commented Sep 23, 2022

justinmk commented Sep 24, 2022 •

edited

Loading

justinmk commented Sep 24, 2022 •

edited

Loading

justinmk Sep 24, 2022

justinmk Sep 24, 2022

clason commented Sep 25, 2022

justinmk commented Sep 26, 2022 •

edited

Loading

justinmk commented Sep 26, 2022

vigoux commented Sep 27, 2022

remove scanner, rewrite grammar.js #16

remove scanner, rewrite grammar.js #16

Conversation

justinmk commented Sep 16, 2022 • edited Loading

justinmk Sep 17, 2022

Choose a reason for hiding this comment

justinmk commented Sep 18, 2022

justinmk commented Sep 19, 2022

vigoux commented Sep 19, 2022

vigoux commented Sep 19, 2022

clason commented Sep 19, 2022 • edited Loading

vigoux commented Sep 19, 2022

clason commented Sep 19, 2022

vigoux commented Sep 19, 2022

clason commented Sep 19, 2022 • edited Loading

justinmk commented Sep 19, 2022 • edited Loading

justinmk commented Sep 19, 2022

clason commented Sep 19, 2022 • edited Loading

This comment was marked as off-topic.

vigoux commented Sep 20, 2022

justinmk commented Sep 20, 2022

This comment was marked as off-topic.

vigoux commented Sep 20, 2022

This comment was marked as off-topic.

vigoux commented Sep 23, 2022

justinmk commented Sep 23, 2022

justinmk commented Sep 24, 2022 • edited Loading

justinmk commented Sep 24, 2022 • edited Loading

justinmk Sep 24, 2022

Choose a reason for hiding this comment

justinmk Sep 24, 2022

Choose a reason for hiding this comment

clason commented Sep 25, 2022

justinmk commented Sep 26, 2022 • edited Loading

Plan

justinmk commented Sep 26, 2022

vigoux commented Sep 27, 2022

justinmk commented Sep 16, 2022 •

edited

Loading

clason commented Sep 19, 2022 •

edited

Loading

clason commented Sep 19, 2022 •

edited

Loading

justinmk commented Sep 19, 2022 •

edited

Loading

clason commented Sep 19, 2022 •

edited

Loading

justinmk commented Sep 24, 2022 •

edited

Loading

justinmk commented Sep 24, 2022 •

edited

Loading

justinmk commented Sep 26, 2022 •

edited

Loading