Tree-Sitter based parser. #261

apozharski · 2024-08-13T12:04:10Z

This PR implements an alternative to the pygments/textmate grammar parsers using tree-sitter and tree-sitter-matlab.

joeced · 2024-08-13T20:24:27Z

Thanks for the contribution. I can review it better if the CI tests are running.
Please add the missing packages in setup.py to allow the tests to run.

joeced · 2024-08-14T13:22:15Z

It looks like you are missing a commit from master branch:

8b5e683
This fixes the numerous error regarding "path".

joeced · 2024-08-16T11:13:29Z

Good work so far! Impressed about the progress. Do you need any help getting tests passing?

apozharski · 2024-08-16T11:30:30Z

I have some more time this afternoon to hunt down the remaining few issues which shouldn't be that bad now that I have re-learned how to get pytest to give me the information I want. I will let you know then if there are any tests that are particularly thorny, though from looking at the remaining failures it looks like primarily small issues with the interface provided by Mat*Parser is a little different from what the documenters expect. (And a few bugs in the code that identifies whether a comment should be attached to a property or not).

apozharski · 2024-08-19T13:34:23Z

Alright it took longer than expected but it seems like all the tests are passing now. There will have to be some changes due to some modifications that the tree-sitter-matlab developer requested to the features/bugfixes we needed (acristoffers/tree-sitter-matlab#15, acristoffers/tree-sitter-matlab#16, acristoffers/tree-sitter-matlab#17, acristoffers/tree-sitter-matlab#18) but they should be very minor.

sphinxcontrib/mat_tree_sitter_parser.py

joeced · 2024-08-26T11:38:52Z

tests/test_matlabify.py

@@ -138,12 +138,19 @@ def test_classes(mod):
    assert isinstance(cls, doc.MatClass)
    assert cls.getter("__name__") == "ClassInheritHandle"
    assert cls.getter("__module__") == "test_data"
-    assert cls.bases == ["handle", "my.super.Class"]
+    assert cls.bases == [("handle",), ("my", "super", "Class")]


Could this be changed back to the original. A list of fully-qualified names?

Yea this can absolutely be changed back. I don't think this broke anything and I was thinking it may be useful for later processing but now with fresh eyes I am unconvinced.

joeced · 2024-08-26T11:40:00Z

tests/test_matlabify.py

-    assert abc.bases == ["ClassInheritHandle", "ClassExample"]
-    assert abc.attrs == {"Abstract": True, "Sealed": True}
+    assert abc.bases == [("ClassInheritHandle",), ("ClassExample",)]
+    assert abc.attrs == {"Abstract": None, "Sealed": None}


Why is the value of the class attribute not parsed?

I believe in this case it was parsed correctly but the matlab code did not assign true and simply used Abstract on its own. I have not yet integrated the existing code for attribute defaults, but I will for consistency.

joeced · 2024-08-26T11:41:36Z

sphinxcontrib/mat_tree_sitter_parser.py

+import re
+
+# rpath = "../../../syscop/software/nosnoc/+nosnoc/Options.m"
+rpath = "/home/anton/tools/matlabdomain/tests/test_data/ClassWithTrailingCommentAfterBases.m"


Hard-coded paths are not great after release :)

:) yep got rid of this. Was using it for quick and dirty testing.

…phinx-contrib#253) * Actually not only enums, quick and dirty start to using textmate parser * a little more boilerplate * property validator parsing * working function parsing without docstrings yet * start enum work * some enum parsing * working parsing for enumeration comments * add handling for block comments to enums * backport enum docstring parsing to properties * remove vestigial file * minor fixes + black

…prs sphinx-contrib#86, sphinx-contrib#88, and sphinx-contrib#90

* CI: Testing for latest Sphinx (8.0) * CI: Fix helper class version checking.

…ation issues remain

apozharski · 2024-09-17T06:55:13Z

Hello @Remi-Gau (also @joeced if you have a chance)! I'd love to have another pair of eyes go over this PR. I am pretty happy with the feature set (I realized this inadvertently addresses #265, see tests involving f_with_input_argument_block.m) and don't particularly want to bloat the diff any more than it already is, but I am open to feedback. Hopefully this would be the last round and we can focus on addressing the open issues.

joeced · 2024-09-17T08:53:24Z

sphinxcontrib/matlab.py

+        table of contents, and can also be used within the
+        :py:meth:`_toc_entry_name` method.
+
+        This method must not be used outwith table of contents generation.


Typo: outwith -> with out

joeced · 2024-09-17T08:57:29Z

Hello @Remi-Gau (also @joeced if you have a chance)! I'd love to have another pair of eyes go over this PR. I am pretty happy with the feature set (I realized this inadvertently addresses #265, see tests involving f_with_input_argument_block.m) and don't particularly want to bloat the diff any more than it already is, but I am open to feedback. Hopefully this would be the last round and we can focus on addressing the open issues.

I looks really nice! I like that tree-sitter looks like it's giving way more structure than pygments. You have my approval :)

apozharski force-pushed the tree-sitter-dev branch from 68f317b to 5912a8f Compare August 14, 2024 11:31

apozharski force-pushed the tree-sitter-dev branch from 0a155d3 to e1ecdfd Compare August 19, 2024 13:22

joeced reviewed Aug 26, 2024

View reviewed changes

sphinxcontrib/mat_tree_sitter_parser.py Show resolved Hide resolved

joeced reviewed Aug 26, 2024

View reviewed changes

sphinxcontrib/mat_tree_sitter_parser.py Show resolved Hide resolved

sphinxcontrib/mat_tree_sitter_parser.py Show resolved Hide resolved

joeced reviewed Aug 26, 2024

View reviewed changes

joeced mentioned this pull request Sep 2, 2024

Autodocument arguments blocks #265

Open

apozharski mentioned this pull request Sep 5, 2024

New maintainer! #266

Open

apozharski force-pushed the tree-sitter-dev branch 2 times, most recently from c929b54 to 6d9405f Compare September 5, 2024 13:19

apozharski added 14 commits September 5, 2024 16:47

Hack for object hierarchy

b7bc006

Initial hack to get enumerations to work

4e60a76

better classdef parsing including changes to MATLAB-language-grammar …

e5384da

…prs sphinx-contrib#86, sphinx-contrib#88, and sphinx-contrib#90

parse function docstring

fee6d03

extract function parser

e75155f

initial work

597ab64

finish integrating mat_textmate_parser with mat_types

54899fb

[skip-ci] some minor changes

3c8c18e

initial work on a tree sitter based parser

bf1e6a2

nearly finished with tree-sitter implementation

3090fa5

everything but events working

b0d603e

working events

f12b0a4

exit early if query returns for block with no elements

522e690

apozharski and others added 15 commits September 5, 2024 16:50

a better attributes query

c55b5d1

Ci: Test on Sphinx 8 / Dev. (sphinx-contrib#259)

46c0e41

* CI: Testing for latest Sphinx (8.0) * CI: Fix helper class version checking.

Fixing nearly all tests in test_parse_mfile

cf88ba4

fix a _lot_ of autodoc

e12307b

temporarily point to tree-sitter-matlab branch on apozharski fork

f1ea4f5

fix old property syntax + update tests

64b2ce2

fixing test_autodoc and test_matlabify, only comment and line continu…

a892a14

…ation issues remain

fixing the last of the tests

a2eae1b

remove dead code

3189e28

address PR comments made by @joeced

86b0fa6

fix

d3b8ace

lint

ae5c357

fix typo in __all__

952af04

remove Pygments dependencies

d4e64f2

py-tree-sitter v0.23.0 has a breaking change for Query.match()

7da1e65

apozharski force-pushed the tree-sitter-dev branch from ceb56d3 to 7da1e65 Compare September 5, 2024 14:53

apozharski changed the base branch from dev-textmate-grammar-for-parsing to master September 5, 2024 14:53

apozharski added 6 commits September 5, 2024 16:56

remove duplicat entry in yaml

a48c236

fix spec printing

1dfa6c7

minor fixes for arg block parsing and a test

d182b9c

fix bug regarding output block argument parsing and add test

aa078e1

remove print and fix test_matlabify

4b3fb89

remove textmate from dev-reqs

6314661

apozharski mentioned this pull request Sep 9, 2024

[WIP] Further improvements to textmate based parser #258

Closed

joeced reviewed Sep 17, 2024

View reviewed changes

fix typo

63781e3

apozharski merged commit 4cd5c51 into sphinx-contrib:master Sep 18, 2024
35 checks passed

apozharski mentioned this pull request Oct 7, 2024

Use textmate grammar instead of pygments #244

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Tree-Sitter based parser. #261

Tree-Sitter based parser. #261

apozharski commented Aug 13, 2024

joeced commented Aug 13, 2024

joeced commented Aug 14, 2024

joeced commented Aug 16, 2024

apozharski commented Aug 16, 2024 •

edited

Loading

apozharski commented Aug 19, 2024 •

edited

Loading

joeced Aug 26, 2024

apozharski Sep 5, 2024

joeced Aug 26, 2024

apozharski Sep 5, 2024

joeced Aug 26, 2024

apozharski Sep 5, 2024

apozharski commented Sep 17, 2024

joeced Sep 17, 2024

joeced commented Sep 17, 2024

Tree-Sitter based parser. #261

Tree-Sitter based parser. #261

Conversation

apozharski commented Aug 13, 2024

joeced commented Aug 13, 2024

joeced commented Aug 14, 2024

joeced commented Aug 16, 2024

apozharski commented Aug 16, 2024 • edited Loading

apozharski commented Aug 19, 2024 • edited Loading

joeced Aug 26, 2024

Choose a reason for hiding this comment

apozharski Sep 5, 2024

Choose a reason for hiding this comment

joeced Aug 26, 2024

Choose a reason for hiding this comment

apozharski Sep 5, 2024

Choose a reason for hiding this comment

joeced Aug 26, 2024

Choose a reason for hiding this comment

apozharski Sep 5, 2024

Choose a reason for hiding this comment

apozharski commented Sep 17, 2024

joeced Sep 17, 2024

Choose a reason for hiding this comment

joeced commented Sep 17, 2024

apozharski commented Aug 16, 2024 •

edited

Loading

apozharski commented Aug 19, 2024 •

edited

Loading