Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Remap operator tokens to K"Identifier" during parsing #474

Open
c42f opened this issue Jul 26, 2024 · 2 comments
Open

Remap operator tokens to K"Identifier" during parsing #474

c42f opened this issue Jul 26, 2024 · 2 comments

Comments

@c42f
Copy link
Member

c42f commented Jul 26, 2024

Having various operator token kinds is relevant during parsing, but they should potentially all be remapped to K"Identifier" afterward because their special status as operators is no longer meaningful once they live in the parse tree - they are essentially just identifiers which will be looked up with the usual scoping rules, just like any other identifier.

Code which processes syntax trees often has to do things like is_operator(x) || kind(x) == K"Identifier". Mapping them to identifiers simplifies things and fixes this issue. For the same reasons, I've also wished for this in JuliaLowering.jl.

One caveat is that this change might be unhelpful for the JuliaSyntax.tokenize() API - need to think about what to do there. (Perhaps operator kinds should be preserved in that case?)

If we do this, we should also do it systematically. For example, to fix https://github.com/JuliaLang/JuliaSyntax.jl/pull/456/files#r1692517031

@fredrikekre
Copy link
Member

Code which processes syntax trees often has to do things like is_operator(x) || kind(x) == K"Identifier".

Ha, good timing for this issue. I commited fredrikekre/Runic.jl@9043278 just a day ago :)

@c42f
Copy link
Member Author

c42f commented Jul 28, 2024

@Keno notes in https://github.com/JuliaLang/JuliaSyntax.jl/pull/456/files#r1693352829

It also seems odd that e.g. isa gets turned into Identifiers, but + does not.

Yes this seems inconsistent.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants