Tooling

[linkpy]

Hello.

I'm very interested in Nelua, but sadly my issue with Lua (or any other dynamic language like it) is the lack of good tooling (mostly autocomplete and symbol browser). So I want to make some extensions for Visual Studio Code to support Nelua, like syntax highlighting (maybe with tree-sitter, as TextMate definitions are too simplistic for Nelua) and most importantly a language extension, which could work very well since Nelua has type hints.

For the first part, I think I can manage, it's just learning about tree-sitter and TextMate definitions to make Nelua syntax highlighting complete. On the second hand, a language extension often uses a LSP (Language Server Protocol) but it is possible to do it with command invocations.

I saw that nelua has a parameter --print-ast and --print-analyzed-ast. It could be possible to add new arguments to give enough information for the language extension to work. For example, --print-symbols could print all of the symbols declared in a file (with they kind, like 'variable', 'function', 'class', etc) after preprocessing (so symbols declared by macros are included).
Another example could be --autocomplete-suggestions which would receive a filepath, and a location (index or line and column, where the cursor is in the editor) and would return a list of possible known values.

Anyway, i'm waiting for your feedback, i really want to use nelua for game developpement and even help to make it more complete.

[edubart]

That would be great! I have made a simple syntax definitions extension for Visual Studio code here https://github.com/edubart/nelua-vscode . But is kinda outdated for some recent additions, I tend to maintain more the sublime-text plugin because I've been personally using it.

I saw that nelua has a parameter --print-ast and --print-analyzed-ast. It could be possible to add new arguments to give enough information for the language extension to work. For example, --print-symbols could print all of the symbols declared in a file (with they kind, like 'variable', 'function', 'class', etc) after preprocessing (so symbols declared by macros are included).
Another example could be --autocomplete-suggestions which would receive a filepath, and a location (index or line and column, where the cursor is in the editor) and would return a list of possible known values.

Good ideia! New commands to help the syntax highlighting like that can added with ease, we just would have to discuss what kind of commands are needed.

But first, looks like you just have discovered the language, I advise to get more familiar with the language and try it out more before trying to create tools. Know that the language is still under development and some planned features are missing.

[linkpy]

Yes, i saw nelua-vscode. It uses TextMate format which isn't good for a language as complex as nelua (or even lua actually), that's why using a simple TextMate format file for obvious syntax (like keywords) is OK, and then using a tree-sitter syntax highligther for more advanced stuff (comments, macro, ...).

Good ideia! New commands to help the syntax highlighting like that can added with ease, we just would have to discuss what kind of commands are needed.

Well, for once the lint command already exists, but to make things simpler maybe adding a --lint-json which returns JSON data instead so parsing it would be simple enough.

Then, i think it would be simpler to move as much stuff towards the extension. Like making a command to give all of the symbols and their scopes from a file, then the extension just cache this so it doesn't have to call back nelua if nothing has changed. The goal would be to get as much information as possible (like fields and methods from a record, variable types and function signatures, etc).

With all of that, a last command is needed : to get the autocomplete suggestion at a given position. If the extension has all of the symbols, nelua just have to return something similar to a type, returning the accepted type (string, stringview, integer, a record, ...) or a valid identifer (myinstance:method, here getting a suggestion after the ':' should return methods-of(type-of myinstance)) basically, and with that the extension can uses the info it has stored about the type of myinstance and propose the methods to the user.

That way is pretty limiting by the way. It will be pretty slow for big files (since nelua has to parse & expand macros each time something is modified), but the LSP protocol looks huge. Maybe it's worth considering making a language server then ? It can be done in another language, like Javascript, with the help of PEG.JS and maybe a JS Lua VM to run macro expensions. Or, maybe it's worth considering splitting up Nelua into "packages" (or maybe just extracting the parts needed), so the AST & Parser can be reused, but that would be annoying to do without a proper package manager (requiring using git submodules).

But first, looks like you just have discovered the language, I advise to get more familiar with the language and try it out more before trying to create tools. Know that the language is still under development and some planned features are missing.

I have a project in mind for this, don't worry 😃

[edubart]

Well, for once the lint command already exists, but to make things simpler maybe adding a --lint-json which returns JSON data instead so parsing it would be simple enough.

I don't like the ideia to return JSON in the Nelua compiler because I don't want to bundle a JSON library inside the sources, because I try to keep the project minimal, but I could print as a Lua table, however I have a better ideia that I will explain below.

Then, i think it would be simpler to move as much stuff towards the extension. Like making a command to give all of the symbols and their scopes from a file, then the extension just cache this so it doesn't have to call back nelua if nothing has changed. The goal would be to get as much information as possible (like fields and methods from a record, variable types and function signatures, etc).

I agree, we should move much stuff as possible to the extension. I have an ideia on how you could access the whole compiler information that I will explain below.

With all of that, a last command is needed : to get the autocomplete suggestion at a given position. If the extension has all of the symbols, nelua just have to return something similar to a type, returning the accepted type (string, stringview, integer, a record, ...) or a valid identifer (myinstance:method, here getting a suggestion after the ':' should return methods-of(type-of myinstance)) basically, and with that the extension can uses the info it has stored about the type of myinstance and propose the methods to the user.

A command for auto complete suggestion is not needed, the ideia I'll explain bellow give ways to do this.

That way is pretty limiting by the way. It will be pretty slow for big files (since nelua has to parse & expand macros each time something is modified),

On the slowness, I think the extension could execute the command asynchronously and use a cache meanwhile (I dunno if Visual Studio Code allows this, please confirm). Also for big projects Nelua will take time to parse a file, because if you want type information it has to parse all the dependent files.

but the LSP protocol looks huge. Maybe it's worth considering making a language server then ? It can be done in another language, like Javascript, with the help of PEG.JS and maybe a JS Lua VM to run macro expensions.

I think a LSP could work good great if you can run it in Lua code, but seems more work, nevertheless I will explain bellow how you could make one, but I suggest first to do something more simple.

Or, maybe it's worth considering splitting up Nelua into "packages" (or maybe just extracting the parts needed), so the AST & Parser can be reused, but that would be annoying to do without a proper package manager (requiring using git submodules).

I don't think that's a good ideia, Nelua cannot be split easily and the language evolves and changes fast, I think this could be only an option the day it Nelua gets out of beta.

It can be done in another language, like Javascript, with the help of PEG.JS and maybe a JS Lua VM to run macro expensions. Or, maybe it's worth considering splitting up Nelua into "packages" (or maybe just extracting the parts needed), so the AST & Parser can be reused, but that would be annoying to do without a proper package manager (requiring using git submodules).

The best way is to do everything in Lua, because in Lua you have full access to the compiler and all its information. How to run Lua it would have to use the bundled Lua 5.4 interpreter with Nelua, it comes with Nelua installation and is the fastest and error prone choice.

The Ideia

The ideia is pretty simple, Nelua is made in Lua, thus you can analyze a file using a lua script and extract any information there,
thus you could do one of these 2 options:

1. Execute a lua script which access all compile information and print back the requested results.
2. Execute a lua server and exchange the results on demand (LSP).

To start easy I suggest to do option 1, later for an improved and faster version if you feels the necessity you could experiment option 2.

If you have interest in this please confirm, then I will setup ways and show an example on how to execute lua script within Nelua and analyze the sources, printing back interesting things like variables per scope, type information, AST, etc. I will design and prepare the ideia in 1-2 days or so and demo for you.

Meanwhile know this, in Nelua itself you already full access to the AST, scope symbols, and types all in the preprocessor, for example:

local Point = @record{x: integer, y: integer}
local a: integer, b: integer = 1, 2

##[[
-- print current ast
print(context.ast)

-- print current scope symbols with type
for i,sym in ipairs(context.scope.symbols) do
  print(i, sym.name, sym.type)
end

-- print Point fields
print('Point fields:')
for i,field in ipairs(context.scope.symbols.Point.value.fields) do
  print(i, field.name, field.type)
end
]]

My ideia is basically an official way to access this in a lua script outside a .nelua file without requiring to compile, so you can do whatever you want, via a command line command like nelua --analyze-script extensionscript.lua myfile.nelua <args> and then the script would print back results for your extension. This script in the future could be a LSP server that runs indefinitely.

[linkpy]

I like the idea, it could work well. Since Nelua's lua VM is just a custom Lua, i guess it's possible to load any kind of lua modules with it, right ? (lkie native lua modules, for example luasocket).

Also, the LSP could be implemented based one https://github.com/sumneko/lua-language-server. Just need to modify it to support Nelua (like removing LPeg to use the compiler's interface, etc).