Reducing parsing time by introducing hash signature mechanism

[stefanos82]

When we execute a Nelua program for the first time, it goes through parsing, preprocessing, analyzing code, generating native code and building it, and executing it.

$ nelua -TV helloworld.nelua 
using config file '/home/stefanos/.config/nelua/neluacfg.lua'
parsed helloworld.nelua (1.5 ms)
skip preprocess helloworld.nelua (0.0 ms)
analyzed (0.1 ms)
generated nelua_cache/helloworld.c
ccache gcc "nelua_cache/helloworld.c" -o "nelua_cache/helloworld" -Wall -fwrapv -g -Wextra -lm
/home/stefanos/code/nelua/overview/nelua_cache/helloworld 
Hello world!

If we re-execute the same command, it goes through the same steps, but uses the cached result(s):

$ nelua -TV helloworld.nelua 
using config file '/home/stefanos/.config/nelua/neluacfg.lua'
parsed helloworld.nelua (1.5 ms)
skip preprocess helloworld.nelua (0.0 ms)
analyzed (0.1 ms)
using cached generated nelua_cache/helloworld.c
using cached binary nelua_cache/helloworld
/home/stefanos/code/nelua/overview/nelua_cache/helloworld 
Hello world!

Ignore the fact that the tested code is just print 'Hello world!' and assume it's a massive project.

Would not it make sense to have a hash signature introduced for each file compilation to compare each build with the previous one, so we could skip unnecessary recompilation cycles?

I already do that for C code with the help of ccache, but imagine doing that for Nelua's actual parsing and preprocessing procedure, how much time it could reduced from going over the same cycle every time you have modified a file, let alone a number of files at once.

[edubart]

Would not it make sense to have a hash signature introduced for each file compilation to compare each build with the previous one, so we could skip unnecessary recompilation cycles?

This is not simple doable and there are many drawbacks in doing that:

1. First the compiler works in a way that even if a file is completely analyzed, the compiler (or the user) may modify the file AST lately while processing other unrelated file, that is, a file is never completely preprocessed/analyzed, this is a design decision to make preprocessing powerful.
2. Even if it's possible in someway to cache partial analyzation, it would require to save it to a file, thus it would require serializing and deserializing the analyzed AST, this would incur a good amount of time, saving and loading a huge AST very fast is not a easy task, I suspect it would cost more time than actually just analyzing.
3. The complexity that such system would cause in the compiler would be huge, and slow down the language development, plus create more burden to maintain, it would also limit the amount of hack-ability in the compiler, all that goes against the goals.

The only possible way to do that without much changes would be doing a hash of all files, that is, the system would not be per file as you suggested, but for a hash of all files. But I don't see much benefit in that, moreover you can achieve the same result using a plain Makefile to check file modification time and skip re-building.

I already do that for C code with the help of ccache, but imagine doing that for Nelua's actual parsing and preprocessing procedure, how much time it could reduced from going over the same cycle every time you have modified a file, let alone a number of files at once.

C design allows that, Nelua design is different, the processing step is way different. Plus in C you can split source code into multiple source files, and compile multiple source files independently, and ccache makes more sense when doing that. This is not doable with Nelua, because the language design requires all files to be preprocessed and analyzed together. And changing these design choices would change the language itself a lot, we would have to kill the powerful preprocessor and compiler hack-ability, and the language would become something else.

Forkmon solution

I've put thoughts into this problem a few times before, and ways I could speed up the compilation to not do redundant parsing and analysis of files while doing live development, because I'm addicted to live development and I don't like slow build times. I've come up with a generic solution that I made as a side project called forkmon.

Check the project lading page, I explain with more details there how this works, I do mention how it works with Nelua in the motivation section. This system could speedup compilation of a large Nelua project by 3x factor in usual live development usage on my tests.

Quick tutorial using Forkmon with Nelua

Quick setup of Forkmon for quick live development in Nelua (Linux users only):

wget -q https://raw.githubusercontent.com/edubart/forkmon/main/bin/linux64/forkmon.so
alias forkmon-nelua='LD_PRELOAD=`pwd`/forkmon.so FORKMON_FILTER=".nelua$" nelua'
echo "print 'hello world'" > hello.nelua
forkmon-nelua hello.nelua

Now split the screen with the terminal and hello.nelua file, then edit hello.nelua and see your code results faster (no need to hit keystrokes to run or switch windows, plus wait less to compile).
Then start using forkmon-nelua alias instead of nelua command for live development.

[stefanos82]

This is not simple doable and there are many drawbacks in doing that:

1. First the compiler works in a way that even if a file is completely analyzed, the compiler (or the user) may modify the file AST lately while processing other unrelated file, that is, a file is never completely preprocessed/analyzed, this is a design decision to make preprocessing powerful.

2. Even if it's possible in someway to cache partial analyzation, it would require to save it to a file, thus it would require serializing and deserializing the analyzed AST, this would incur a good amount of time, saving and loading a huge AST very fast is not a easy task, I suspect it would cost more time than actually just analyzing.

3. The complexity that such system would cause in the compiler would be huge, and slow down the language development, plus create more burden to maintain, it would also limit the amount of hack-ability in the compiler, all that goes against the goals.

I complete forgot about the possibility to edit AST during preprocessing... 🤦

Forkmon solution

I've put thoughts into this problem a few times before, and ways I could speed up the compilation to not do redundant parsing and analysis of files while doing live development, because I'm addicted to live development and I don't like slow build times. I've come up with a generic solution that I made as a side project called forkmon.

Check the project lading page, I explain with more details there how this works, I do mention how it works with Nelua in the motivation section. This system could speedup compilation of a large Nelua project by 3x factor in usual live development usage on my tests.

I have seen this project of yours and clearly you were inspired by nodemon.

Quick tutorial using Forkmon with Nelua

Quick setup of Forkmon for quick live development in Nelua (Linux users only):

wget -q https://raw.githubusercontent.com/edubart/forkmon/main/bin/linux64/forkmon.so
alias forkmon-nelua='LD_PRELOAD=`pwd`/forkmon.so FORKMON_FILTER=".nelua$" nelua'
echo "print 'hello world'" > hello.nelua
forkmon-nelua hello.nelua

Now split the screen with the terminal and hello.nelua file, then edit hello.nelua and see your code results faster (no need to hit keystrokes to run or switch windows, plus wait less to compile).
Then start using forkmon-nelua alias instead of nelua command for live development.

Do you think it could get merged as part of Nelua and be added as a flag, like nelua --start-watch, nelua --stop-watch to work as background service?

It could be handy to have such flags set in my .config/nelua/neluacfg.lua; total automation!

[stefanos82]

By the way, you need to update the Makefile to include the prebuilt output, in case was generated by the user.

[stefanos82]

This is an interesting output, at least to me:

Do you think if all preprocessing was taking place as one step and then all parsing as one step as well, could benefit Nelua?

It reminded me how C2 language is doing its code analysis in contrast to C: