Chevrotain packages required at runtime

[dhowe]

I'm wondering what subset of the Chevrotain packages from the list below are actually required at runtime for use in a js library (esm):

./node_modules/chevrotain/lib/src/api.js
./node_modules/chevrotain/lib/src/version.js
./node_modules/chevrotain/lib/src/parse/parser/parser.js
./node_modules/chevrotain/lib/src/scan/lexer_public.js
./node_modules/chevrotain/lib/src/scan/tokens_public.js
./node_modules/chevrotain/lib/src/parse/grammar/lookahead.js
./node_modules/chevrotain/lib/src/parse/grammar/llk_lookahead.js
./node_modules/chevrotain/lib/src/parse/errors_public.js
./node_modules/chevrotain/lib/src/parse/exceptions_public.js
./node_modules/chevrotain/lib/src/scan/lexer_errors_public.js
./node_modules/@chevrotain/gast/lib/src/api.js
./node_modules/@chevrotain/cst-dts-gen/lib/src/api.js
./node_modules/chevrotain/lib/src/diagrams/render_public.js
./node_modules/@chevrotain/utils/lib/src/api.js
./node_modules/chevrotain/lib/src/parse/grammar/follow.js
./node_modules/chevrotain/lib/src/parse/grammar/gast/gast_resolver_public.js
./node_modules/chevrotain/lib/src/parse/parser/traits/recoverable.js
./node_modules/chevrotain/lib/src/parse/parser/traits/looksahead.js
./node_modules/chevrotain/lib/src/parse/parser/traits/tree_builder.js
./node_modules/chevrotain/lib/src/parse/parser/traits/lexer_adapter.js
./node_modules/chevrotain/lib/src/parse/parser/traits/recognizer_api.js
./node_modules/chevrotain/lib/src/parse/parser/traits/recognizer_engine.js
./node_modules/chevrotain/lib/src/parse/parser/traits/error_handler.js
./node_modules/chevrotain/lib/src/parse/parser/traits/context_assist.js
./node_modules/chevrotain/lib/src/parse/parser/traits/gast_recorder.js
./node_modules/chevrotain/lib/src/parse/parser/traits/perf_tracer.js
./node_modules/chevrotain/lib/src/parse/parser/utils/apply_mixins.js
./node_modules/chevrotain/lib/src/parse/grammar/checks.js
./node_modules/chevrotain/lib/src/scan/lexer.js
./node_modules/chevrotain/lib/src/scan/tokens.js
./node_modules/chevrotain/lib/src/scan/reg_exp_parser.js
./node_modules/chevrotain/lib/src/parse/grammar/interpreter.js
./node_modules/chevrotain/lib/src/parse/grammar/rest.js
./node_modules/@chevrotain/gast/lib/src/model.js
./node_modules/@chevrotain/gast/lib/src/visitor.js
./node_modules/@chevrotain/gast/lib/src/helpers.js
./node_modules/@chevrotain/cst-dts-gen/lib/src/model.js
./node_modules/@chevrotain/cst-dts-gen/lib/src/generate.js

[msujew]

I believe Chevrotain needs everything except for the @chevrotain/cst-dts-gen package and the chevrotain/lib/src/diagrams/render_public.js file.

[dhowe]

Besides manually removing these packages, is there a way to include only those needed for the runtime ?

[msujew]

We had an issue tracking bundle size improvements related to that, see #1697. But that was basically the only thing that was left in there. Since the main chevrotain ESM export also exports the diagram renderer, I don't believe there's an easy way to tree-shake the part of out any bundle.

[bd82]

What is the requirement / need / constraints ?

Chevrotain it approx 140kb bundled and minified.
I am sure a little bit more size optimization be achieved, but I don't think any more significant optimizations are possible...

ESM and tree-shaking

@msujew
Is not a major point in favor of using ESM and import / export syntax to enable tree shaking algorithms to follow the dependency graph and only bundle what is actually used the the application?

Runtime deps

Assuming ESM import/export indeed allows for better tree-shacking, you could try to bundle your grammar (esbuild/rollup/webpack) and inspect the bundle contents (e.g run it without minification) to see if the DTS generation APIs are present or not...

There may also be other APIs which could potentially only be needed at design time, for example some validations during grammar initialization could be deferred and only be invoked during automated testing.

• To speed up parser initialization.

But cutting out these parts out of the "runtime" would be much harder as their invocation condition is dynamic.

[jindong-zhannng]

Hi @bd82, may I know are there some examples that demonstrate how much tree shaking can help on reducing bundle size in context of chevrotain? I am building a dsl parser that will run in browser so the size is critical.

I'm evaluating the approximate size of the parser if built using chevrotain. And I noticed the size of @toml-tools/parser is incredibly small, I wonder if this is a usual case. Thanks.

[jindong-zhannng]

Ok, I think I got it. After I re-test it in my local machine, the actual minified bundle size of @toml-tools/parser with esbuild is 137.3 kb. Seems the result in bundlephobia is not accurate due to some bugs...

Considering @toml-tools/parser is really pretty simple, it seems like every parser built with chevrotain will always contain around 100k weight which is hard to be further optimized.

[bd82]

Hello @jindong-zhannng

To decrease the size of bundled Chevrotain farther more dynamic bundling logic would be required.
Tree shaking is not enough, as it is not that components are not imported,

Some features may mostly be used during "design time" and could be conditionally skipped at runtime (e.g. various validations) but the bundling tool has no way to identify this.

Another scenario are features which are not used in a specific grammar, but once
again this information is not represented in the files imports graph.

One last scenario would be persisting (and loading) the results of the "grammar recording" phase instead of performing it at runtime, thus removing the related code.

I'm not sure how much could be reduced, but I do not expect a huge percentage.
However, I'm fairly certain refactoring to enable implementing various "lean custom builds" pre-sets
would be too much investment for too little gain.

You may be able to experiment with this a bit by patching
Chevrotain and removing imports/invocations of design time only flows (e.g grammar validations).
and/or code related to unused features in your grammar (e.g CST building if you are instead using embedded actions).