[for reference] all work done which is not in original repo

[GerHobbelt]

Since a lot has been done and several of these features are tough to 'extract cleanly' to produce 'simple' patches (they won't be simple anyway), the list of differences (features and fixes in the derived repo):

[to be completed]

Main features

• full Unicode support (okay, astral codepoints are hairy and only partly supported) in lexer and parser

  • lexer can handle XRegExp \pXXX unicode regex atoms, e.g. \p{Alphabetic}

    • jison auto-expands and re-combines these when used inside regex set expressions in macros, e.g.

      ALPHA                                   [{UNICODE_LETTER}a-zA-Z_]
      

      will be reduced to the equivalent of

      ALPHA                                   [{UNICODE_LETTER}_]
      

      hence you don't need to worry your regexes will include duplicate characters in regex [...] set expressions.

  • parser rule names can be Unicode identifiers (you're not limited to US ASCII there).

• lexer macros can be used inside regex set expressions (in other macros and/or lexer rules); the lexer will barf a hairball (i.e. throw an informative error) when the macro cannot be expanded to represent a character set without causing counter-intuitive results), e.g. this is a legal series of lexer macros now:

  ASCII_LETTER                            [a-zA-z]
  UNICODE_LETTER                          [\p{Alphabetic}{ASCII_LETTER}]
  ALPHA                                   [{UNICODE_LETTER}_]
  DIGIT                                   [\p{Number}]
  WHITESPACE                              [\s\r\n\p{Separator}]
  ALNUM                                   [{ALPHA}{DIGIT}]
  
  NAME                                    [{ALPHA}](?:[{ALNUM}-]*{ALNUM})?
  ID                                      [{ALPHA}]{ALNUM}*
  

• the parser generator produces optimized parse kernels: any feature you do not use in your grammar (e.g. error rule driven error recovery or @elem location info tracking) is rigorously stripped from the generated parser kernel, producing the fastest possible parser engine.

• you can define a custom written lexer in the grammar definition file's %lex ... /lex section in case you find the standard lexer is too slow to your liking on otherwise insufficient. (This is done by specifying a no-rules lexer with the custom lexer placed in the lexer trailing action code block.)

• you can %include action code chunks from external files, in case you find that the action code blurbs obscure the grammar's / lexer's definition. Use this when you have complicated/extensive action code for rules or a large amount of 'trailing code' ~ code following the %% end-of-ruleset marker.

• CLI: -c 2 -- you now have the choice between two different table compression algorithms:

  • mode 2 creates the smallest tables,
  • mode 1 is the one available in 'vanilla jison' and
  • mode 0 is 'no compression what-so-ever'

Minor 'Selling Points'

• you can produce parsers which do not include a try ... catch wrapper for that last bit of speed and/or when you want to handle errors in surrounding userland code.

• all errors are thrown using a parser and lexer-specific Error-derived class which allows userland code to discern which type of error (and thus: available extra error information!) is being processed via a simple/fast instanceof check for either of them.

• the jison CLI tool will print additional error information when a grammar parse error occurred (derived off / closely related to Add detail grammar error output #321 and Stop silencing useful information. #258)

• the jison CLI tool will print parse table statistics when requested (-I commandline switch) so you can quickly see how much table space your grammar is consuming. Handy when you are optimizing your grammar to reduce the number of states per parse for performance reasons.

• includes [a derivative or close relative of] Added support for ES2015 module generation #326, Pass moduleMain option to generator #316, Support %options ranges in the grammar. #302, Fix moduleName (and other options?) being lost #290, #282,#283 - support specifying moduleName on CLI, support generate names... #284

• fixes

  • TypeError: undefined is not an object (evaluating 'this.yy.parser') #358 (crashes on this.yy.parser missing errors)
  • Fix issue where the location of the past lexer input is reported on error #356 (wrong input attached to error)
  • this.lexer is undefined #353 (crashes on this.yy.lexer missing errors)
  • Always remove the token stack label #352 (token_stack label issue: jison-gho's way of code stripping does depend on labels at all, so the issue is moot now)
  • [feature req] add bison YYRECOVERING support #349 (YYRECOVERING macro support -- should work, fingers crossed 😉)
  • Getting "this.performAction.call is not a function" when trying to parse text #348 (performAction invocation trouble)
  • definitions inconsistencies between lex and jison #333 (lexer recognizes literal regex parts without quotes whenever possible),
  • [email protected] throws non-Error #328 (all errors are Error-derived instances with a text message and extra info attached),
  • lexer.pushState does not work as expected (or intended?) #317 (?not sure?),
  • "Jison is not defined" when running generated parser in debug mode #313,
  • Support ranges in locations #301,
  • is there error recovery? #299 (with minor additional abilities compared to vanilla jison, e.g. configurable error recovery search depth),
  • Feature Request: Report unused rules to improve refactoring of large grammars #296 (unused grammar rules are reported and nuked, i.e. not included in the generated output),
  • Add parameter in the command-line tools to support specifying moduleName #282,
  • Not work for JSON format #276 (and we support JSON5 format besides!),
  • EBNF unable to get at sub-expression #254,
  • Is it possible to inspect the state stack in helper functions? #239 (all parser stacks are available in all grammar rule action code via yyvstack, yysstack, etc. -- documented in the documented grammar file's top API documenting comment chunk),
  • How to extract values from nested EBNF clauses? #233 (EBNF rewriting to BNF now works; see also the wiki),
  • Typos in documentation #231,
  • Implementing own parser.yy.parseError function throws error #218 (and parseError can now produce a return value for the parser to return to the calling userland code),
  • Any plan to support epsilon? #210,
  • Is there a way to include another file just in the middle of the current one? #175 (kind of..., we now support %include filepath statements in stead of any code chunk),
  • Changing lexer mode from parser fails #165 (kind of... now jison does not fetch look-ahead when the rule reduce action doesn't need it; it requires intimate understanding of your grammar and the way this LALR grammar engine handles it, but you can once again code 'lexer hacks' from inside parser rules' action code. Shudder or rejoice, depending on your mental make-up ;-) ),
  • need way to distinguish errors #138 (instanceof of parser and lexer error class),
  • Support %initial-action #121 (indirectly, you can now do this by writing an action code chunk for an initial 'epsilon' rule and get this behaviour that way)

Where is this thing heading?

• using recast et al to help analyze rule action code to help code-strip both parser and lexer to produce fast parse/lex runs. Currently only the parser gets analyzed (a tad roughly) to strip costly operations from the parser run-time to make it fast / efficient.
• also note migrate towards using a monorepo: ES6/rollup/etc. is a horror otherwise GerHobbelt/jison#16: moving towards a babel-style monorepo. This work has now completed (oct-nov 2017: jison-gho releases 0.6.1-200+)