Conflicts in optional production with leading optional

[ericprud]

Cc: @RubenVerborgh

This grammar yields conflicts on p2_Opt and Flag_Star:

Foo:
       p1 p2_Opt EOF	{ return { p1: $1, p2: $2 } } ;

p1:
      'P1' ;

p2_Opt:
      	
    | p2 ;

p2:
      Flag_Star 'P2'	-> $1.concat(' ', $2)
    ;

Flag_Star:
      	-> '0th flag'
    | Flag_Star 'Flag'	-> $1.concat(' ', $2)
    ;

If I change Foo to be

Foo:
      p1 EOF	{ return { p1: $1, p2: 'default p2' } }
    | p1 p2 EOF	{ return { p1: $1, p2: $2 } } ;

the error messages go away and the parser works, but that means my parser requires a lot of idiosyncratic hand maintenance. I suspect that the calculation of the state is examining only the current production when it should recursively examine the referenced productions. Note that this grammar is valid in yacc:

Foo:
      p1 p2_Opt	;

p2_Opt:
      	
    | p2	;

p1:
    TOK_P1	;

p2:
    Flag_Star TOK_P2	;

Flag_Star:
    	
    | Flag_Star TOK_Flag	;

with the obvious lex:

([\t\n\r ])+
"P1"		{ return TOK_P1; }
"P2"		{ return TOK_P2; }
"Flag"		{ return TOK_Flag; }
<<EOF>>		{ yyterminate();}