Question: Access/Match Last Consumed Character(s) of Input

[sanssecours]

Issue Description

I wrote a very basic parser for a small subset of YAML using PEGTL. While I was able to construct rules for most of the YAML specification without too much problems, there is a particular grammar rule that my code currently does not handle. The spec defines the rule ns-plain-char(c) as:

ns-plain-char(c) ::= (ns-plain-safe(c) - ":" - "#")               | 
                     (/* An ns-char preceding */ "#")             | 
                     (":" /* Followed by an ns-plain-safe(c) */ )

. The problem here is the comment “An ns-char preceding”. This part of the rule specifies that the last consumed character should match the rule ns-char.

One option to work around this issue would be to create a custom rule that checks the last character in the input. However, as far as I can tell, the input class only provides methods to check the text the parser did not consume yet (peek_char, peek_byte). Do you have an idea how I can solve this problem without too much hassle?

[ColinH]

Scanning backwards is not properly supported because the usual (?) way with PEGs is to make use of the infinite look-ahead and disambiguate in advance. Or you rework the grammar in other ways, by deferring decisions to as late as possible.

Now you would presumable prefer to use the YAML grammar as is, without major refactoring.

The memory_input now also remembers where it started parsing, so you could actually check whether there are bytes between begin() and current(), and implement a kind of backward-scanning UTF-8 decoding function.

So it's a hassle, but it's possible, and I couldn't say whether there is a more elegant solution without thinking about it some more.

[sanssecours]

Thank you for the detailed answer. I used the second approach you described in my project. As far as I can tell it seems to work.

[ColinH]

How are your bridging the gap between the input's current() method that usually returns a const char* and knows nothing about UTF-8, and the 21/32-bit code units you are comparing with?

[sanssecours]

I guess I do not do that 😄. I just assumed current() returns a pointer to the last character in UTF-32 form. At least that would be what it would return in my perfect version of this world.

Is there an easy method to handle the encoding?

[sanssecours]

I changed the rule ns_char_preceding in my latest commit. Now the rule should work correctly, if the input is encoded in UTF-8.

[ColinH]

There's no easy way (that we supply), glad that you managed to make it work (even if the hard way).