Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Investigate accepting a stream of tokens as input #202

Open
tzlaine opened this issue Oct 24, 2024 · 3 comments
Open

Investigate accepting a stream of tokens as input #202

tzlaine opened this issue Oct 24, 2024 · 3 comments

Comments

@tzlaine
Copy link
Collaborator

tzlaine commented Oct 24, 2024

Not sure exactly how this should work -- should it take some abstract notion of a token, a lexrtl token, something I create, a CTRE-style thing, or what?

But it definitely deserves at least some investigation. Some parsers are very hard to write (or even ambiguous) without lexing first.
tzlaine added a commit that referenced this issue Nov 5, 2024
@tzlaine
Initial sketch of a lexer for Boost.Parser, based on using CTRE as an…
eaa53d5 
… external

dependency.  Lots of TODOs, somewhat thin testing, but the basics work.

See #202.
tzlaine added a commit that referenced this issue Nov 5, 2024
@tzlaine
Add a range adaptor for token_view; token_view -> tokens_view. Break
f5f77d8 
detail::make_input_subrange() out into its own header for reuse.

See #202.
tzlaine added a commit that referenced this issue Nov 5, 2024
@tzlaine
Add a range adaptor for token_view; token_view -> tokens_view. Break
4ac5244 
detail::make_input_subrange() out into its own header for reuse.

See #202.
tzlaine added a commit that referenced this issue Nov 7, 2024
@tzlaine
Make tokens_view regular by removing tokens_view::tokens_, which creates
f08f668 
internal references.

See #202.
tzlaine added a commit that referenced this issue Nov 7, 2024
@tzlaine
Add a token cache to tokens_view; grab a bunch of tokens at a time when
e09f527 
advancing past the end of the cache.  Filter out whitespace tokens entirely.
Make tokens_view noncopyable+nonmovable.

See #202.
tzlaine added a commit that referenced this issue Nov 7, 2024
@tzlaine
Add support for an externally-supplied token cache.
08b8214 
See #202.
tzlaine added a commit that referenced this issue Nov 7, 2024
@tzlaine
Change all the existing lex runs to compare their output against an e…
10aa943 
…xpected

sequence of tokens.

See #202.
tzlaine added a commit that referenced this issue Nov 8, 2024
@tzlaine
Specify the true/false and number token_specs in the adobe lexer test…
8854328 
… code to

use a bool and a double as its value, repsectively, instead of a string_view.

See #202.
tzlaine added a commit that referenced this issue Nov 8, 2024
@tzlaine
Add much longer lexing test; fix errors.
52d9cb5 
See #202.
tzlaine added a commit that referenced this issue Nov 8, 2024
@tzlaine
Switch from double to long double as token's floating point represent…
c1cc177 
…ation.

Therer's room for it, since it's in a union with a string_view anyway.

See #202.
tzlaine added a commit that referenced this issue Nov 8, 2024
@tzlaine
Use bitfields to reduce the size of token by a few bytes.
9e8da80 
See #202.
tzlaine added a commit that referenced this issue Nov 8, 2024
@tzlaine
Factor out a lot of the lexer.cpp tests into a separate file.
4ca0766 
See #202.
tzlaine added a commit that referenced this issue Nov 8, 2024
@tzlaine
Extend the lexer tests, addressing some of the testing TODOs; fix err…
6fdab1d 
…ors.

See #202.
tzlaine added a commit that referenced this issue Nov 8, 2024
@tzlaine
Extend the lexer tests, addressing some of the testing TODOs; fix err…
6a9a4b1 
…ors.

See #202.
tzlaine added a commit that referenced this issue Nov 8, 2024
@tzlaine
Address the remaining non-documentation TODOs in the lexer header and…
c1746bc 
… tests.

See #202.
tzlaine added a commit that referenced this issue Nov 9, 2024
@tzlaine
Initial, partial sketch of token_parser.
9380c75 
See #202.
tzlaine added a commit that referenced this issue Nov 9, 2024
@tzlaine
Initial, partial sketch of token_parser.
0c81549 
See #202.
tzlaine added a commit that referenced this issue Nov 9, 2024
@tzlaine
Initial, partial sketch of token_parser.
aac4dac 
See #202.
tzlaine added a commit that referenced this issue Nov 9, 2024
@tzlaine
Sketch in support for matching tokens against expected values in toke…
bd9c7a8 
…n_parser.

See #202.
tzlaine added a commit that referenced this issue Nov 9, 2024
@tzlaine
Sketch in operator() on token_spec_t.
ba2d757 
See #202.
tzlaine added a commit that referenced this issue Nov 9, 2024
@tzlaine
Completely rethink the relationship among token_parser, token_spec_t,…
c624681 
… and

parser_interface.  token_spec is now a variable template that generates a
parser_interface wrapping a token_parser, which parameterized on the
token_spec_t.  This way, a single token_spec use can be used to specify how to
lex, and how to parse.

See #202.
tzlaine added a commit that referenced this issue Nov 9, 2024
@tzlaine
Add token_parser::operator()(range) to support matchign against token…
93e0aa2 
…s that

producestring_views.

See #202.
tzlaine added a commit that referenced this issue Nov 9, 2024
@tzlaine
Add token_parser::operator()(range) to support matchign against token…
43b6a2f 
…s that

producestring_views.

See #202.
tzlaine added a commit that referenced this issue Nov 10, 2024
@tzlaine
Put detail::make_input_subrange back into parer.hpp.
364d8ae 
See #202.
tzlaine added a commit that referenced this issue Nov 10, 2024
@tzlaine
Reorganize the lexer-related header inclusion scheme slightly. lexer.…
b093b52 
…hpp is

now required to come before parser.hpp.

See #202.
tzlaine added a commit that referenced this issue Nov 10, 2024
@tzlaine
Sketch in the printing implemetnation for token_parser.
dd5ffe7 
See #202.
tzlaine added a commit that referenced this issue Nov 10, 2024
@tzlaine
Alter detail::make_input_subrange() to return its input unchanged whe…
0935656 
…n the

range's value type is a specializtion of token.

See #202.
tzlaine added a commit that referenced this issue Nov 10, 2024
@tzlaine
Change token<> to include the position of the start of the token in the
c6331b2 
underlying sequence, and change the way that the error handler is invoked, so
that it detects token iterators, and passes iterators into the underlying
range to the error handler, instead of the token iterators.

See #202.
tzlaine added a commit that referenced this issue Nov 10, 2024
@tzlaine
Add a std::string data member, and an override of what() to parser_er…
ff22ab1 
…ror, to

ensure that the messge is preserved exactly when translating exceptions in
failed lexer parsing.

See #202.
tzlaine added a commit that referenced this issue Nov 10, 2024
@tzlaine
Fix multiple thinkos in tokens_view.
a55603c 
See #202.
tzlaine added a commit that referenced this issue Nov 29, 2024
@tzlaine
Switch from double to long double as token's floating point represent…
6ed1152 
…ation.

Therer's room for it, since it's in a union with a string_view anyway.

See #202.
tzlaine added a commit that referenced this issue Nov 29, 2024
@tzlaine
Use bitfields to reduce the size of token by a few bytes.
ad64fb6 
See #202.
tzlaine added a commit that referenced this issue Nov 29, 2024
@tzlaine
Factor out a lot of the lexer.cpp tests into a separate file.
391bb2b 
See #202.
tzlaine added a commit that referenced this issue Nov 29, 2024
@tzlaine
Extend the lexer tests, addressing some of the testing TODOs; fix err…
ee8ab13 
…ors.

See #202.
tzlaine added a commit that referenced this issue Nov 29, 2024
@tzlaine
Address the remaining non-documentation TODOs in the lexer header and…
358adf2 
… tests.

See #202.
tzlaine added a commit that referenced this issue Nov 29, 2024
@tzlaine
Initial, partial sketch of token_parser.
f00f4df 
See #202.
tzlaine added a commit that referenced this issue Nov 29, 2024
@tzlaine
Sketch in support for matching tokens against expected values in toke…
fbc21ef 
…n_parser.

See #202.
tzlaine added a commit that referenced this issue Nov 29, 2024
@tzlaine
Sketch in operator() on token_spec_t.
1a405f8 
See #202.
tzlaine added a commit that referenced this issue Nov 29, 2024
@tzlaine
Completely rethink the relationship among token_parser, token_spec_t,…
e419ef2 
… and

parser_interface.  token_spec is now a variable template that generates a
parser_interface wrapping a token_parser, which parameterized on the
token_spec_t.  This way, a single token_spec use can be used to specify how to
lex, and how to parse.

See #202.
tzlaine added a commit that referenced this issue Nov 29, 2024
@tzlaine
Add token_parser::operator()(range) to support matchign against token…
416607c 
…s that

producestring_views.

See #202.
tzlaine added a commit that referenced this issue Nov 29, 2024
@tzlaine
Put detail::make_input_subrange back into parer.hpp.
793e519 
See #202.
tzlaine added a commit that referenced this issue Nov 29, 2024
@tzlaine
Reorganize the lexer-related header inclusion scheme slightly. lexer.…
b6c1229 
…hpp is

now required to come before parser.hpp.

See #202.
tzlaine added a commit that referenced this issue Nov 29, 2024
@tzlaine
Sketch in the printing implemetnation for token_parser.
298bae0 
See #202.
tzlaine added a commit that referenced this issue Nov 29, 2024
@tzlaine
Alter detail::make_input_subrange() to return its input unchanged whe…
8830dfe 
…n the

range's value type is a specializtion of token.

See #202.
tzlaine added a commit that referenced this issue Nov 29, 2024
@tzlaine
Change token<> to include the position of the start of the token in the
092a761 
underlying sequence, and change the way that the error handler is invoked, so
that it detects token iterators, and passes iterators into the underlying
range to the error handler, instead of the token iterators.

See #202.
tzlaine added a commit that referenced this issue Nov 29, 2024
@tzlaine
Add a std::string data member, and an override of what() to parser_er…
1421592 
…ror, to

ensure that the messge is preserved exactly when translating exceptions in
failed lexer parsing.

See #202.
tzlaine added a commit that referenced this issue Nov 29, 2024
@tzlaine
Fix multiple thinkos in tokens_view.
2b518bc 
See #202.
tzlaine added a commit that referenced this issue Nov 29, 2024
@tzlaine
Add a smoke test for using token views as input to parse(); fix errors.
67c3ec1 
See #202.
tzlaine added a commit that referenced this issue Nov 29, 2024
@tzlaine
Add cases to tracing.cpp to cover the printing of token parsers; fix …
00510ed 
…errors.

See #202.
tzlaine added a commit that referenced this issue Nov 29, 2024
@tzlaine
Remove the support for token_view::iterator -> underlying iterator tr…
f298bfe 
…anslation

in *parse_impl(); add support to the error handlers and their support
functions directly instead.  There are simply too many APIs there that need
the translation to leave it to other code.

Add lex_error exception type, and add support for all the APIs that used to
take a parse_error param to now take either a parse_error or a lex_error.

Throw lex_error from failed parsing of lexed tokens in detail::make_token().

See #202.
tzlaine added a commit that referenced this issue Nov 29, 2024
@tzlaine
Plumb a pointer to the tokens_view through to the parse context; change
9dbf302 
seq_parser to use the tokens_view pointer from the contet to trim the cache at
an expectation point.

See #202.
tzlaine added a commit that referenced this issue Nov 29, 2024
@tzlaine
Macro guard private members of tokens_view from inclusion in Doxygen …
e654c9f 
…reference

entry.

See #202.
tzlaine added a commit that referenced this issue Nov 29, 2024
@tzlaine
Preclude skipper *parse() overloads from token parsing with a static_…
178d62a 
…assert.

See #202.
tzlaine added a commit that referenced this issue Nov 29, 2024
@tzlaine
Documentation pass on the changes made so far to suupport token parsing.
9670a92 
See #202.
tzlaine added a commit that referenced this issue Nov 29, 2024
@tzlaine
Documentation pass on the changes made so far to suupport token parsing.
39b96df 
See #202.
tzlaine added a commit that referenced this issue Nov 29, 2024
@tzlaine
Documentation pass on the changes made so far to support token parsing.
8cc16ef 
See #202.
tzlaine added a commit that referenced this issue Nov 30, 2024
@tzlaine
Documentation pass on the changes made so far to support token parsing.
92c4993 
See #202.
tzlaine added a commit that referenced this issue Nov 30, 2024
@tzlaine
Fix error detected in tests of token caching introduced in doc exampl…
3176c6f 
…es. Make

sure not to copy tokens_view within the *parse() call graph.

See #202.
tzlaine added a commit that referenced this issue Nov 30, 2024
@tzlaine
Add static_assert(std::integral<T> || std::floating_point<T>) to
87e00a1 
detail::token_with_value, just in case.

See #202.
tzlaine added a commit that referenced this issue Nov 30, 2024
@tzlaine
Slight simplification of code trying to detect token iterators.
eab7e82 
See #202.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@tzlaine @jefftrull