Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Keyword list rework #2743

Merged
merged 8 commits into from
Nov 21, 2023
Merged

Keyword list rework #2743

merged 8 commits into from
Nov 21, 2023

Conversation

P-E-P
Copy link
Member

@P-E-P P-E-P commented Nov 15, 2023 •
edited
Loading

Some keyword are missing, the list lack edition separation and some suffix do not fit every context very well.

  • Add missing keyword
  • Change keyword suffixes
  • Add some edition mechanism
  • Replace raw keyword call with they constexpr counterpart

Fixes #2735

@P-E-P P-E-P added this to the GCC 14.1 release milestone Nov 15, 2023
@P-E-P P-E-P self-assigned this Nov 15, 2023
@P-E-P P-E-P force-pushed the keyword_rework branch 2 times, most recently from b1a014b to e6b1b1b Compare November 16, 2023 14:52
@P-E-P P-E-P requested a review from CohenArthur November 16, 2023 15:26
@P-E-P P-E-P marked this pull request as ready for review November 16, 2023 15:26
CohenArthur
CohenArthur approved these changes Nov 21, 2023
View reviewed changes
Copy link
Member

@CohenArthur CohenArthur left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

that looks like an improvement, thanks :) I guess later down the line we can think about adding more specific behavior to 2018 vs 2015 keywords, and other editions

gcc/rust/lex/rust-token.h
RS_TOKEN (LAST_TOKEN, "<last-token-marker>")

// Contains all token types. Crappy implementation via x-macros.
enum TokenId
{
#define RS_TOKEN(name, _) name,
#define RS_TOKEN_KEYWORD(x, y) RS_TOKEN (x, y)
#define RS_TOKEN_KEYWORD_2015(x, y) RS_TOKEN (x, y)
#define RS_TOKEN_KEYWORD_2018 RS_TOKEN_KEYWORD_2015
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

:D

P-E-P added 8 commits November 21, 2023 12:43
@P-E-P
Replace TOK suffix with KW
76d2260 
TOK suffix was chosen to disambiguate some identifiers with c++ reserved
keyword. Even though this list lies within the rust-token header, this
macro is used in many context sometimes unrelated with the lexer and
tokens. This TOK suffix may appear surprising in such context.

gcc/rust/ChangeLog:

	* lex/rust-token.h (enum PrimitiveCoreType): Change keyword suffix from
	tok to kw.
	* ast/rust-ast-collector.cc (TokenCollector::visit): Update suffix to
	match the new declaration.
	* lex/rust-lex.cc (Lexer::parse_raw_identifier): Likewise.
	* parse/rust-parse-impl.h (can_tok_start_type): Likewise.
	(Parser::parse_item): Likewise.
	(Parser::parse_vis_item): Likewise.
	(Parser::parse_extern_crate): Likewise.
	(Parser::parse_function): Likewise.
	(Parser::parse_function_qualifiers): Likewise.
	(Parser::parse_struct): Likewise.
	(Parser::parse_enum): Likewise.
	(Parser::parse_static_item): Likewise.
	(Parser::parse_trait_item): Likewise.
	(Parser::parse_inherent_impl_item): Likewise.
	(Parser::parse_trait_impl_item): Likewise.
	(Parser::parse_extern_block): Likewise.
	(Parser::parse_external_item): Likewise.
	(Parser::parse_stmt): Likewise.
	(Parser::parse_return_expr): Likewise.
	(Parser::parse_match_expr): Likewise.
	(Parser::parse_type): Likewise.
	(Parser::parse_for_prefixed_type): Likewise.
	(Parser::parse_type_no_bounds): Likewise.
	(Parser::parse_stmt_or_expr): Likewise.
	* parse/rust-parse.cc (peculiar_fragment_match_compatible): Likewie.
	* util/rust-token-converter.cc (convert): Likewise.

Signed-off-by: Pierre-Emmanuel Patry <[email protected]>
@P-E-P
Add edition separation for keywords
38789c8 
It might be required in the future to get only the keywords from a
specific edition. To do so we need a mean to differentiate keywords based
on their edition. This commit changes the existing keyword macro to
allow such behavior.

gcc/rust/ChangeLog:

	* lex/rust-token.h (enum PrimitiveCoreType): Change enum macro calls.
	(RS_TOKEN_KEYWORD): Remove generic token keyword macro.
	(RS_TOKEN_KEYWORD_2015): Introduce keywords for edition 2015.
	(RS_TOKEN_KEYWORD_2018): Likewise with edition 2018.
	* lex/rust-token.cc (RS_TOKEN_KEYWORD): Remove old macro definition.
	(RS_TOKEN_KEYWORD_2015): Replace with 2015 definition...
	(RS_TOKEN_KEYWORD_2018): ... and 2018 definition.
	* util/rust-keyword-values.cc (RS_TOKEN_KEYWORD):  Likewise.
	(RS_TOKEN_KEYWORD_2015): Likewise.
	(RS_TOKEN_KEYWORD_2018): Likewise.
	* util/rust-keyword-values.h (RS_TOKEN_KEYWORD): Likewise.
	(RS_TOKEN_KEYWORD_2015): Likewise.
	(RS_TOKEN_KEYWORD_2018): Likewise.

Signed-off-by: Pierre-Emmanuel Patry <[email protected]>
@P-E-P
Treat underscore as a keyword
c19bfca 
Make the underscore token a 2015 keyword.

gcc/rust/ChangeLog:

	* lex/rust-token.h (enum PrimitiveCoreType): Change macro for
	underscore in token list.

Signed-off-by: Pierre-Emmanuel Patry <[email protected]>
@P-E-P
Add await keyword
2fc3625 
The 2018 edition await keyword was missing from the keyword list.

gcc/rust/ChangeLog:

	* lex/rust-token.h (enum PrimitiveCoreType): Add await keyword
	definition.

Signed-off-by: Pierre-Emmanuel Patry <[email protected]>
@P-E-P
Replace some keyword raw values
28ee769 
Raw values cannot be understood easily by most tools. This commit replace
some raw values with their variable counterpart.

gcc/rust/ChangeLog:

	* ast/rust-ast-collector.cc (TokenCollector::visit): Replace raw value
	with keyword call.
	* ast/rust-ast.h: Likewise.
	* parse/rust-parse-impl.h (Parser::parse_path_ident_segment): Likewise.
	(Parser::parse_macro_match_fragment): Likewise.
	(Parser::parse_extern_crate): Likewise.
	(Parser::parse_use_tree): Likewise.
	(Parser::parse_const_item): Likewise.
	(Parser::parse_literal_expr): Likewise.
	(Parser::parse_maybe_named_param): Likewise.
	(Parser::parse_pattern_no_alt): Likewise.
	(Parser::left_denotation): Likewise.
	(Parser::parse_path_in_expression_pratt): Likewise.

Signed-off-by: Pierre-Emmanuel Patry <[email protected]>
@P-E-P
Add a list of weak keyword
e102067 
Retrieving a weak keyword value is done using raw values. Introducing a
list of weak keywords means this could change.

gcc/rust/ChangeLog:

	* util/rust-keyword-values.h (class WeakKeywords): Add new class with
	weak keyword constexpr.

Signed-off-by: Pierre-Emmanuel Patry <[email protected]>
@P-E-P
Replace some weak keyword raw value with constexpr
aba1b08 
Raw values may have typos or contain error, replacing those will
improve the codebase.

gcc/rust/ChangeLog:

	* ast/rust-ast-collector.cc (TokenCollector::visit): Replace raw value.
	* parse/rust-parse-impl.h (Parser::is_macro_rules_def): Likewise.
	(Parser::parse_item): Likewise.
	(Parser::parse_vis_item): Likewise.
	(Parser::parse_macro_rules_def): Likewise.
	(Parser::parse_union): Likewise.
	(Parser::parse_trait_impl_item): Likewise.
	(Parser::parse_stmt): Likewise.
	(Parser::parse_stmt_or_expr): Likewise.

Signed-off-by: Pierre-Emmanuel Patry <[email protected]>
@P-E-P
Introduce a proper keyword list
92745b8 
The old "keyword" list was used for the lexer, and could therefore not
be used with keyword spanning over multiple tokens as those tokens should
remain lexed as is. Hence the introduction of a new list macro for
keyword exclusive tasks. This also means we can no longer match a token
id for each keyword. The token id map has been renamed to keep it's
properties.

gcc/rust/ChangeLog:

	* lex/rust-lex.cc (Lexer::classify_keyword): Update keyword map name.
	* lex/rust-token.h (enum PrimitiveCoreType): Remove some deprecated
	comments.
	* util/rust-keyword-values.cc (get_keywords): Update the keyword map
	name.
	(RS_TOKEN): Define as empty
	(RS_TOKEN_KEYWORD_2015): Add the emission value.
	(RS_TOKEN_KEYWORD_2018): Likewise.
	* util/rust-keyword-values.h (RS_KEYWORD_LIST): Introduce the keyword
	list.
	(RS_TOKEN_KEYWORD_2018): Define multiple new keywords.

Signed-off-by: Pierre-Emmanuel Patry <[email protected]>
@P-E-P P-E-P enabled auto-merge November 21, 2023 12:19
@P-E-P P-E-P added this pull request to the merge queue Nov 21, 2023
Merged via the queue into Rust-GCC:master with commit 34479e5 Nov 21, 2023
9 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@CohenArthur CohenArthur CohenArthur approved these changes

Assignees

@P-E-P P-E-P

Labels
enhancement
Projects
Project Pineapple
Status: Done
Milestone
GCC 14.1 release
Development

Successfully merging this pull request may close these issues.

Add missing reserved keyword
2 participants
@P-E-P @CohenArthur