Skip to content

Split the parser implementation into multiple files #4322

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
P-E-P wants to merge 4 commits into
base: master
Choose a base branch
from
Open

Split the parser implementation into multiple files #4322

P-E-P wants to merge 4 commits into from
+14,822 −14,623

Conversation

@P-E-P
Copy link
Member

@P-E-P P-E-P commented Dec 3, 2025

That 14 kloc file isn't practical.

P-E-P added 3 commits December 3, 2025 23:45
@P-E-P
Move is_right_angle_tok to util namespace
96eca4f 
gcc/rust/ChangeLog:

	* parse/rust-parse-impl.h (is_right_angle_tok): Move function ...
	(Parser::parse_generic_params_in_angles): Change call.
	(Parser::parse_for_lifetimes): Likewise.
	(Parser::parse_path_generic_args): Likewise.
	* parse/rust-parse-utils.h (is_right_angle_tok): ... to here.

Signed-off-by: Pierre-Emmanuel Patry <[email protected]>
@P-E-P
Move function to utils namespace
5dfc6d1 
gcc/rust/ChangeLog:

	* parse/rust-parse-impl.h (can_tok_start_type): Move function from ...
	* parse/rust-parse-utils.h (can_tok_start_type): ... to here.

Signed-off-by: Pierre-Emmanuel Patry <[email protected]>
@P-E-P
Split parser implementation into several files
96739e3 
gcc/rust/ChangeLog:

	* parse/rust-parse-impl.h (Parser::parse_inner_attributes): Move to
	rust-parse-impl-XXX.hxx implementation file.
	(Parser::parse_doc_comment): Likewise. Change return type to
	AttributeBody.
	(Parser::parse_inner_attribute): Likewise.
	(Parser::parse_attribute_body): Likewise.
	(Parser::parse_identifier_or_keyword_token): Likewise.
	(Parser::parse_outer_attributes): Likewise.
	(Parser::is_macro_rules_def): Likewise.
	(Parser::parse_simple_path): Likewise.
	(Parser::parse_item): Likewise.
	(Parser::parse_simple_path_segment): Likewise.
	(Parser::parse_path_ident_segment): Likewise.
	(Parser::parse_vis_item): Likewise.
	(Parser::parse_attr_input): Likewise.
	(Parser::parse_async_item): Likewise.
	(Parser::parse_delim_token_tree): Likewise.
	(Parser::parse_macro_rules_def): Likewise.
	(Parser::parse_decl_macro_def): Likewise.
	(Parser::parse_token_tree): Likewise.
	(Parser::parse_visibility): Likewise.
	(Parser::parse_module): Likewise.
	(Parser::parse_outer_attribute): Likewise.
	(Parser::parse_extern_crate): Likewise.
	(Parser::parse_use_decl): Likewise.
	(Parser::parse_use_tree): Likewise.
	(Parser::parse_function): Likewise.
	(Parser::parse_macro_invocation): Likewise.
	(Parser::parse_macro_rule): Likewise.
	(Parser::parse_function_qualifiers): Likewise.
	(Parser::parse_macro_matcher): Likewise.
	(Parser::parse_generic_params_in_angles): Likewise.
	(Parser::parse_macro_match): Likewise.
	(Parser::parse_generic_param): Likewise.
	(Parser::parse_macro_match_fragment): Likewise.
	(Parser::parse_macro_match_repetition): Likewise.
	(Parser::parse_generic_params): Likewise.
	(Parser::parse_lifetime_params): Likewise.
	(Parser::parse_lifetime_params_objs): Likewise.
	(Parser::parse_lifetime_param): Likewise.
	(Parser::parse_type_params): Likewise.
	(Parser::parse_type_param): Likewise.
	(Parser::parse_function_params): Likewise.
	(Parser::parse_function_param): Likewise.
	(Parser::parse_function_return_type): Likewise.
	(Parser::parse_where_clause): Likewise.
	(Parser::parse_lifetime_where_clause_item): Likewise.
	(Parser::parse_for_lifetimes): Likewise.
	(Parser::parse_type_param_bounds): Likewise.
	(Parser::parse_type_param_bound): Likewise.
	(Parser::parse_trait_bound): Likewise.
	(Parser::parse_lifetime_bounds): Likewise.
	(Parser::parse_lifetime): Likewise.
	(Parser::lifetime_from_token): Likewise.
	(Parser::parse_external_type_item): Likewise.
	(Parser::parse_type_alias): Likewise.
	(Parser::parse_struct): Likewise.
	(Parser::parse_struct_fields): Likewise.
	(Parser::parse_struct_field): Likewise.
	(Parser::parse_tuple_fields): Likewise.
	(Parser::parse_tuple_field): Likewise.
	(Parser::parse_enum): Likewise.
	(Parser::parse_enum_items): Likewise.
	(Parser::parse_enum_item): Likewise.
	(Parser::parse_union): Likewise.
	(Parser::parse_const_item): Likewise.
	(Parser::parse_static_item): Likewise.
	(Parser::parse_trait): Likewise.
	(Parser::parse_trait_item): Likewise.
	(Parser::parse_trait_type): Likewise.
	(Parser::parse_trait_const): Likewise.
	(Parser::parse_impl): Likewise.
	(Parser::parse_inherent_impl_item): Likewise.
	(Parser::parse_trait_impl_item): Likewise.
	(Parser::parse_extern_block): Likewise.
	(Parser::parse_external_item): Likewise.
	(Parser::parse_stmt): Likewise.
	(Parser::parse_let_stmt): Likewise.
	(Parser::parse_generic_arg): Likewise.
	(Parser::parse_path_generic_args): Likewise.
	(Parser::parse_generic_args_binding): Likewise.
	(Parser::parse_self_param): Likewise.
	(Parser::parse_expr_stmt): Likewise.
	(Parser::parse_anon_const): Likewise.
	(Parser::parse_const_block_expr): Likewise.
	(Parser::parse_grouped_expr): Likewise.
	(Parser::parse_type_path): Likewise.
	(Parser::parse_closure_expr): Likewise.
	(Parser::parse_literal_expr): Likewise.
	(Parser::parse_box_expr): Likewise.
	(Parser::parse_return_expr): Likewise.
	(Parser::parse_try_expr): Likewise.
	(Parser::parse_break_expr): Likewise.
	(Parser::parse_continue_expr): Likewise.
	(Parser::parse_type_path_segment): Likewise.
	(Parser::parse_loop_label): Likewise.
	(Parser::parse_type_path_function): Likewise.
	(Parser::parse_if_expr): Likewise.
	(Parser::parse_path_in_expression): Likewise.
	(Parser::parse_path_expr_segment): Likewise.
	(Parser::parse_if_let_expr): Likewise.
	(Parser::parse_loop_expr): Likewise.
	(Parser::parse_qualified_path_in_type): Likewise.
	(Parser::parse_labelled_loop_expr): Likewise.
	(Parser::parse_match_expr): Likewise.
	(Parser::parse_match_arm): Likewise.
	(Parser::parse_match_arm_patterns): Likewise.
	(Parser::parse_async_block_expr): Likewise.
	(Parser::parse_array_expr): Likewise.
	(Parser::parse_closure_param): Likewise.
	(Parser::parse_type): Likewise.
	(Parser::parse_paren_prefixed_type): Likewise.
	(Parser::parse_for_prefixed_type): Likewise.
	(Parser::parse_maybe_named_param): Likewise.
	(Parser::parse_reference_type_inner): Likewise.
	(Parser::parse_reference_type): Likewise.
	(Parser::parse_raw_pointer_type): Likewise.
	(Parser::parse_slice_or_array_type): Likewise.
	(Parser::parse_type_no_bounds): Likewise.
	(Parser::parse_paren_prefixed_type_no_bounds): Likewise.
	(Parser::parse_literal_or_range_pattern): Likewise.
	(Parser::parse_range_pattern_bound): Likewise.
	(Parser::parse_pattern): Likewise.
	(Parser::parse_pattern_no_alt): Likewise.
	(Parser::parse_reference_pattern): Likewise.
	(Parser::parse_grouped_or_tuple_pattern): Likewise.
	(Parser::parse_slice_pattern): Likewise.
	(Parser::parse_identifier_pattern): Likewise.
	(Parser::parse_ident_leading_pattern): Likewise.
	(Parser::parse_struct_pattern_elems): Likewise.
	(Parser::parse_struct_pattern_field): Likewise.
	(Parser::unexpected_token): Likewise.
	(Parser::skip_after_semicolon): Likewise.
	(Parser::skip_token): Likewise.
	(Parser::maybe_skip_token): Likewise.
	(Parser::expect_token): Likewise.
	(Parser::skip_after_end): Likewise.
	(Parser::skip_after_end_block): Likewise.
	(Parser::skip_after_next_block): Likewise.
	(Parser::skip_after_end_attribute): Likewise.
	(Parser::done_end_or_else): Likewise.
	(Parser::done_end): Likewise.
	* parse/rust-parse.h: Change function return type to AttributeBody
	instead of a tuple (parse_doc_comment). Move enum binding powers from
	impl file.
	* parse/rust-parse-impl-attribute.hxx: New file.
	* parse/rust-parse-impl-macro.hxx: New file.
	* parse/rust-parse-impl-path.hxx: New file.
	* parse/rust-parse-impl-pattern.hxx: New file.
	* parse/rust-parse-impl-ttree.hxx: New file.
	* parse/rust-parse-impl-utils.hxx: New file.
	* parse/rust-parse-impl-expr.hxx: New file.

Signed-off-by: Pierre-Emmanuel Patry <[email protected]>
@P-E-P P-E-P force-pushed the parser_split_file branch from e7b8d88 to 96739e3 Compare December 4, 2025 01:24
@P-E-P P-E-P requested a review from CohenArthur December 4, 2025 01:25
@P-E-P
Change parser template implementation file ext
cc79982 
We have a few more template implementation files within the rust frontend
that use the hxx extension to make their content clear and highlight the
missing header guards.

gcc/rust/ChangeLog:

	* parse/rust-parse-impl-lexer.cc: Fix included file name.
	* parse/rust-parse-impl-macro.cc: Likewise.
	* parse/rust-parse-impl-proc-macro.cc: Likewise.
	* parse/rust-parse-impl.h: Move to...
	* parse/rust-parse-impl.hxx: ...here.

Signed-off-by: Pierre-Emmanuel Patry <[email protected]>
@P-E-P P-E-P requested review from philberty and powerboat9 December 4, 2025 12:24
@P-E-P
Copy link
Member Author

P-E-P commented Dec 4, 2025

@philberty @CohenArthur @powerboat9 I'm not asking for a deep review since this is mostly code moving around. I'd like to get your feelings around this PR. And more specifically the last two commits 96739e3 and cc79982

Do we want to split the parser ? Do we want to keep it as is ?

Do we want to rename the template implementation file to make sure it won't be included like any other header files (there is no header guard) ?

@CohenArthur
Copy link
Member

CohenArthur commented Dec 5, 2025

I don't have strong feelings one way or another. does this help with things like clangd/other LSPs to work on the parser? if yes then I'm more than happy to split it

@P-E-P
Copy link
Member Author

P-E-P commented Dec 5, 2025

I don't have strong feelings one way or another. does this help with things like clangd/other LSPs to work on the parser? if yes then I'm more than happy to split it

If I remember correctly the LSP was indeed giving up at the end of the file but I think this has been resolved with a recent clangd release.

powerboat9
powerboat9 approved these changes Dec 5, 2025
View reviewed changes
Copy link
Collaborator

@powerboat9 powerboat9 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@powerboat9 powerboat9 powerboat9 approved these changes

@CohenArthur CohenArthur Awaiting requested review from CohenArthur

@philberty philberty Awaiting requested review from philberty

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@P-E-P @CohenArthur @powerboat9