Tokenizer/PHP: fix tokens missing/file content being removed by attribute retokenization #1280
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
In light of an upcoming change, let's also test the token code/content of the first token _after_ the attribute sequence to safeguard that no token duplication is being introduced via the array_slicing and dicing done in the PHP < 8.0 "comment to attribute" retokenization.
…bute retokenization This commit fixes the issue detailed in ticket 1279, where in case of an unfinished (parse error) attribute, the contents on the same line as the attribute opener would be removed from the token stream when PHPCS is run on PHP < 8.0. On PHP < 8.0, attributes tokenize as comments and the Tokenizer reparses the contents of the comment to make it a usable token stream. As things were, if the attribute closer could not be found, the `PHP::parsePhpAttribute()` method would return `null` instead of returning the array of (reparsed) tokens which were on the same line as the attribute opener, which meant the "comment content"/attribute content was effectively removed from the token stream. The fix includes changing the return type of the (private) `PHP::parsePhpAttribute()` method from `array|null` to `array`, which also allows for simplifying some code handling the return value of this method. Includes additional tests and improvements to the pre-existing parse error test. Fixes 1279
This was referenced Sep 22, 2025
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
Tokenizer/AttributesTest: move parse error test to own file
Tokenizer/AttributesTest: expand test coverage
In light of an upcoming change, let's also test the token code/content of the first token after the attribute sequence to safeguard that no token duplication is being introduced via the array_slicing and dicing done in the PHP < 8.0 "comment to attribute" retokenization.
Tokenizer/PHP: fix tokens missing/file content being removed by attribute retokenization
This commit fixes the issue detailed in ticket #1279, where in case of an unfinished (parse error) attribute, the contents on the same line as the attribute opener would be removed from the token stream when PHPCS is run on PHP < 8.0.
On PHP < 8.0, attributes tokenize as comments and the Tokenizer reparses the contents of the comment to make it a usable token stream.
As things were, if the attribute closer could not be found, the
PHP::parsePhpAttribute()method would returnnullinstead of returning the array of (reparsed) tokens which were on the same line as the attribute opener, which meant the "comment content"/attribute content was effectively removed from the token stream.The fix includes changing the return type of the (private)
PHP::parsePhpAttribute()method fromarray|nulltoarray, which also allows for simplifying some code handling the return value of this method.Includes additional tests and improvements to the pre-existing parse error test.
Suggested changelog entry
Fixed: Tokenizer/PHP: on PHP < 8.0, an unclosed attribute (parse error) could end up removing some tokens from the token stream.
Related issues/external references
Fixes #1279