Skip to content

Commit

Permalink
Tokenizer/PHP: efficiency fix
Browse files Browse the repository at this point in the history
Reminder: the PHP::processAdditional()` method walks _back_ from the end of the file to the beginning.

The type handling retokenization layer is triggered for each `&`, `|` and `)` the tokenizer encounters.

When something is recognized as a valid type declaration, the relevant tokens will all be retokenized in one go the first time the type handling layer is triggered, which means that - as the type tokens will have been retokenized already -, the type handling layer will not be triggered again for any of the other type related tokens in the type.

However, if the type is *not* recognized as a valid type, the type handling layer will keep getting retriggered and will (correctly) keep concluding this is not a valid type.

The change in this PR, prevents the type handling layer from doing any work when it is retriggered on a token which was previously already seen and concluded to be, either not part of a type or part of an invalid type.

This should make the tokenizer marginally faster for complex types containing an error, like `(A&B|(C&D)|(E&F)`.
  • Loading branch information
jrfnl committed May 21, 2024
1 parent 07477a7 commit 87c3936
Showing 1 changed file with 12 additions and 1 deletion.
13 changes: 12 additions & 1 deletion src/Tokenizers/PHP.php
Original file line number Diff line number Diff line change
Expand Up @@ -2602,7 +2602,9 @@ protected function processAdditional()

$this->createAttributesNestingMap();

$numTokens = count($this->tokens);
$numTokens = count($this->tokens);
$lastSeenTypeToken = $numTokens;

for ($i = ($numTokens - 1); $i >= 0; $i--) {
// Check for any unset scope conditions due to alternate IF/ENDIF syntax.
if (isset($this->tokens[$i]['scope_opener']) === true
Expand Down Expand Up @@ -3038,6 +3040,12 @@ protected function processAdditional()
|| $this->tokens[$i]['code'] === T_BITWISE_AND
|| $this->tokens[$i]['code'] === T_CLOSE_PARENTHESIS
) {
if ($lastSeenTypeToken < $i) {
// We've already examined this code to check if it is a type declaration and concluded it wasn't.
// No need to do it again.
continue;
}

/*
Convert "|" to T_TYPE_UNION or leave as T_BITWISE_OR.
Convert "&" to T_TYPE_INTERSECTION or leave as T_BITWISE_AND.
Expand Down Expand Up @@ -3250,6 +3258,9 @@ protected function processAdditional()
break;
}//end for

// Remember the last token we examined as part of the (non-)"type declaration".
$lastSeenTypeToken = $x;

if ($confirmed === false
&& $suspectedType === 'property or parameter'
&& isset($this->tokens[$i]['nested_parenthesis']) === true
Expand Down

0 comments on commit 87c3936

Please # to comment.