Skip to content

Commit 72a66aa

Browse files
committed
PHP 8.1 | Tokenizer/PHP: bug fix for overeager explicit octal notation backfill
Follow up on 3481. Just like for all other type of integer notations, if a numeric literal separator is used, it is not allowed between the prefix and the actual number. ```php // This is fine. $b = 0b1_0; $o = 0o6_3; // This is an invalid use of the numeric literal separator. $b = 0b_10; $o = 0o_63; ``` This PR fixes the backfill for explicit octal notation to NOT backfill these type of invalid sequences as the inconsistent tokenization across PHP versions which that causes, can create havoc in sniffs. Includes adding additional unit tests.
1 parent 154a688 commit 72a66aa

File tree

5 files changed

+64
-1
lines changed

5 files changed

+64
-1
lines changed

src/Tokenizers/PHP.php

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -728,7 +728,8 @@ protected function tokenize($string)
728728
&& (isset($tokens[($stackPtr + 1)]) === true
729729
&& is_array($tokens[($stackPtr + 1)]) === true
730730
&& $tokens[($stackPtr + 1)][0] === T_STRING
731-
&& strtolower($tokens[($stackPtr + 1)][1][0]) === 'o')
731+
&& strtolower($tokens[($stackPtr + 1)][1][0]) === 'o'
732+
&& $tokens[($stackPtr + 1)][1][1] !== '_')
732733
) {
733734
$finalTokens[$newStackPtr] = [
734735
'code' => T_LNUMBER,

tests/Core/Tokenizer/BackfillExplicitOctalNotationTest.inc

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -5,3 +5,12 @@ $foo = 0o137041;
55

66
/* testExplicitOctalCapitalised */
77
$bar = 0O137041;
8+
9+
/* testExplicitOctalWithNumericSeparator */
10+
$octal = 0o137_041;
11+
12+
/* testInvalid1 */
13+
$foo = 0o_137;
14+
15+
/* testInvalid2 */
16+
$foo = 0O_41;

tests/Core/Tokenizer/BackfillExplicitOctalNotationTest.php

Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -62,6 +62,27 @@ public function dataExplicitOctalNotation()
6262
'value' => '0O137041',
6363
],
6464
],
65+
[
66+
[
67+
'marker' => '/* testExplicitOctalWithNumericSeparator */',
68+
'type' => 'T_LNUMBER',
69+
'value' => '0o137_041',
70+
],
71+
],
72+
[
73+
[
74+
'marker' => '/* testInvalid1 */',
75+
'type' => 'T_LNUMBER',
76+
'value' => '0',
77+
],
78+
],
79+
[
80+
[
81+
'marker' => '/* testInvalid2 */',
82+
'type' => 'T_LNUMBER',
83+
'value' => '0',
84+
],
85+
],
6586
];
6687

6788
}//end dataExplicitOctalNotation()

tests/Core/Tokenizer/BackfillNumericSeparatorTest.inc

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -77,6 +77,12 @@ $testValue = 107_925_284 .88;
7777
/* testInvalid10 */
7878
$testValue = 107_925_284/*comment*/.88;
7979

80+
/* testInvalid11 */
81+
$foo = 0o_137;
82+
83+
/* testInvalid12 */
84+
$foo = 0O_41;
85+
8086
/*
8187
* Ensure that legitimate calculations are not touched by the backfill.
8288
*/

tests/Core/Tokenizer/BackfillNumericSeparatorTest.php

Lines changed: 26 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -336,6 +336,32 @@ public function dataNoBackfill()
336336
],
337337
],
338338
],
339+
[
340+
'/* testInvalid11 */',
341+
[
342+
[
343+
'code' => T_LNUMBER,
344+
'content' => '0',
345+
],
346+
[
347+
'code' => T_STRING,
348+
'content' => 'o_137',
349+
],
350+
],
351+
],
352+
[
353+
'/* testInvalid12 */',
354+
[
355+
[
356+
'code' => T_LNUMBER,
357+
'content' => '0',
358+
],
359+
[
360+
'code' => T_STRING,
361+
'content' => 'O_41',
362+
],
363+
],
364+
],
339365
[
340366
'/* testCalc1 */',
341367
[

0 commit comments

Comments
 (0)