Skip to content

Commit faa7e87

Browse files
fabpotfabpot
committed
feature #4775 Add getOperatorTokens() to ExpressionParserInterface to separate operator token registration from parser identity (fabpot)
This PR was merged into the 3.x branch. Discussion ---------- Add getOperatorTokens() to ExpressionParserInterface to separate operator token registration from parser identity Closes #4767 Closes #4774 Commits ------- e5eb95d Add getOperatorTokens() to ExpressionParserInterface to separate operator token registration from parser identity
2 parents 86e4384 + e5eb95d commit faa7e87

File tree

10 files changed

+115
-29
lines changed

10 files changed

+115
-29
lines changed

‎CHANGELOG‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,6 @@
11
# 3.24.0 (2026-XX-XX)
22

3+
* Deprecate not implementing the `getOperatorTokens()` method in `ExpressionParserInterface` implementations
34
* Deprecate passing a non-`AbstractExpression` node to `ParserTwig\Node\Expression\Binary\MatchesBinary` constructor
45
* Deprecate passing a non-`AbstractExpression` node to `Parser::setParent()`
56
* Add support for renaming variables in object destructuring (`{name: userName} = user`)

‎doc/deprecated.rst‎

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -485,3 +485,13 @@ Operators
485485

486486
* The ``Twig\OperatorPrecedenceChange`` class is deprecated as of Twig 3.21,
487487
use ``Twig\ExpressionParser\PrecedenceChange`` instead.
488+
489+
* Not implementing the ``getOperatorTokens()`` method in
490+
``Twig\ExpressionParser\ExpressionParserInterface`` implementations is
491+
deprecated as of Twig 3.24. This method will be added to the interface in
492+
Twig 4.0. It returns the operator token strings that the expression parser
493+
handles (used by the Lexer and the parser registry). If your custom
494+
expression parser extends ``Twig\ExpressionParser\AbstractExpressionParser``,
495+
the default implementation returns ``[$this->getName(), ...$this->getAliases()]``.
496+
Override it if your parser doesn't handle operator tokens (return ``[]``) or if
497+
the operator tokens differ from the parser name.

‎doc/operators_precedence.rst‎

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -92,12 +92,12 @@
9292
+------------+------------------+---------+---------------+-------------------------------------------------------------------+
9393
| 0 | ``(`` | prefix | n/a | Explicit group expression (a) |
9494
+------------+------------------+---------+---------------+-------------------------------------------------------------------+
95-
| | ``literal`` | | | A literal value (boolean, string, number, sequence, mapping, ...) |
96-
+------------+------------------+---------+---------------+-------------------------------------------------------------------+
9795
| | ``?`` | infix | Left | Conditional operator (a ? b : c) |
9896
+------------+------------------+---------+---------------+-------------------------------------------------------------------+
9997
| | ``=`` | | Right | Assignment operator |
10098
+------------+------------------+---------+---------------+-------------------------------------------------------------------+
99+
| | ``literal`` | prefix | n/a | A literal value (boolean, string, number, sequence, mapping, ...) |
100+
+------------+------------------+---------+---------------+-------------------------------------------------------------------+
101101

102102
When a precedence will change in 4.0, the new precedence is indicated by the arrow ``=>``.
103103

@@ -196,9 +196,9 @@ Here is the same table for Twig 4.0 with adjusted precedences:
196196
+------------+------------------+---------+---------------+-------------------------------------------------------------------+
197197
| 0 | ``(`` | prefix | n/a | Explicit group expression (a) |
198198
+------------+------------------+---------+---------------+-------------------------------------------------------------------+
199-
| | ``literal`` | | | A literal value (boolean, string, number, sequence, mapping, ...) |
200-
+------------+------------------+---------+---------------+-------------------------------------------------------------------+
201199
| | ``?`` | infix | Left | Conditional operator (a ? b : c) |
202200
+------------+------------------+---------+---------------+-------------------------------------------------------------------+
203201
| | ``=`` | | Right | Assignment operator |
204202
+------------+------------------+---------+---------------+-------------------------------------------------------------------+
203+
| | ``literal`` | prefix | n/a | A literal value (boolean, string, number, sequence, mapping, ...) |
204+
+------------+------------------+---------+---------------+-------------------------------------------------------------------+

‎src/ExpressionParser/AbstractExpressionParser.php‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -27,4 +27,9 @@ public function getAliases(): array
2727
{
2828
return [];
2929
}
30+
31+
public function getOperatorTokens(): array
32+
{
33+
return [$this->getName(), ...$this->getAliases()];
34+
}
3035
}

‎src/ExpressionParser/ExpressionParserInterface.php‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -11,6 +11,14 @@
1111

1212
namespace Twig\ExpressionParser;
1313

14+
/**
15+
* @method list<string> getOperatorTokens() Returns the operator token strings that this expression parser handles.
16+
* These are the strings that should be recognized as operator tokens by the Lexer,
17+
* and used to look up the parser in the registry.
18+
* For most parsers, this returns the name and aliases. Parsers that don't handle
19+
* operator tokens (like LiteralExpressionParser) should return an empty array.
20+
* This method will be added to the interface in Twig 4.0.
21+
*/
1422
interface ExpressionParserInterface
1523
{
1624
public function __toString(): string;

‎src/ExpressionParser/ExpressionParsers.php‎

Lines changed: 33 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -54,10 +54,9 @@ public function add(array $parsers): static
5454
// throw new \InvalidArgumentException(\sprintf('Precedence for "%s" must be between 0 and 512, got %d.', $parser->getName(), $parser->getPrecedence()));
5555
}
5656
$interface = $parser instanceof PrefixExpressionParserInterface ? PrefixExpressionParserInterface::class : InfixExpressionParserInterface::class;
57-
$this->parsersByName[$interface][$parser->getName()] = $parser;
5857
$this->parsersByClass[$parser::class] = $parser;
59-
foreach ($parser->getAliases() as $alias) {
60-
$this->parsersByName[$interface][$alias] = $parser;
58+
foreach (self::getOperatorTokensFor($parser) as $token) {
59+
$this->parsersByName[$interface][$token] = $parser;
6160
}
6261
}
6362

@@ -90,9 +89,22 @@ public function getByName(string $interface, string $name): ?ExpressionParserInt
9089

9190
public function getIterator(): \Traversable
9291
{
92+
$seen = [];
9393
foreach ($this->parsersByName as $parsers) {
94-
// we don't yield the keys
95-
yield from $parsers;
94+
foreach ($parsers as $parser) {
95+
$id = spl_object_id($parser);
96+
if (!isset($seen[$id])) {
97+
$seen[$id] = true;
98+
yield $parser;
99+
}
100+
}
101+
}
102+
foreach ($this->parsersByClass as $parser) {
103+
$id = spl_object_id($parser);
104+
if (!isset($seen[$id])) {
105+
$seen[$id] = true;
106+
yield $parser;
107+
}
96108
}
97109
}
98110

@@ -124,4 +136,20 @@ public function getPrecedenceChanges(): \WeakMap
124136

125137
return $this->precedenceChanges;
126138
}
139+
140+
/**
141+
* @internal
142+
*
143+
* @return array<string>
144+
*/
145+
public static function getOperatorTokensFor(ExpressionParserInterface $parser): array
146+
{
147+
if (method_exists($parser, 'getOperatorTokens')) {
148+
return $parser->getOperatorTokens();
149+
}
150+
151+
trigger_deprecation('twig/twig', '3.24', 'Not implementing the "getOperatorTokens()" method in "%s" is deprecated. This method will be part of the "%s" interface in 4.0.', $parser::class, ExpressionParserInterface::class);
152+
153+
return [$parser->getName(), ...$parser->getAliases()];
154+
}
127155
}

‎src/ExpressionParser/Prefix/LiteralExpressionParser.php‎

Lines changed: 6 additions & 19 deletions
Original file line numberDiff line numberDiff line change
@@ -30,8 +30,6 @@
3030
*/
3131
final class LiteralExpressionParser extends AbstractExpressionParser implements PrefixExpressionParserInterface, ExpressionParserDescriptionInterface
3232
{
33-
private string $type = 'literal';
34-
3533
public function parse(Parser $parser, Token $token): AbstractExpression
3634
{
3735
$stream = $parser->getStream();
@@ -41,41 +39,30 @@ public function parse(Parser $parser, Token $token): AbstractExpression
4139
switch ($token->getValue()) {
4240
case 'true':
4341
case 'TRUE':
44-
$this->type = 'constant';
45-
4642
return new ConstantExpression(true, $token->getLine());
4743

4844
case 'false':
4945
case 'FALSE':
50-
$this->type = 'constant';
51-
5246
return new ConstantExpression(false, $token->getLine());
5347

5448
case 'none':
5549
case 'NONE':
5650
case 'null':
5751
case 'NULL':
58-
$this->type = 'constant';
59-
6052
return new ConstantExpression(null, $token->getLine());
6153

6254
default:
63-
$this->type = 'variable';
64-
6555
return new ContextVariable($token->getValue(), $token->getLine());
6656
}
6757

6858
// no break
6959
case $token->test(Token::NUMBER_TYPE):
7060
$stream->next();
71-
$this->type = 'constant';
7261

7362
return new ConstantExpression($token->getValue(), $token->getLine());
7463

7564
case $token->test(Token::STRING_TYPE):
7665
case $token->test(Token::INTERPOLATION_START_TYPE):
77-
$this->type = 'string';
78-
7966
return $this->parseStringExpression($parser);
8067

8168
case $token->test(Token::PUNCTUATION_TYPE):
@@ -96,7 +83,6 @@ public function parse(Parser $parser, Token $token): AbstractExpression
9683
if (preg_match(Lexer::REGEX_NAME, $token->getValue(), $matches) && $matches[0] == $token->getValue()) {
9784
// in this context, string operators are variable names
9885
$stream->next();
99-
$this->type = 'variable';
10086

10187
return new ContextVariable($token->getValue(), $token->getLine());
10288
}
@@ -109,7 +95,12 @@ public function parse(Parser $parser, Token $token): AbstractExpression
10995

11096
public function getName(): string
11197
{
112-
return $this->type;
98+
return 'literal';
99+
}
100+
101+
public function getOperatorTokens(): array
102+
{
103+
return [];
113104
}
114105

115106
public function getDescription(): string
@@ -153,8 +144,6 @@ private function parseStringExpression(Parser $parser)
153144

154145
private function parseSequenceExpression(Parser $parser)
155146
{
156-
$this->type = 'sequence';
157-
158147
$stream = $parser->getStream();
159148
$stream->expect(Token::OPERATOR_TYPE, '[', 'A sequence element was expected');
160149

@@ -185,8 +174,6 @@ private function parseSequenceExpression(Parser $parser)
185174

186175
private function parseMappingExpression(Parser $parser)
187176
{
188-
$this->type = 'mapping';
189-
190177
$stream = $parser->getStream();
191178
$stream->expect(Token::PUNCTUATION_TYPE, '{', 'A mapping element was expected');
192179

‎src/Lexer.php‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -13,6 +13,7 @@
1313
namespace Twig;
1414

1515
use Twig\Error\SyntaxError;
16+
use Twig\ExpressionParser\ExpressionParsers;
1617

1718
/**
1819
* @author Fabien Potencier <fabien@symfony.com>
@@ -527,7 +528,7 @@ private function getOperatorRegex(): string
527528
{
528529
$expressionParsers = [];
529530
foreach ($this->env->getExpressionParsers() as $expressionParser) {
530-
$expressionParsers = array_merge($expressionParsers, [$expressionParser->getName()], $expressionParser->getAliases());
531+
$expressionParsers = array_merge($expressionParsers, ExpressionParsers::getOperatorTokensFor($expressionParser));
531532
}
532533

533534
$expressionParsers = array_combine($expressionParsers, array_map('strlen', $expressionParsers));

‎tests/ExpressionParserTest.php‎

Lines changed: 33 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -27,7 +27,10 @@
2727
use Twig\Environment;
2828
use Twig\Error\RuntimeError;
2929
use Twig\Error\SyntaxError;
30+
use Twig\ExpressionParser\InfixExpressionParserInterface;
31+
use Twig\ExpressionParser\Prefix\LiteralExpressionParser;
3032
use Twig\ExpressionParser\Prefix\UnaryOperatorExpressionParser;
33+
use Twig\ExpressionParser\PrefixExpressionParserInterface;
3134
use Twig\Extension\AbstractExtension;
3235
use Twig\Loader\ArrayLoader;
3336
use Twig\Node\Expression\ArrayExpression;
@@ -796,6 +799,36 @@ public static function getBindingPowerTests(): iterable
796799
yield '= stronger than logical' => ['{% do a = false or true %}{{ a }}', '{% do a = (false or true) %}{{ a }}', eval('$a = false || true; return $a;')];
797800
yield '= stronger than ternary' => ['{% do c = 4 ? 0 : -1 %}{{ c }}', '{% do c = (4 ? 0 : -1) %}{{ c }}', eval('return 4 ? 0 : -1;')];
798801
}
802+
803+
public function testLiteralExpressionParserGetOperatorTokensReturnsEmptyArray()
804+
{
805+
$env = new Environment(new ArrayLoader());
806+
$parser = $env->getExpressionParsers()->getByClass(LiteralExpressionParser::class);
807+
808+
$this->assertSame([], $parser->getOperatorTokens());
809+
$this->assertSame('literal', $parser->getName());
810+
}
811+
812+
public function testExpressionParserGetOperatorTokensDefaultBehavior()
813+
{
814+
$env = new Environment(new ArrayLoader());
815+
816+
foreach ($env->getExpressionParsers() as $parser) {
817+
if ($parser instanceof LiteralExpressionParser) {
818+
continue;
819+
}
820+
$expected = [$parser->getName(), ...$parser->getAliases()];
821+
$this->assertSame($expected, $parser->getOperatorTokens(), \sprintf('getOperatorTokens() for %s should return name + aliases.', $parser::class));
822+
}
823+
}
824+
825+
public function testLiteralIsNotRegisteredAsOperator()
826+
{
827+
// Ensure "literal" is not in the operator registry
828+
$env = new Environment(new ArrayLoader());
829+
$this->assertNull($env->getExpressionParsers()->getByName(PrefixExpressionParserInterface::class, 'literal'));
830+
$this->assertNull($env->getExpressionParsers()->getByName(InfixExpressionParserInterface::class, 'literal'));
831+
}
799832
}
800833

801834
class NotReadyFunctionExpression extends FunctionExpression

‎tests/LexerTest.php‎

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -454,6 +454,19 @@ public function testFilterAndAttributeNamedAfterOperator()
454454

455455
// add a dummy assertion here to satisfy PHPUnit, the only thing we want to test is that the code above
456456
// can be executed without throwing any exceptions
457+
}
458+
459+
public function testLiteralIsNotAnOperator()
460+
{
461+
// "literal" is the name of the LiteralExpressionParser but should not be treated as an operator token
462+
$template = '{{ literal }}';
463+
464+
$lexer = new Lexer(new Environment(new ArrayLoader()));
465+
$stream = $lexer->tokenize(new Source($template, 'index'));
466+
$stream->expect(Token::VAR_START_TYPE);
467+
$stream->expect(Token::NAME_TYPE, 'literal');
468+
$stream->expect(Token::VAR_END_TYPE);
469+
457470
$this->addToAssertionCount(1);
458471
}
459472

0 commit comments

Comments
 (0)