Skip to content

Commit d7658d2

Browse files
nicolas-grekasnicolas-grekas
committed
bug symfony#26169 [Routing] Handle very large set of dynamic routes (nicolas-grekas)
This PR was merged into the 4.1-dev branch. Discussion ---------- [Routing] Handle very large set of dynamic routes | Q | A | ------------- | --- | Branch? | master | Bug fix? | yes | New feature? | no | BC breaks? | no | Deprecations? | no | Tests pass? | yes | Fixed tickets | - | License | MIT | Doc PR | - Follow up of symfony#26059; allows handling very long lists of dynamic routes. Allows running https://github.com/tyler-sommer/php-router-benchmark seamlessly. BTW, here are the result of running this benchmark: R3 extension is not loaded. Skipping initialization for "Worst-case matching" test using R3. R3 extension is not loaded. Skipping initialization for "First route matching" test using R3. ## Worst-case matching This benchmark matches the last route and unknown route. It generates a randomly prefixed and suffixed route in an attempt to thwart any optimization. 1,000 routes each with 9 arguments. This benchmark consists of 10 tests. Each test is executed 1,000 times, the results pruned, and then averaged. Values that fall outside of 3 standard deviations of the mean are discarded. Test Name | Results | Time | + Interval | Change --------- | ------- | ---- | ---------- | ------ Symfony4 - unknown route (1000 routes) | 995 | 0.0000085699 | +0.0000000000 | baseline Symfony4 - last route (1000 routes) | 999 | 0.0000086754 | +0.0000001055 | 1% slower FastRoute - unknown route (1000 routes) | 980 | 0.0000305154 | +0.0000219455 | 256% slower FastRoute - last route (1000 routes) | 999 | 0.0000529922 | +0.0000444223 | 518% slower Pux PHP - unknown route (1000 routes) | 972 | 0.0003162730 | +0.0003077032 | 3591% slower Pux PHP - last route (1000 routes) | 999 | 0.0004376847 | +0.0004291148 | 5007% slower Aura v2 - unknown route (1000 routes) | 976 | 0.0138277517 | +0.0138191818 | 161253% slower Aura v2 - last route (1000 routes) | 989 | 0.0138914190 | +0.0138828491 | 161996% slower ## First route matching This benchmark tests how quickly each router can match the first route. 1,000 routes each with 9 arguments. This benchmark consists of 5 tests. Each test is executed 1,000 times, the results pruned, and then averaged. Values that fall outside of 3 standard deviations of the mean are discarded. Test Name | Results | Time | + Interval | Change --------- | ------- | ---- | ---------- | ------ FastRoute - first route | 999 | 0.0000016928 | +0.0000000000 | baseline Pux PHP - first route | 999 | 0.0000017381 | +0.0000000453 | 3% slower Symfony4 - first route | 997 | 0.0000029818 | +0.0000012890 | 76% slower Aura v2 - first route | 977 | 0.0000376436 | +0.0000359508 | 2124% slower Commits ------- ee8b201 [Routing] Handle very large set of dynamic routes
2 parents 4d6c481 + ee8b201 commit d7658d2

File tree

3 files changed

+2880
-25
lines changed

3 files changed

+2880
-25
lines changed

‎src/Symfony/Component/Routing/Matcher/Dumper/PhpMatcherDumper.php

Lines changed: 51 additions & 25 deletions
Original file line numberDiff line numberDiff line change
@@ -27,6 +27,7 @@
2727
class PhpMatcherDumper extends MatcherDumper
2828
{
2929
private $expressionLanguage;
30+
private $signalingException;
3031

3132
/**
3233
* @var ExpressionFunctionProviderInterface[]
@@ -87,12 +88,8 @@ public function addExpressionLanguageProvider(ExpressionFunctionProviderInterfac
8788

8889
/**
8990
* Generates the code for the match method implementing UrlMatcherInterface.
90-
*
91-
* @param bool $supportsRedirections Whether redirections are supported by the base class
92-
*
93-
* @return string Match method as PHP code
9491
*/
95-
private function generateMatchMethod($supportsRedirections)
92+
private function generateMatchMethod(bool $supportsRedirections): string
9693
{
9794
// Group hosts by same-suffix, re-order when possible
9895
$matchHost = false;
@@ -132,18 +129,27 @@ public function match(\$rawPathinfo)
132129

133130
/**
134131
* Generates PHP code to match a RouteCollection with all its routes.
135-
*
136-
* @param RouteCollection $routes A RouteCollection instance
137-
* @param bool $supportsRedirections Whether redirections are supported by the base class
138-
*
139-
* @return string PHP code
140132
*/
141-
private function compileRoutes(RouteCollection $routes, $supportsRedirections, $matchHost)
133+
private function compileRoutes(RouteCollection $routes, bool $supportsRedirections, bool $matchHost): string
142134
{
143135
list($staticRoutes, $dynamicRoutes) = $this->groupStaticRoutes($routes, $supportsRedirections);
144136

145137
$code = $this->compileStaticRoutes($staticRoutes, $supportsRedirections, $matchHost);
146-
$code .= $this->compileDynamicRoutes($dynamicRoutes, $supportsRedirections, $matchHost);
138+
$chunkLimit = count($dynamicRoutes);
139+
140+
while (true) {
141+
try {
142+
$this->signalingException = new \RuntimeException('PCRE compilation failed: regular expression is too large');
143+
$code .= $this->compileDynamicRoutes($dynamicRoutes, $supportsRedirections, $matchHost, $chunkLimit);
144+
break;
145+
} catch (\Exception $e) {
146+
if (1 < $chunkLimit && $this->signalingException === $e) {
147+
$chunkLimit = 1 + ($chunkLimit >> 1);
148+
continue;
149+
}
150+
throw $e;
151+
}
152+
}
147153

148154
if ('' === $code) {
149155
$code .= " if ('/' === \$pathinfo) {\n";
@@ -275,13 +281,14 @@ private function compileStaticRoutes(array $staticRoutes, bool $supportsRedirect
275281
* matching-but-failing subpattern is blacklisted by replacing its name by "(*F)", which forces a failure-to-match.
276282
* To ease this backlisting operation, the name of subpatterns is also the string offset where the replacement should occur.
277283
*/
278-
private function compileDynamicRoutes(RouteCollection $collection, bool $supportsRedirections, bool $matchHost): string
284+
private function compileDynamicRoutes(RouteCollection $collection, bool $supportsRedirections, bool $matchHost, int $chunkLimit): string
279285
{
280286
if (!$collection->all()) {
281287
return '';
282288
}
283289
$code = '';
284290
$state = (object) array(
291+
'regex' => '',
285292
'switch' => '',
286293
'default' => '',
287294
'mark' => 0,
@@ -301,11 +308,13 @@ private function compileDynamicRoutes(RouteCollection $collection, bool $support
301308
return '';
302309
};
303310

311+
$chunkSize = 0;
304312
$prev = null;
305313
$perModifiers = array();
306314
foreach ($collection->all() as $name => $route) {
307315
preg_match('#[a-zA-Z]*$#', $route->compile()->getRegex(), $rx);
308-
if ($prev !== $rx[0] && $route->compile()->getPathVariables()) {
316+
if ($chunkLimit < ++$chunkSize || $prev !== $rx[0] && $route->compile()->getPathVariables()) {
317+
$chunkSize = 1;
309318
$routes = new RouteCollection();
310319
$perModifiers[] = array($rx[0], $routes);
311320
$prev = $rx[0];
@@ -326,8 +335,10 @@ private function compileDynamicRoutes(RouteCollection $collection, bool $support
326335
$routes->add($name, $route);
327336
}
328337
$prev = false;
329-
$code .= "\n {$state->mark} => '{^(?'";
330-
$state->mark += 4;
338+
$rx = '{^(?';
339+
$code .= "\n {$state->mark} => ".self::export($rx);
340+
$state->mark += strlen($rx);
341+
$state->regex = $rx;
331342

332343
foreach ($perHost as list($hostRegex, $routes)) {
333344
if ($matchHost) {
@@ -340,8 +351,9 @@ private function compileDynamicRoutes(RouteCollection $collection, bool $support
340351
$hostRegex = '[^/]*+';
341352
$state->hostVars = array();
342353
}
343-
$state->mark += 3 + $prev + strlen($hostRegex);
344-
$code .= "\n .".self::export(($prev ? ')' : '')."|{$hostRegex}(?");
354+
$state->mark += strlen($rx = ($prev ? ')' : '')."|{$hostRegex}(?");
355+
$code .= "\n .".self::export($rx);
356+
$state->regex .= $rx;
345357
$prev = true;
346358
}
347359

@@ -358,8 +370,19 @@ private function compileDynamicRoutes(RouteCollection $collection, bool $support
358370
}
359371
if ($matchHost) {
360372
$code .= "\n .')'";
373+
$state->regex .= ')';
374+
}
375+
$rx = ")$}{$modifiers}";
376+
$code .= "\n .'{$rx}',";
377+
$state->regex .= $rx;
378+
379+
// if the regex is too large, throw a signaling exception to recompute with smaller chunk size
380+
set_error_handler(function ($type, $message) { throw $this->signalingException; });
381+
try {
382+
preg_match($state->regex, '');
383+
} finally {
384+
restore_error_handler();
361385
}
362-
$code .= "\n .')$}{$modifiers}',";
363386
}
364387

365388
if ($state->default) {
@@ -403,7 +426,7 @@ private function compileDynamicRoutes(RouteCollection $collection, bool $support
403426
* @param \stdClass $state A simple state object that keeps track of the progress of the compilation,
404427
* and gathers the generated switch's "case" and "default" statements
405428
*/
406-
private function compileStaticPrefixCollection(StaticPrefixCollection $tree, \stdClass $state, int $prefixLen = 0)
429+
private function compileStaticPrefixCollection(StaticPrefixCollection $tree, \stdClass $state, int $prefixLen = 0): string
407430
{
408431
$code = '';
409432
$prevRegex = null;
@@ -413,10 +436,12 @@ private function compileStaticPrefixCollection(StaticPrefixCollection $tree, \st
413436
if ($route instanceof StaticPrefixCollection) {
414437
$prevRegex = null;
415438
$prefix = substr($route->getPrefix(), $prefixLen);
416-
$state->mark += 3 + strlen($prefix);
417-
$code .= "\n .".self::export("|{$prefix}(?");
439+
$state->mark += strlen($rx = "|{$prefix}(?");
440+
$code .= "\n .".self::export($rx);
441+
$state->regex .= $rx;
418442
$code .= $this->indent($this->compileStaticPrefixCollection($route, $state, $prefixLen + strlen($prefix)));
419443
$code .= "\n .')'";
444+
$state->regex .= ')';
420445
$state->markTail += 1;
421446
continue;
422447
}
@@ -434,8 +459,9 @@ private function compileStaticPrefixCollection(StaticPrefixCollection $tree, \st
434459
$hasTrailingSlash = $hasTrailingSlash && (!$methods || isset($methods['GET']));
435460
$state->mark += 3 + $state->markTail + $hasTrailingSlash + strlen($regex) - $prefixLen;
436461
$state->markTail = 2 + strlen($state->mark);
437-
$code .= "\n .";
438-
$code .= self::export(sprintf('|%s(*:%s)', substr($regex, $prefixLen).($hasTrailingSlash ? '?' : ''), $state->mark));
462+
$rx = sprintf('|%s(*:%s)', substr($regex, $prefixLen).($hasTrailingSlash ? '?' : ''), $state->mark);
463+
$code .= "\n .".self::export($rx);
464+
$state->regex .= $rx;
439465
$vars = array_merge($state->hostVars, $vars);
440466

441467
if (!$route->getCondition() && (!is_array($next = $routes[1 + $i] ?? null) || $regex !== $next[1])) {
@@ -472,7 +498,7 @@ private function compileStaticPrefixCollection(StaticPrefixCollection $tree, \st
472498
/**
473499
* A simple helper to compiles the switch's "default" for both static and dynamic routes.
474500
*/
475-
private function compileSwitchDefault(bool $hasVars, string $routesKey, bool $matchHost, bool $supportsRedirections, bool $checkTrailingSlash)
501+
private function compileSwitchDefault(bool $hasVars, string $routesKey, bool $matchHost, bool $supportsRedirections, bool $checkTrailingSlash): string
476502
{
477503
if ($hasVars) {
478504
$code = <<<EOF

0 commit comments

Comments
 (0)