ErrorHandler causes extreme performance issues with broad source entries

[acbramley]

Q	A
PHPUnit version	11.5.46
PHP version	8.4.6
Installation Method	Composer

Summary

After upgrading to PHPUnit 11.5 (from 10.5) our overall test suite was almost 1 hour slower. This is a large test suite for a Drupal website. Drilling down further I found even a basic unit test went from fractions of a second, to 45 seconds for a single test case.

When debugging this test case I noticed a deprecation was being issued via the Mockery library, and this was resulting in millions of function calls, with the following stack trace:

When debugging into the SourceFilter, I noticed $directories = $this->aggregateDirectories($source->excludeDirectories()); in this case was a single valued array, with /data/./ as the only key.

This is the root of the codebase, essentially meaning that every deprecation issued by a test was resulting in an iteration over every single file in the codebase.

I then found that our source entry in phpunit.xml had this:

  <source>
    <include>
      <directory>./app/modules/custom</directory>
      <directory>./app/profiles/custom</directory>
      <directory>./src</directory>
    </include>
    <exclude>
      <directory suffix="Test.php">./</directory>
      <directory suffix="TestBase.php">./</directory>
    </exclude>
  </source>

Changing to this fixed it:

  <source>
    <include>
      <directory>./app/modules/custom</directory>
      <directory>./app/profiles/custom</directory>
      <directory>./src</directory>
    </include>
    <exclude>
      <directory suffix="Test.php">./app/*/custom/*/tests</directory>
      <directory suffix="TestBase.php">./app/*/custom/*/tests</directory>
    </exclude>
  </source>

Current behavior

Very slow tests when deprecations are triggered when include/exclude directories are too broad.

How to reproduce

?

Expected behavior

No performance degradation upon upgrading phpunit.

This report is in part just to raise awareness of this issue, and perhaps see if there's anything we could do on phpunit's end to ensure aggregated directories do not result in scanning everything.

[sebastianbergmann]

This is probably a duplicate of #6114.

[sebastianbergmann]

every deprecation issued by a test was resulting in an iteration over every single file in the codebase

The first deprecation (E_DEPRECATED, E_USER_DEPRECATED) triggered while a test is running causes an iteration over every single file in the codebase. This is a slow operation that @dantleech is improving in #6114. However, this work is only performed once. Unless your tests use process isolation as we're not sharing the result of this expensive operation between parent process and child processes.

I then found that our source entry in phpunit.xml had this:

  <source>
    <include>
      <directory>./app/modules/custom</directory>
      <directory>./app/profiles/custom</directory>
      <directory>./src</directory>
    </include>
    <exclude>
      <directory suffix="Test.php">./</directory>
      <directory suffix="TestBase.php">./</directory>
    </exclude>
  </source>

Changing to this fixed it:

  <source>
    <include>
      <directory>./app/modules/custom</directory>
      <directory>./app/profiles/custom</directory>
      <directory>./src</directory>
    </include>
    <exclude>
      <directory suffix="Test.php">./app/*/custom/*/tests</directory>
      <directory suffix="TestBase.php">./app/*/custom/*/tests</directory>
    </exclude>
  </source>

You must not put your test sources onto the list configured through <source>. If you follow the best practice of having separate root directories for your tested code, usually src, and test code, usually tests, then this is in most cases as easy as only configuring src and its subdirectories onto the list. If you mix tested code and test code in your directory structure then you must exclude your test code as you just did.

[acbramley]

@sebastianbergmann thanks for the explanation, I am not using the RunTestsInSeparateProcesses attribute anywhere at the moment, but that is something that has been added to all Drupal core non-unit tests (Functional/Kernel) for various reasons.