Ruleset: remove test specific code

The PHPCS 2.7.1 release introduced a change to how sniff tests are run.
Originally, sniff tests would be run in the context of their ruleset. This could lead to parts of the sniff being untestable, like if the ruleset would exclude a error code or contained custom property settings.

As of PHPCS 2.7.1, sniff tests are run in isolation and no longer take the ruleset of the standard they belong to into account, which solves the above problem.

This change was introduced via commit 3df85dc.

Unfortunately, the way this change was made was via a change to the `Ruleset` class, not by changing the test framework.

It basically meant that the `Ruleset` class would now behave differently when the following two conditions were met:
1. It was being used in a test context (`PHP_CODESNIFFER_IN_TESTS` constant defined).
2. Sniff restrictions were applied via the Config, either by using the CLI `--sniffs` argument or by setting the `Config::$sniffs` property directly.

In other words, this now created a new problem when testing parts of the `Ruleset` class which would need the `Config::$sniffs` property to be set.
It also makes writing various other tests for the framework more difficult as there are plenty of times when tests would benefit from the `Config::$sniffs` property being respected. In other words, this led to work-arounds being needed in various other places in the tests, either by emulating the sniff selection or by bypassing the `Ruleset` conditions and doing the sniff selection in a test specific ruleset XML fixture file.

Aside from the above, it also makes the barrier to entry for contributors to write tests for PHPCS that much higher as the behaviour is non-intuitive.

All in all, time to get rid of the condition in the `Ruleset` class.

Now, of course, we do still want to test sniffs in isolation, so to do that the `AbstractSniffUnitTest` class has been updated to on-the-fly create ruleset XML files which only include a single sniff without any customizations.

I considered using `php://memory` or `php://temp` instead of writing the file to disk, however, neither of those work with `file_get_contents()`, which is used to read the XML ruleset files in the `Ruleset` class, so unfortunately, those temporary file writes cannot be avoided without making bigger changes, which is outside the scope of this PR.

Now, I wondered about the performance impact of writing a file to disk for every sniff being tested, and while there is a small, but noticeable, difference when running on Windows (20 vs 27 seconds for `--filter Standards`), IMO this difference is acceptable when offset against the hours and hours of dev time lost trying to debug why perfectly valid tests weren't working due to `--sniffs` not being respected in the tests.

And to be sure, I've also tested the new test isolation mechanism with an external standard which is using the PHPCS native test framework and it looks to be working fine.

Note: even though the `AbstractSniffUnitTest` will clean up the temporary file after each test, if the test run would crash, it could be possible for the file to remain on disk. With this in mind, the file name for the temporary file has been added to the `.gitignore` file.

Related to 966

Author: jrfnl

.gitignore

@@ -8,3 +8,4 @@
 composer.lock
 phpstan.neon
 /node_modules/
+/tests/Standards/sniffStnd.xml

src/Ruleset.php

@@ -201,22 +201,6 @@ public function __construct(Config $config)
                 Autoload::addSearchPath(dirname($standard), $namespace);
             }
 
-            if (defined('PHP_CODESNIFFER_IN_TESTS') === true && empty($restrictions) === false) {
-                // In unit tests, only register the sniffs that the test wants and not the entire standard.
-                foreach ($restrictions as $restriction) {
-                    $sniffs = array_merge($sniffs, $this->expandRulesetReference($restriction, dirname($standard)));
-                }
-
-                if (empty($sniffs) === true) {
-                    // Sniff reference could not be expanded, which probably means this
-                    // is an installed standard. Let the unit test system take care of
-                    // setting the correct sniff for testing.
-                    return;
-                }
-
-                break;
-            }
-
             if (PHP_CODESNIFFER_VERBOSITY === 1) {
                 echo "Registering sniffs in the $standardName standard... ";
                 if (count($config->standards) > 1 || PHP_CODESNIFFER_VERBOSITY > 2) {

tests/Standards/AbstractSniffUnitTest.php

@@ -26,18 +26,43 @@ abstract class AbstractSniffUnitTest extends TestCase
 {
 
     /**
-     * Cache for the Config object.
+     * Ruleset template with placeholders.
      *
-     * @var \PHP_CodeSniffer\Tests\ConfigDouble
+     * @var string
      */
-    private static $config;
+    private const RULESET_TEMPLATE = <<<'TEMPLATE'
+<?xml version="1.0"?>
+<ruleset xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" name="[STANDARDNAME]" xsi:noNamespaceSchemaLocation="../../phpcs.xsd">
+    <description>Temporary ruleset used by the AbstractSniffUnitTest class.</description>
+
+    <rule ref="[SNIFFFILEREF]"/>
+
+</ruleset>
+TEMPLATE;
+
+    /**
+     * Placeholders used in the ruleset template which need to be replaced.
+     *
+     * @var array<string>
+     */
+    private const SEARCH_FOR = [
+        '[STANDARDNAME]',
+        '[SNIFFFILEREF]',
+    ];
+
+    /**
+     * Location where the temporary ruleset file will be saved.
+     *
+     * @var string
+     */
+    private const RULESET_FILENAME = __DIR__.'/sniffStnd.xml';
 
     /**
-     * Cache for Ruleset objects.
+     * Cache for the Config object.
      *
-     * @var array<string, \PHP_CodeSniffer\Ruleset>
+     * @var \PHP_CodeSniffer\Tests\ConfigDouble
      */
-    private static $rulesets = [];
+    private static $config;
 
     /**
      * Extensions to disregard when gathering the test files.
@@ -52,6 +77,18 @@ abstract class AbstractSniffUnitTest extends TestCase
     ];
 
 
+    /**
+     * Clean up temporary ruleset file.
+     *
+     * @return void
+     */
+    public static function tearDownAfterClass(): void
+    {
+        @unlink(self::RULESET_FILENAME);
+
+    }//end tearDownAfterClass()
+
+
     /**
      * Get a list of all test files to check.
      *
@@ -103,7 +140,8 @@ protected function shouldSkipTest()
      * Tests the extending classes Sniff class.
      *
      * @return void
-     * @throws \PHPUnit\Framework\Exception
+     *
+     * @throws \PHP_CodeSniffer\Exceptions\RuntimeException
      */
     final public function testSniff()
     {
@@ -122,6 +160,23 @@ final public function testSniff()
         // Get a list of all test files to check.
         $testFiles = $this->getTestFiles($testFileBase);
 
+        $sniffFile = preg_replace('`[/\\\\]Tests[/\\\\]`', DIRECTORY_SEPARATOR.'Sniffs'.DIRECTORY_SEPARATOR, $testFileBase);
+        $sniffFile = str_replace('UnitTest.', 'Sniff.php', $sniffFile);
+
+        if (file_exists($sniffFile) === false) {
+            $this->fail(sprintf('ERROR: Sniff file %s for test %s does not appear to exist', $sniffFile, static::class));
+        }
+
+        $replacements    = [
+            $standardName,
+            $sniffFile,
+        ];
+        $rulesetContents = str_replace(self::SEARCH_FOR, $replacements, self::RULESET_TEMPLATE);
+
+        if (file_put_contents(self::RULESET_FILENAME, $rulesetContents) === false) {
+            throw new RuntimeException('Failed to write custom ruleset file');
+        }
+
         if (isset(self::$config) === true) {
             $config = self::$config;
         } else {
@@ -130,31 +185,11 @@ final public function testSniff()
             self::$config  = $config;
         }
 
-        $config->standards = [$standardName];
+        $config->standards = [self::RULESET_FILENAME];
         $config->sniffs    = [$sniffCode];
         $config->ignored   = [];
 
-        if (isset(self::$rulesets[$standardName]) === false) {
-            $ruleset = new Ruleset($config);
-            self::$rulesets[$standardName] = $ruleset;
-        }
-
-        $ruleset = self::$rulesets[$standardName];
-
-        $sniffFile = preg_replace('`[/\\\\]Tests[/\\\\]`', DIRECTORY_SEPARATOR.'Sniffs'.DIRECTORY_SEPARATOR, $testFileBase);
-        $sniffFile = str_replace('UnitTest.', 'Sniff.php', $sniffFile);
-
-        if (file_exists($sniffFile) === false) {
-            $this->fail(sprintf('ERROR: Sniff file %s for test %s does not appear to exist', $sniffFile, static::class));
-        }
-
-        $sniffClassName = substr(get_class($this), 0, -8).'Sniff';
-        $sniffClassName = str_replace('\Tests\\', '\Sniffs\\', $sniffClassName);
-        $sniffClassName = Common::cleanSniffClass($sniffClassName);
-
-        $restrictions = [strtolower($sniffClassName) => true];
-        $ruleset->registerSniffs([$sniffFile], $restrictions, []);
-        $ruleset->populateTokenListeners();
+        $ruleset = new Ruleset($config);
 
         $failureMessages = [];
         foreach ($testFiles as $testFile) {