Tests: refactor the test framework

As things were, the test framework used a setup with custom test suites, which was based on PHPUnit 2/3 and only still worked by accident, not by design. This type of setup was deprecated in PHPUnit 4 and support has been removed completely in PHPUnit 10.

This commit updates the test framework to start using the test suite setup via the PHPUnit config file instead and removes all classes related to the custom test suite setup.

Notes:
* As all test suites are now declared in the PHPUnit config file, the tests can now be run by just calling `phpunit`.
    No need to explicitly pass the `tests/AllTests.php` file anymore.
* As part of the test suite setup, all files needed by the tests were preloaded and the autoloader was removed once the test suites had been set up. This looks to have been designed like this originally to allow for the tests to be run in a PEAR setup.
    Various global variables were used in this process and referred to in the test cases.
    Those global variables are now no longer declared and all uses of these have been removed.
* The aforementioned preloading previously could cause issues when running tests for external standards and/or when running the tests for PHPCS itself when various external standards were registered in `installed_paths`.
    To work around this, PR squizlabs/PHP_CodeSniffer 1146 introduced a `PHPCS_IGNORE_TESTS` environment variable to explicitly ignore other standards than the one(s) being tested.
    With the changes now made, it is no longer needed to set this environment variable (once a test suite for an external standard has been made compatible with the PHPCS 4.0 test framework).
* As there is now no place from which to call the `printPHPCodeSnifferTestOutput()` method anymore, the method has been removed and the "# sniff test files generated # unique error codes; # were fixable (#%)" summary line will no longer be printed at the end of a test run.
    A number of global variables were declared and written to by the tests to allow for that output previously.
    Those global variables are now no longer declared and assignments to these has been removed.
* The previously used global variables containing an instance of the `Config` class and an array of instances of the `Ruleset` class have also been removed. This information is now stored in `private static` properties in the `AbstractSniffUnitTest` class.
* To ensure autoloading of the tests and other files also works for tests for external standards using the PHPCS native test framework and loading the PHPCS native test bootstrap from _their_ test bootstrap, all standards registered in the applicable `CodeSniffer.conf` file will be registered with the autoloader.

Information on how to upgrade test suites for external standards which are based on the PHPCS native test framework will be included in the PHPCS 4.0 upgrade guide.

Fixes 25

Refs:
* sebastianbergmann/phpunit 3213
* sebastianbergmann/phpunit 4460

Author: jrfnl

.github/workflows/quicktest.yml

@@ -60,14 +60,14 @@ jobs:
 
       - name: 'PHPUnit: run the full test suite'
         if: ${{ matrix.os != 'windows-latest' }}
-        run: php "vendor/bin/phpunit" tests/AllTests.php --no-coverage
+        run: php "vendor/bin/phpunit" --no-coverage
 
       - name: 'PHPUnit: run tests which may have different outcomes on Windows'
         if: ${{ matrix.os == 'windows-latest' }}
-        run: php "vendor/bin/phpunit" tests/AllTests.php --group Windows --no-coverage
+        run: php "vendor/bin/phpunit" --group Windows --no-coverage
 
       - name: 'PHPUnit: run select tests in CBF mode'
-        run: php "vendor/bin/phpunit" tests/AllTests.php --group CBF --exclude-group nothing --no-coverage
+        run: php "vendor/bin/phpunit" --group CBF --exclude-group nothing --no-coverage
         env:
           PHP_CODESNIFFER_CBF: '1'
 

.github/workflows/test.yml

@@ -204,11 +204,11 @@ jobs:
 
       - name: 'PHPUnit: run the full test suite without code coverage'
         if: ${{ matrix.skip_tests != true }}
-        run: php "vendor/bin/phpunit" tests/AllTests.php --no-coverage
+        run: php "vendor/bin/phpunit" --no-coverage
 
       - name: 'PHPUnit: run select tests in CBF mode'
         if: ${{ matrix.skip_tests != true }}
-        run: php "vendor/bin/phpunit" tests/AllTests.php --group CBF --exclude-group nothing --no-coverage
+        run: php "vendor/bin/phpunit" --group CBF --exclude-group nothing --no-coverage
         env:
           PHP_CODESNIFFER_CBF: '1'
 
@@ -307,13 +307,13 @@ jobs:
       - name: "Run the unit tests with code coverage"
         if: ${{ matrix.os != 'windows-latest' }}
         run: >
-          php "vendor/bin/phpunit" tests/AllTests.php
+          php "vendor/bin/phpunit"
           ${{ steps.phpunit_version.outputs.VERSION >= '9.3' && '--coverage-cache ./build/phpunit-cache' || '' }}
 
       - name: "Run select tests in CBF mode with code coverage"
         if: ${{ matrix.os != 'windows-latest' }}
         run: >
-          php "vendor/bin/phpunit" tests/AllTests.php
+          php "vendor/bin/phpunit"
           ${{ steps.phpunit_version.outputs.VERSION >= '9.3' && '--coverage-cache ./build/phpunit-cache' || '' }}
           --group CBF --exclude-group nothing --coverage-clover build/logs/clover-cbf.xml
         env:
@@ -322,7 +322,7 @@ jobs:
       - name: "Run the unit tests which may have different outcomes on Windows with code coverage"
         if: ${{ matrix.os == 'windows-latest' }}
         run: >
-          php "vendor/bin/phpunit" tests/AllTests.php --group Windows
+          php "vendor/bin/phpunit" --group Windows
           ${{ steps.phpunit_version.outputs.VERSION >= '9.3' && '--coverage-cache ./build/phpunit-cache' || '' }}
 
       - name: "Upload coverage results to Coveralls (normal run)"

composer.json

@@ -59,15 +59,15 @@
         ],
         "test": [
             "Composer\\Config::disableProcessTimeout",
-            "@php ./vendor/phpunit/phpunit/phpunit tests/AllTests.php --no-coverage"
+            "@php ./vendor/phpunit/phpunit/phpunit --no-coverage"
         ],
         "coverage": [
             "Composer\\Config::disableProcessTimeout",
-            "@php ./vendor/phpunit/phpunit/phpunit tests/AllTests.php -d max_execution_time=0"
+            "@php ./vendor/phpunit/phpunit/phpunit -d max_execution_time=0"
         ],
         "coverage-local": [
             "Composer\\Config::disableProcessTimeout",
-            "@php ./vendor/phpunit/phpunit/phpunit tests/AllTests.php --coverage-html ./build/coverage-html -d max_execution_time=0"
+            "@php ./vendor/phpunit/phpunit/phpunit --coverage-html ./build/coverage-html -d max_execution_time=0"
         ],
         "build": [
             "Composer\\Config::disableProcessTimeout",

phpunit.xml.dist

@@ -13,8 +13,17 @@
     forceCoversAnnotation="true"
     >
     <testsuites>
-        <testsuite name="PHP_CodeSniffer Test Suite">
-            <file>tests/AllTests.php</file>
+        <testsuite name="PHPCS_Core">
+            <directory>./tests/Core/</directory>
+        </testsuite>
+        <testsuite name="PHPCS_Sniffs">
+            <directory>./src/Standards/Generic/Tests/</directory>
+            <directory>./src/Standards/PEAR/Tests/</directory>
+            <directory>./src/Standards/PSR1/Tests/</directory>
+            <directory>./src/Standards/PSR2/Tests/</directory>
+            <directory>./src/Standards/PSR12/Tests/</directory>
+            <directory>./src/Standards/Squiz/Tests/</directory>
+            <directory>./src/Standards/Zend/Tests/</directory>
         </testsuite>
     </testsuites>