[TASK] Rework chapter about the runTests.sh and build/testing mechanism (#340)

* [TASK] Rework chapter about the runTests.sh and build/testing mechanism

* Update Documentation/HandlingAPatch/CherryPick.rst

* [TASK] Code review changes

* Apply suggestions from code review

---------

Co-authored-by: Stefan Frömken <[email protected]>

Author: garvinhicking

Documentation/HandlingAPatch/CherryPick.rst

@@ -51,16 +51,20 @@ to cherry-pick it from the review system into your local git repository.
 
 4. Clean up your local repository
 
-   Save your local changes beforehand, if you have any.
-   
+   Save your local changes beforehand, if you have any. Otherwise the
+   `git reset` performed after that would delete local changes, which
+   may happen if you work on different patches simultaneously with the
+   same repository directory.
+
    .. code-block:: bash
 
       git stash save 'comment-your-changes'
 
    .. code-block:: bash
 
+      git fetch --all
       git reset --hard origin/main
-      git pull
+      git pull --rebase
 
 5. Execute the command (git cherry-pick)
 
@@ -77,5 +81,7 @@ to cherry-pick it from the review system into your local git repository.
 6. Cleanup your TYPO3 installation
 
    Depending on the changes made by the patch, you may have to apply some changes
-   to your TYPO3 installation: see :ref:`cleanup-typo3`.
+   to your TYPO3 installation as well. Also, if the last time you pulled from the
+   GitHub repository is some time ago, you may need to pull the most recent
+   dependencies. And you may need to rebuild the CSS/JS assets. See :ref:`cleanup-typo3`.
 

Documentation/HandlingAPatch/CleanupTypo3.rst

@@ -19,8 +19,9 @@ Before :ref:`Cherry-picking a patch <cherry-pick-a-patch>` or starting a new pat
 .. code-block:: bash
    :caption: shell command
 
+   git fetch --all
    git reset --hard origin/main
-   git pull
+   git pull --rebase
 
 
 .. _cleanup-typo3:
@@ -40,7 +41,9 @@ any case or if in doubt, you can safely perform all steps.
    :caption: shell command
 
    Build/Scripts/runTests.sh -s clean
+   # For DDEV environments, prefix with `ddev ...`
    bin/typo3 cache:flush
+   bin/typo3 cache:warmup
 
 **Changes in composer.json**:
 
@@ -64,9 +67,16 @@ Delete browser cache or
 `hard refresh <https://www.filecloud.com/blog/2015/03/tech-tip-how-to-do-hard-refresh-in-browsers/>`__
 (e.g. CTRL + F5)
 
+Also you may need to create the JS/CSS assets:
+
+   .. code-block:: bash
+
+      ./Build/Scripts/runTests.sh -s buildCss
+      ./Build/Scripts/runTests.sh -s buildJavascript
+
 **Changes in DB schema (ext_tables.sql)**:
 
-Maintenance: Analyze Database Structure, Apply selected changes.
+:guilabel:`Maintenance: Analyze Database Structure, Apply selected changes.`
 
 .. image:: /Images/ManualScreenshots/analyze.svg
    :class: with-shadow

Documentation/Setup/Prerequisites.rst

@@ -12,17 +12,18 @@ Required:
 
 *  Git
 *  Once you have setup the Git repository, it is advised to look at the listed
-   dependencies for  :file:`runTests.sh` by running
-   :bash:`Build/Scripts/runTests.sh -h` (see :ref:`runTests_sh`).
+   dependencies (basically: Docker) for  :file:`runTests.sh` (see :ref:`runTests_sh`).
 
 Recommended:
 
 *  IDE with TYPO3 plugins, such as PhpStorm or Visual Studio Code. While you
    can also use a simple editor, a properly setup IDE will help with
    code-completion, marking errors and applying coding guidelines.
-*  DDEV for setting up the TYPO3 installation. Alternative methods
-   are possible, e.g. using a locally installed web server and database. DDEV
+*  `DDEV <https://ddev.com>`__ for setting up the PHP/Database web environment as a base for the TYPO3
+   installation. Alternative methods
+   are possible (XAMPP, LAMP, MAMP, WAMP, custom docker-compose) or even using a custom
+   locally installed web server and database. DDEV
    provides custom configurations for TYPO3 which can be used for core
-   development as well.
+   development as well, and is an easy-to-use layer on top of docker-compose.
 
 Information on using these tools is covered in detail later on this chapter.

Documentation/Setup/SetupTypo3.rst

@@ -37,16 +37,17 @@ composer install
 Run composer install in the same directory you cloned the TYPO3 CMS core
 repository.
 
-It is recommended to use runTests.sh for this. The "direct command" is an
-alternative. You only need to run one of these!
+It is recommended to use :file:`runTests.sh` for this (see :ref:`testing-core`). The "direct command" is an
+alternative, but it requires your local system to have proper PHP and Composer versions ready to use.
+You only need to run one of these!
 
 
 ..  note::
 
-    While working with the TYPO3 sources composer can not detect the TYPO3 version of your
+    While working with the TYPO3 sources, Composer may not detect the TYPO3 version of your
     cloned project because there was none. Before you run `composer install` may need to export
     the `COMPOSER_ROOT_VERSION environment variable <https://getcomposer.org/doc/03-cli.md#composer-root-version>`__.
-    Here you need to set a full version string matching the TYPO3 version of your clone.
+    Here you need to set a full version string matching the TYPO3 version of your Git clone.
 
     Example:
 

Documentation/Testing/CoreTesting.rst

@@ -14,12 +14,13 @@ you a better understanding of testing within TYPO3's Core. A full Core git check
 to run tests in TYPO3.
 
 Core development is most likely bound to the Core `main` branch - backporting patches to older
-branches are usually handled by Core maintainers and often don't affect other Core contributors.
+branches is usually handled by Core maintainers ("Mergers") and often doesn't affect other Core contributors.
 
-Note, the main script
-`Build/Scripts/runTests.sh <https://github.com/typo3/typo3/blob/main/Build/Scripts/runTests.sh>`_
-is relatively new. It works best when executed on a Linux based host but can be run under macOS and
-Windows with some performance drawbacks on macOS.
+The main entry point to perform testing and build-related actions is the helper
+`Build/Scripts/runTests.sh <https://github.com/typo3/typo3/blob/main/Build/Scripts/runTests.sh>`_.
+It works best when executed on a Linux based host but can be run under macOS and
+Windows with some filesystem performance drawbacks on macOS. It utilizes Docker containers,
+and more details can be found in :ref:`Using runTests.sh <t3contribute:runTests_sh>`.
 
 Additionally, it *is* possible to execute tests on a local system without using Docker. Depending on
 which test suite is executed, developers may need to configure their environments to run the
@@ -37,7 +38,7 @@ Many developers are familiar with `Docker <https://www.docker.com/>`_. As outlin
 reliable environment to run tests and also remove the need to manage niche dependencies on your local
 environment for tests such as "execute functional test 'X' using PostgreSQL with xdebug".
 
-Git, docker and docker-compose are all required. For standalone test execution, a local installation of
+Git and docker (or podman) are required. For standalone test execution, a local installation of
 PHP is not required. You can even `composer install` a Core by calling `Build/Scripts/runTests.sh -s
 composerInstall` in a container.
 
@@ -57,7 +58,7 @@ Windows can rely on WSL to have a decent docker version, too.
 Quick start
 ===========
 
-From now on, it is assumed that git, docker and docker-compose are available with the most up-to-date release
+From now on, it is assumed that git and docker (or podman) are available with the most up-to-date release
 running on the host system. Executing the basic Core unit test suite boils down to:
 
 ..  code-block:: shell
@@ -80,27 +81,25 @@ Overview
 So what just happened? We cloned a Core, Composer installed dependencies and executed Core
 unit tests. Let's have a look at some more details: :file:`runTests.sh` is a shell script that figures out
 which test suite with which options a user wants to execute, does some error handling for broken
-combinations, writes the file `Build/testing-docker/local/.env` according to its findings and then executes a
-couple of `docker-compose` commands to prepare containers, runs tests and stops containers after execution
-again.
-
-A Core developer doing this for the first time may notice `docker-compose` pulling several container images
-before continuing. These are the dependent images needed to execute certain jobs. For instance the
-container `typo3gmbh/php74 <https://hub.docker.com/r/typo3gmbh/php72/>`_ may be fetched. Its definition
-can be found at `TYPO3 GmbH GitHub <https://github.com/TYPO3GmbH/infra-bamboo-remote-agent>`_.
-These are the exact same containers Bamboo based testing is executed in. In Bamboo, the combination of
-:file:`Build/bamboo/src/main/java/core/PreMergeSpec.java` and :file:`Build/testing-docker/bamboo/docker-compose.yml`
-specifies what Bamboo executes for patches pushed to the review system. On local testing, this is the
-combination of :file:`Build/Scripts/runTests.sh`, :file:`Build/testing-docker/local/.env` (created by
-runTests.sh) and
-`Build/testing-docker/local/docker-compose.yml <https://github.com/typo3/typo3/blob/main/Build/testing-docker/local/docker-compose.yml>`_.
-
-Whats impressive is that :file:`runTests.sh` can do everything locally that Bamboo executes as `pre-merge
-<https://bamboo.typo3.com/browse/CORE-GTC>`_ tests at the same time. It's just that the combinations of tests
-and splitting to different jobs is slightly different, for instance Bamboo does multiple tests in
-the "integration" test at once that are single "check" suites in :file:`runTests.sh`. But if a patch is
-pushed to Bamboo and it complains about something being broken, it is possible to replay and fix the
-failing suite locally, then push an updated patch and hopefully enable the Bamboo test to pass.
+combinations and then uses local docker commands to run specific containers with specific options. Using
+these containers, the actions are performed, and the containers are stopped after execution.
+
+A Core developer doing this for the first time may notice that several container images
+will be pulled before continuing. These are the dependent images needed to execute certain jobs. For instance,
+a container providing the specific PHP-version may be fetched. The same containers are used for the
+TYPO3 CI GitLab Pipeline, even utilizing the same `runTests.sh` script. What's impressive is that
+you can locally run the same tests like a fully-fledged CI server..
+
+The GitLab CI Pipeline is maintained through the Ansible infrastructure found on
+`https://git.typo3.org/typo3/CI/testing-infrastructure/-/tree/main/ansible?ref_type=heads`__, and the Pipeline
+itself is set up through `https://github.com/TYPO3/typo3/tree/main/Build/gitlab-ci/`__.
+
+Compared to your local execution it's just that the combinations of tests
+and splitting to different jobs which is slightly different, for instance GitLab CI paralelly performs multiple
+tests with more complex version matrixes (PHP and Databases).
+
+If a patch is pushed to GitLab and it complains about something being broken, it is possible to replay and fix the
+failing suite locally, then push an updated patch and hopefully enable the tests to pass.
 
 
 A runTests.sh run
@@ -111,37 +110,23 @@ Let's pick a :file:`runTests.sh` example and have a closer look:
 ..  code-block:: shell
 
     lolli@apoc /var/www/local/cms/Web $ Build/Scripts/runTests.sh -s functional typo3/sysext/core/Tests/Functional/Authentication/
-    Using driver: mysqli
-    Creating network "local_default" with the default driver
-    Creating local_mariadb_1      ...  done
-    Creating local_memcached1-5_1 ...  done
-    Creating local_redis4_1       ...  done
-    Creating local_prepare_functional_mariadb_run ...  done
-    Waiting for database start...
-    Database is up
-    Creating local_functional_mariadb_run ...  done
-    PHP 8.0.14 (cli) (built: Dec 18 2021 02:58:33) ( NTS )
-    PHPUnit 9.5.10 by Sebastian Bergmann and contributors.
-
-    ........................................................          56 / 56 (100%)
-
-    Time: 00:24.864, Memory: 101.00 MB
-
-    OK (56 tests, 162 assertions)
-    Stopping local_redis4_1       ...  done
-    Stopping local_mariadb_1      ...  done
-    Stopping local_memcached1-5_1 ...  done
-    Removing local_functional_mariadb_run_d03a24bcf25c         ...  done
-    Removing local_prepare_functional_mariadb_run_4648d92e8e32 ...  done
-    Removing local_redis4_1                                    ...  done
-    Removing local_mariadb_1                                   ...  done
-    Removing local_memcached1-5_1                              ...  done
-    Removing network local_default
+    PHPUnit 11.2.5 by Sebastian Bergmann and contributors.
+
+    Runtime:       PHP 8.2.19
+    Configuration: /Users/garvin/TYPO3/typo3-core-bugreproduce-base/typo3-core/Build/phpunit/FunctionalTests.xml
+
+    ................................................................. 65 / 67 ( 97%)
+    ..                                                                67 / 67 (100%)
+
+    Time: 00:12.077, Memory: 103.00 MB
+
+    OK (67 tests, 176 assertions)
 
     ###########################################################################
     Result of functional
-    PHP: 8.0
-    DBMS: mariadb  version 10.3  driver mysqli
+    Container runtime: docker
+    PHP: 8.2
+    DBMS: sqlite
     SUCCESS
     ###########################################################################
 
@@ -152,11 +137,12 @@ Let's pick a :file:`runTests.sh` example and have a closer look:
 
 The command asks :file:`runTests.sh` to execute the "functional" test suite `-s functional` and to not execute all
 available tests but only those within `typo3/sysext/core/Tests/Functional/Authentication/`. The script first
-starts the containers it needs: Redis, memcached and a MariaDB. All in one network. It then waits until
-the MariaDB container opens its database port, then starts a PHP 8.0 container and calls phpunit to execute
-the tests. phpunit executes only one test in this case, that one is green. The containers and networks are then
-removed again. Note the exit code of :file:`runTests.sh` (`echo $?`) is identical to the exit code of the phpunit
-call: If phpunit reports green, :file:`runTests.sh` returns 0, and if phpunit is red, the exit code would be non zero.
+starts the containers it needs: Redis, memcached (previously also MariaDB by default, which is now using
+SQLite instead, due to less dependencies). All in one network. It then starts a PHP 8.2 container and calls
+phpunit from there to execute the tests. phpunit executes only one test in this case, that one is green. The containers
+and networks are then removed again. Note the exit code of :file:`runTests.sh` (`echo $?`) is identical to the exit
+code of the phpunit call: If phpunit reports green, :file:`runTests.sh` returns 0, and if phpunit is red, the exit code
+would be non zero.
 
 
 ..  _testing-core-examples:
@@ -172,7 +158,7 @@ are not valid:
 
     lolli@apoc /var/www/local/cms/Web $ Build/Scripts/runTests.sh -h
     TYPO3 Core test runner. Execute acceptance, unit, functional and other test suites in
-    a docker based test environment. Handles execution of single test files, sending
+    a container based test environment. Handles execution of single test files, sending
     xdebug information to a local IDE and more.
     ...
 
@@ -181,14 +167,14 @@ tests, but there is more:
 
 ..  code-block:: shell
 
-    # Execute the unit test suite with PHP 7.4
-    Build/Scripts/runTests.sh -s unit -p 7.4
+    # Execute the unit test suite with PHP 8.3
+    Build/Scripts/runTests.sh -s unit -p 8.3
 
     # Execute some backend acceptance tests
     Build/Scripts/runTests.sh -s acceptance typo3/sysext/core/Tests/Acceptance/Backend/Topbar/
 
-    # Execute some functional tests with PHP 8.0 and postgres DBMS
-    Build/Scripts/runTests.sh -s functional -p 8.0 -d postgres typo3/sysext/core/Tests/Functional/Package/
+    # Execute some functional tests with PHP 8.2 and postgres DBMS
+    Build/Scripts/runTests.sh -s functional -p 8.2 -d postgres typo3/sysext/core/Tests/Functional/Package/
 
     # Execute the cgl fixer
     Build/Scripts/runTests.sh -s cglGit
@@ -199,8 +185,11 @@ tests, but there is more:
 As shown there are various combinations available. Just go ahead, read the help output and play around.
 There are tons of further test suites to try.
 
-One interesting detail should be mentioned: :file:`runTests.sh` uses `typo3gmbh/phpXY <https://hub.docker.com/r/typo3gmbh/>`_
-as main PHP containers. Those are loosely maintained and may be updated. Use the command
+Also note that you can use the `-b` option to switch between `docker` and `podman` container execution,
+with `podman` being the default (when available).
+
+One interesting detail should be mentioned: :file:`runTests.sh` uses several containers from
+`https://github.com/orgs/TYPO3/packages`_ for PHP and JavaScript environments. Use the command
 `Build/Scripts/runTests.sh -u` to fetch the latest versions of these containers.
 
 ..  index::
@@ -216,9 +205,9 @@ To speed up test execution, the PHP extension `xdebug` is not usually loaded.
 However, to allow debugging tests and system under tests, it is possible to
 activate xdebug and send debug output to a local IDE. We'll use PhpStorm for this example.
 
-Let's verify our PhpStorm debug settings first. Go to File > Settings > Languages & Frameworks > PHP
-> Debug. Make sure "Can accept external connections" is enabled, remember the port if it is not the
-default port(9000) and also raise "Max. simultaneous connections" to two or three. Note remote debugging
+Let's verify our PhpStorm debug settings first. Go to :guilabel:`File > Settings > Languages & Frameworks > PHP
+> Debug`. Make sure "Can accept external connections" is enabled, remember the port if it is not the
+default port (9000) and also raise "Max. simultaneous connections" to two or three. Note remote debugging
 may impose a security risk since everyone on the network can send debug streams to your host.
 
 ..  figure:: PhpstormXdebugSettings.png
@@ -243,7 +232,7 @@ stops at this line and opens the debug window.
 
 ..  code-block:: shell
 
-    Build/Scripts/runTests.sh -x -s functional -p 7.4 -d postgres typo3/sysext/core/Tests/Functional/Package/RuntimeActivatedPackagesTest.php
+    Build/Scripts/runTests.sh -x -s functional -p 8.1 -d postgres typo3/sysext/core/Tests/Functional/Package/RuntimeActivatedPackagesTest.php
 
 The important flag here is `-x`! This is available for unit and functional testing. It enables xdebug
 in the PHP container and sends all debug information to port 9000 of the host system. If a local PhpStorm
@@ -256,3 +245,23 @@ Additionally, it may be useful to activate "Break at first line in PHP scripts"
 mounts the local path to the same location within the container, so path mapping is not needed. PhpStorm
 also comes with a `guide <https://www.jetbrains.com/help/phpstorm/configuring-xdebug.html>`_ how to set up
 debugging.
+
+Building
+========
+
+Luckily, `runTests.sh` also helps us to build JavaScript and CSS assets:
+
+..  code-block:: shell
+
+    Build/Scripts/runTests.sh -s buildJavaScript
+    Build/Scripts/runTests.sh -s buildCss
+
+Again, this utilizes all the needed containers for the proper NodeJS environment, so you have
+zero local dependencies on properly building.
+
+You can also run a watch task thanks to the full integration of npm command execution:
+
+..  code-block:: shell
+
+    Build/Scripts/runTests.sh -s npm -- run watch
+

Documentation/Testing/History.rst

@@ -309,3 +309,15 @@ major pre-condition, the broken constant TYPO3_MODE is finally gone (deprecated
 Further core versions can drop that constant and extensions will have to drop their usage, too.
 So with v12, TYPO3_MODE will be gone, and TYPO3 can create cool features from this. So again, the
 core testing paved the way for new opportunities and TYPO3 usages.
+
+2024
+====
+
+TYPO3 currently uses 4 dedicated "bare bone" Servers to perform CI tasks. This hardware performs
+11.513 unit tets in ~15 seconds. A typical pre-merge pipeline runtime is at ~5 minutes with 2 permutations
+of acceptance tests, 3 permutations of ~7500 functional tests, 3 permutations of unit tests plus
+statical code analysis, linting, build checks.
+
+The 4 servers are provisioned using Ansible:
+`https://git.typo3.org/typo3/CI/testing-infrastructure/-/tree/main/ansible?ref_type=heads`__ and
+Pipelines configured in `https://github.com/TYPO3/typo3/tree/main/Build/gitlab-ci/`__.