Dev: Ensure tests are run in random

[paulbalandan]

PHP Version

8.5.0

CodeIgniter4 Version

4.7.1-dev

Per ChatGPT, tests need to run in random to enjoy the following benefits:

✔️ It reveals hidden dependencies
✔️ Forces true test isolation
✔️ Prevents flaky, misleading test results
✔️ Makes your test suite production-realistic

This issue has been initially tackled by #7717, but due to other complications it didn't fare well. As this is not a fairly easy task, I'm proposing we do it on pieces. Currently, our test suite runs in default order but should be random at some point (i.e., executionOrder="random").

Here's the plan, instead of one big PR changing all tests, we can do it by component (HTTP, Cache, Database, etc.). Then, after all tests run green on randomised testing, we set executionOrder in phpunit.xml to "random".

Please reference this issue for any PRs addressing pieces of this.

• API: refactor: make random-order API tests deterministic #9983
• AutoReview: test: indicate components that already pass random execution tests #10073
• Autoloader: test: indicate components that already pass random execution tests #10073
• Cache
• CLI:
  • refactor: make random-order CLI tests deterministic #9998
  • refactor: fix dependency on test execution order #10014
• Commands
• Config
• Cookie: test: indicate components that already pass random execution tests #10073
• Database
• DataCaster
• DataConverter
• Debug
• Email
• Encryption
• Entity
• Events: test: indicate components that already pass random execution tests #10073
• Files: test: indicate components that already pass random execution tests #10073
• Filters
• Format: test: indicate components that already pass random execution tests #10073
• Helpers
• Honeypot
• HotReloader: test: indicate components that already pass random execution tests #10073
• HTTP
• I18n
• Images
• Language: test: indicate components that already pass random execution tests #10073
• Log: test: indicate components that already pass random execution tests #10073
• Pager: test: indicate components that already pass random execution tests #10073
• Publisher: test: indicate components that already pass random execution tests #10073
• RESTful: test: indicate components that already pass random execution tests #10073
• Router
  • test: fix determinism of CacheTest #9967
• Security
  • refactor: fix Security test fail by itself #9969
  • test: indicate components that already pass random execution tests #10073
• Session
• Test
• Throttle: test: indicate components that already pass random execution tests #10073
• Typography: test: indicate components that already pass random execution tests #10073
• Validation
• View: test: indicate components that already pass random execution tests #10073

[neznaika0]

I also mentioned the randomization of tests, the answer was the same - it takes a lot of time.

I can't imagine how to edit Database tests, this is the most difficult stage.

[paulbalandan]

I also think randomising database tests would be the most difficult piece.