Add Selenium tests to continuous integration

[RedDragonWebDesign]

There is a ton of technical debt in this legacy code. Enough to keep a full time dev busy for a year upgrading everything.

I pretty much have no time to dedicate to this, which is why the repo has been inactive. There is very little public interest in this repo either, with only 9 stars and like 1 ticket/PR a year.

I think the place to start is to set up some automated tests that test against each PHP version. Then work from the bottom up fixing things broken in each version. So first PHP 7.0, then PHP 7.1, etc. Until the scripts run with no feature breakage and no fatal errors on the current PHP version, PHP 8.3.

Each PHP version should be paired with a version of MySQL that was typical for usage with that PHP version. Since MySQL versions can also cause breaking changes.

• 2015 / PHP 7.0 / MySQL 5.7
• 2016 / PHP 7.1 / MySQL 5.7
• 2017 / PHP 7.2 / MySQL 5.7
• 2018 / PHP 7.3 / MySQL 8.0
• 2019 / PHP 7.4 / MySQL 8.0
• 2020 / PHP 8.0 / MySQL 8.0
• 2021 / PHP 8.1 / MySQL 8.0
• 2022 / PHP 8.2 / MySQL 8.0
• 2023 / PHP 8.3 / MySQL 8.3

Now, what kind of tests? I'm not going to write proper unit and integration tests for this repo. That'd be too much work. We need something that can be set up quickly and detect broad changes. In my opinion, the options are...

• Selenium tests - a type of testing where you program a browser to click around, and assert that certain strings and elements are found. a lot of work to write though, and can be flaky
• pixel tests - a type of testing that takes a screenshot of pages and looks for any visual differences
• HTML snapshot tests - wrap all the entry point files in a function that outputs an HTML string, then call that function in the entry points and in tests. record what strings are "normally" output, and compare those against what is currently output after making code changes

I think trying to fix every deprecation, notice, and warning would be too much work as well. So we should set the test's error reporting to only report errors. We can put off fixing deprecations until the syntax is actually deprecated and causes a breakage.

[RedDragonWebDesign]

After thinking about this for a day, am leaning towards...

• picking one PHP/MySQL version at a time for CI, not multiple versions
  • right now another dev and I are using PHP 8.1 for our localhosts, and BlueThrust isn't as broken as I thought when running that, so that may be a good version to use.
  • we can gradually increment the version over time, once the current version is in good shape in terms of errors and broken features
  • not planning on backwards compatibility. i think we can assume that most web servers have an option to select a modern PHP version. and I want to be able to use modern syntax like ?? in the code
• Selenium (with WebDriverIO framework)
  • should visit every page signed in
  • should visit every page signed out
  • should specifically test some features and happy paths
    • installer
    • forum
    • log in
    • log out
    • the console has some good groupings of pages/links that make sense. each grouping should get a test that checks its major features

[deepend-tildeclub]

This seems like a solid plan. My server is currently running PHP 8.1, specifically version 8.1.20. We could consider upgrading to PHP 8.2 to extend the support for PHP updates. It's worth noting that security updates for PHP 8.1 are scheduled to end in November, although this may vary depending on the distribution's support.

[RedDragonWebDesign]

• install Selenium framework WebDriverIO (c8c9fb8). so far I like it way better than just Selenium since it handles installing the correct Chrome WebDriver version for you, and also forces you to organize the files well
• get WebDriverIO working in continuous integration (attempted in d2ca4f8, needs debugging)
  • hard code the PHP version to 8.1 and MySQL version to 8.0 for now
• tests to add
  • PHP error detection (set error_reporting to maximum including errors/warnings/notices/deprecations. once a page is cleared of errors, add it to a list of clean pages to test so no errors ever come back) (maybe visit the pages, then at the end of that test, check the PHP error log and fail if it's not blank)
    • while logged out for X pages
    • while logged in for X pages
  • test the installer
  • test logging in
  • test logging out
  • test forum (viewing board, making thread, replying to thread)

[deepend-tildeclub]

• install Selenium framework WebDriverIO (c8c9fb8). so far I like it way better than just Selenium since it handles installing the correct Chrome WebDriver version for you, and also forces you to organize the files well
• get WebDriverIO working in continuous integration (attempted in d2ca4f8, needs debugging)

Guessing an issue with your WebDriverIO is that your wdio-github.conf.js is using CommonJS syntax. while your package.json suggests your setup to use ES Modules.