Automatic regression testing

[PatrickvL]

It would be very valuable if the kernel tests could be ran as part of a build process, so that projects like Cxbx-Reloaded, xqemu, OpenXbox or other parties can somehow run these kernel tests, and compare it against a reference output, so that regressions can be spotted directly after making changes.

[Luca1991]

The problem here is that Travis (our CI) is running Linux, and CxBx-Reloaded is currently Windows only.
It might be possible to install wine, running tests and the upload the results, but this will be very hacky.

[PatrickvL]

Ouch, that could prove difficult. But what if we'd do it the other way around - let the Cxbx-Reloaded AppVeyor build run the most recent kernel test suite?

[Luca1991]

Yeah, this might be possible. It should be possible to expand the appveyor build script to download the latest build of the kernel test suite and run it just after compiling cxbx.

If I'm not wrong, the code written by @Fisherman166 avoid the creation of the log file when running on emu. This is an issue in this case, as we need to create the log file regardless of our running platform. I might be wrong on this... I'll check tomorrow.

[Fisherman166]

It is true that only the real hardware creates a logfile. Attempting to run the same code that works on the Xbox hardware on Cxbx-Reloaded will cause errors, likely because the virtual file system in the Cxbx kernel is not fully implemented. Until the Cxbx file system is more robust, I don't think there is a better way to get output from the kernel test suite than redirecting Cxbx kernel output to a krnldebug.txt file.

[RadWolfie]

Using xqemu or any emulators that require kernel bios will require self-hosted CI. Any cloud service to run the test with required kernel bios are likely not permitted. But I'm not a lawyer. 🤷‍♂️