Split CI test execution into parallel jobs (OSOE-730)

[Piedone]

This is something for the future, when we'll have a lot more (UI) tests, I just wanted to record the idea.

Currently, the Tests step in the CI build takes 8-10 minutes under Ubuntu. We have 71 UI tests that take up the bulk of this time. Once we'll have a lot more and, we've exhausted other speed-up options, and we want to make this step faster, we could split the Tests step into two or more parallel jobs. I'd only do this for the Ubuntu build-and-test-larger-runners job.

This requires us to have at least 3 jobs: One that does everything up to Build and Static Code Analysis. Then, it'd upload the build artifacts. Two other (parallel) jobs then would be triggered, download the artifacts, and execute the tests.

We need to employ Cancel Workflow on Failure for each of these jobs still.

This would allow us to have two separate VMs running the tests in parallel.

Jira issue

[Piedone]

We could also split test execution to use multiple GitHub-hosted free (4-core) Ubuntu instances and get roughly the same workflow runtime (or perhaps even better, since we can have any number of instances, so theoretically the shortest test execution needs to be as long as the longest-running test).

The challenges I see here:

• How to determine which tests to keep together. The goal would be to have each job run for roughly the same time, and ideally, this would be managed automatically. A primitive first approach would be to split the two UI tests projects into two jobs (and do the unit tests together with the longer one).
• Artifacts, workflow telemetry, the test report will all be split too. Is this an issue?

[Piedone]

We'd need a self-adjusting mechanism here, that'll run roughly the same amount of tests on each runner (ideally, this would take into account how long each tests takes to balance them, but for now, just running the same number of tests seems good enough). Then, we'd only configure the number of runners, or even better, the maximum number of tests on a runner (and then the workflow would spin up as many runners as necessary).

This can be achieved with a dynamically generated JSON matrix value. See e.g.:

• https://medium.com/@luuk-mes/how-to-run-dynamic-github-action-workflows-35692957ef94
• https://developer-friendly.blog/blog/2024/03/09/github-actions-dynamic-matrix/#step-3-build-the-docker-images
• https://frontside.com/blog/2022-12-12-dynamic-github-action-jobs/