Support Multiple Submissions from Hosts: Research Runner and Reporter Codebase

[kittenkamala]

Research the existing code base and configuration for reporting test results to identify all the specific points that need to be updated in the reporter and runner code. Document that here to help facilitate a solution for supporting multiple test submissions from hosts.

[kittenkamala]

https://github.com/WordPress/phpunit-test-reporter/blob/master/src/class-restapi.php#L116

[kittenkamala]

result-set.php makes a table / template that test results get pushed to

result data is being iterated through here: https://github.com/WordPress/phpunit-test-reporter/blob/master/parts/result-set.php#L34

from the result data the post author is being grabbed here: https://github.com/WordPress/phpunit-test-reporter/blob/master/parts/result-set.php#L43

[kittenkamala]

In the runner, results get put in an array, json encoded and sent to the api:

https://github.com/WordPress/phpunit-test-runner/blob/master/functions.php#L106

[pfefferle]

What about starting with a simple env var like export WPT_HOSTING_PACKAGE=Cloud Hosting? I think it is no problem to have multiple runners for multiple hosting packages.

[getsource]

@pfefferle Thanks for the note! I think that makes sense. I like that it would make it really easy for hosts to add/specify another environment.

Agreed it isn't a problem! What do you think should be the next step?

On getting it to the reporter, we could roll it up into $env, a different existing field, or add a new one.

Some details on the $env option:
$env is curated information in JSON about the environment where tests are run. It's currently generated in a couple of places. It doesn't contain any info from the.env file, and because it is sometimes run on a different machine, that'd probably have to be injected somehow.

[pfefferle]

Semantically $env would be perfect. get_env_details() would be a good place to start, but then it will be ignored for the env.json file? Perhaps the reporter.php should add the info: https://github.com/WordPress/phpunit-test-runner/blob/67347895ad59d94abec722c69753a8184daa4aa8/report.php#L47

[pfefferle]

What speaks against using $env is, that it is a very dominant parameter, that will also be used to group and filter the output (a hoster will be represented with one, two or more item types in a revision segment, with the name of the hosting type), so it might be better to use a more prominent field?!