Enhancement: Merge summary reports from shards

[Vivek-Kini]

First of all, thanks a lot for getting this summary. Its very useful snapshot after the gha completes Test execution for a quick view.

We usually use the test matrix to run our tests and hence we have different shards within the GHA CI workflow. Is there a possibility where we can merge these different sections of reporting to be merged into one consolidated view report so that one github summary section covers everything?

[estruyf]

If you shard your tests, you should be able to collect all shards and run the merge-reports command.

npx playwright merge-reports --config=merge.config.ts ./blob-reports

The config (merge.config.ts):

import type { GitHubActionOptions } from "@estruyf/github-actions-reporter";

export default {
  testDir: "./tests",
  reporter: [
    ['@estruyf/github-actions-reporter', <GitHubActionOptions>{
      title: 'All test results',
      showError: true
    }],
  ],
};

[Vivek-Kini]

Thanks, I do merge reports and then have a final html report available to analyze from blobs(generated by shards).

However, this reporter (gha reporter) does take the results to display on GitHub summary after the individual shard's execute and not on merged final report.

[estruyf]

@Vivek-Kini that is why you should create a separate configuration which you use for the merge-reports command. It should be handled by Playwright that way.

How are you currently doing this for the HTML report?

[Vivek-Kini]

@Vivek-Kini that is why you should create a separate configuration which you use for the merge-reports command. It should be handled by Playwright that way.

How are you currently doing this for the HTML report?

currently, we have setup for each shards to create blob reports and then after all the shards are completed, I create one HTML report after merging. However, gha reporter creates summary based on the test execution on shards.. so I see the individual summaries.

Rather, if there's a specific input file which can be provided to gha reporter which can provide summary would be helpful.

[ajworkos]

Sorry to hijack, but kind of related

@estruyf when I run this exact command with the same configuration

the command succeeds, but I don't get a github comment. Is there any other configuration that I need to add?