test(core): merge all package test reports #6261
Conversation
Problem: - CI test reporting only captures the last package's results - Running `npm run testE2E` only preserves toolkit's report.xml, losing other package results Solution: - Generate individual report.xml files per subproject - Consolidate all existing package reports into root .test-reports/report.xml
|
/runIntegrationTests |
|
|
/runIntegrationTests |
|
What are the tradeoffs for merging the report versus having one per extension? Curious because for code coverage we have separate ones. |
|
If you don't merge them you can't see them in the ui here: https://d1ihu6zq92vp9p.cloudfront.net/487310e6-ebfc-4fef-a0b1-b441147b3c23/report.html. That's really the issue i'm trying to solve. It's pretty annoying when an amazon q test fails and then you have to manually look through the logs to find the problem I don't think that restriction applies to codecov since you can upload multiple different reports at the same time |
|
/runIntegrationTests |
jpinkney-aws
force-pushed
the
merge-reports
branch
from
e312707 to
924b90e
Compare
|
/runIntegrationTests |
|
/runIntegrationTests |
1 similar comment
|
/runIntegrationTests |
jpinkney-aws
force-pushed
the
merge-reports
branch
from
f7570f4 to
5d3ade7
Compare
| . buildspec/shared/common.sh | ||
| 2>&1 xvfb-run npm test --silent | run_and_report 2 \ | ||
| 2>&1 xvfb-run npm test --silent; npm run mergeReports -- "$?" | run_and_report 2 \ | ||
| 'rejected promise not handled' \ |
There was a problem hiding this comment.
Based on a local test of this command:
2>&1 echo aaa; echo bbb | grep x
I suspect the above ; may not correctly pipe all the output from npm test to run_and_report as intended.
To confirm that the piping to run_and_report is still working correctly, can you try temporarily changing this search patter to something that always outputs, like:
| 'rejected promise not handled' \ | |
| 'deleteTestTempDirs' \ |
then CI should fail.
If it doesn't, we can fix the issue by changing the pipeline to something like (note: whitespace near the braces is required):
{ 2>&1 xvfb-run npm test --silent; npm run mergeReports -- "$?"; } | run_and_report 2 \
There was a problem hiding this comment.
Just tested and that was indeed the case. I've switched to your suggestion and it seems to work as expected
|
/runIntegrationTests |
|
/runIntegrationTests |
## Problem - CI test reporting only captures the last package's results - Running `npm run testE2E` only preserves toolkit's report.xml, losing other package results ## Solution - Generate individual report.xml files per subproject - Consolidate all existing package reports into root .test-reports/report.xml - This is done in the buildspecs themselves rather than directly in the package.json (the original approach to solve this problem) because mac/linux and windows have two different ways of getting the last error code, resulting in a complicated package.json for something thats only needed in our codebuilds --- - Treat all work as PUBLIC. Private `feature/x` branches will not be squash-merged at release time. - Your code changes must meet the guidelines in [CONTRIBUTING.md](https://github.com/aws/aws-toolkit-vscode/blob/master/CONTRIBUTING.md#guidelines). License: I confirm that my contribution is made under the terms of the Apache 2.0 license.
## Problem - CI test reporting only captures the last package's results - Running `npm run testE2E` only preserves toolkit's report.xml, losing other package results ## Solution - Generate individual report.xml files per subproject - Consolidate all existing package reports into root .test-reports/report.xml - This is done in the buildspecs themselves rather than directly in the package.json (the original approach to solve this problem) because mac/linux and windows have two different ways of getting the last error code, resulting in a complicated package.json for something thats only needed in our codebuilds --- - Treat all work as PUBLIC. Private `feature/x` branches will not be squash-merged at release time. - Your code changes must meet the guidelines in [CONTRIBUTING.md](https://github.com/aws/aws-toolkit-vscode/blob/master/CONTRIBUTING.md#guidelines). License: I confirm that my contribution is made under the terms of the Apache 2.0 license.
3 participants
Problem
npm run testE2Eonly preserves toolkit's report.xml, losing other package resultsSolution
feature/xbranches will not be squash-merged at release time.License: I confirm that my contribution is made under the terms of the Apache 2.0 license.