(Help required) Tests for nextest partitions#274
Open
TriplEight wants to merge 39 commits intotaiki-e:mainfrom
Open
(Help required) Tests for nextest partitions#274TriplEight wants to merge 39 commits intotaiki-e:mainfrom
TriplEight wants to merge 39 commits intotaiki-e:mainfrom
Conversation
taiki-e
reviewed
May 6, 2023
Owner
taiki-e
left a comment
There was a problem hiding this comment.
Thanks for the PR! I haven't looked at the details yet, but left a comment about the CI issue for now.
Co-authored-by: Taiki Endo <te316e89@gmail.com>
Author
|
@taiki-e I've a question: when the tests are done on every partition, it runs By far I've managed to merge them as follows: But then |
a820cec to
59cba33
Compare
2a48173 to
6f4700a
Compare
932a369 to
67a45bf
Compare
a648a60 to
0815d6a
Compare
69bcef2 to
566baff
Compare
1436eb2 to
a23e1dd
Compare
e172f19 to
93840f8
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
cc #271 (comment)
Here are two sample integration tests:
It shows that the coverage counted in partitions is not full. And can be a good example of how to do it.
The first step is to run tests and partition coverage (reports are used here only to illustrate that coverage on partitions should not be expected to be full). Then it archives the coverage data (@taiki-e please check if it's only the needed data, I left some debug output for that) to pass it to...
The second step. It merges and generates the coverage. I don't know if this is done correctly, please check.
It might also be a good idea to check this against some fixtures.
--archive-file.This is to show that nextest archive when run with `--partition' and llvm-cov partial reports belong to the different stages of CI. And also serves as an example of how to do this.
The first step just runs nextest to create an archive and pass it to the partitions.
The second step is to partition the tests and run coverage like 2/2.
The third step just runs llvm-cov to merge the coverage from the partitions and create a report.
However, I couldn't get 2/3 to work. There's a lot of comments on my results.
I suspect that #265 should also address the
--extract-toflag, maybe the problem is that (#271 (comment)) llvm-cov setstarget/llvm-cov-targetas a default and https://github.com/taiki-e/cargo-llvm-cov/pull/266/files#diff-36dcce9bb960d87a9efde38a0309b8e3e552a4193594bb2065ced1e749c9b743R251 uses it to extract nextest artifacts too. See the comments for details.Please feel free to contribute to this PR, it's open.
You can see the previous CI runs in TriplEight#2
When this is resolved, I'll write some documentation to https://nexte.st/book/test-coverage.html
cc @sunshowers, your help would be greatly appreciated.