Skip to content

[Backend Tester] Add CSV report generation #12741

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 71 commits into from
Aug 12, 2025
Merged

[Backend Tester] Add CSV report generation #12741

merged 71 commits into from
Aug 12, 2025
+322 −15

Conversation

GregoryComer
Copy link
Member

@GregoryComer GregoryComer commented Jul 23, 2025 •
edited
Loading

Add some initial CSV report generation, detailing results and parameters for each individual test. Delegation statistics and such will come next. I've also added a basic test for the report generation, which I will expand upon in this stack.

Here's some sample output from running add tests for XNNPACK:

Test ID,Test Case,Backend,Flow,Result,Dtype
test_add_dtype_float32_xnnpack,test_add_dtype,xnnpack,xnnpack,Success (Delegated),torch.float32
test_add_dtype_float32_xnnpack_static_int8,test_add_dtype,xnnpack,xnnpack_static_int8,Success (Delegated),torch.float32
test_add_f32_alpha_xnnpack,test_add_f32_alpha,xnnpack,xnnpack,Fail (Quantize),
test_add_f32_alpha_xnnpack_static_int8,test_add_f32_alpha,xnnpack,xnnpack_static_int8,Fail (Quantize),
test_add_f32_bcast_first_xnnpack,test_add_f32_bcast_first,xnnpack,xnnpack,Success (Delegated),
test_add_f32_bcast_first_xnnpack_static_int8,test_add_f32_bcast_first,xnnpack,xnnpack_static_int8,Success (Delegated),
test_add_f32_bcast_second_xnnpack,test_add_f32_bcast_second,xnnpack,xnnpack,Success (Delegated),
test_add_f32_bcast_second_xnnpack_static_int8,test_add_f32_bcast_second,xnnpack,xnnpack_static_int8,Success (Delegated),
test_add_f32_bcast_unary_xnnpack,test_add_f32_bcast_unary,xnnpack,xnnpack,Success (Delegated),
test_add_f32_bcast_unary_xnnpack_static_int8,test_add_f32_bcast_unary,xnnpack,xnnpack_static_int8,Success (Delegated),

@GregoryComer
Update
f120e70 
[ghstack-poisoned]
@GregoryComer
Update
0fb85e6 
[ghstack-poisoned]
@GregoryComer
Update
4d8d844 
[ghstack-poisoned]
@GregoryComer
Update
dc12b40 
[ghstack-poisoned]
@GregoryComer
Update
ead0616 
[ghstack-poisoned]
@GregoryComer
Update
0f13676 
[ghstack-poisoned]
@GregoryComer
Update
b0b01f2 
[ghstack-poisoned]
@GregoryComer
Update
8b9c9ef 
[ghstack-poisoned]
@GregoryComer
Update
06bf03a 
[ghstack-poisoned]
@GregoryComer
Update
2f8f49b 
[ghstack-poisoned]
@GregoryComer
Update
8ca7766 
[ghstack-poisoned]
@GregoryComer
Update
bffb95f 
[ghstack-poisoned]
@GregoryComer
Update
d21492b 
[ghstack-poisoned]
@GregoryComer
Update
e2c4ea5 
[ghstack-poisoned]
@GregoryComer
Update
8230848 
[ghstack-poisoned]
@GregoryComer
Update
2a1f564 
[ghstack-poisoned]
@GregoryComer
Update
b35e7b1 
[ghstack-poisoned]
@GregoryComer
Update
5c4c6ce 
[ghstack-poisoned]
@GregoryComer
Update
9397803 
[ghstack-poisoned]
@GregoryComer
Update
9dfeb5a 
[ghstack-poisoned]
@GregoryComer
Update
ff5c4a5 
[ghstack-poisoned]
@GregoryComer
Update
42a5de5 
[ghstack-poisoned]
@GregoryComer
Update
402d8f5 
[ghstack-poisoned]
@GregoryComer
Update
34d3ab3 
[ghstack-poisoned]
@GregoryComer
Update
1105e04 
[ghstack-poisoned]
@GregoryComer
Update
482bd21 
[ghstack-poisoned]
@GregoryComer
Update
ea548b7 
[ghstack-poisoned]
@GregoryComer
Update
4108f54 
[ghstack-poisoned]
@GregoryComer
Update
7ef236b 
[ghstack-poisoned]
@GregoryComer
Update
4a58c9d 
[ghstack-poisoned]
@GregoryComer
Update
bcb697c 
[ghstack-poisoned]
GregoryComer added a commit that referenced this pull request Jul 23, 2025
@GregoryComer
[Backend Tester] Add CSV report generation
42f383f 
ghstack-source-id: 17e6147
ghstack-comment-id: 3105325555
Pull-Request: #12741
@GregoryComer
Update
11a5a02 
[ghstack-poisoned]
@GregoryComer
Update
244b146 
[ghstack-poisoned]
@GregoryComer
Update
de21ac2 
[ghstack-poisoned]
@GregoryComer GregoryComer mentioned this pull request Jul 24, 2025
Open
@GregoryComer
Update
4ae840d 
[ghstack-poisoned]
@GregoryComer
Update
710ea49 
[ghstack-poisoned]
@GregoryComer
Update
32f54b0 
[ghstack-poisoned]
@GregoryComer
Update
2eb59fc 
[ghstack-poisoned]
@GregoryComer
Update
5cc4941 
[ghstack-poisoned]
This was referenced Jul 24, 2025
Open
Open
GregoryComer added a commit to GregoryComer/executorch that referenced this pull request Jul 31, 2025
@GregoryComer
[Backend Tester] Add CSV report generation
a006ab2 
ghstack-source-id: 6264a78
ghstack-comment-id: 3105325555
Pull-Request: pytorch#12741
@GregoryComer
Update
dd09555 
[ghstack-poisoned]
@GregoryComer
Update
f1db3a0 
[ghstack-poisoned]
This was referenced Aug 8, 2025
Open
Open
Open
@GregoryComer
Update
f261355 
[ghstack-poisoned]
@GregoryComer
Update
c3a24f9 
[ghstack-poisoned]
This was referenced Aug 12, 2025
Open
Open
Open
Open
Open
Open
Base automatically changed from gh/GregoryComer/87/head to main August 12, 2025 16:41
@GregoryComer GregoryComer merged commit 923c1b4 into main Aug 12, 2025
102 of 198 checks passed
@GregoryComer GregoryComer deleted the gh/GregoryComer/88/head branch August 12, 2025 16:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@digantdesai digantdesai digantdesai approved these changes

@cccclai cccclai Awaiting requested review from cccclai cccclai is a code owner

Assignees
No one assigned
Labels
CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed.
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@GregoryComer @digantdesai @facebook-github-bot