[Backend Tester] Add CSV report generation #12741

GregoryComer · 2025-07-23T01:28:29Z

Add some initial CSV report generation, detailing results and parameters for each individual test. Delegation statistics and such will come next. I've also added a basic test for the report generation, which I will expand upon in this stack.

Here's some sample output from running add tests for XNNPACK:

Test ID,Test Case,Backend,Flow,Result,Dtype
test_add_dtype_float32_xnnpack,test_add_dtype,xnnpack,xnnpack,Success (Delegated),torch.float32
test_add_dtype_float32_xnnpack_static_int8,test_add_dtype,xnnpack,xnnpack_static_int8,Success (Delegated),torch.float32
test_add_f32_alpha_xnnpack,test_add_f32_alpha,xnnpack,xnnpack,Fail (Quantize),
test_add_f32_alpha_xnnpack_static_int8,test_add_f32_alpha,xnnpack,xnnpack_static_int8,Fail (Quantize),
test_add_f32_bcast_first_xnnpack,test_add_f32_bcast_first,xnnpack,xnnpack,Success (Delegated),
test_add_f32_bcast_first_xnnpack_static_int8,test_add_f32_bcast_first,xnnpack,xnnpack_static_int8,Success (Delegated),
test_add_f32_bcast_second_xnnpack,test_add_f32_bcast_second,xnnpack,xnnpack,Success (Delegated),
test_add_f32_bcast_second_xnnpack_static_int8,test_add_f32_bcast_second,xnnpack,xnnpack_static_int8,Success (Delegated),
test_add_f32_bcast_unary_xnnpack,test_add_f32_bcast_unary,xnnpack,xnnpack,Success (Delegated),
test_add_f32_bcast_unary_xnnpack_static_int8,test_add_f32_bcast_unary,xnnpack,xnnpack_static_int8,Success (Delegated),

[ghstack-poisoned]

GregoryComer · 2025-07-23T01:28:30Z

Stack from ghstack (oldest at bottom):

pytorch-bot · 2025-07-23T01:28:32Z

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/12741

📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

❌ 1 New Failure, 2 Pending

As of commit c3a24f9 with merge base 3a02146 ():

NEW FAILURE - The following job has failed:

Build documentation / build (buck2) / Build doc (gh)
At least one of the pre-conditions you specified did not hold

This comment was automatically generated by Dr. CI and updates every 15 minutes.

ghstack-source-id: ec13121 ghstack-comment-id: 3105325555 Pull-Request: #12741

digantdesai · 2025-07-23T15:53:57Z

backends/test/suite/tests/test_reporting.py

@@ -0,0 +1,101 @@
+import torch


digantdesai · 2025-07-23T15:54:47Z

backends/test/suite/tests/test_reporting.py

+]
+
+class Reporting(unittest.TestCase):
+    def test_csv_report_simple(self):


Nit: tsv works better usually

I don't have a strong opinion on this, personally. Let's chat tomorrow. I'm good to switch it to TSV if you think it would be better.

[ghstack-poisoned]

ghstack-source-id: f67c54b ghstack-comment-id: 3105325555 Pull-Request: #12741

[ghstack-poisoned]

ghstack-source-id: 058e3ca ghstack-comment-id: 3105325555 Pull-Request: #12741

[ghstack-poisoned]

ghstack-source-id: 8dd5a49 ghstack-comment-id: 3105325555 Pull-Request: #12741

[ghstack-poisoned]

ghstack-source-id: 4f76de1 ghstack-comment-id: 3105325555 Pull-Request: #12741

[ghstack-poisoned]

ghstack-source-id: 17e6147 ghstack-comment-id: 3105325555 Pull-Request: #12741

[ghstack-poisoned]

ghstack-source-id: 6264a78 ghstack-comment-id: 3105325555 Pull-Request: pytorch#12741

[ghstack-poisoned]

Update

19760fc

[ghstack-poisoned]

GregoryComer requested a review from cccclai as a code owner July 23, 2025 01:28

facebook-github-bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Jul 23, 2025

GregoryComer added a commit that referenced this pull request Jul 23, 2025

[Backend Tester] Add CSV report generation

391b923

ghstack-source-id: ec13121 ghstack-comment-id: 3105325555 Pull-Request: #12741

GregoryComer requested a review from digantdesai July 23, 2025 01:30

digantdesai reviewed Jul 23, 2025

View reviewed changes

backends/test/suite/tests/test_reporting.py Outdated

@@ -0,0 +1,101 @@

import torch

Copy link

Contributor

digantdesai Jul 23, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you!

digantdesai reviewed Jul 23, 2025

View reviewed changes

digantdesai approved these changes Jul 23, 2025

View reviewed changes

Update

5d437b1

[ghstack-poisoned]

GregoryComer added a commit that referenced this pull request Jul 23, 2025

[Backend Tester] Add CSV report generation

60fc879

ghstack-source-id: f67c54b ghstack-comment-id: 3105325555 Pull-Request: #12741

Update

e2ea2a3

[ghstack-poisoned]

GregoryComer added a commit that referenced this pull request Jul 23, 2025

[Backend Tester] Add CSV report generation

af471be

ghstack-source-id: 058e3ca ghstack-comment-id: 3105325555 Pull-Request: #12741

Update

3976629

[ghstack-poisoned]

GregoryComer added a commit that referenced this pull request Jul 23, 2025

[Backend Tester] Add CSV report generation

21208f3

ghstack-source-id: 8dd5a49 ghstack-comment-id: 3105325555 Pull-Request: #12741

Update

cdd15c1

[ghstack-poisoned]

GregoryComer added a commit that referenced this pull request Jul 23, 2025

[Backend Tester] Add CSV report generation

625c942

ghstack-source-id: 4f76de1 ghstack-comment-id: 3105325555 Pull-Request: #12741

Update

bcb697c

[ghstack-poisoned]

GregoryComer added a commit that referenced this pull request Jul 23, 2025

[Backend Tester] Add CSV report generation

42f383f

ghstack-source-id: 17e6147 ghstack-comment-id: 3105325555 Pull-Request: #12741

Update

de21ac2

[ghstack-poisoned]

GregoryComer mentioned this pull request Jul 24, 2025

[Backend Tester] Add tensor error statistic reporting #12809

Open

GregoryComer added 2 commits July 24, 2025 10:20

Update

32f54b0

[ghstack-poisoned]

Update

5cc4941

[ghstack-poisoned]

This was referenced Jul 24, 2025

[Backend Tester] Report quantization and lowering times #12838

Open

[Backend Tester] Report delegation statistics #12846

Open

GregoryComer added a commit to GregoryComer/executorch that referenced this pull request Jul 31, 2025

[Backend Tester] Add CSV report generation

a006ab2

ghstack-source-id: 6264a78 ghstack-comment-id: 3105325555 Pull-Request: pytorch#12741

Update

f1db3a0

[ghstack-poisoned]

This was referenced Aug 8, 2025

[Backend Tester] Report PTE size #13249

Open

[Backend Tester] Add portable test flow #13250

Open

[Backend Tester] Clean up a few test issues #13258

Draft

Update

c3a24f9

[ghstack-poisoned]

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[Backend Tester] Add CSV report generation #12741

[Backend Tester] Add CSV report generation #12741

GregoryComer commented Jul 23, 2025 •

edited

Loading

Uh oh!

GregoryComer commented Jul 23, 2025 •

edited

Loading

Uh oh!

pytorch-bot bot commented Jul 23, 2025 •

edited

Loading

Uh oh!

digantdesai Jul 23, 2025

Uh oh!

digantdesai Jul 23, 2025

Uh oh!

GregoryComer Aug 11, 2025

Uh oh!

Uh oh!

[Backend Tester] Add CSV report generation #12741

Are you sure you want to change the base?

[Backend Tester] Add CSV report generation #12741

Conversation

GregoryComer commented Jul 23, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

GregoryComer commented Jul 23, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pytorch-bot bot commented Jul 23, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/12741

❌ 1 New Failure, 2 Pending

Uh oh!

digantdesai Jul 23, 2025

Choose a reason for hiding this comment

Uh oh!

digantdesai Jul 23, 2025

Choose a reason for hiding this comment

Uh oh!

GregoryComer Aug 11, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

GregoryComer commented Jul 23, 2025 •

edited

Loading

GregoryComer commented Jul 23, 2025 •

edited

Loading

pytorch-bot bot commented Jul 23, 2025 •

edited

Loading