[Backend Tester] Add subtest index field #13311

GregoryComer · 2025-08-12T04:57:15Z

Some test functions run multiple test cases. Add a subtest index field to disambiguate these in the report.

[ghstack-poisoned]

GregoryComer · 2025-08-12T04:57:16Z

Stack from ghstack (oldest at bottom):

pytorch-bot · 2025-08-12T04:57:19Z

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/13311

📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

❌ 8 New Failures, 2 Unrelated Failures

As of commit 88d1dd3 with merge base d7ecd87 ():

NEW FAILURES - The following jobs have failed:

Build documentation / build (buck2) / Build doc (gh)
At least one of the pre-conditions you specified did not hold
Lint / lintrunner / linux-job (gh)
>>> Lint for backends/test/suite/models/__init__.py:
pull / unittest-arm-backend-with-no-fvp (test_pytest_ops) / linux-job (gh)
RuntimeError: Command docker exec -t 161b6591a7e9db37989718832530d9d26dde7006182fb0a4452422dd03accc55 /exec failed with exit code 1
trunk / test-arm-backend (test_pytest_ops_ethosu_fvp) / linux-job (gh)
RuntimeError: Command docker exec -t c8467145171c6be14e2d050d4003f268981594607e6afb541dbdfbe4caeda7d5 /exec failed with exit code 1
trunk / test-arm-backend (test_smaller_stories_llama) / linux-job (gh)
RuntimeError: Command docker exec -t b25e4a424557065286d8b8081849ff7a2e3549eb2b29b22fc2d1665f6187a917 /exec failed with exit code 1
trunk / test-models-arm-zephyr (add) / linux-job (gh)
RuntimeError: Command docker exec -t f3086640528d25a6c3dee23f3755a11313750e609c592d0d18ca914e7669d53d /exec failed with exit code 1
trunk / test-models-arm-zephyr (mv2) / linux-job (gh)
RuntimeError: Command docker exec -t f11196dfdcec35967b250a14c4a8848a49b4326772f7c28733e7b2e6cc9a5bd2 /exec failed with exit code 1
trunk / test-models-arm-zephyr (softmax) / linux-job (gh)
RuntimeError: Command docker exec -t 9a499bf9d7f0e6a23c89110b35444ae1347037a827496a0f80280cccd2dcfae5 /exec failed with exit code 1

BROKEN TRUNK - The following jobs failed but were present on the merge base:

👉 Rebase onto the `viable/strict` branch to avoid these failures

pull / unittest-arm-backend-with-no-fvp (test_pytest_models) / linux-job (gh) (trunk failure)
backends/arm/test/models/stable_diffusion/test_vae_AutoencoderKL.py::TestAutoencoderKL::test_AutoencoderKL_tosa_MI
trunk / test-arm-backend (test_pytest_models_ethosu_fvp) / linux-job (gh) (trunk failure)
backends/arm/test/models/stable_diffusion/test_vae_AutoencoderKL.py::TestAutoencoderKL::test_AutoencoderKL_tosa_MI

This comment was automatically generated by Dr. CI and updates every 15 minutes.

ghstack-source-id: e22a46b ghstack-comment-id: 3177697272 Pull-Request: #13311

[ghstack-poisoned]

digantdesai · 2025-08-12T11:48:53Z

backends/test/suite/models/__init__.py

@@ -119,6 +119,7 @@ def run_model_test(
        flow,
        context.test_name,
        context.test_base_name,
+        0, # subtest_index - currently unused for model tests


So the hierarchy is
Test ID -> test -> subtest?

The naming is a bit confusing. I'm open to suggestions. The test_base_name is the test name (test_add_dtype, for example). Test name adds flow and parameterization (test_add_dtype_float32_xnnpack). The subtest index corresponds to a single python unittest function that runs multiple test cases.

[ghstack-poisoned]

GregoryComer added 2 commits August 11, 2025 21:12

Update

2c4488f

[ghstack-poisoned]

Update

491ec2b

[ghstack-poisoned]

GregoryComer added a commit that referenced this pull request Aug 12, 2025

[Backend Tester] Add subtest index field

ef0f7bd

ghstack-source-id: e22a46b ghstack-comment-id: 3177697272 Pull-Request: #13311

meta-cla bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Aug 12, 2025

GregoryComer requested a review from digantdesai August 12, 2025 04:59

GregoryComer mentioned this pull request Aug 12, 2025

[Backend Tester] Reduce log verbosity / spam #13312

Open

GregoryComer added 6 commits August 11, 2025 22:24

Update

fc1c2a7

[ghstack-poisoned]

Update

285e058

[ghstack-poisoned]

Update

605a0ad

[ghstack-poisoned]

Update

90ce443

[ghstack-poisoned]

Update

1b9315e

[ghstack-poisoned]

Update

94d89c4

[ghstack-poisoned]

GregoryComer mentioned this pull request Aug 12, 2025

[Backend Tester] Seed based on test name #13313

Open

GregoryComer marked this pull request as ready for review August 12, 2025 06:12

GregoryComer requested a review from cccclai as a code owner August 12, 2025 06:12

GregoryComer added 2 commits August 11, 2025 23:19

Update

1e3e79c

[ghstack-poisoned]

Update

1c136d7

[ghstack-poisoned]

digantdesai reviewed Aug 12, 2025

View reviewed changes

digantdesai approved these changes Aug 12, 2025

View reviewed changes

GregoryComer added 11 commits August 12, 2025 11:26

Update

dd30127

[ghstack-poisoned]

Update

4216805

[ghstack-poisoned]

Update

49944fa

[ghstack-poisoned]

Update

60f3932

[ghstack-poisoned]

Update

f4df9c9

[ghstack-poisoned]

Update

832cea5

[ghstack-poisoned]

Update

51c392a

[ghstack-poisoned]

Update

f066d46

[ghstack-poisoned]

Update

778f7cd

[ghstack-poisoned]

Update

c19a8ea

[ghstack-poisoned]

Update

88d1dd3

[ghstack-poisoned]

GregoryComer requested review from larryliu0820, kirklandsign and SS-JIA as code owners August 12, 2025 23:22

GregoryComer changed the base branch from gh/GregoryComer/121/head to gh/GregoryComer/120/head August 12, 2025 23:22

GregoryComer mentioned this pull request Aug 13, 2025

[Backend Tester] Add test flow CLI arg #13360

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[Backend Tester] Add subtest index field #13311

[Backend Tester] Add subtest index field #13311

Uh oh!

GregoryComer commented Aug 12, 2025 •

edited

Loading

Uh oh!

GregoryComer commented Aug 12, 2025 •

edited

Loading

Uh oh!

pytorch-bot bot commented Aug 12, 2025 •

edited

Loading

Uh oh!

digantdesai Aug 12, 2025

Uh oh!

GregoryComer Aug 12, 2025

Uh oh!

Uh oh!

[Backend Tester] Add subtest index field #13311

Are you sure you want to change the base?

[Backend Tester] Add subtest index field #13311

Uh oh!

Conversation

GregoryComer commented Aug 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

GregoryComer commented Aug 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pytorch-bot bot commented Aug 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/13311

❌ 8 New Failures, 2 Unrelated Failures

Uh oh!

digantdesai Aug 12, 2025

Choose a reason for hiding this comment

Uh oh!

GregoryComer Aug 12, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

GregoryComer commented Aug 12, 2025 •

edited

Loading

GregoryComer commented Aug 12, 2025 •

edited

Loading

pytorch-bot bot commented Aug 12, 2025 •

edited

Loading