[Backend Tester] Reduce log verbosity / spam #13312

GregoryComer · 2025-08-12T05:12:12Z

When running tests, deprecation warnings for export_for_training and "internal consistency verification was requested but not available" create a large amount of spam in the CLI and drown out actual test info during the run. Neither warning is particularly useful. I've suppressed the the export_for_training warning in the backend tester only and switched the tester to not request internal consistency verification since it's not compiled into pybindings by default. This gives much cleaner CLI output.

Before:

test_add_dtype_float32_xnnpack_static_int8_per_channel (test_add.Add.test_add_dtype_float32_xnnpack_static_int8_per_channel) ... /home/gregory/src/executorch/src/executorch/backends/test/harness/stages/quantize.py:50: FutureWarning: `torch.export.export_for_training` is deprecated and will be removed in PyTorch 2.10. Please use `torch.export.export` instead, which is functionally equivalent.
  captured_graph = export_for_training(artifact, inputs, strict=True).module()
/home/gregory/miniconda3/envs/executorch/lib/python3.12/site-packages/torchao/quantization/pt2e/utils.py:818: FutureWarning: `torch.export.export_for_training` is deprecated and will be removed in PyTorch 2.10. Please use `torch.export.export` instead, which is functionally equivalent.
  aten_pattern = torch.export.export_for_training(
/home/gregory/miniconda3/envs/executorch/lib/python3.12/site-packages/torchao/quantization/pt2e/utils.py:818: FutureWarning: `torch.export.export_for_training` is deprecated and will be removed in PyTorch 2.10. Please use `torch.export.export` instead, which is functionally equivalent.
  aten_pattern = torch.export.export_for_training(
/home/gregory/miniconda3/envs/executorch/lib/python3.12/site-packages/torchao/quantization/pt2e/utils.py:818: FutureWarning: `torch.export.export_for_training` is deprecated and will be removed in PyTorch 2.10. Please use `torch.export.export` instead, which is functionally equivalent.
  aten_pattern = torch.export.export_for_training(
/home/gregory/miniconda3/envs/executorch/lib/python3.12/site-packages/torchao/quantization/pt2e/utils.py:818: FutureWarning: `torch.export.export_for_training` is deprecated and will be removed in PyTorch 2.10. Please use `torch.export.export` instead, which is functionally equivalent.
... (it logs like 20x per test)

After:

test_add_dtype_float32_xnnpack (test_add.Add.test_add_dtype_float32_xnnpack) ... ok
test_add_dtype_float32_xnnpack_static_int8_per_channel (test_add.Add.test_add_dtype_float32_xnnpack_static_int8_per_channel) ... ok
test_add_f32_alpha_xnnpack (test_add.Add.test_add_f32_alpha_xnnpack) ... ok
test_add_f32_alpha_xnnpack_static_int8_per_channel (test_add.Add.test_add_f32_alpha_xnnpack_static_int8_per_channel) ... ERROR
...

[ghstack-poisoned]

GregoryComer · 2025-08-12T05:12:13Z

Stack from ghstack (oldest at bottom):

pytorch-bot · 2025-08-12T05:12:15Z

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/13312

📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

❌ 3 New Failures, 1 Unrelated Failure

As of commit c583fa1 with merge base d7ecd87 ():

NEW FAILURES - The following jobs have failed:

Build documentation / build (buck2) / Build doc (gh)
At least one of the pre-conditions you specified did not hold
Lint / lintrunner / linux-job (gh)
>>> Lint for backends/test/suite/models/__init__.py:
pull / unittest-arm-backend-with-no-fvp (test_pytest_ops) / linux-job (gh)
RuntimeError: Command docker exec -t ee904f05c74fa077f270f3fffe24a99d402306a4946afc8b804cce3f92a15049 /exec failed with exit code 1

BROKEN TRUNK - The following job failed but were present on the merge base:

👉 Rebase onto the `viable/strict` branch to avoid these failures

pull / unittest-arm-backend-with-no-fvp (test_pytest_models) / linux-job (gh) (trunk failure)
backends/arm/test/models/stable_diffusion/test_vae_AutoencoderKL.py::TestAutoencoderKL::test_AutoencoderKL_tosa_MI

This comment was automatically generated by Dr. CI and updates every 15 minutes.

ghstack-source-id: bbefbff ghstack-comment-id: 3177730920 Pull-Request: #13312

[ghstack-poisoned]

ghstack-source-id: ffceed5 ghstack-comment-id: 3177730920 Pull-Request: #13312

[ghstack-poisoned]

ghstack-source-id: ffceed5 ghstack-comment-id: 3177730920 Pull-Request: #13312

[ghstack-poisoned]

ghstack-source-id: bbefbff ghstack-comment-id: 3177730920 Pull-Request: #13312

[ghstack-poisoned]

digantdesai · 2025-08-12T11:51:32Z

backends/test/harness/stages/serialize.py

@@ -39,7 +40,9 @@ def graph_module(self) -> None:

    def run_artifact(self, inputs):
        inputs_flattened, _ = tree_flatten(inputs)
-        executorch_module = _load_for_executorch_from_buffer(self.buffer)
+        executorch_module = _load_for_executorch_from_buffer(
+            self.buffer, program_verification=Verification.Minimal


what does this do? i.e. Minimal ==> "do the verification but don't print warnings"?

This doesn't actually cause any change in verification, because the extended verification feature isn't compiled into pybindings by default. So the runtime logs a warning because it was requested but not available. This change updates the code to not ask for extended verification.

Here's the relevant code from the runtime. The "InternalConsistency verification requested but not available " log is the one contributing to log spam.

executorch/runtime/executor/program.cpp

Lines 123 to 136 in d7ecd87

#if ET_ENABLE_PROGRAM_VERIFICATION

EXECUTORCH_SCOPE_PROF("Program::verify_internal_consistency");

flatbuffers::Verifier verifier(

reinterpret_cast<const uint8_t*>(program_data->data()),

program_data->size());

bool ok = executorch_flatbuffer::VerifyProgramBuffer(verifier);

ET_CHECK_OR_RETURN_ERROR(

ok,

InvalidProgram,

"Verification failed; data may be truncated or corrupt");

#else

ET_LOG(

Info, "InternalConsistency verification requested but not available");

#endif

digantdesai · 2025-08-12T11:51:35Z

backends/test/suite/runner.py

@@ -249,6 +250,10 @@ def build_test_filter(args: argparse.Namespace) -> TestFilter:
 def runner_main():
    args = parse_args()

+    # Suppress deprecation warnings for export_for_training, as it generates a
+    # lot of log spam. We don't really need the warning here.
+    warnings.simplefilter("ignore", category=FutureWarning)


FYI PyTest has this nice feature where they count same warnings and at the end print warning summary https://docs.pytest.org/en/stable/how-to/capture-warnings.html

I want to look at using pytest or another framework once we get the core harness and suite landed. I originally went with unittest, in part, because staying closer to unittest made it easier to run within buck/Meta-internal. But if that constraint is lifted, we can probably leverage pytest for some nicer features without having to make too many code changes.

[ghstack-poisoned]

Update

375b076

[ghstack-poisoned]

GregoryComer added a commit that referenced this pull request Aug 12, 2025

[Backend Tester] Reduce log verbosity / spam

af8e238

ghstack-source-id: bbefbff ghstack-comment-id: 3177730920 Pull-Request: #13312

meta-cla bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Aug 12, 2025

GregoryComer requested a review from digantdesai August 12, 2025 05:16

Update

cfbc408

[ghstack-poisoned]

GregoryComer added a commit that referenced this pull request Aug 12, 2025

[Backend Tester] Reduce log verbosity / spam

0400e6c

ghstack-source-id: ffceed5 ghstack-comment-id: 3177730920 Pull-Request: #13312

Update

67f60e4

[ghstack-poisoned]

GregoryComer added a commit that referenced this pull request Aug 12, 2025

[Backend Tester] Reduce log verbosity / spam

7f1004a

ghstack-source-id: ffceed5 ghstack-comment-id: 3177730920 Pull-Request: #13312

Update

d79d1a8

[ghstack-poisoned]

GregoryComer added a commit that referenced this pull request Aug 12, 2025

[Backend Tester] Reduce log verbosity / spam

82ca7ce

ghstack-source-id: bbefbff ghstack-comment-id: 3177730920 Pull-Request: #13312

GregoryComer mentioned this pull request Aug 12, 2025

[Backend Tester] Seed based on test name #13313

Open

GregoryComer marked this pull request as ready for review August 12, 2025 06:12

GregoryComer requested a review from cccclai as a code owner August 12, 2025 06:12

GregoryComer added 2 commits August 11, 2025 23:19

Update

6043952

[ghstack-poisoned]

Update

780519f

[ghstack-poisoned]

digantdesai reviewed Aug 12, 2025

View reviewed changes

digantdesai approved these changes Aug 12, 2025

View reviewed changes

GregoryComer added 6 commits August 12, 2025 11:26

Update

092b8d1

[ghstack-poisoned]

Update

fe201b8

[ghstack-poisoned]

Update

40819b9

[ghstack-poisoned]

Update

ab5415f

[ghstack-poisoned]

Update

b4a8428

[ghstack-poisoned]

Update

c583fa1

[ghstack-poisoned]

GregoryComer mentioned this pull request Aug 13, 2025

[Backend Tester] Add test flow CLI arg #13360

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[Backend Tester] Reduce log verbosity / spam #13312

[Backend Tester] Reduce log verbosity / spam #13312

Uh oh!

GregoryComer commented Aug 12, 2025 •

edited

Loading

Uh oh!

GregoryComer commented Aug 12, 2025 •

edited

Loading

Uh oh!

pytorch-bot bot commented Aug 12, 2025 •

edited

Loading

Uh oh!

digantdesai Aug 12, 2025

Uh oh!

GregoryComer Aug 12, 2025 •

edited

Loading

Uh oh!

digantdesai Aug 12, 2025

Uh oh!

GregoryComer Aug 12, 2025

Uh oh!

Uh oh!

	#if ET_ENABLE_PROGRAM_VERIFICATION
	EXECUTORCH_SCOPE_PROF("Program::verify_internal_consistency");
	flatbuffers::Verifier verifier(
	reinterpret_cast<const uint8_t*>(program_data->data()),
	program_data->size());
	bool ok = executorch_flatbuffer::VerifyProgramBuffer(verifier);
	ET_CHECK_OR_RETURN_ERROR(
	ok,
	InvalidProgram,
	"Verification failed; data may be truncated or corrupt");
	#else
	ET_LOG(
	Info, "InternalConsistency verification requested but not available");
	#endif

[Backend Tester] Reduce log verbosity / spam #13312

Are you sure you want to change the base?

[Backend Tester] Reduce log verbosity / spam #13312

Uh oh!

Conversation

GregoryComer commented Aug 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

GregoryComer commented Aug 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pytorch-bot bot commented Aug 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/13312

❌ 3 New Failures, 1 Unrelated Failure

Uh oh!

digantdesai Aug 12, 2025

Choose a reason for hiding this comment

Uh oh!

GregoryComer Aug 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

digantdesai Aug 12, 2025

Choose a reason for hiding this comment

Uh oh!

GregoryComer Aug 12, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

GregoryComer commented Aug 12, 2025 •

edited

Loading

GregoryComer commented Aug 12, 2025 •

edited

Loading

pytorch-bot bot commented Aug 12, 2025 •

edited

Loading

GregoryComer Aug 12, 2025 •

edited

Loading