Arm backend: Make TOSA backend NCHW-compatible #12994

AdrianLundell · 2025-07-30T14:16:53Z

Moves the needed input/output transposes into the delegated graph to run on Ethos-U rather than requiring the EthosUBackend to implement transposes on CPU.
Renames the annotate_channels_last_dim_order_pass to to_tosa_memory_format_pass since to be more descriptive and future proof.

This changes additionally enables running multiple batches since the EthosU transpose supports that natively, whereas the CPU implementation did not.

cc @digantdesai @freddan80 @per @zingo @oscarandersson8218

pytorch-bot · 2025-07-30T14:16:57Z

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/12994

📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

❌ 5 New Failures, 1 Unrelated Failure

As of commit 8c66a7b with merge base d87766e ():

NEW FAILURES - The following jobs have failed:

Build documentation / build (buck2) / Build doc (gh)
At least one of the pre-conditions you specified did not hold
pull / unittest-arm-backend-with-no-fvp (test_pytest_ops) / linux-job (gh)
RuntimeError: Command docker exec -t d1336c3accbc82d952e7e09b2ebc162fed35239b10fb5c43a8f96251a1c146f4 /exec failed with exit code 1
trunk / test-arm-backend (test_pytest_ops_ethosu_fvp) / linux-job (gh)
RuntimeError: Command docker exec -t 531ecf953221d0c39e0abb42b5c5e14298cc294660063edafd8be70ea3454774 /exec failed with exit code 1
trunk / test-huggingface-transformers-coreml (olmo-1b|coreml_fp32_gpu|--quantize) / macos-job (gh)
RuntimeError: Command bash /Users/runner/work/_temp/exec_script failed with exit code 1
trunk / test-huggingface-transformers-coreml (qwen3-1.7b|xnnpack|--quantize) / macos-job (gh)
RuntimeError: Command bash /Users/runner/work/_temp/exec_script failed with exit code 138

BROKEN TRUNK - The following job failed but were present on the merge base:

👉 Rebase onto the `viable/strict` branch to avoid these failures

Apple / build-benchmark-app / macos-job (gh) (trunk failure)
RuntimeError: Command bash /Users/runner/work/_temp/exec_script failed with exit code 65

This comment was automatically generated by Dr. CI and updates every 15 minutes.

AdrianLundell · 2025-07-30T14:19:11Z

@digantdesai This is a quite big change, do you need to run it internally before merging?

digantdesai

Glad to see so many xfails removed. Left some comments.

backends/arm/_passes/to_tosa_memory_format_pass.py

backends/arm/test/passes/test_to_tosa_memory_format.py

digantdesai · 2025-07-31T10:28:08Z

backends/arm/test/passes/test_to_tosa_memory_format.py

+@common.parametrize("module", modules)
+def test_to_tosa_memory_format_tosa_MI_functional(module):
+    # Also run the actual pass pipeline to ensure functional correctness.
+    pipeline = TosaPipelineMI[input_t](module, module.get_inputs(), [])


No FVP tests?

My intention here was to only test the logic of the pass with as little other things that could go wrong as possible ( quantization and fvps), we do also test the pass on FVP in other places since the change affects all modules we run tests on. Sounds ok?

zingo · 2025-07-31T15:20:35Z

This will unfortunately need a rebase after the TOSA 0.80.1 removal :(

- Moves the needed input/output transposes into the delegated graph to run on Ethos-U rather than requiring the EthosUBackend to implement transposes on CPU. - Renames the annotate_channels_last_dim_order_pass to to_tosa_memory_format_pass since to be more descriptive and future proof. This changes additionally enables running muliple batches since the EtohsU transpose supports that natively, whereas the CPU implementation did not. Change-Id: I676e5915b15cbcc370a03d70bfa2ea2fe20b2210 Signed-off-by: Adrian Lundell <[email protected]>

facebook-github-bot · 2025-08-06T12:09:44Z

@digantdesai has imported this pull request. If you are a Meta employee, you can view this in D79720805.

…nge-1031069

Sebastian-Larsson · 2025-08-11T11:55:03Z

@digantdesai has imported this pull request. If you are a Meta employee, you can view this in D79720805.

@digantdesai Any update on this?

Change-Id: I98f211d2e44ce68c6a1d03ee6941abf84108b3e5

digantdesai · 2025-08-14T14:38:47Z

Hey @Sebastian-Larsson Apologies for the delay. Internal tests got broken couple of days back so they are masking this signal once I merge #13221 hopefully I we can rebase this and pull internally again. 🙏

…nge-1031069

facebook-github-bot · 2025-08-20T16:44:37Z

@digantdesai has imported this pull request. If you are a Meta employee, you can view this in D79720805.

Change-Id: Ib9e678c9471b861df141becfd2eb9511964fb627 Signed-off-by: Adrian Lundell <[email protected]>

digantdesai · 2025-08-21T12:10:10Z

Exception: An error occurred when running the 'ToTosaMemoryFormatPass'

Getting this on the internal CI
Haven't looked deeper yet

AdrianLundell · 2025-08-21T12:23:08Z

Exception: An error occurred when running the 'ToTosaMemoryFormatPass'

Getting this on the internal CI Haven't looked deeper yet

Have you tried the latest version? I have fixed one issue from the moving of the transpose to TOSA dialect change and that passed our internal CI at least.

AdrianLundell · 2025-08-21T14:22:32Z

The sigmoid error can be seen on other PR:s as well so it is not related to this patch.

facebook-github-bot · 2025-08-22T18:31:03Z

@digantdesai has imported this pull request. If you are a Meta employee, you can view this in D79720805.

AdrianLundell · 2025-08-25T10:40:29Z

I see that you are getting internal errors, anything I can help with?

digantdesai · 2025-08-26T11:56:22Z

Actually its ok. Feel free to merge.

digantdesai · 2025-08-26T15:19:47Z

Seems like it broke backends/arm/test/ops/test_sigmoid_16bit.py::test_sigmoid_tosa_INT_add_sigmoid[rand]?

AdrianLundell · 2025-08-26T15:42:10Z

Seems like it yes, I thought it was flakyness since we had similar accuracy errors in the sigmoids before but apparently it always fails, sorry about that.

It was previously an xfail though so it is just a matter of adding it back, I can do it first thing tomorrow morning if you do not do it.

Fix issue introduced in #12994 Signed-off-by: Adrian Lundell <[email protected]>

AdrianLundell requested a review from digantdesai as a code owner July 30, 2025 14:16

AdrianLundell added partner: arm For backend delegation, kernels, demo, etc. from the 3rd-party partner, Arm ciflow/trunk release notes: arm Changes to the ARM backend delegate labels Jul 30, 2025

meta-cla bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Jul 30, 2025

digantdesai reviewed Jul 31, 2025

View reviewed changes

digantdesai approved these changes Aug 1, 2025

View reviewed changes

AdrianLundell force-pushed the change-1031069 branch from 61fa369 to 9c5ef40 Compare August 6, 2025 12:01

AdrianLundell mentioned this pull request Aug 6, 2025

Arm backend: Do not partition noop subgraphs #12995

Merged

Merge branch 'main' of https://github.com/pytorch/executorch into cha…

884e7f4

…nge-1031069

AdrianLundell and others added 3 commits August 13, 2025 12:21

Merge branch 'main' into change-1031069

97bc972

Merge branch 'main' into change-1031069

6454c19

Add cosh xfail

1b3d5db

Change-Id: I98f211d2e44ce68c6a1d03ee6941abf84108b3e5

AdrianLundell and others added 2 commits August 20, 2025 14:03

Merge branch 'main' of https://github.com/pytorch/executorch into cha…

c35e43a

…nge-1031069

Merge branch 'main' into change-1031069

8ff0158

AdrianLundell and others added 2 commits August 21, 2025 13:05

Arm backend: Fix merge issue

3e07401

Change-Id: Ib9e678c9471b861df141becfd2eb9511964fb627 Signed-off-by: Adrian Lundell <[email protected]>

Merge branch 'main' into change-1031069

8c66a7b

oscarandersson8218 merged commit 51de606 into pytorch:main Aug 26, 2025
588 of 602 checks passed

AdrianLundell mentioned this pull request Aug 27, 2025

Arm backend: Add back sigmoid_16bit_add xfail #13724

Merged

mansnils pushed a commit that referenced this pull request Aug 27, 2025

Arm backend: Add back sigmoid_16bit_add xfail (#13724)

39054a2

Fix issue introduced in #12994 Signed-off-by: Adrian Lundell <[email protected]>

AdrianLundell deleted the change-1031069 branch September 4, 2025 12:16

Arm backend: Make TOSA backend NCHW-compatible #12994

Arm backend: Make TOSA backend NCHW-compatible #12994

Uh oh!

Conversation

AdrianLundell commented Jul 30, 2025 • edited by pytorch-bot bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pytorch-bot bot commented Jul 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/12994

❌ 5 New Failures, 1 Unrelated Failure

Uh oh!

AdrianLundell commented Jul 30, 2025

Uh oh!

digantdesai left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

digantdesai Jul 31, 2025

Choose a reason for hiding this comment

Uh oh!

AdrianLundell Jul 31, 2025

Choose a reason for hiding this comment

Uh oh!

zingo commented Jul 31, 2025

Uh oh!

facebook-github-bot commented Aug 6, 2025

Uh oh!

Sebastian-Larsson commented Aug 11, 2025

Uh oh!

digantdesai commented Aug 14, 2025

Uh oh!

facebook-github-bot commented Aug 20, 2025

Uh oh!

digantdesai commented Aug 21, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

AdrianLundell commented Aug 21, 2025

Uh oh!

AdrianLundell commented Aug 21, 2025

Uh oh!

facebook-github-bot commented Aug 22, 2025

Uh oh!

AdrianLundell commented Aug 25, 2025

Uh oh!

digantdesai commented Aug 26, 2025

Uh oh!

Uh oh!

digantdesai commented Aug 26, 2025

Uh oh!

AdrianLundell commented Aug 26, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

AdrianLundell commented Jul 30, 2025 •

edited by pytorch-bot bot

Loading

pytorch-bot bot commented Jul 30, 2025 •

edited

Loading

digantdesai commented Aug 21, 2025 •

edited

Loading