[XNNPACK] Support 2d Transposed Convolution in XNNPACK delegate #7514

AkiSakurai · 2025-01-05T10:55:27Z

Add support for Transposed Convolution in XNNPACK delegate.
The test is copied from conv2d.
Some patterns are not quantized by XNNPACKQuantizer, and the quantization check is skipped.
Skip fusion of ReLU if there are multiple uses of the 2d Transposed Convolution node.

pytorch-bot · 2025-01-05T10:55:31Z

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/7514

📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

✅ No Failures

As of commit 731506f with merge base 3f9324c ():
💚 Looks good so far! There are no failures yet. 💚

This comment was automatically generated by Dr. CI and updates every 15 minutes.

pytorch-bot · 2025-01-05T10:57:11Z

Didn't find following labels among repository labels: release notes: Support 2d Transposed Convolution in XNNPACK delegate

pytorch-bot · 2025-01-05T11:00:22Z

Didn't find following labels among repository labels: backends

AkiSakurai · 2025-01-05T11:00:59Z

@pytorchbot label "release notes: backends"

digantdesai

This is brilliant. Thanks for adding this. This was on our list for a while. Great quality PR!

Added a few nits, and comments. I am happy with this, let's fix these and we can merge this.

In the meanwhile, let me run some more internal tests to make sure we are looking good.

Thanks again.

backends/xnnpack/_passes/fuse_activation_pass.py

digantdesai · 2025-01-10T03:55:44Z

backends/xnnpack/test/ops/test_conv_transpose2d.py

+            self._test(
+                ConvTranspose2d(bias=has_bias),
+                quant_config=get_symmetric_quantization_config(),
+                check_quantized=False,  # XNNPackQuantizer does not this pattern


hmm, we should be able to add support for it, here

digantdesai · 2025-01-10T03:58:30Z

backends/xnnpack/test/ops/test_conv_transpose2d.py

+        self._test(ConvTranspose2d(groups=2, in_channels=2, out_channels=6))
+
+    def test_fp32_conv_transpose2d_bn(self):
+        class ConvTranspose2dBatchNorm(torch.nn.Module):


move this out or use the more complex version here?

digantdesai · 2025-01-10T04:01:53Z

backends/xnnpack/test/ops/test_linear.py

 except:
    torchao_installed = False

+# Set higher recompile limit to avoid exception on over-recompilation in tests


How this is related to deconv? Just ran into this when running linear test?

digantdesai · 2025-01-10T04:08:42Z

backends/xnnpack/test/ops/test_conv_transpose2d.py

@@ -0,0 +1,430 @@
+# Copyright (c) Meta Platforms, Inc. and affiliates.


Thank you for adding these tests. I assume it was non-trivial to overload conv tests for transpose case?

digantdesai · 2025-01-10T04:22:02Z

backends/xnnpack/operators/node_visitor.py

            fp32_static_weights: XNN_FLAG_FP32_STATIC_WEIGHTS for fp16 conv
+            swap_in_out_for_transpose_weights: bool to indicate whether tensor shape should be
+                permuted and reshape from (inc, oc/groups, height, width) to  (oc, inc/groups, height, width)
+            groups: number of groups for swap_in_out_for_transpose_weights


assert groups == 1 when the new flag is false.

digantdesai · 2025-01-10T04:23:50Z

backends/xnnpack/operators/node_visitor.py

            else:
                assert f"Unsupported weight per channel quantization axis for depthwise conv2d: {quant_params.axis}, expecting 0."

+        if swap_in_out_for_transpose_weights and (


Nit: can we cleanly merge with depthwise clause above, OK if it gets harder to read.

backends/xnnpack/operators/node_visitor.py

digantdesai · 2025-01-10T04:30:48Z

backends/xnnpack/_passes/fuse_batch_norm_with_conv.py

+        kernel_shape = get_shape(kernel_node)
+        stride = cast(List[int], conv.args[3])
+
+        if is_transpose and (


Nit: Add a comment about kernel shape and strides matching

facebook-github-bot · 2025-01-10T04:33:26Z

@digantdesai has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

digantdesai · 2025-01-10T16:44:17Z

backends/xnnpack/test/ops/test_conv_transpose2d.py

+from typing import Optional
+
+import executorch.extension.pybindings.portable_lib  # noqa[F401]
+import executorch.kernels.quantized  # noqa[F401]


When doing local testing I am running into ModuleNotFoundError: No module named 'executorch.kernels'. I guess we need this to run the quantized ops when not lowered to XNNPACK. Can we skip running in the case of quantized graph and stop at serializing, perhaps that will eliminate the need to include this module?

…tion fusion

mcr229

@AkiSakurai Thank you for this contribution! This is super valuable, and appreciate the work put in to this, and this is the first external operator contribution from a user! Would love to hear some feedback on what it was like working in the code base. Any ways to improve our code quality, readability, documentation would all be great to hear.

What prompted the contribution? Personal feature enablement? Model enablement?
What part of the development flow took the longest setting up?
How easy was it to test and verify your code changes?
What part of the code was the hardest to navigate and the most confusing? Quantization? Passes? Schema Serialization?
What(if any) parts of the code were easy to navigate and understand?

backends/xnnpack/_passes/fuse_activation_pass.py

mcr229 · 2025-01-13T18:55:33Z

backends/xnnpack/_passes/tag_implicit_q_dq_pass.py

+        op_name = cast(torch._ops.OpOverload, node.target).name()
+
+        # Weight and Input should both be quantized
+        if op_name == exir_ops.edge.aten.convolution.default.name():


what was the reasoning for this? I imagine it should've returned true in the previous implementation as well?

The issue arises when a non-quantized operation is interleaved between two quantized operations. That operation also match the dequantize-op-quantize pattern. However, an operation with quantized input and float weight is not supported by XNNPACK.

mcr229 · 2025-01-13T19:22:40Z

backends/xnnpack/partition/config/gemm_configs.py

+            and weight_quant_params.per_channel
+            and (groups > 1 or weight_quant_params.axis != 1)
+        ):
+            why(


pretty neat way of checking this constraint

digantdesai

Thanks. The quantized import is no longer complaining.

digantdesai · 2025-01-14T03:04:19Z

@mcr229 - Added #7640 FYI.

digantdesai · 2025-01-14T03:08:01Z

@AkiSakurai I think you broke this test in a good way. Can you please fix it? 🙏

digantdesai · 2025-01-14T03:08:21Z

Once this is fixed, I can merge this.

facebook-github-bot · 2025-01-14T16:32:07Z

@digantdesai has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

* Support Transposed Convolution in XNNPACK delegate * Apply suggestions * Remove invalid restriction for transpose convolution batch normalization fusion * fix size analysis tool test

digantdesai · 2025-02-06T03:59:33Z

FYI #8090

facebook-github-bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Jan 5, 2025

pytorch-bot bot added the release notes: backends label Jan 5, 2025

mergennachin requested review from digantdesai and mcr229 January 8, 2025 13:10

mergennachin added the module: xnnpack Issues related to xnnpack delegation and the code under backends/xnnpack/ label Jan 8, 2025

digantdesai reviewed Jan 10, 2025

View reviewed changes

AkiSakurai added 2 commits January 12, 2025 10:01

Support Transposed Convolution in XNNPACK delegate

db8a015

Apply suggestions

26cd168

AkiSakurai force-pushed the convtranspose2d branch from 5b1b565 to 26cd168 Compare January 12, 2025 09:13

Remove invalid restriction for transpose convolution batch normaliza…

0dbeb61

…tion fusion

mcr229 approved these changes Jan 13, 2025

View reviewed changes

digantdesai approved these changes Jan 14, 2025

View reviewed changes

digantdesai mentioned this pull request Jan 14, 2025

Add support for quantized deconv #7640

Closed

fix size analysis tool test

731506f

digantdesai added release notes: xnnpack Changes to the XNNPack backend delegate and removed release notes: backends [DO NOT USE] labels Jan 14, 2025

digantdesai merged commit 63e6136 into pytorch:main Jan 14, 2025
6 checks passed

		@@ -0,0 +1,430 @@
		# Copyright (c) Meta Platforms, Inc. and affiliates.

[XNNPACK] Support 2d Transposed Convolution in XNNPACK delegate #7514

[XNNPACK] Support 2d Transposed Convolution in XNNPACK delegate #7514

Uh oh!

Conversation

AkiSakurai commented Jan 5, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pytorch-bot bot commented Jan 5, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/7514

✅ No Failures

Uh oh!

pytorch-bot bot commented Jan 5, 2025

Uh oh!

pytorch-bot bot commented Jan 5, 2025

Uh oh!

AkiSakurai commented Jan 5, 2025

Uh oh!

digantdesai left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

facebook-github-bot commented Jan 10, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mcr229 left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

digantdesai left a comment

Choose a reason for hiding this comment

Uh oh!

digantdesai commented Jan 14, 2025

Uh oh!

digantdesai commented Jan 14, 2025

Uh oh!

digantdesai commented Jan 14, 2025

Uh oh!

facebook-github-bot commented Jan 14, 2025

Uh oh!

Uh oh!

digantdesai commented Feb 6, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

AkiSakurai commented Jan 5, 2025 •

edited

Loading

pytorch-bot bot commented Jan 5, 2025 •

edited

Loading

mcr229 left a comment •

edited

Loading

digantdesai commented Feb 6, 2025 •

edited

Loading