Incorrect Observer Sharing/Derivation at Conv-ReLU+ Residual with Arm Ethos Quantizer

[Ninja91]

🐛 Describe the bug

There's a bug in PyTorch 2.0 quantization with ARM quantizer where observers are incorrectly getting shared/derived across different operations (add, permute, relu). Reason most likely being add requires inputs to share observer which when merged with relu and permute observer sharing results in this bug. Since the ReLU observer is shared and observes negative values, incorrect quantization parameters(qparams) are computed (zp!=-128). The Relu fusion is impacted as discussed here which impacts model delegation to ARM U55 backend.

Compare quantized model and see qparams:
With bug triggered:

class SampleModel(nn.Module):
    def __init__(self, in_channels, out_channels, kernel_size, input_dim):
        super().__init__()
        self.conv1 = nn.Conv1d(in_channels, out_channels, kernel_size)
        self.relu = torch.nn.ReLU()
        self.linear = nn.Linear(out_channels, out_channels)

    def forward(self, x):
        # Permute: (N, T, C) -> (N, C, T)
        x = x.permute(0, 2, 1)
        # Conv1D
        x = self.conv1(x)
        # ReLU
        x = self.relu(x)
        # Permute back: (N, C, T) -> (N, T, C)
        x = x.permute(0, 2, 1)
        # Residual connection
        x = x + self.linear(x)
        return x

# Input shape: (1, 10, 3)
model = SampleModel(in_channels=3, out_channels=16, kernel_size=3, input_dim=10)

def _get_ethos_quantizer(
    compile_spec: List[CompileSpec],
) -> Union[Quantizer, TorchQuantizer]:
    from executorch.backends.arm.quantizer import (
        EthosUQuantizer,
        get_symmetric_quantization_config,
    )

    quantizer = EthosUQuantizer(compile_spec=compile_spec)
    quantization_config = get_symmetric_quantization_config(is_per_channel=False)
    quantizer.set_global(quantization_config)
    return quantizer

# Apply post-training quantization with ARM U55 recipe
quantized_model = post_train_quantize(model, calibration_data, quantizer=_get_ethos_quantizer(get_u55_compile_spec()))

With bug not triggered for linear connection:

class SampleModel(nn.Module):
    def __init__(self, in_channels, out_channels, kernel_size, input_dim):
        super().__init__()
        self.conv1 = nn.Conv1d(in_channels, out_channels, kernel_size)
        self.relu = torch.nn.ReLU()
        self.linear = nn.Linear(out_channels, out_channels)

    def forward(self, x):
        # Permute: (N, T, C) -> (N, C, T)
        x = x.permute(0, 2, 1)
        # Conv1D
        x = self.conv1(x)
        # ReLU
        x = self.relu(x)
        # Permute back: (N, C, T) -> (N, T, C)
        x = x.permute(0, 2, 1)
        # Linear connection instead of residual connection
        x = self.linear(x)
        return x

# Input shape: (1, 10, 3)
model = SampleModel(in_channels=3, out_channels=16, kernel_size=3, input_dim=10)
# Apply post-training quantization with ARM U55 recipe
# Apply post-training quantization with ARM U55 recipe
quantized_model = post_train_quantize(model, calibration_data, quantizer=_get_ethos_quantizer(get_u55_compile_spec()))

Although strange that this hasn't been uncovered with other residual networks, but IMO we need to reconsider certain Shared and Derived quantization spec and optimizations to fix this bug. This is a blocker and if there's no immediate resolution, discussion on short term fixes will also be meaningful.
cc: @digantdesai , @3l1, @Tessil

Versions

Python version: 3.12.11+meta (3.12:55fee9c, Jun 03 2025, 15:41:33) [Clang 19.1.2
Python platform: Linux-5.19.0-0_fbk12_hardened_11583_g0bef9520ca2b-x86_64-with-glibc2.34
Is CUDA available: False
CUDA runtime version: No CUDA
CUDA_MODULE_LOADING set to: N/A
GPU models and configuration: No CUDA
Nvidia driver version: No CUDA
cuDNN version: No CUDA
Is XPU available: False
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

cc @digantdesai @freddan80 @per @zingo @oscarandersson8218

[digantdesai]

Thanks for the detailed description and a repro. Let me (or someone from Arm ET Ethos team) take a look. 👍

[gggekov]

Thanks for raising the issue and the detailed analysis @Ninja91 .
I agree with you we have a problem in how we annotate the torch.ops.aten.add.Tensor in /executorch/backends/arm/quantizer/quantization_annotator.py . I believe the right thing to do is to have a different quant specs on the two inputs of the add, rescale the Conv1D+ReLU and do the add. Let me dig more into that.

[JakeStevens]

Reason most likely being add requires inputs to share observer which when merged with relu and permute observer sharing results in this bug

This is more general; I am seeing what I believe to be the same issue (or at least related) with no add, but rather a concat

[gggekov]

Looking in backends/arm/quantizer/quantization_annotator.py

• Linear is annotated with QuantizationSpec which makes sense because it should have its own quant parameters
• Both concat and add are annotated with a SharedQuantizationSpec for the case when adding/concatenating different tensors. For add, we are using QuantizationSpec for the first input(permute, which has the same quant parameters as ReLU) and SharedQuantizationSpec for the second input(linear). The SharedQuantizationSpec is shared among the output of the add, permute and linear, so the observer will see more values than just the [0;+inf) range of the ReLU. I believe that's why we are getting the same zp & qmin on linear as on the ReLU(and zp!=qmin).

[Ninja91]

Yep, @gggekov summarized it well. We would like to identify a quick fix for now and hopefully this bug will track and resolve in a long term one. For the time being, would it make sense to remove the SharedQuantizationSpec. If so, what would be the effects? Any other short term solutions?

[Ninja91]

I tried to keep each input using it's own input spec and same for output at add op and I get this failure:

RuntimeError: Non-passthrough operation could not run on NPU.

[gggekov]

@Ninja91
Are you still gettingRuntimeError: Non-passthrough operation could not run on NPU. with the latest Vela when annotating ADD with its own input & output observers, without SharedQuantizationSpec? I am not getting an error anymore on the tip of main.

Also, is it important to do the lowering for U55 or you can do it for U85 as well?

I believe a fix to your issue is more complicated than annotating ADD with independent observers as if it was MUL. For MUL, multiplying two fp numbers in the quantised domain is scale_1(int8_1-zp_1) * scale_2(int8_2-zp_2) = scale_1scale_2(int8_1-zp_1)*(int8_2-zp_2), so you have a common scale for MUL. We are having some logic in node visitors to figure out the smallest scale(see here and here) but there are cases when annotating add with independent observers doesn't work on u55(though i should caveat annotating add with independent observers works on almost all test cases on u85 in our internal tests).

Also, for completeness about your model - the conv_1d and the ReLU are not quantised because first, during annotation, we are annotating conv1d and relu as if we are going to fuse them later during the lowerings, then, still at annotation time, we are getting zp!=qmin on ADD because of the shared observer so the conv_1d and ReLU are not fused in the to_edge_transform_and_lower. And lastly, in tosa_supported_operators.py, the only input to the relu is conv_1d, there isn't a DQ node because we thought conv_1d and relu were going to be fused so the conv_1d and relu combination is left in FP32.

I think a short term fix will be to enable you to quantise the conv1d and relu properly, without fusing, perhaps on U85 to begin with. A long term solution will require to fix the logic around the SharedObserver on ops with multiple inputs(concat, add, etc) and that should solve the fusing too. I need more time to think about both of these scenarios and get input from our team.

[Ninja91]

Lowering to both U55 and U85 are important. I am on main and using Vela 4.3 and still see the error. What would the short term fix look like? If it means to have dq-conv-q-dq-relu-q, I don't think that would work accurately because the observer after relu is still being shared by linear due to the add/cat SharedQuantizationSpec.