Fix dequantize_affine before iOS18

[metascroy]

dequantize_affine does not work correctly before iOS18, when the op gets translated to constexpr_affine_dequantize instead of constexpr_blockwise_shift_scale.

This is because the axis should be None (default value) instead of -1. In this case, the axis gets inferred from the shape of scales.

This is similar to the change in the ExecuTorch PR: pytorch/executorch#13896

[metascroy]

@YifanShenSZ can you have a look?

[YifanShenSZ]

Hey Scott, first to confirm your use case: Do you want weight-only quantization, or activation quantization?

Some background about coreml ops that might be helpful:

1. The constexpr_ ops are for weight-only quantization, that's why they are "const expression"s: the quantized weight are known at compile time
2. Among const expr ops, Blockwise shift scale is new in iOS18, affine dequantize was new in iOS16
3. Dequantize is for activation quantization, and was new in iOS17

[metascroy]

@YifanShenSZ this is for weight-only quantization, which is represented as dequantize_affine (applied to the weights) followed by linear/embedding.

The existing registration works for iOS18 (blockwise scale shift), but has a bug in affine_dequantize (iOS16). The bug is the axis is incorrect. This doesn't affect iOS18 because axis is not used in blockwise scale shift.

Dequantize is for activation quantization, and was new in iOS17

I assume you mean "quantize" here is used for activation quantization?

[YifanShenSZ]

@YifanShenSZ this is for weight-only quantization, which is represented as dequantize_affine (applied to the weights) followed by linear/embedding.

The existing registration works for iOS18 (blockwise scale shift), but has a bug in affine_dequantize (iOS16). The bug is the axis is incorrect. This doesn't affect iOS18 because axis is not used in blockwise scale shift.

I see. So in this case, you will want

1. For iOS 18 and higher, translate to constexpr_blockwise_shift_scale
2. For iOS 16 and 17, translate to constexpr_affine_dequantize
3. For iOS 15 and earlier, error out

Dequantize is for activation quantization, and was new in iOS17

I assume you mean "quantize" here is used for activation quantization?

Yeah I mean quantize and dequantize, which are for activation quantization (i.e. quantize / dequantize variables) and introduced in iOS 17

[metascroy]

@YifanShenSZ this is for weight-only quantization, which is represented as dequantize_affine (applied to the weights) followed by linear/embedding.
The existing registration works for iOS18 (blockwise scale shift), but has a bug in affine_dequantize (iOS16). The bug is the axis is incorrect. This doesn't affect iOS18 because axis is not used in blockwise scale shift.

I see. So in this case, you will want

1. For iOS 18 and higher, translate to constexpr_blockwise_shift_scale
2. For iOS 16 and 17, translate to constexpr_affine_dequantize
3. For iOS 15 and earlier, error out

Dequantize is for activation quantization, and was new in iOS17

I assume you mean "quantize" here is used for activation quantization?

Yeah I mean quantize and dequantize, which are for activation quantization (i.e. quantize / dequantize variables) and introduced in iOS 17

Yes, but the function _utils._construct_constexpr_dequant_op already does this translation. I'm just fixing the arg passed to it for axis (axis is only used by _utils._construct_constexpr_dequant_op before iOS18, where it translates the op to constexpr_affine_dequantize; it is not used for iOS18, where the op gets translated to constexpr_blockwise_shift_scale). The CI only tests on iOS18, so I fix the arg for axis and add a test for iOS16.

[YifanShenSZ]

Got it, then yes this is correct. Let's hear what CI has to say

CI 🔴 https://gitlab.com/coremltools1/coremltools/-/commit/c47235b01c27d101c373ce06b03f1758d3457557/pipelines

[YifanShenSZ]

nit: lower case "i", i.e. iOS

[YifanShenSZ]

CI 🔴 https://gitlab.com/coremltools1/coremltools/-/commit/30b74a15f3f3d6fef6b7ffbbfb1b79b2fcb0bfb6/pipelines

[YifanShenSZ]

nit: you might missed minimum_deployment_target?

[YifanShenSZ]

@metascroy please rebase on top of latest main + address comment, then let's try CI again

[YifanShenSZ]

CI 🔴 https://gitlab.com/coremltools1/coremltools/-/commit/30b74a15f3f3d6fef6b7ffbbfb1b79b2fcb0bfb6/pipelines

because we support torch up to 2.7.1 😮‍💨 please rebase on top of latest main to pick up #2591

[metascroy]

CI 🔴 https://gitlab.com/coremltools1/coremltools/-/commit/30b74a15f3f3d6fef6b7ffbbfb1b79b2fcb0bfb6/pipelines

because we support torch up to 2.7.1 😮‍💨 please rebase on top of latest main to pick up #2591

rebased

[metascroy]

@YifanShenSZ can this be merged?

[YifanShenSZ]

rebased

Thanks 🙏 Kicked off new CI https://gitlab.com/coremltools1/coremltools/-/commit/fb4eb5207bcc205d1e8ac9da83106a3ec7926581/pipelines 🔴

[YifanShenSZ]

CI 🔴 due to the typo IOS -> iOS

    class TestPyTorchQuantizationOps(TorchQuantizationBaseTest):
../../envs/coremltools-py3.10/lib/python3.10/site-packages/coremltools/converters/mil/frontend/torch/test/test_torch_quantization_ops.py:278: in TestPyTorchQuantizationOps
    itertools.product(compute_units, [True, False], [ct.target.IOS16, ct.target.IOS17]),
../../envs/coremltools-py3.10/lib/python3.10/enum.py:437: in __getattr__
    raise AttributeError(name) from None
E   AttributeError: IOS16. Did you mean: 'iOS16'?

[metascroy]

iOS16

Updated

[YifanShenSZ]

✅ CI https://gitlab.com/coremltools1/coremltools/-/commit/2b7f8edceabc203fe72bd38a92a18eed2092f1ef/pipelines

[YifanShenSZ]

Many thanks for your contribution to coremltools! 🙏