Add NPU (Ascend) backend support for INT4 weight-only quantization workflow #3172
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
🔗 Helpful Links🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/ao/3172
Note: Links to docs will display an error until the docs builds have been completed. This comment was automatically generated by Dr. CI and updates every 15 minutes. |
meta-cla
bot
added
the
CLA Signed
FFFrog
reviewed
Oct 14, 2025
Comment on lines
29
to
33
| try: | ||
| import torch_npu | ||
| except ImportError: | ||
| torch_npu = None | ||
|
|
There was a problem hiding this comment.
PyTorch provide Autoload mechinasm, so we do not need to import it explicitly.
FFFrog
reviewed
Oct 14, 2025
Comment on lines
43
to
44
| @unittest.skipIf(torch_npu is None, "torch_npu is not available") | ||
| @unittest.skipIf(not torch_npu.npu.is_available(), "NPU not available") |
There was a problem hiding this comment.
Suggested change
| @unittest.skipIf(torch_npu is None, "torch_npu is not available") | |
| @unittest.skipIf(not torch_npu.npu.is_available(), "NPU not available") | |
| @unittest.skipIf(torch.accelerator.current_accelerator(True).type == "npu" and torch.accelerator.is_available(), "NPU not available") |
FFFrog
reviewed
Oct 14, 2025
Comment on lines
45
to
48
| @unittest.skipIf( | ||
| version.parse(torch_npu.__version__) < version.parse("2.7.1rc1"), | ||
| "Need torch_npu 2.7.1rc1+", | ||
| ) |
There was a problem hiding this comment.
We can remove it because there are some strcit version mapping between PyTorch and Torch_NPU
FFFrog
reviewed
Oct 14, 2025
| ) | ||
|
|
||
| assert int_data.dtype == torch.int32, ( | ||
| f"torch_npu.npu_convert_weight_to_int4pack expects `int32` dtype" |
There was a problem hiding this comment.
Suggested change
| f"torch_npu.npu_convert_weight_to_int4pack expects `int32` dtype" | |
| f"torch.ops.npu.npu_convert_weight_to_int4pack expects `int32` dtype" |
FFFrog
reviewed
Oct 14, 2025
| ) | ||
|
|
||
| assert int_data.shape[-1] % 8 == 0, ( | ||
| f"torch_npu.npu_convert_weight_to_int4pack expects last dim must be aligned to 8,but got {int_data.shape[-1]}" |
There was a problem hiding this comment.
Suggested change
| f"torch_npu.npu_convert_weight_to_int4pack expects last dim must be aligned to 8,but got {int_data.shape[-1]}" | |
| f"torch.ops.npu.npu_convert_weight_to_int4pack expects last dim must be aligned to 8,but got {int_data.shape[-1]}" |
|
Hi @jcaip @jerryzh168 , please help to review it, thanks! |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Related to #3044
Summary
This PR adds NPU (Ascend) backend support for the INT4 weight-only quantization workflow.
It introduces a new tensor subclass,
Int4PlainInt32TensorNPU, aligned with the existingInt4PlainInt32Tensorfor theplain_int32packing format.Environment
Files changed
Modified
torchao/quantization/__init__.pytorchao/quantization/quant_api.pytorchao/quantization/quantize_/workflows/__init__.pyAdded
torchao/quantization/quantize_/workflows/int4/int4_plain_int32_tensor_npu.pytest/quantization/quantize_/workflows/int4/test_int4_plain_int32_tensor_npu.pyImplementation Overview
Int4PlainInt32TensorNPUto enable NPU backend support for INT4 weight-only quantization.quant_api.pyfor dispatch.__init__.pyfiles to ensure proper import and exposure.Test Case
test/quantization/quantize_/workflows/int4/test_int4_plain_int32_tensor_npu.py