Skip to content

Add coreml quant recipes #13265

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
wants to merge 8 commits into
base: gh/abhinaykukkadapu/4/head
Choose a base branch
from
Open

Add coreml quant recipes #13265

wants to merge 8 commits into from
+800 −174

Conversation

abhinaykukkadapu
Copy link
Contributor

@abhinaykukkadapu abhinaykukkadapu commented Aug 10, 2025 •
edited
Loading

Adds coreml quant recipes after FP32/16 recipes added in #13121

Recipes added:

  1. PT2E_INT8_STATIC
  2. PT2E_INT8_WEIGHT_ONLY
  3. INT4_WEIGHT_ONLY_PER_CHANNEL
  4. INT4_WEIGHT_ONLY_PER_GROUP
  5. INT8_WEIGHT_ONLY_PER_CHANNEL
  6. INT8_WEIGHT_ONLY_PER_GROUP
  7. CODEBOOK_WEIGHT_ONLY

@abhinaykukkadapu
Update
ebc30b2 
[ghstack-poisoned]
@abhinaykukkadapu
Copy link
Contributor Author

abhinaykukkadapu commented Aug 10, 2025 •
edited
Loading

Stack from ghstack (oldest at bottom):

@pytorch-bot PyTorch Bot
Copy link

pytorch-bot bot commented Aug 10, 2025 •
edited
Loading

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/13265

Note: Links to docs will display an error until the docs builds have been completed.

This comment was automatically generated by Dr. CI and updates every 15 minutes.

@meta-cla meta-cla bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Aug 10, 2025
abhinaykukkadapu added a commit that referenced this pull request Aug 10, 2025
@abhinaykukkadapu
Add coreml quant recipes
bf7413d 
ghstack-source-id: 1bb73e0
ghstack-comment-id: 3172341606
Pull-Request: #13265
@abhinaykukkadapu abhinaykukkadapu mentioned this pull request Aug 10, 2025
Open
@abhinaykukkadapu
Update
1207a46 
[ghstack-poisoned]
abhinaykukkadapu added a commit that referenced this pull request Aug 10, 2025
@abhinaykukkadapu
Add coreml quant recipes
801d78a 
ghstack-source-id: 76f6fc8
ghstack-comment-id: 3172341606
Pull-Request: #13265
@abhinaykukkadapu abhinaykukkadapu requested review from kimishpatel and metascroy and removed request for shoumikhin, cccclai and mcr229 August 10, 2025 03:52
abhinaykukkadapu
abhinaykukkadapu commented Aug 10, 2025
View reviewed changes
abhinaykukkadapu added a commit that referenced this pull request Aug 10, 2025
@abhinaykukkadapu
Add coreml quant recipes
c0c32b6 
ghstack-source-id: 76f6fc8
ghstack-comment-id: 3172341606
Pull-Request: #13265
@abhinaykukkadapu
Update
1137ee9 
[ghstack-poisoned]
abhinaykukkadapu added a commit that referenced this pull request Aug 10, 2025
@abhinaykukkadapu
Add coreml quant recipes
7f79b99 
ghstack-source-id: d147f71
ghstack-comment-id: 3172341606
Pull-Request: #13265
@abhinaykukkadapu
Update
0e422bf 
[ghstack-poisoned]
abhinaykukkadapu added a commit that referenced this pull request Aug 11, 2025
@abhinaykukkadapu
Add coreml quant recipes
8882d4a 
ghstack-source-id: cdcd86d
ghstack-comment-id: 3172341606
Pull-Request: #13265
metascroy
metascroy reviewed Aug 11, 2025
View reviewed changes
metascroy
metascroy reviewed Aug 11, 2025
View reviewed changes
metascroy
metascroy reviewed Aug 11, 2025
View reviewed changes
metascroy
metascroy reviewed Aug 11, 2025
View reviewed changes
metascroy
metascroy reviewed Aug 11, 2025
View reviewed changes
metascroy
metascroy reviewed Aug 11, 2025
View reviewed changes
metascroy
metascroy reviewed Aug 11, 2025
View reviewed changes
backends/apple/coreml/recipes/coreml_recipe_provider.py
)
elif recipe_type == CoreMLRecipeType.INT4_WEIGHT_ONLY_PER_GROUP:
group_size = kwargs.pop("group_size", 32)
return self._build_torchao_quantized_recipe(
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What is the compute precision used in these quantization recipes?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@metascroy It is FLOAT16, see _get_coreml_lowering_recipe function defaults.

metascroy
metascroy reviewed Aug 11, 2025
View reviewed changes
@abhinaykukkadapu
Update
b5fd482 
[ghstack-poisoned]
abhinaykukkadapu added a commit that referenced this pull request Aug 12, 2025
@abhinaykukkadapu
Add coreml quant recipes
7658344 
ghstack-source-id: c72ea76
ghstack-comment-id: 3172341606
Pull-Request: #13265
@abhinaykukkadapu abhinaykukkadapu mentioned this pull request Aug 12, 2025
Open
@abhinaykukkadapu
Update
925cf69 
[ghstack-poisoned]
abhinaykukkadapu added a commit that referenced this pull request Aug 12, 2025
@abhinaykukkadapu
Add coreml quant recipes
c2dc8c1 
ghstack-source-id: 622ea48
ghstack-comment-id: 3172341606
Pull-Request: #13265
@abhinaykukkadapu
Update
e33390f 
[ghstack-poisoned]
abhinaykukkadapu added a commit that referenced this pull request Aug 12, 2025
@abhinaykukkadapu
Add coreml quant recipes
4961772 
ghstack-source-id: 3f0920d
ghstack-comment-id: 3172341606
Pull-Request: #13265
digantdesai
digantdesai reviewed Aug 12, 2025
View reviewed changes
backends/apple/coreml/recipes/coreml_recipe_types.py Outdated
# All TorchAO recipes accept filter_fn kwarg to control which layers are quantized
# INT4 Weight-only Quantization, per-channel (axis=0)
# Additional kwargs: filter_fn (default: None - quantizes linear layers)
INT4_WEIGHT_ONLY_PER_CHANNEL = "coreml_int4_weight_only_per_channel"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Add torchao somewhere in the name?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Will add something like TORCHAO_ INT4_WEIGHT_ONLY_PER_CHANNEL format. I'm also modifying the xnnpack enums as well to this format.

metascroy
metascroy reviewed Aug 12, 2025
View reviewed changes
backends/apple/coreml/recipes/coreml_recipe_types.py Outdated
INT8_WEIGHT_ONLY_PER_GROUP = "coreml_int8_weight_only_per_group"

## Codebook/Palettization Quantization
# Additional kwargs: bits (1-8, default: 3), block_size (default: [-1, 16]),
Copy link
Contributor

@metascroy metascroy Aug 12, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure we should have a default bitwidth/block_size for codebook. Similarly, I'm not sure if we should have default group_size for per-group.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it makes sense to remove both defaults, bit width and block_size

metascroy
metascroy reviewed Aug 12, 2025
View reviewed changes
backends/apple/coreml/recipes/coreml_recipe_types.py Outdated
INT4_WEIGHT_ONLY_PER_CHANNEL = "coreml_int4_weight_only_per_channel"
# INT4 Weight-only Quantization, per-group
# Additional kwargs: group_size (default: 32), filter_fn (default: None - quantizes linear layers)
INT4_WEIGHT_ONLY_PER_GROUP = "coreml_int4_weight_only_per_group"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm wondering if all the torchao recipes should use the same default filter_fn to be less confusing.

Right now, codebook is using nn.Linear/nn.Embedding, but the other ones are using None (defaults to linear).

Maybe make None default for all (linear only) or make linear/embedding default for all?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

To keep it broad, may be, linear/embedding default enables more usecases?

@metascroy
Copy link
Contributor

Overall looks good. Address comments then feel free to merge.

@abhinaykukkadapu
Update
cb7440e 
[ghstack-poisoned]
abhinaykukkadapu added a commit that referenced this pull request Aug 12, 2025
@abhinaykukkadapu
Add coreml quant recipes
f6b8bd9 
ghstack-source-id: 6efef94
ghstack-comment-id: 3172341606
Pull-Request: #13265
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@digantdesai digantdesai digantdesai left review comments

@metascroy metascroy metascroy approved these changes

@kimishpatel kimishpatel Awaiting requested review from kimishpatel

Assignees
No one assigned
Labels
CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed.
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@abhinaykukkadapu @metascroy @digantdesai