Skip to content

Use a symmetric quantization with no clipping error to improve llama perplexity #5163

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 1 commit into
base: main
Choose a base branch
from
Open

Use a symmetric quantization with no clipping error to improve llama perplexity #5163

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .ci/docker/ci_commit_pins/torchao.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1 +1 @@
0916b5b29b092afcbf2b898caae49abe80662bac
c6abf2bd576828dc8ed175fba2c4c1d0d3681a1d
5 changes: 4 additions & 1 deletion examples/models/llama2/source_transformation/quantize.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -73,9 +73,12 @@ def quantize(
if group_size is None:
raise Exception("For 8da4w quantization, group size must be specified.")
from torchao.quantization.quant_api import Int8DynActInt4WeightQuantizer
from torchao.quantization.quant_primitives import MappingType

model = Int8DynActInt4WeightQuantizer(
precision=torch_dtype, groupsize=group_size
precision=torch_dtype,
groupsize=group_size,
mapping_type=MappingType.SYMMETRIC_NO_CLIPPING_ERR,
).quantize(model)
if verbose:
print("quantized model:", model)
Expand Down
Loading