This repository was archived by the owner on Sep 10, 2025. It is now read-only.

Replace WeightOnlyInt8Linear with TorchAO int8_weight_only quantization#1328

Closed

vmpuri wants to merge 84 commits intomainfrom

torchao_int8_weight_only

Commits on Oct 24, 2024

Replace WeightOnlyInt8Linear with TorchAO int8_weight_only quantization

vmpuri
authored and
vmpuri
committed

Commits on Nov 12, 2024

Merge branch 'main' into torchao_int8_weight_only
vmpuri
authored

Commits on Feb 4, 2025

fix: enforce python version install requirements (#1345)

leseb
authored and
vmpuri
committed
Remove last references to use_distributed argument (#1353)

authored and
vmpuri
committed
Add cstdint to tokenizer (missing include) (#1339)

authored and
vmpuri
committed
Setup a SIGINT handler to gracefully exit the program once the user presses ctrl+c (#1352)

authored and
vmpuri
committed
Update cli.py to make --device/--dtype pre-empt quantize dict-specified values (#1359)

authored and
vmpuri
committed
Update Caching logic to only trigger on the first inference sample (#1369)

Jack-Khuu
authored and
vmpuri
committed
Minor typo + Update install_requirements.sh to support python 3.10 >= (#1368)

Jack-Khuu
authored and
vmpuri
committed
fix: Remove dup gguf dependency (#1371)

leseb
authored and
vmpuri
committed
Bug Fix: Check for explicit cli device (fast) (#1374)

Jack-Khuu
authored and
vmpuri
committed
fix: do not print perf stat when NaN (#1375)

leseb
authored and
vmpuri
committed
fix: Fail gracefully when "model" arg is missing when downloading (#1372)

leseb
authored and
vmpuri
committed
Ignore tokens per sec from jit_compile iteration (#1378)

authored and
vmpuri
committed
Download fix (#1366)

authored and
vmpuri
committed
Update builder.py (#1387)

mikekgfb
authored and
vmpuri
committed
Add multimodal to possible tests (#1382)

mikekgfb
authored and
vmpuri
committed
Fix typo in RuntimeException in builder.py (#1386)

authored and
vmpuri
committed
Bug fix: Enable fast to override quantize json (#1377)

Jack-Khuu
authored and
vmpuri
committed
Changing the referenced AAR so that it uses the AAR from the docs (#1390)

infil00p
authored and
vmpuri
committed
Typo fixes in native-execution.md (#1394)

mikekgfb
authored and
vmpuri
committed
Improvements for readability in ADVANCED-USERS.md (#1393)

mikekgfb
authored and
vmpuri
committed
Update multimodal.md to exercise server as part of test (#1391)

mikekgfb
authored and
vmpuri
committed
Update quantization.md link to quantize.py (#1392)

Jack-Khuu
authored and
vmpuri
committed
Bump torch pin to 20241010 (#1400)

larryliu0820
authored and
vmpuri
committed
Use pytorch-labs/tokenizers and remove tokenizer/ (#1401)

larryliu0820
authored and
vmpuri
committed
Update PT Pin to 1013 (#1407)

Jack-Khuu
authored and
vmpuri
committed
Update docs for max-autotune usage (#1405)

authored and
vmpuri
committed
Update run-docs to include `run-docs native` (#1403)

authored and
vmpuri
committed
Update README.md to run and query server during test (#1384)

authored and
vmpuri
committed
Update run-docs to enable `run-docs evaluation` (#1383)

authored and
vmpuri
committed
Revert "Use pytorch-labs/tokenizers and remove tokenizer/ (#1401)" (#1414)

Jack-Khuu
authored and
vmpuri
committed
Update README.md (whitespace) (#1412)

mikekgfb
authored and
vmpuri
committed
Update evaluation.md to include AOTI (#1411)

mikekgfb
authored and
vmpuri
committed
Update ADVANCED-USERS.md (#1396)

authored and
vmpuri
committed
Bump PT pin to 20241028 (#1419)

Jack-Khuu
authored and
vmpuri
committed
Avoid curl fails due to server startup time in CI(#1418)

authored and
vmpuri
committed
Add torchao mps ops (#1415)

manuelcandales
authored and
vmpuri
committed
Multi Pin Bumps across PT/AO/tune/ET: pt dev20241213 (#1367)

Jack-Khuu
authored and
vmpuri
committed
Update int4pack related in torchchat gguf (#1404)

authored and
vmpuri
committed
update torchao pin: optimized shaders (#1428)

manuelcandales
authored and
vmpuri
committed
Update install_requirements.sh to tune + pt/pt dev20241218 (#1426)

Jack-Khuu
authored and
vmpuri
committed
Add Granite code support (#1336)

authored and
vmpuri
committed
Fix 3.2 11B inference, by updating padded_collate_tiled_images_and_mask args (#1431)

Jack-Khuu
authored and
vmpuri
committed
Integrate distributed inference with chat/server (#1381)

authored and
vmpuri
committed
Granite 3.0 / 3.1 dense support (#1432)

gabe-l-hart
authored and
vmpuri
committed
Fix typo in quantize.py (#1434)

mikekgfb
authored and
vmpuri
committed
Update sh -> bash in quantization.md (#1437)

mikekgfb
authored and
vmpuri
committed
Output explicit selection of /bin/bash as interpreter for test scripts generated from .md with updown.py (#1421)

authored and
vmpuri
committed
Fix how stream flag is read from request (#1441)

mreso
authored and
vmpuri
committed
[retry] Use pytorch-labs/tokenizers and remove tokenizer/ (#1401) (#1443)

larryliu0820
authored and
vmpuri
committed
Update README.md to include granite (#1445)

mikekgfb
authored and
vmpuri
committed
Create local-model.md (#1448)

mikekgfb
authored and
vmpuri
committed
Update evaluation.md (#1442)

mikekgfb
authored and
vmpuri
committed
Create distributed.md (#1438)

authored and
vmpuri
committed
[aoti] Remove need for -l in cmake (#1159)

angelayi
authored and
vmpuri
committed
Bumping ET Pin to Jan16 2025 (#1459)

Jack-Khuu
authored and
vmpuri
committed
Fix typo in quantize.py (#1461)

authored and
vmpuri
committed
Update run-readme-pr-mps.yml for typo (#1460)

authored and
vmpuri
committed
Add Intel XPU device support to generate and serve (#1361)

authored and
vmpuri
committed
Create run-readme-pr-linuxaarch64 (#1350)

authored and
vmpuri
committed
Bump test-readme-mps-macos timeout (#1451)

mikekgfb
authored and
vmpuri
committed
Update torch/tune/vision pins to 1/19/25 (#1467)

Jack-Khuu
authored and
vmpuri
committed
Add warning in PTEModel when not defined (#1468)

Jack-Khuu
authored and
vmpuri
committed
Add attention_backend as a configurable option (#1456)

authored and
vmpuri
committed
Update import of sdpa_with_kv_cache to custom_ops (#1470)

Jack-Khuu
authored and
vmpuri
committed
Typo: Fix generate signature type hint for attention_backend (#1471)

Jack-Khuu
authored and
vmpuri
committed
chat: Change role to user for user prompts (#1447)

authored and
vmpuri
committed
Update run-readme-pr-linuxaarch64.yml to use correct runner (#1469)

Jack-Khuu
authored and
vmpuri
committed
Increment start_pos by encoded size in generate (#1462)

authored and
vmpuri
committed
Explicitly turning off pybindings for ExecuTorch unless requested (#1475)

Jack-Khuu
authored and
vmpuri
committed
Replace RMSNorm by nn.RMSNorm (#1464)

manuelcandales
authored and
vmpuri
committed
Update aoti calls to utilize new export and packaging APIs (#1455)

authored and
vmpuri
committed
Update numpy requirements to no longer upper bound on 2.0 (#1479)

Jack-Khuu
authored and
vmpuri
committed
Add evaluation, multimodal, native tests to run-readme-pr-macos.yml (#1409)

authored and
vmpuri
committed
Add evaluation, multimodal, native tests to run-readme-pr-mps.yml (#1410)

authored and
vmpuri
committed
Force run-readme-pr-macos.yml to use CPU instead of incorrectly loading to MPS (#1417)

authored and
vmpuri
committed
Add distributed tests to run-readme-pr.yml (#1466)

mikekgfb
authored and
vmpuri
committed
Update run-docs to avoid code duplication (#1439)

authored and
vmpuri
committed
Add `export --output-snapshot-path snap.tc`, and `--snapshot-path snap.tc` (#1465)

authored and
vmpuri
committed
Update check_gibberish to check for aspell availability(#1487)

mikekgfb
authored and
vmpuri
committed
Add DeepSeek R1 Distill 8B (#1488)

Jack-Khuu
authored and
vmpuri
committed
Replace WeightOnlyInt8Linear with TorchAO int8_weight_only quantization
vmpuri
committed
Fallback to original quantization if float16
vmpuri
committed