Skip to content

[Transform] QuIP Modifier #1648

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
wants to merge 58 commits into
base: main
Choose a base branch
from
Open

[Transform] QuIP Modifier #1648

wants to merge 58 commits into from
+261 −1

Conversation

kylesayrs
Copy link
Collaborator

@kylesayrs kylesayrs commented Jul 15, 2025 •
edited
Loading

Purpose

  • Enable quip-style transforms

Prerequisites

Changes

  • Added quip_example.py to examples folder
    • As made clear in the disclaimer, this example requires minimum versions of compressed-tensors and transformers to run
  • Added QuIPModifier which handles the construction of a quip-style transform config

Testing

  • Added modifier serialization and correctness tests

Evaluation

Evaluation performed by @brian-dellabetta

Evals on Llama 3.2 1B with Quip (num_fewshot 8, limit 1000 to be compatible with results here) :

Strat gsm8k,strict gsm8k_llama,strict
FP16 .352 .323
Quip .348 .322
W4A16 .180 .017
Quip+W4A16 .213 .141

Follow Ups

  • Infer data free pipeline, even if a transform modifier is included
  • Modify example to use GPTQ once basic evaluation has been performed

kylesayrs and others added 30 commits June 23, 2025 19:34
@kylesayrs @brian-dellabetta
wip
ba617db 
Signed-off-by: Kyle Sayers <[email protected]>
@kylesayrs @brian-dellabetta
use random-hadamard, add correctness tests
2f5b1c8 
Signed-off-by: Kyle Sayers <[email protected]>
@kylesayrs @brian-dellabetta
add correctness test, note that precision makes a large difference
3aa35e7 
Signed-off-by: Kyle Sayers <[email protected]>
@brian-dellabetta
add on lifecycle methods
b6c088e 
Signed-off-by: Brian Dellabetta <[email protected]>
@brian-dellabetta
Merge branch 'main' into kylesayrs/transform-modifier
d1eb2a1
@brian-dellabetta
TransformModifier with SpinQuant R1&R2
3207124 
Signed-off-by: Brian Dellabetta <[email protected]>
@brian-dellabetta
spinquant and quip_online, running but outputting gibberish
a88ca3c 
Signed-off-by: Brian Dellabetta <[email protected]>
@brian-dellabetta
updated example
5bd51df 
Signed-off-by: Brian Dellabetta <[email protected]>
@brian-dellabetta
DummyModel script
3c216dd 
Signed-off-by: Brian Dellabetta <[email protected]>
@kylesayrs
implement fuse_norm_linears
bbcdc8c 
Signed-off-by: Kyle Sayers <[email protected]>
@kylesayrs
Merge branch 'kylesayrs/fuse-helpers' into bdellabe/transform-modifier
bd7f4d5
@kylesayrs
R1 working
f5c2150 
Signed-off-by: Kyle Sayers <[email protected]>
@kylesayrs
add r2, increase precision
dc5c30c 
Signed-off-by: Kyle Sayers <[email protected]>
@kylesayrs
spinquant modifier
7172c26 
Signed-off-by: Kyle Sayers <[email protected]>
@kylesayrs
remove space
9298e82 
Signed-off-by: Kyle Sayers <[email protected]>
@kylesayrs
use iterable
f77226d 
Signed-off-by: Kyle Sayers <[email protected]>
@kylesayrs
add rotation validation
fdb64b5 
Signed-off-by: Kyle Sayers <[email protected]>
@kylesayrs
embedding fusion
5daa2d5 
Signed-off-by: Kyle Sayers <[email protected]>
@kylesayrs
add missing norm fusion
0e9af7b 
Signed-off-by: Kyle Sayers <[email protected]>
@kylesayrs
use norm mappings
fce83be 
Signed-off-by: Kyle Sayers <[email protected]>
@kylesayrs
break into separate files
a979f8a 
Signed-off-by: Kyle Sayers <[email protected]>
@kylesayrs
small cleanup
4cab29e 
Signed-off-by: Kyle Sayers <[email protected]>
@kylesayrs
cleanup
f1cc987 
Signed-off-by: Kyle Sayers <[email protected]>
@kylesayrs
more cleanup
a7bb2e2 
Signed-off-by: Kyle Sayers <[email protected]>
@kylesayrs
make new weight on cpu
0cf0188 
Signed-off-by: Kyle Sayers <[email protected]>
@kylesayrs
standardize, make modifier serializable
53ea307 
Signed-off-by: Kyle Sayers <[email protected]>
@kylesayrs
add compress model script
4b4257f 
Signed-off-by: Kyle Sayers <[email protected]>
@kylesayrs
use untie_word_embeddings
dc7ac1a 
Signed-off-by: Kyle Sayers <[email protected]>
@kylesayrs
style
8542f8d 
Signed-off-by: Kyle Sayers <[email protected]>
@kylesayrs
better registery logic
b1e637e 
Signed-off-by: Kyle Sayers <[email protected]>
brian-dellabetta and others added 6 commits July 17, 2025 14:16
@brian-dellabetta
style fixes
f2ef7cf 
Signed-off-by: Brian Dellabetta <[email protected]>
@brian-dellabetta
remove hardcoded pipeline logic
d0e5bc5 
Signed-off-by: Brian Dellabetta <[email protected]>
@brian-dellabetta
docstrings
31ac8e9 
Signed-off-by: Brian Dellabetta <[email protected]>
@brian-dellabetta
stylefixes
a4abb3d 
Signed-off-by: Brian Dellabetta <[email protected]>
@kylesayrs @brian-dellabetta
implement quip
490b987 
Signed-off-by: Kyle Sayers <[email protected]>
@kylesayrs @brian-dellabetta
add example, cleanup
ac7dbcd 
Signed-off-by: Kyle Sayers <[email protected]>
@brian-dellabetta brian-dellabetta force-pushed the kylesayrs/transform-quip-modifier branch from 7805321 to ac7dbcd Compare July 25, 2025 18:41
@brian-dellabetta
Copy link
Collaborator

Evals on Llama 3.2 1B with Quip (num_fewshot 8, limit 1000 to be compatible with results here) :

Strat gsm8k,strict gsm8k_llama,strict
FP16 .352 .323
Quip .348 .322
W4A16 .180 .017
Quip+W4A16 .213 .141

@kylesayrs
update quip example
a5d3ddc 
Signed-off-by: Kyle Sayers <[email protected]>
@kylesayrs kylesayrs changed the base branch from bdellabe/transform-modifier to main August 4, 2025 15:28
@kylesayrs kylesayrs marked this pull request as ready for review August 5, 2025 17:17
brian-dellabetta
brian-dellabetta previously approved these changes Aug 5, 2025
View reviewed changes
Copy link
Collaborator

@brian-dellabetta brian-dellabetta left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

beautiful

@kylesayrs
remove extra file
ec83dc4 
Signed-off-by: Kyle Sayers <[email protected]>
dsikka
dsikka requested changes Aug 7, 2025
View reviewed changes
Copy link
Collaborator

@dsikka dsikka left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

very nice - can we fix the example?

src/llmcompressor/pipelines/data_free/pipeline.py
@@ -27,5 +28,9 @@ def __call__(
:param dataloader: loads data for calibration
:param dataset_args: dataset arguments relevant to pipelines
"""
# some ops are still performed on the model by modifiers
# we want those ops to occur on the GPU
dispatch_for_generation(model)
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That means we can leverage more than one gpu for data free cases, including weight-only RTN schemes?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Technically yes, although the weights are still calibrated in a synchronous for loop, so there's no speedup gained from the extra gpus

@kylesayrs
update example
1f5ce4c 
Signed-off-by: Kyle Sayers <[email protected]>
@kylesayrs
move tests files
7324f4b 
Signed-off-by: Kyle Sayers <[email protected]>
@kylesayrs
remove calib dataset, add note
7d34cca 
Signed-off-by: Kyle Sayers <[email protected]>
@kylesayrs
add targets field
972f59f 
Signed-off-by: Kyle Sayers <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@gemini-code-assist gemini-code-assist[bot] gemini-code-assist[bot] left review comments

@dsikka dsikka dsikka requested changes

@brian-dellabetta brian-dellabetta brian-dellabetta left review comments

Requested changes must be addressed to merge this pull request.

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@kylesayrs @brian-dellabetta @dsikka