Skip to content

[GPTQ] Add inversion fallback#1283

Merged
dsikka merged 4 commits intomainfrom
kylesayrs/gptq-inversion-fallback
Apr 1, 2025
Merged

[GPTQ] Add inversion fallback#1283
dsikka merged 4 commits intomainfrom
kylesayrs/gptq-inversion-fallback

Conversation

@kylesayrs
Copy link
Collaborator

Purpose

  • Given the increasing size of large language models (such as DeepSeek-R1 which contains 45034 linear layers), the likelihood that any of the hessian inversions will spontaneously fail is significant
  • These changes cause the GPTQ algorithm to fall back to RTN for any layers which fail hessian inversion

Changes

  • Implement fallback by setting hessian value to identity matrix if inversion fails

Signed-off-by: Kyle Sayers <kylesayrs@gmail.com>
@github-actions
Copy link

👋 Hi! Thank you for contributing to llm-compressor. Please add the ready label when the PR is ready for review.

Note: This is required to complete the testing suite, please only add the label once the PR is code complete and local testing has been performed.

Copy link
Collaborator

@brian-dellabetta brian-dellabetta left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

cool!

@kylesayrs kylesayrs added the ready When a PR is ready for review label Mar 27, 2025
rahul-tuli
rahul-tuli previously approved these changes Apr 1, 2025
Signed-off-by: Kyle Sayers <kylesayrs@gmail.com>
@kylesayrs kylesayrs dismissed stale reviews from rahul-tuli and brian-dellabetta via 56a5f14 April 1, 2025 16:44
@dsikka dsikka enabled auto-merge (squash) April 1, 2025 16:53
@dsikka dsikka merged commit 0873f28 into main Apr 1, 2025
8 checks passed
@dsikka dsikka deleted the kylesayrs/gptq-inversion-fallback branch April 1, 2025 17:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ready When a PR is ready for review

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants