FEAT add GraLoRA #2851

yeonjoon-jung01 · 2025-10-20T01:24:36Z

Summary

We opened the initial PR for the GraLoRA method (a granular low-rank adaptation that improves expression power and outlier handling, selected as a NeurIPS 2025 Spotlight), based on #2636

Test Codes

pytest tests/test_gralora.py -v
pytest tests/test_config.py -k "Gralora" -v
pytest tests/test_decoder_models.py -k "Gralora" -v
pytest tests/test_custom_models.py -k "Gralora" -v

BenjaminBossan

Thanks for contributing GraLoRA to PEFT. The method looks interesting and the implementation generally looks good.

I have added a bunch of comments, but many of these are just due to your fork being a bit older. We have simplified PEFT now so that you can remove a bunch of code, I have marked the code that can be deleted.

Apart from the comments that I added, to complete this PR, let's work on:

Extend tests: Add tests to test_custom_models.py, test_encoder_decoder_models.py, test_feature_extraction_models.py, and test_seq_classifier.py
Also, let's add documentation and ideally also at least one example.
Optional, but highly recommended: Add an experiment to our PEFT method comparison suite.

src/peft/tuners/gralora/config.py

src/peft/tuners/gralora/layer.py

src/peft/tuners/gralora/model.py

tests/test_gralora.py

BenjaminBossan · 2025-10-23T09:58:01Z

@yeonjoon-jung01 Please ping me when you're finished so that I know that I can give this another review. Also, if possible, please avoid force pushes or rebases, as those make reviews harder.

…hts parameter for flexible initialization

…est coverage

…ight calculation.

…ce, and more intuitive hybrid_r handling.

yeonjoon-jung01 · 2025-10-24T04:44:19Z

@yeonjoon-jung01 Please ping me when you're finished so that I know that I can give this another review. Also, if possible, please avoid force pushes or rebases, as those make reviews harder.

@BenjaminBossan I’ve finished updating the code 🙂. I saw your message a bit late — I had already rebased the branch to sync with the main stream, just in case there might be any conflicts. I’ll make sure to avoid force pushes or rebases from now on. Sorry about that!

yeonjoon-jung01 · 2025-10-24T10:15:13Z

@BenjaminBossan I have also resolved the previously missed features.

I’ve extended the test coverage to include test_custom_models.py, test_encoder_decoder_models.py, test_feature_extraction_models.py, and test_seq_classifier.py.
Additionally, I’ve added corresponding documentation and example code.

BenjaminBossan

Thanks for the updates to the PR. I did another review round, please check.

Also, before committing your changes, please call make style. Ensure that you have the correct version of ruff installed (0.12.12).

docs/source/package_reference/gralora.md

src/peft/tuners/gralora/config.py

src/peft/tuners/gralora/layer.py

tests/test_gralora.py

yeonjoon-jung01 · 2025-10-24T18:23:25Z

@BenjaminBossan I’ve resolved all of your comments and applied the suggested changes. The main update is that I removed tests/test_gralora.py and integrated the related test cases into the existing test_initialization and test_custom_models files, including additional scenarios for Hybrid GraLoRA.

HuggingFaceDocBuilderDev · 2025-10-27T11:42:29Z

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

BenjaminBossan · 2025-10-27T11:44:55Z

@yeonjoon-jung01 Could you please run make style?

yeonjoon-jung01 · 2025-10-27T12:40:55Z

@BenjaminBossan I have run make style updated the documentations

BenjaminBossan

Thanks for the updates. A test is failing. Check my comment on possible solutions.

src/peft/tuners/gralora/layer.py

yeonjoon-jung01 · 2025-10-28T12:28:11Z

@BenjaminBossan Do you think there’s any additional code I should test or update?

BenjaminBossan

Thanks for the updates, the change looks good.

I focused on the examples this time and found a few issues. Some are possibly due to some recent changes in transformers, not sure, but we should update them so that the examples run out of the box.

Moreover, I ran an experiment with GraLoRA on the PEFT MetaMath benchmark. I used the default settings for the config and in general, the results compare to LoRA rank 32, with similar memory usage and training time. However, the final test accuracy fell slightly short, attaining 46.2% compared to 48.2% with LoRA. If you have any suggestion for better GraLoRA hyper-parameters for this experiment, feel free to check them in as an experiment. Otherwise, we can also work the defaults.

examples/gralora_finetuning/README.md

BenjaminBossan · 2025-10-28T15:55:19Z

examples/gralora_finetuning/gralora_finetuning.py

+        per_device_eval_batch_size=batch_size,
+        warmup_steps=100,
+        weight_decay=0.01,
+        logging_dir="./logs",


This argument errors for me, does it work for you?

It worked for me! I guess it’s a version dependency issue. I used transformers==4.57.1, and the error did not occur.

BenjaminBossan · 2025-10-28T15:55:38Z

examples/gralora_finetuning/gralora_finetuning.py

+    parser.add_argument("--learning_rate", type=float, default=1e-4, help="Learning rate")
+    parser.add_argument("--cutoff_len", type=int, default=512, help="Cutoff length for tokenization")
+    parser.add_argument("--val_set_size", type=int, default=500, help="Validation set size")
+    parser.add_argument("--quantize", action="store_true", help="Use quantization")


Quantizing wouldn't work, right?

I have checked that quantization works fine with the GraLoRA method.

yeonjoon-jung01 · 2025-10-29T03:40:07Z

Thanks for the updates, the change looks good.

I focused on the examples this time and found a few issues. Some are possibly due to some recent changes in transformers, not sure, but we should update them so that the examples run out of the box.

Moreover, I ran an experiment with GraLoRA on the PEFT MetaMath benchmark. I used the default settings for the config and in general, the results compare to LoRA rank 32, with similar memory usage and training time. However, the final test accuracy fell slightly short, attaining 46.2% compared to 48.2% with LoRA. If you have any suggestion for better GraLoRA hyper-parameters for this experiment, feel free to check them in as an experiment. Otherwise, we can also work the defaults.

@BenjaminBossan Could you please try with learning rate 2e-4 instead of the default 1e-4 for GraLoRA?

…tegration

yeonjoon-jung01 · 2025-10-29T10:21:45Z

Thanks for the updates, the change looks good.
I focused on the examples this time and found a few issues. Some are possibly due to some recent changes in transformers, not sure, but we should update them so that the examples run out of the box.
Moreover, I ran an experiment with GraLoRA on the PEFT MetaMath benchmark. I used the default settings for the config and in general, the results compare to LoRA rank 32, with similar memory usage and training time. However, the final test accuracy fell slightly short, attaining 46.2% compared to 48.2% with LoRA. If you have any suggestion for better GraLoRA hyper-parameters for this experiment, feel free to check them in as an experiment. Otherwise, we can also work the defaults.

@BenjaminBossan Could you please try with learning rate 2e-4 instead of the default 1e-4 for GraLoRA?

@BenjaminBossan I’ve tested the method on my side with both rank 32 and 64, and in both cases, it achieved higher performance than LoRA. If you’d like, I can also commit the configuration and result JSON files, or you’re welcome to try reproducing it on your end.
I used a learning rate of 2e-4 for both cases, as GraLoRA is more robust to outliers, making a higher learning rate manageable. For the other settings, I used rank 32 with alpha 64 and rank 64 with alpha 128. All other parameters were kept at their default values.

yeonjoon-jung01 mentioned this pull request Oct 20, 2025

Support for GraLoRA #2636

Closed

BenjaminBossan requested changes Oct 22, 2025

View reviewed changes

yeonjoon-jung01 force-pushed the gralora_support branch from 134e6f0 to a24d156 Compare October 23, 2025 06:34

yeonjoon-jung01 and others added 5 commits October 24, 2025 00:00

feat: Add Gralora configuration and basic implementation

6dfa24e

ENH Support merge/unmerge in GraLoRA functionality; support init_weig…

bfa1ef7

…hts parameter for flexible initialization

TST Add test suite for GraLoRA.

9813b17

FIX & TEST: Fix GraLoRA bugs in get_peft_config_as_dict and improve t…

c1fe6c4

…est coverage

Refactor GraLoRA weight computation to improve efficiency in delta-we…

4f1444f

…ight calculation.

yeonjoon-jung01 force-pushed the gralora_support branch from c53ffce to 2618a8a Compare October 23, 2025 15:21

yeonjoon-jung01 added 2 commits October 24, 2025 00:29

Refactor GraLoRA code for clearer documentation, simplified inheritan…

9431502

…ce, and more intuitive hybrid_r handling.

Update test code for the GraLoRA method

dec25f5

yeonjoon-jung01 force-pushed the gralora_support branch from 2618a8a to dec25f5 Compare October 23, 2025 15:31

yeonjoon-jung01 requested a review from BenjaminBossan October 24, 2025 01:36

ADD: documentations, examples, and test code for GraLoRA method

925ad72

BenjaminBossan requested changes Oct 24, 2025

View reviewed changes

REFACTOR: integrate GraLoRA tests into existing test files

3f69d8f

yeonjoon-jung01 requested a review from BenjaminBossan October 24, 2025 18:24

UPDATE document format in GraLoRA

430e896

BenjaminBossan requested changes Oct 27, 2025

View reviewed changes

src/peft/tuners/gralora/layer.py Outdated Show resolved Hide resolved

FIX CPU casting in GraLoRA get_delta_weight function

351877f

yeonjoon-jung01 requested a review from BenjaminBossan October 27, 2025 14:18

BenjaminBossan requested changes Oct 28, 2025

View reviewed changes

HaohanTsao mentioned this pull request Oct 29, 2025

DOCS: update README for GraLoRA finetuning with correct SFTTrainer integration yeonjoon-jung01/peft#2

Merged

DOCS: update README for GraLoRA finetuning with correct SFTTrainer in…

a1c944a

…tegration

Uh oh!

FEAT add GraLoRA #2851

Are you sure you want to change the base?

FEAT add GraLoRA #2851

Conversation

yeonjoon-jung01 commented Oct 20, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Test Codes

Uh oh!

BenjaminBossan left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

BenjaminBossan commented Oct 23, 2025

Uh oh!

yeonjoon-jung01 commented Oct 24, 2025

Uh oh!

yeonjoon-jung01 commented Oct 24, 2025

Uh oh!

BenjaminBossan left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

yeonjoon-jung01 commented Oct 24, 2025

Uh oh!

HuggingFaceDocBuilderDev commented Oct 27, 2025

Uh oh!

BenjaminBossan commented Oct 27, 2025

Uh oh!

yeonjoon-jung01 commented Oct 27, 2025

Uh oh!

BenjaminBossan left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

yeonjoon-jung01 commented Oct 28, 2025

Uh oh!

BenjaminBossan left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

BenjaminBossan Oct 28, 2025

Choose a reason for hiding this comment

Uh oh!

yeonjoon-jung01 Oct 29, 2025

Choose a reason for hiding this comment

Uh oh!

BenjaminBossan Oct 28, 2025

Choose a reason for hiding this comment

Uh oh!

yeonjoon-jung01 Oct 29, 2025

Choose a reason for hiding this comment

Uh oh!

yeonjoon-jung01 commented Oct 29, 2025

Uh oh!

yeonjoon-jung01 commented Oct 29, 2025

Uh oh!

Reviewers

Assignees

yeonjoon-jung01 commented Oct 20, 2025 •

edited

Loading