Feature request

Paper: https://arxiv.org/pdf/2404.10292
Reference code: https://github.com/JT-Sun/Filtering-WoRA

Motivation (WoRA)
We propose integrating WoRA as a new PEFT adapter to provide a lightweight, mergeable way to (1) learn a weighted direction that blends pretrained weights with a low-rank update and (2) decouple magnitude from direction for stability and control, as described in the WWW25 accepted paper.

Key innovations:

• Weighted direction (learnable α, β). Learn the trade-off between the base direction and the low-rank update before normalization.
• Normalized update. Column-wise normalization of the combined direction yields stable optimization.
• Drop-in & mergeable. Same training/inference ergonomics as LoRA-style adapters; merge/unmerge is supported.

Method (WoRA)
We follow DoRA’s magnitude–direction decoupling (m is not our contribution). Our contribution is the weighted direction with learnable α,β before normalization.

Figures
WoRA methodology:

Geometric view (weighted direction with α, β):

Compare the performance:

Your contribution

The implementation in https://github.com/JT-Sun/Filtering-WoRA is based on peft, and we would be pleased submit a pull request, welcoming any suggestions or guidance on this.