PermuLoRA

[JBedichek]

Hey guys!

Thanks for everything you do first of all, I'm super grateful for your organization. You enable me and so many others to chase our dreams by making this magic so accessible.

I wrote a paper, under review at ICML, not on arxiv yet. I tried a LoRA variant of the general idea and it seems pretty effective relative to base LoRA (and is compatible with other LoRA architectures like adaLoRA). It reduced perplexity 10% relative to base LoRA (On DistilGPT2). I was thinking people might want to use it, I was wondering if there was any chance of getting it integrated?

I attached my implementation, and the paper (abstract is a bit verbose, and doesn't mention the LoRA variant, only the fully connected variant).

PermuLayerFinal.pdf
PermuLoRA_module.txt

Is there something I can do to integrate this into the glorious PEFT?

[BenjaminBossan]

Thank you for your kind words. Regarding the integration of the PermuLoRA method: I would suggest to wait for the reviews, as they may contain suggestions that lead you to make changes to the implementation. Moreover, I would expect the reviewers to ask for more extensive comparisons with other PEFT methods (hopefully easily done with PEFT) and on larger models and datasets (could be more difficult).

To explain, we had some mixed experience in the past with being too eager to add new methods, whose implementations were then changed after the PEFT integration. We want to avoid this, as this breaks backwards compatibility.

[JBedichek]

Makes complete sense, will do.