Where can I find the existing community experiment results for method_comparision?

[2catycm]

I've noticed that there is recently made available a new folder called method_comparision in this awesome library.

After reading the README of this folder, I've learnt how to do new experiment.

Howerver, I wonder if there are existing results and the result gradio page for us to refer to.

[2catycm]

https://github.com/huggingface/peft/tree/main/method_comparison

[githubnemo]

Not yet, we are currently working on verifying the results and automating the result creation. I think it'll be ready soon :)

[2catycm]

Hi @githubnemo, thanks for your kind reply. I think your work is significant and vital, thanks for your efforts to make a good benchmark for different peft methods.

As you mentioned that the results are verifying now, I would like to know if the results are valid and accurate if I try to run a new experiment following the readme.

I am currently trying to propose a new peft method and want to compare its performance with other SOTA methods on some convincing and stable benchmarks. Would you happen to have any suggestions? Thanks.

[githubnemo]

There were already some pull requests that used the MetaMathQA task to compare their results with LoRA as a baseline and to debug some aspects (like memory consumption). I think in that regard the code is already good enough. So, yes, go ahead.

[2catycm]

There were already some pull requests that used the MetaMathQA task to compare their results with LoRA as a baseline and to debug some aspects (like memory consumption). I think in that regard the code is already good enough. So, yes, go ahead.

Thanks for your guidance! I've successfully run an experiment of LoRA with LLaMA-3.2-3B on MetaMathQA based on your code, and it yields an test accuracy of 0.484 on GSM8k. Is it a reasonable result?

[2catycm]

There were already some pull requests that used the MetaMathQA task to compare their results with LoRA as a baseline and to debug some aspects (like memory consumption). I think in that regard the code is already good enough. So, yes, go ahead.

I am also curious about how to test the accuracy of a full finetuning process for reference. Do I need to make a new folder and have a config in it? Or shall I disable the code around get_peft_model? I don't know what the elegant practice is.

In fact, I am trying to propose a new peft method that is currently not in peft library. Abstractly, my method can be employed by calling a python function model = prepare_model_by_the_new_peft_method(model, config_dict_for_the_method). So I want to disable the peft scripts to the normal full finetuning process and then inject that function to the code, while keeping the training params that is friendly to peft methods like LORA or my new method. Do you have any suggestions on this? Thanks.

[githubnemo]

You can find the results of all the benchmarks in this PR: #2593
Your result seems roughly in line with these numbers.

If you want to propose a new PEFT method I'd suggest to open a new issue that explains the nature of the method and its potential impact (who is it for, why is it good, where is it published). There we can also answer potential questions regarding testing and benchmarking.