reproducing Table 1

[xenoppy]

Hello!

We directly cloned this repository, performed whitening and decomposition, and fine-tuned using dataset yahma/alpaca-cleaned. Unfortunately, the performance we obtained differs significantly from the results reported in Table 1.

May I ask whether you applied any additional methods beyond those described in the paper and repository when conducting these experiments? Also, would it be possible for you to share the fine-tuning hyperparameters used for Table 1?

For reference, here is the command we used in our testing:

python SVDLLM.py --model jeffwan/llama-7b-hf --step 1 --ratio 0.2 --whitening_nsamples 256 --dataset wikitext2 --seed 3 --model_seq_len 2048 --save_path . 
python utils/LoRA.py --prune_model $svded_model_name --data_path yahma/alpaca-cleaned --output_dir $first_half_dir  --lora_target_modules q_u_proj,k_u_proj,v_u_proj,o_u_proj,gate_u_proj,down_u_proj,up_u_proj --lora_r 8 --num_epochs 3 --learning_rate 1e-4 --batch_size 512
python SVDLLM.py --step 4 --model_path $svded_model_name --lora  $first_half_dir
python LoRA_test.py --prune_model ${first_half_dir}/merge.pt --data_path yahma/alpaca-cleaned --output_dir $second_half_dir  --lora_target_modules q_v_proj,k_v_proj,v_v_proj,o_v_proj,gate_v_proj,down_v_proj,up_v_proj --lora_r 8 --num_epochs 3 --learning_rate 1e-4 --batch_size 512

Thank you very much for your time and for sharing such an interesting piece of work. I greatly appreciate any guidance you could provide.