LoRI results on Hex-PHI

[pjlintw]

I ran LoRI on the safety benchmark using saferpaca_llama3_r_32.sh.
On Hex-PHI, I obtained 90.6 accuracy for LoRI-D and 95.0 for LoRI-S.

These numbers are slightly different from the reported scores in the paper (92.8 for LoRI-D and 93.8 for LoRI-S).
Could you clarify whether the reported results are averaged over multiple runs or due to different hyperparameters?

Thanks!

[juzhengz]

Hi @pjlintw! The results you obtained are consistent with our expectations. The slight variance in numbers (and the fact that your LoRI-S score is actually higher than reported) comes from two main sources of randomness in the current implementation:

• In the provided scripts, we did not enforce a fixed global seed for matrix $A$ initialization. As a result, every run operates in a slightly different random subspace. While the performance is robust, small fluctuations like the ones you observed are expected between different random seeds.
• The evaluation on Hex-PHI involves text generation. There is inherent randomness in the model's outputs due to sampling, which can also contribute to slight variations in the final accuracy scores.