recover questions #2
Description
I use llama3.1-8b as base model, and your checkpoint in the README as fine_tuned model, but when I run "python pome.py", I met these questions
"
weight_diff = (params.data - params_v.data).to(torch.float64)
RuntimeError: The size of tensor a (128257) must match the size of tensor b (128256) at non-singleton dimension 0
U, S, Vh = torch.linalg.svd(weight_diff, full_matrices=False)
RuntimeError: linalg.svd: The input tensor A must have at least 2 dimensions.
"
I fix it by jumping these layers and get a POME checkpoint, but the model performance before POME is " gsm8k acc==== 0.8066717210007581", after POME is "gsm8k acc==== 0.6527672479150872".