Evaluation on Glue using Lora

[paulzyzy]

您好，我在测试lora时，训练没有遇到问题，但是evaluation的时候有遇到如下报错。 其中moe_peft.json是我用指令直接generate出来的。
同时，我用lora在glue的mrpc上训练了两次，一次epoch2 另一次epoch30。 epoch30我能看到多训练了28个epoch并且checkpoint都有存储，但是最后得到的jason file里的evaluation结果一模一样，请问这可能是哪里出问题了吗？非常感谢您的解答和帮助
[
{
"adapter_name": "mrpc_0",
"task_name": "glue:mrpc",
"date_time": "2025-04-02 12:35:43",
"metrics": {
"accuracy": 0.6838235294117647,
"f1": 0.8122270742358079
},
"training_steps": 918
}
]

[
{
"adapter_name": "mrpc_lora",
"task_name": "glue:mrpc",
"date_time": "2025-04-02 14:58:31",
"metrics": {
"accuracy": 0.6838235294117647,
"f1": 0.8122270742358079
},
"training_steps": 13770
}
]

python moe_peft.py --base_model TinyLlama/TinyLlama_v1.1 --evaluate --config moe_peft.json --fp16:
Traceback (most recent call last):
File "/home/yuz23046/MoE-PEFT/moe_peft.py", line 281, in
moe_peft.evaluate(
File "/home/yuz23046/miniconda3/envs/peft_moe/lib/python3.12/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/home/yuz23046/MoE-PEFT/moe_peft/evaluator.py", line 324, in evaluate
return compute_result(model, configs, save_file)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/yuz23046/MoE-PEFT/moe_peft/evaluator.py", line 227, in compute_result
compute_results = config.metric.compute()
^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/yuz23046/MoE-PEFT/moe_peft/tasks/common.py", line 42, in compute
return self.metric.compute()
^^^^^^^^^^^^^^^^^^^^^^
File "/home/yuz23046/miniconda3/envs/peft_moe/lib/python3.12/site-packages/evaluate/module.py", line 467, in compute
output = self._compute(**inputs, **compute_kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/yuz23046/.cache/huggingface/modules/evaluate_modules/metrics/evaluate-metric--glue/05234ba7acc44554edcca0978db5fa3bc600eeee66229abe79ff9887eacaf3ed/glue.py", line 148, in _compute
return acc_and_f1(predictions, references)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/yuz23046/.cache/huggingface/modules/evaluate_modules/metrics/evaluate-metric--glue/05234ba7acc44554edcca0978db5fa3bc600eeee66229abe79ff9887eacaf3ed/glue.py", line 89, in acc_and_f1
f1 = float(f1_score(y_true=labels, y_pred=preds))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/yuz23046/miniconda3/envs/peft_moe/lib/python3.12/site-packages/sklearn/utils/_param_validation.py", line 216, in wrapper
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/home/yuz23046/miniconda3/envs/peft_moe/lib/python3.12/site-packages/sklearn/metrics/_classification.py", line 1324, in f1_score
return fbeta_score(
^^^^^^^^^^^^
File "/home/yuz23046/miniconda3/envs/peft_moe/lib/python3.12/site-packages/sklearn/utils/_param_validation.py", line 189, in wrapper
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/home/yuz23046/miniconda3/envs/peft_moe/lib/python3.12/site-packages/sklearn/metrics/_classification.py", line 1517, in fbeta_score
_, _, f, _ = precision_recall_fscore_support(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/yuz23046/miniconda3/envs/peft_moe/lib/python3.12/site-packages/sklearn/utils/_param_validation.py", line 189, in wrapper
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/home/yuz23046/miniconda3/envs/peft_moe/lib/python3.12/site-packages/sklearn/metrics/_classification.py", line 1830, in precision_recall_fscore_support
labels = _check_set_wise_labels(y_true, y_pred, average, labels, pos_label)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/yuz23046/miniconda3/envs/peft_moe/lib/python3.12/site-packages/sklearn/metrics/_classification.py", line 1613, in _check_set_wise_labels
raise ValueError(
ValueError: Target is multiclass but average='binary'. Please choose another average setting, one of [None, 'micro', 'macro', 'weighted'].

[paulzyzy]

在mnli的训练结束的evaluation中有如下报错，并且最后没有存包含evaluation数据的 .jason file：
Traceback (most recent call last):
File "/home/yuz23046/MoE-PEFT/moe_peft.py", line 291, in
moe_peft.train(
File "/home/yuz23046/MoE-PEFT/moe_peft/trainer.py", line 357, in train
_perform_evaluate(
File "/home/yuz23046/MoE-PEFT/moe_peft/trainer.py", line 261, in _perform_evaluate
results = evaluate(configs=evaluate_configs, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/yuz23046/miniconda3/envs/peft_moe/lib/python3.12/site-packages/torch/utils/contextlib.py", line 116, in decorate_context
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/home/yuz23046/MoE-PEFT/moe_peft/evaluator.py", line 275, in evaluate
prepare_tasks(model, tokenizer, configs)
File "/home/yuz23046/MoE-PEFT/moe_peft/evaluator.py", line 96, in prepare_tasks
config.prepare(tokenizer, model.device)
File "/home/yuz23046/MoE-PEFT/moe_peft/evaluator.py", line 71, in prepare
self.data = self.dataload_fn(tokenizer)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/yuz23046/MoE-PEFT/moe_peft/evaluator.py", line 32, in dataload_fn
data = self.task.loading_data(False, self.data_path)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/yuz23046/MoE-PEFT/moe_peft/tasks/common.py", line 140, in loading_data
data = data[self.subset_map[0] if is_train else self.subset_map[1]]
~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/yuz23046/miniconda3/envs/peft_moe/lib/python3.12/site-packages/datasets/dataset_dict.py", line 82, in getitem
return super().getitem(k)
^^^^^^^^^^^^^^^^^^^^^^
KeyError: 'validation'