The results were inconsistent between the two evaluations using the inference model classification

[Xjl200204121]

The results were inconsistent when the inference model of DeepSeek-R1 was used for classification evaluation, and the DeepSeek-Chat model did not have this problem. What is the reason for this?

[e06084]

Thanks for flagging this problem.

Can you share us with your config? I try script as below which run successfully：

from dingo.exec import Executor
from dingo.io import InputArgs


def classify_topic():
    input_data = {
        "eval_group": "test",
        "input_path": "/Users/chupei/code/dingo/test/data/test_sft_jsonl.jsonl",  # local filesystem dataset
        "save_data": True,
        "save_correct": True,
        "dataset": "local",
        "data_format": "jsonl",
        "column_content": "question",
        "log_level": "INFO",
        "custom_config": {
            "prompt_list": ["PromptClassifyTopic"],
            "llm_config": {
                "LLMClassifyTopic": {
                    "key": "xxxx",
                    "api_url": "https://api.deepseek.com/v1",
                    "model": "deepseek-reasoner",
                }
            }
        }
    }
    input_args = InputArgs(**input_data)
    executor = Executor.exec_map["local"](input_args)
    result = executor.execute()
    print(result)


if __name__ == '__main__':
    classify_topic()