[BUG]For the source-deployed coze-loop using the Deepseek-v3 model to build an evaluator, an error occurs when using the evaluator.

[sujingheng]

📋 CheckList

• I have searched existing issues to avoid duplicates
• I am using a recently maintained version of Coze Loop
• I have provided all required information
• I understand this is a bug report and not a feature request
• I have submitted this report in English (otherwise it will not be processed)

🐛 Bug Description

When creating an evaluator, clicking 'Debug' sometimes returns a normal response, but sometimes displays an error: 'Invalid model output.
When testing with this evaluator, checking the call trace also shows 'Invalid model output.
On the official website, using the web version to build prompts and evaluators in the same way does not result in errors.

Question: Does the official Doubao model's evaluator output have a fixed format requirement? If so, what is this format?
Additionally, must the evaluator be built exclusively with the Doubao model? Are other models incompatible (e.g., a locally deployed coze-loop)?

🔄 Steps to Reproduce

1. Using source-deployed coze-loop
2. Employing locally hosted DS-v3 model
3. When constructing the evaluator and clicking debug, intermittent model output errors occur during multiple runs

✅ Expected Behavior

no error

❌ Actual Behavior

invalid output from model,code=601205015 message=invalid output from model

🚨 Severity

None

🔧 Component

None

💻 Environment

No response

🔧 Go Environment

No response

📋 Logs

📝 Additional Context

No response

[lsy357]

It seems the issue is caused by the evaluator model outputting a score in a non-numeric format. Although the output format is constrained in the evaluator prompt, different models may perform differently, leading to unstable evaluation results.

Why this problem doesn’t occur in the commercial version:

1. The evaluator’s output format prompt might be different. You can refer to the trace and modify this part in the open-source configuration to match the commercial version.
2. Compare the model configurations in the open-source and commercial versions, such as the temperature setting.

[lsy357]

@tpfz Have we made the optimizations to the parsing of the evaluator’s results regarding this issue?