EvaluationAgent review is failing

[gganapavarapu]

Query: "Hello!"

Agent Response: "General response Hello! How can I assist you today? Remember, I'm here as an Asset Lifecycle Management agent to help with any queries related to the lifecycle of your assets."

Characteristic form: "This question is a general greeting, and the agent should respond in a friendly and welcoming manner."

Response from Evaluation Agent:

review={'status': 'Error', 'reasoning': 'Failed to produce valid JSON after 3 attempts.', 'suggestions': 'Review the prompt and refine the LLM response strategy.'}

Expected response:

{
"task_completion": true/false,
"data_retrieval_accuracy": true/false,
"generalized_result_verification": true/false,
"agent_sequence_correct": true/false,
"clarity_and_justification": true/false,
"hallucinations": true/false,
"suggestions": "Optional. Actions or improvements for rectifying the response if applicable."
}

[jtrayfield]

could not duplicate

`(venv) bigboote:alm-agent jtray$ python evaluation/evaluation.py --query-file foo.json
Namespace(query_file='foo.json', query_ids=[], query_type='single')
1: started,
General response Hello! How can I assist you today? Remember, I'm here as an Asset Lifecycle Management agent to help with any queries related to the lifecycle of your assets.
FAILURE: 1 failed: Hello!
Traceback (most recent call last):
File "/Users/jtray/git/alm-agent/evaluation/evaluation.py", line 227, in main
res = evaluate_single_query(query, rsp)
File "/Users/jtray/git/alm-agent/evaluation/evaluation.py", line 122, in evaluate_single_query
for agents in expected["agents"]:
~~~~~~~~^^^^^^^^^^
KeyError: 'agents'

slack message: id, agent set equal, result valid

successful completions: 0 / 1
agent set equal: 0 / 1
result valid: 0 / 1

id, agent set equal, expected agents, actual agents, result valid, query
`

[DhavalRepo18]

@jtrayfield this issue cannot be reproducible. After three trial, we the model is not responding.
@gganapavarapu Did you clone the repo and used the code. if yes then we can create a separate branch and provide a robust solution.

[DhavalRepo18]

@jtrayfield where does your RObustEvaluator sit? you run 5 times and then do MajorityVote.

[jtrayfield]

@DhavalRepo18 AssetOpsBench/src/evaluation_agent/agent.py