The accuracy matching code in evaluate.py is as follows: agent_match = actual_agent in pred["predicted_agent"]; step_match = actual_step in pred["predicted_step"]. If the diagnosed step is 1 and the golden step is 10, it will still be determined as a match. Is this a bug?
The accuracy matching code in evaluate.py is as follows: agent_match = actual_agent in pred["predicted_agent"]; step_match = actual_step in pred["predicted_step"]. If the diagnosed step is 1 and the golden step is 10, it will still be determined as a match. Is this a bug?