Is there any way to get the reason for the ragas score?

[divyamuralidharan-ai]

[ X ] I have checked the documentation and related resources and couldn't resolve my bug.

Describe the bug
Not able to see the reason for ragas score.

Ragas version: 0.3.2
Python version: 3.11.6

Code to Reproduce
for trace in predictions:
# --- build reference
reference = infer_reference(trace["overall_input"])
print("============reference===============")
print(reference)
reference_tool_calls = sorted(
[ToolCall(name=tool, args={}) for tool in reference.get("expected_tools", [])],
key=lambda x: x.name
)
actual_tool_called = sorted(
[ToolCall(name=tool, args={}) for tool in set(trace["tool_calls"])],
key=lambda x: x.name
)

    # --- multi-turn sample
    sample = MultiTurnSample(
        user_input=[
            RHMessage(content=trace["overall_input"]),
            RAMessage(content=trace["overall_output"], tool_calls=actual_tool_called),
        ],
        reference_tool_calls=reference_tool_calls,
        reference=reference["reference"],
        reference_topics=[reference["reference_topic"]]
    )

    # --- single-turn sample
    test_data = SingleTurnSample(
        user_input=trace["overall_input"],
        response=trace["overall_output"]
    )

    # --- run evaluations
    dataset_multi = EvaluationDataset(samples=[sample])
    # dataset_single = EvaluationDataset(samples=[test_data])

    multi_results = evaluate(dataset=dataset_multi, metrics=multi_turn_metrics)

Error trace
Not getting reason for ragas score.

Expected behavior
Need to see both score and reason

[anistark]

Hi @divyamuralidharan-ai,

Great question! Yes, you can access the reasoning for Ragas scores. The way to access it depends on which metrics you're using and how you're accessing the results.
You can access via EvaluationResult.scores or convert to DataFrame and inspect.
If you're using custom metrics, you can return MetricResult objects that include reasoning.
Ref: here for MetricResult.

LLM-based metrics (like AspectCritic, AgentGoalAccuracy) automatically generate reasoning.
Non-LLM metrics (like ToolCallAccuracy, ToolCallF1) return numeric scores without reasoning.

df = multi_results.to_pandas()
print(df.columns.tolist())

for col in df.columns:
    if 'reason' in col:
        print(f"\n{col}:")
        print(df[col].tolist())

@Rahul2512Chauhan had some thoughts on this as well.

cc: @jjmachan

[Rahul2512Chauhan]

Thanks for clarifying, @anistark Got it now — LLM metrics already generate reasoning, and non-LLM ones are numeric-only.

I was curious — would it make sense to add some form of lightweight explanation for non-LLM metrics too or is it better to keep them purely numeric?

[divyamuralidharan-ai]

hi @anistark ,
I am using AgentGoalAccuracy. ToolCallAccuracy and TopicAdherence. How can i get the reason for this.

[anistark]

hi @anistark , I am using AgentGoalAccuracy. ToolCallAccuracy and TopicAdherence. How can i get the reason for this.

ToolCallAccuracy don't suppose will return a reasoning. For AgentGoalAccuracy and TopicAdherence, you can do something like this:

for i, trace in enumerate(results.traces):
    row = {'sample_index': i}

    row.update(trace.scores)

    if 'agent_goal_accuracy' in trace:
        aga_trace = trace['agent_goal_accuracy']
        if 'CompareOutcomePrompt' in aga_trace:
            output = aga_trace['CompareOutcomePrompt']['output']
            row['aga_reason'] = output.get('reason', '')
        if 'InferGoalOutcomePrompt' in aga_trace:
            output = aga_trace['InferGoalOutcomePrompt']['output']
            row['aga_user_goal'] = output.get('user_goal', '')
            row['aga_end_state'] = output.get('end_state', '')

df_reasoning = pd.DataFrame(reasoning_data)
df_reasoning.to_string()

For topic adherence, You can get for topics and classifications in it similarly create the dataframe and get output.

[github-actions]

It seems the issue was answered, closing this now.

[jkhetan]

I am trying to pull the explanation for faithfulness using the evaluate() call. I run the following code from above:
df = multi_results.to_pandas()
print(df.columns.tolist())

for col in df.columns:
if 'reason' in col:
print(f"\n{col}:")
print(df[col].tolist())

I get a blank output, meaning, reason is not available. Am I missing anything? Any help will be greatly appreciated.