Skip to content

Commit 2319dce

Browse files
authored
fix: add reference tool call to required cols (#1580)
1 parent 400f243 commit 2319dce

File tree

2 files changed

+5
-6
lines changed

2 files changed

+5
-6
lines changed

docs/concepts/metrics/available_metrics/agents.md

Lines changed: 3 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -71,10 +71,9 @@ scorer = TopicAdherenceScore(mode="recall")
7171
`ToolCallAccuracy` is a metric that can be used to evaluate the performance of the LLM in identifying and calling the required tools to complete a given task. This metric needs `user_input` and `reference_tool_calls` to evaluate the performance of the LLM in identifying and calling the required tools to complete a given task. The metric is computed by comparing the `reference_tool_calls` with the Tool calls made by the AI. The values range between 0 and 1, with higher values indicating better performance.
7272

7373
```python
74+
from ragas.metrics import ToolCallAccuracy
7475
from ragas.dataset_schema import MultiTurnSample
7576
from ragas.messages import HumanMessage,AIMessage,ToolMessage,ToolCall
76-
from ragas.metrics import ToolCallAccuracy
77-
7877

7978
sample = [
8079
HumanMessage(content="What's the weather like in New York right now?"),
@@ -89,7 +88,7 @@ sample = [
8988
AIMessage(content="75°F is approximately 23.9°C.")
9089
]
9190

92-
sampl2 = MultiTurnSample(
91+
sample = MultiTurnSample(
9392
user_input=sample,
9493
reference_tool_calls=[
9594
ToolCall(name="weather_check", args={"location": "New York"}),
@@ -98,7 +97,7 @@ sampl2 = MultiTurnSample(
9897
)
9998

10099
scorer = ToolCallAccuracy()
101-
await metric.multi_turn_ascore(sample)
100+
await scorer.multi_turn_ascore(sample)
102101
```
103102

104103
The tool call sequence specified in `reference_tool_calls` is used as the ideal outcome. If the tool calls made by the AI does not the the order or sequence of the `reference_tool_calls`, the metric will return a score of 0. This helps to ensure that the AI is able to identify and call the required tools in the correct order to complete a given task.

src/ragas/metrics/_tool_call_accuracy.py

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -20,7 +20,7 @@ class ToolCallAccuracy(MultiTurnMetric):
2020
default_factory=lambda: {
2121
MetricType.MULTI_TURN: {
2222
"user_input",
23-
"reference",
23+
"reference_tool_calls",
2424
}
2525
}
2626
)
@@ -61,7 +61,7 @@ def is_sequence_aligned(
6161
async def _multi_turn_ascore(
6262
self, sample: MultiTurnSample, callbacks: Callbacks
6363
) -> float:
64-
assert sample.reference_tool_calls is not None, "Reference is not set"
64+
assert sample.reference_tool_calls is not None, "Reference tool calls is not set"
6565

6666
pred_tool_calls = []
6767
for item in sample.user_input:

0 commit comments

Comments
 (0)