Skip to content

Commit 8a56282

Browse files
authored
Fix: json format inconsistency in context recall (prompt for statemen… (#776)
There is a small inconsistency in the prompt for statement creation / attribution for the context recall metric. The output key `classification` is expected to be an array of attributed statements (one or more statements). This is defined correctly in the first example, but in the two following examples the array brackets are missing. This inconsistency may not lead to any problems when working with powerful llms (e.g. with OpenAI) most of the time, but I noticed some confusion about the expected output when running the original code with some other smaller models. This way it is more clear and consistent, that one **or more** statements are expected to be returned.
1 parent 4c31c0f commit 8a56282

File tree

1 file changed

+14
-10
lines changed

1 file changed

+14
-10
lines changed

src/ragas/metrics/_context_recall.py

Lines changed: 14 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -52,21 +52,25 @@
5252
"question": """who won 2020 icc world cup?""",
5353
"context": """The 2022 ICC Men's T20 World Cup, held from October 16 to November 13, 2022, in Australia, was the eighth edition of the tournament. Originally scheduled for 2020, it was postponed due to the COVID-19 pandemic. England emerged victorious, defeating Pakistan by five wickets in the final to clinch their second ICC Men's T20 World Cup title.""",
5454
"answer": """England""",
55-
"classification": {
56-
"statement_1": "England won the 2022 ICC Men's T20 World Cup.",
57-
"reason": "From context it is clear that England defeated Pakistan to win the World Cup.",
58-
"Attributed": "1",
59-
},
55+
"classification": [
56+
{
57+
"statement_1": "England won the 2022 ICC Men's T20 World Cup.",
58+
"reason": "From context it is clear that England defeated Pakistan to win the World Cup.",
59+
"Attributed": "1",
60+
}
61+
],
6062
},
6163
{
6264
"question": """What is the primary fuel for the Sun?""",
6365
"context": """NULL""",
6466
"answer": """Hydrogen""",
65-
"classification": {
66-
"statement_1": "The Sun's primary fuel is hydrogen.",
67-
"reason": "The context contains no information",
68-
"Attributed": "0",
69-
},
67+
"classification": [
68+
{
69+
"statement_1": "The Sun's primary fuel is hydrogen.",
70+
"reason": "The context contains no information",
71+
"Attributed": "0",
72+
}
73+
],
7074
},
7175
],
7276
input_keys=["question", "context", "answer"],

0 commit comments

Comments
 (0)