Skip to content

Commit 7b64a27

Browse files
Fix shared examples bug in FactualCorrectness metric (#2016)
- Fixes #1961 This commit fixes an issue where initializing a FactualCorrectness metric had side effects on other metrics created afterward. The problem was caused by PydanticPrompt using a class-level examples list shared across all instances. By explicitly setting ```py self.claim_decomposition_prompt.examples = [] ``` before extending with specific examples, we ensure each FactualCorrectness instance gets its own isolated examples list, preventing cross-contamination with other metrics.
1 parent b7cf854 commit 7b64a27

File tree

1 file changed

+10
-2
lines changed

1 file changed

+10
-2
lines changed

src/ragas/metrics/_factual_correctness.py

Lines changed: 10 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -193,12 +193,20 @@ class FactualCorrectness(MetricWithLLM, SingleTurnMetric):
193193
beta: float = 1.0
194194
atomicity: t.Literal["low", "high"] = "low"
195195
coverage: t.Literal["low", "high"] = "low"
196-
claim_decomposition_prompt: PydanticPrompt = ClaimDecompositionPrompt()
197-
nli_prompt: PydanticPrompt = NLIStatementPrompt()
196+
claim_decomposition_prompt: PydanticPrompt = field(
197+
default_factory=ClaimDecompositionPrompt
198+
)
199+
nli_prompt: PydanticPrompt = field(default_factory=NLIStatementPrompt)
198200
language: str = "english"
199201

200202
def __post_init__(self):
201203
value = f"{self.atomicity}_atomicity_{self.coverage}_coverage"
204+
205+
# This creates a new instance-specific examples list, isolating
206+
# changes to just this instance and preventing cross-contamination
207+
# with other metrics.
208+
self.claim_decomposition_prompt.examples = []
209+
202210
for item in DecompositionType:
203211
if item.value == value:
204212
self.claim_decomposition_prompt.examples.extend(

0 commit comments

Comments
 (0)