Skip to content

Commit 4bae2b0

Browse files
Add optional use_effective_order parameter to BLEU score (#1862)
The original BLEU score fails for answers shorter than four tokens because the number of 4-grams is zero. Adding the use_effective_order parameter allows users to obtain scores by considering only n-grams up to the number of tokens in the answer when ( N < 4 ). This feature is already implemented in sacreBLEU, so we only need to add the parameter to the BLEU object in ragas. The default is set to False to maintain previous behavior. --------- Co-authored-by: Shahules786 <[email protected]>
1 parent 02a16f2 commit 4bae2b0

File tree

1 file changed

+2
-1
lines changed

1 file changed

+2
-1
lines changed

src/ragas/metrics/_bleu_score.py

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -15,6 +15,7 @@ class BleuScore(SingleTurnMetric):
1515
default_factory=lambda: {MetricType.SINGLE_TURN: {"reference", "response"}}
1616
)
1717
language: str = "english"
18+
kwargs: t.Dict[str, t.Any] = field(default_factory=dict)
1819

1920
def __post_init__(self):
2021
try:
@@ -41,7 +42,7 @@ async def _single_turn_ascore(
4142

4243
reference = [[reference] for reference in reference_sentences]
4344
response = response_sentences
44-
score = self.corpus_bleu(response, reference).score / 100
45+
score = self.corpus_bleu(response, reference, **self.kwargs).score / 100
4546
assert isinstance(score, float), "Expecting a float"
4647
return score
4748

0 commit comments

Comments
 (0)