Skip to content

Commit e967181

Browse files
committed
docs: update SummaryScore metric documentation to collections-based API
- Add new collections-based API example using SummaryScore from ragas.metrics.collections - Include synchronous usage note with .score() method - Add custom configuration example for length_penalty and coeff parameters - Move legacy example to Legacy Metrics API section with deprecation warning - Preserve all conceptual explanations and formulas - Tested example code and verified it produces expected output
1 parent 48265b4 commit e967181

File tree

1 file changed

+53
-3
lines changed

1 file changed

+53
-3
lines changed

docs/concepts/metrics/available_metrics/summarization_score.md

Lines changed: 53 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -2,7 +2,7 @@
22

33
## Summarization Score
44

5-
`SummarizationScore` metric gives a measure of how well the summary (`response`) captures the important information from the `retrieved_contexts`. The intuition behind this metric is that a good summary shall contain all the important information present in the context(or text so to say).
5+
The **Summarization Score** metric measures how well a summary (`response`) captures the important information from the `reference_contexts`. The intuition behind this metric is that a good summary should contain all the important information present in the context.
66

77
We first extract a set of important keyphrases from the context. These keyphrases are then used to generate a set of questions. The answers to these questions are always `yes(1)` for the context. We then ask these questions to the summary and calculate the summarization score as the ratio of correctly answered questions to the total number of questions.
88

@@ -27,7 +27,55 @@ $$
2727
\text{conciseness score}*\text{coeff}
2828
$$
2929

30-
## Example
30+
### Example
31+
32+
```python
33+
from openai import AsyncOpenAI
34+
from ragas.llms import llm_factory
35+
from ragas.metrics.collections import SummaryScore
36+
37+
# Setup LLM
38+
client = AsyncOpenAI()
39+
llm = llm_factory("gpt-4o-mini", client=client)
40+
41+
# Create metric
42+
scorer = SummaryScore(llm=llm)
43+
44+
# Evaluate
45+
result = await scorer.ascore(
46+
reference_contexts=[
47+
"A company is launching a new product, a smartphone app designed to help users track their fitness goals. The app allows users to set daily exercise targets, log their meals, and track their water intake. It also provides personalized workout recommendations and sends motivational reminders throughout the day."
48+
],
49+
response="A company is launching a fitness tracking app that helps users set exercise goals, log meals, and track water intake, with personalized workout suggestions and motivational reminders."
50+
)
51+
print(f"Summary Score: {result.value}")
52+
```
53+
54+
Output:
55+
56+
```
57+
Summary Score: 0.6423387096775146
58+
```
59+
60+
!!! note "Synchronous Usage"
61+
If you prefer synchronous code, you can use the `.score()` method instead of `.ascore()`:
62+
63+
```python
64+
result = scorer.score(
65+
reference_contexts=[...],
66+
response="..."
67+
)
68+
```
69+
70+
71+
## Legacy Metrics API
72+
73+
The following examples use the legacy metrics API pattern. For new projects, we recommend using the collections-based API shown above.
74+
75+
!!! warning "Deprecation Timeline"
76+
This API will be deprecated in version 0.4 and removed in version 1.0. Please migrate to the collections-based API shown above.
77+
78+
### Example with SingleTurnSample
3179

3280
```python
3381
from ragas.dataset_schema import SingleTurnSample
@@ -44,7 +92,9 @@ sample = SingleTurnSample(
4492
scorer = SummarizationScore(llm=evaluator_llm)
4593
await scorer.single_turn_ascore(sample)
4694
```
47-
Output
95+
96+
Output:
97+
4898
```
4999
0.6423387096775146
50100
```

0 commit comments

Comments
 (0)