You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
docs: update SummaryScore metric documentation to collections-based API
- Add new collections-based API example using SummaryScore from ragas.metrics.collections
- Include synchronous usage note with .score() method
- Add custom configuration example for length_penalty and coeff parameters
- Move legacy example to Legacy Metrics API section with deprecation warning
- Preserve all conceptual explanations and formulas
- Tested example code and verified it produces expected output
Copy file name to clipboardExpand all lines: docs/concepts/metrics/available_metrics/summarization_score.md
+53-3Lines changed: 53 additions & 3 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -2,7 +2,7 @@
2
2
3
3
## Summarization Score
4
4
5
-
`SummarizationScore` metric gives a measure of how well the summary (`response`) captures the important information from the `retrieved_contexts`. The intuition behind this metric is that a good summary shall contain all the important information present in the context(or text so to say).
5
+
The **Summarization Score** metric measures how well a summary (`response`) captures the important information from the `reference_contexts`. The intuition behind this metric is that a good summary should contain all the important information present in the context.
6
6
7
7
We first extract a set of important keyphrases from the context. These keyphrases are then used to generate a set of questions. The answers to these questions are always `yes(1)` for the context. We then ask these questions to the summary and calculate the summarization score as the ratio of correctly answered questions to the total number of questions.
8
8
@@ -27,7 +27,55 @@ $$
27
27
\text{conciseness score}*\text{coeff}
28
28
$$
29
29
30
-
## Example
30
+
### Example
31
+
32
+
```python
33
+
from openai import AsyncOpenAI
34
+
from ragas.llms import llm_factory
35
+
from ragas.metrics.collections import SummaryScore
36
+
37
+
# Setup LLM
38
+
client = AsyncOpenAI()
39
+
llm = llm_factory("gpt-4o-mini", client=client)
40
+
41
+
# Create metric
42
+
scorer = SummaryScore(llm=llm)
43
+
44
+
# Evaluate
45
+
result =await scorer.ascore(
46
+
reference_contexts=[
47
+
"A company is launching a new product, a smartphone app designed to help users track their fitness goals. The app allows users to set daily exercise targets, log their meals, and track their water intake. It also provides personalized workout recommendations and sends motivational reminders throughout the day."
48
+
],
49
+
response="A company is launching a fitness tracking app that helps users set exercise goals, log meals, and track water intake, with personalized workout suggestions and motivational reminders."
50
+
)
51
+
print(f"Summary Score: {result.value}")
52
+
```
53
+
54
+
Output:
55
+
56
+
```
57
+
Summary Score: 0.6423387096775146
58
+
```
59
+
60
+
!!! note "Synchronous Usage"
61
+
If you prefer synchronous code, you can use the `.score()` method instead of `.ascore()`:
62
+
63
+
```python
64
+
result = scorer.score(
65
+
reference_contexts=[...],
66
+
response="..."
67
+
)
68
+
```
69
+
70
+
71
+
## Legacy Metrics API
72
+
73
+
The following examples use the legacy metrics API pattern. For new projects, we recommend using the collections-based API shown above.
74
+
75
+
!!! warning "Deprecation Timeline"
76
+
This API will be deprecated in version 0.4 and removed in version 1.0. Please migrate to the collections-based API shown above.
0 commit comments