Skip to content

Commit 20e3e2e

Browse files
committed
docs: update string metrics (ExactMatch, StringPresence, NonLLMStringSimilarity) to collections API
- Updated NonLLMStringSimilarity with collections API example and configuration section - Updated ExactMatch with collections API example - Updated StringPresence with collections API example - Added synchronous usage notes for all three metrics - Moved legacy examples to Legacy Metrics API section with deprecation warnings - All examples tested and verified to work correctly
1 parent 5046604 commit 20e3e2e

File tree

1 file changed

+148
-4
lines changed

1 file changed

+148
-4
lines changed

docs/concepts/metrics/available_metrics/traditional.md

Lines changed: 148 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -5,6 +5,60 @@
55
`NonLLMStringSimilarity` metric measures the similarity between the reference and the response using traditional string distance measures such as Levenshtein, Hamming, and Jaro. This metric is useful for evaluating the similarity of `response` to the `reference` text without relying on large language models (LLMs). The metric returns a score between 0 and 1, where 1 indicates a perfect match between the response and the reference. This is a non LLM based metric.
66

77
### Example
8+
9+
```python
10+
from ragas.metrics.collections import NonLLMStringSimilarity, DistanceMeasure
11+
12+
# Create metric (no LLM/embeddings needed)
13+
scorer = NonLLMStringSimilarity(distance_measure=DistanceMeasure.LEVENSHTEIN)
14+
15+
# Evaluate
16+
result = await scorer.ascore(
17+
reference="The Eiffel Tower is located in Paris.",
18+
response="The Eiffel Tower is located in India."
19+
)
20+
print(f"NonLLM String Similarity Score: {result.value}")
21+
```
22+
23+
Output:
24+
25+
```
26+
NonLLM String Similarity Score: 0.8918918918918919
27+
```
28+
29+
!!! note "Synchronous Usage"
30+
If you prefer synchronous code, you can use the `.score()` method instead of `.ascore()`:
31+
32+
```python
33+
result = scorer.score(
34+
reference="The Eiffel Tower is located in Paris.",
35+
response="The Eiffel Tower is located in India."
36+
)
37+
```
38+
39+
### Configuration
40+
41+
You can choose from available string distance measures from `DistanceMeasure`. Here is an example of using Hamming distance:
42+
43+
```python
44+
scorer = NonLLMStringSimilarity(distance_measure=DistanceMeasure.HAMMING)
45+
```
46+
47+
Available distance measures include:
48+
- `DistanceMeasure.LEVENSHTEIN` (default)
49+
- `DistanceMeasure.HAMMING`
50+
- `DistanceMeasure.JARO`
51+
- `DistanceMeasure.JARO_WINKLER`
52+
53+
### Legacy Metrics API
54+
55+
The following examples use the legacy metrics API pattern. For new projects, we recommend using the collections-based API shown above.
56+
57+
!!! warning "Deprecation Timeline"
58+
This API will be deprecated in version 0.4 and removed in version 1.0. Please migrate to the collections-based API shown above.
59+
60+
#### Example with SingleTurnSample
61+
862
```python
963
from ragas.dataset_schema import SingleTurnSample
1064
from ragas.metrics._string import NonLLMStringSimilarity
@@ -17,12 +71,14 @@ sample = SingleTurnSample(
1771
scorer = NonLLMStringSimilarity()
1872
await scorer.single_turn_ascore(sample)
1973
```
20-
Output
74+
75+
Output:
76+
2177
```
2278
0.8918918918918919
2379
```
2480

25-
One can choose from available string distance measures from `DistanceMeasure`. Here is an example of using Hamming distance.
81+
#### Example with Different Distance Measure
2682

2783
```python
2884
from ragas.metrics._string import NonLLMStringSimilarity, DistanceMeasure
@@ -183,8 +239,50 @@ Output:
183239
```
184240

185241
## Exact Match
242+
186243
The `ExactMatch` metric checks if the response is exactly the same as the reference text. It is useful in scenarios where you need to ensure that the generated response matches the expected output word-for-word. For example, arguments in tool calls, etc. The metric returns 1 if the response is an exact match with the reference, and 0 otherwise.
187244

245+
### Example
246+
247+
```python
248+
from ragas.metrics.collections import ExactMatch
249+
250+
# Create metric (no LLM/embeddings needed)
251+
scorer = ExactMatch()
252+
253+
# Evaluate
254+
result = await scorer.ascore(
255+
reference="Paris",
256+
response="India"
257+
)
258+
print(f"Exact Match Score: {result.value}")
259+
```
260+
261+
Output:
262+
263+
```
264+
Exact Match Score: 0.0
265+
```
266+
267+
!!! note "Synchronous Usage"
268+
If you prefer synchronous code, you can use the `.score()` method instead of `.ascore()`:
269+
270+
```python
271+
result = scorer.score(
272+
reference="Paris",
273+
response="India"
274+
)
275+
```
276+
277+
### Legacy Metrics API
278+
279+
The following examples use the legacy metrics API pattern. For new projects, we recommend using the collections-based API shown above.
280+
281+
!!! warning "Deprecation Timeline"
282+
This API will be deprecated in version 0.4 and removed in version 1.0. Please migrate to the collections-based API shown above.
283+
284+
#### Example with SingleTurnSample
285+
188286
```python
189287
from ragas.dataset_schema import SingleTurnSample
190288
from ragas.metrics import ExactMatch
@@ -197,14 +295,58 @@ sample = SingleTurnSample(
197295
scorer = ExactMatch()
198296
await scorer.single_turn_ascore(sample)
199297
```
200-
Output
298+
299+
Output:
300+
201301
```
202302
0.0
203303
```
204304

205305
## String Presence
306+
206307
The `StringPresence` metric checks if the response contains the reference text. It is useful in scenarios where you need to ensure that the generated response contains certain keywords or phrases. The metric returns 1 if the response contains the reference, and 0 otherwise.
207308

309+
### Example
310+
311+
```python
312+
from ragas.metrics.collections import StringPresence
313+
314+
# Create metric (no LLM/embeddings needed)
315+
scorer = StringPresence()
316+
317+
# Evaluate
318+
result = await scorer.ascore(
319+
reference="Eiffel Tower",
320+
response="The Eiffel Tower is located in India."
321+
)
322+
print(f"String Presence Score: {result.value}")
323+
```
324+
325+
Output:
326+
327+
```
328+
String Presence Score: 1.0
329+
```
330+
331+
!!! note "Synchronous Usage"
332+
If you prefer synchronous code, you can use the `.score()` method instead of `.ascore()`:
333+
334+
```python
335+
result = scorer.score(
336+
reference="Eiffel Tower",
337+
response="The Eiffel Tower is located in India."
338+
)
339+
```
340+
341+
### Legacy Metrics API
342+
343+
The following examples use the legacy metrics API pattern. For new projects, we recommend using the collections-based API shown above.
344+
345+
!!! warning "Deprecation Timeline"
346+
This API will be deprecated in version 0.4 and removed in version 1.0. Please migrate to the collections-based API shown above.
347+
348+
#### Example with SingleTurnSample
349+
208350
```python
209351
from ragas.dataset_schema import SingleTurnSample
210352
from ragas.metrics import StringPresence
@@ -216,7 +358,9 @@ sample = SingleTurnSample(
216358
scorer = StringPresence()
217359
await scorer.single_turn_ascore(sample)
218360
```
219-
Output
361+
362+
Output:
363+
220364
```
221365
1.0
222366
```

0 commit comments

Comments
 (0)