You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
docs: update string metrics (ExactMatch, StringPresence, NonLLMStringSimilarity) to collections API
- Updated NonLLMStringSimilarity with collections API example and configuration section
- Updated ExactMatch with collections API example
- Updated StringPresence with collections API example
- Added synchronous usage notes for all three metrics
- Moved legacy examples to Legacy Metrics API section with deprecation warnings
- All examples tested and verified to work correctly
Copy file name to clipboardExpand all lines: docs/concepts/metrics/available_metrics/traditional.md
+148-4Lines changed: 148 additions & 4 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -5,6 +5,60 @@
5
5
`NonLLMStringSimilarity` metric measures the similarity between the reference and the response using traditional string distance measures such as Levenshtein, Hamming, and Jaro. This metric is useful for evaluating the similarity of `response` to the `reference` text without relying on large language models (LLMs). The metric returns a score between 0 and 1, where 1 indicates a perfect match between the response and the reference. This is a non LLM based metric.
6
6
7
7
### Example
8
+
9
+
```python
10
+
from ragas.metrics.collections import NonLLMStringSimilarity, DistanceMeasure
The following examples use the legacy metrics API pattern. For new projects, we recommend using the collections-based API shown above.
56
+
57
+
!!! warning "Deprecation Timeline"
58
+
This API will be deprecated in version 0.4 and removed in version 1.0. Please migrate to the collections-based API shown above.
59
+
60
+
#### Example with SingleTurnSample
61
+
8
62
```python
9
63
from ragas.dataset_schema import SingleTurnSample
10
64
from ragas.metrics._string import NonLLMStringSimilarity
@@ -17,12 +71,14 @@ sample = SingleTurnSample(
17
71
scorer = NonLLMStringSimilarity()
18
72
await scorer.single_turn_ascore(sample)
19
73
```
20
-
Output
74
+
75
+
Output:
76
+
21
77
```
22
78
0.8918918918918919
23
79
```
24
80
25
-
One can choose from available string distance measures from `DistanceMeasure`. Here is an example of using Hamming distance.
81
+
#### Example with Different Distance Measure
26
82
27
83
```python
28
84
from ragas.metrics._string import NonLLMStringSimilarity, DistanceMeasure
@@ -183,8 +239,50 @@ Output:
183
239
```
184
240
185
241
## Exact Match
242
+
186
243
The `ExactMatch` metric checks if the response is exactly the same as the reference text. It is useful in scenarios where you need to ensure that the generated response matches the expected output word-for-word. For example, arguments in tool calls, etc. The metric returns 1 if the response is an exact match with the reference, and 0 otherwise.
187
244
245
+
### Example
246
+
247
+
```python
248
+
from ragas.metrics.collections import ExactMatch
249
+
250
+
# Create metric (no LLM/embeddings needed)
251
+
scorer = ExactMatch()
252
+
253
+
# Evaluate
254
+
result =await scorer.ascore(
255
+
reference="Paris",
256
+
response="India"
257
+
)
258
+
print(f"Exact Match Score: {result.value}")
259
+
```
260
+
261
+
Output:
262
+
263
+
```
264
+
Exact Match Score: 0.0
265
+
```
266
+
267
+
!!! note "Synchronous Usage"
268
+
If you prefer synchronous code, you can use the `.score()` method instead of `.ascore()`:
269
+
270
+
```python
271
+
result = scorer.score(
272
+
reference="Paris",
273
+
response="India"
274
+
)
275
+
```
276
+
277
+
### Legacy Metrics API
278
+
279
+
The following examples use the legacy metrics API pattern. For new projects, we recommend using the collections-based API shown above.
280
+
281
+
!!! warning "Deprecation Timeline"
282
+
This API will be deprecated in version 0.4 and removed in version 1.0. Please migrate to the collections-based API shown above.
283
+
284
+
#### Example with SingleTurnSample
285
+
188
286
```python
189
287
from ragas.dataset_schema import SingleTurnSample
190
288
from ragas.metrics import ExactMatch
@@ -197,14 +295,58 @@ sample = SingleTurnSample(
197
295
scorer = ExactMatch()
198
296
await scorer.single_turn_ascore(sample)
199
297
```
200
-
Output
298
+
299
+
Output:
300
+
201
301
```
202
302
0.0
203
303
```
204
304
205
305
## String Presence
306
+
206
307
The `StringPresence` metric checks if the response contains the reference text. It is useful in scenarios where you need to ensure that the generated response contains certain keywords or phrases. The metric returns 1 if the response contains the reference, and 0 otherwise.
207
308
309
+
### Example
310
+
311
+
```python
312
+
from ragas.metrics.collections import StringPresence
313
+
314
+
# Create metric (no LLM/embeddings needed)
315
+
scorer = StringPresence()
316
+
317
+
# Evaluate
318
+
result =await scorer.ascore(
319
+
reference="Eiffel Tower",
320
+
response="The Eiffel Tower is located in India."
321
+
)
322
+
print(f"String Presence Score: {result.value}")
323
+
```
324
+
325
+
Output:
326
+
327
+
```
328
+
String Presence Score: 1.0
329
+
```
330
+
331
+
!!! note "Synchronous Usage"
332
+
If you prefer synchronous code, you can use the `.score()` method instead of `.ascore()`:
333
+
334
+
```python
335
+
result = scorer.score(
336
+
reference="Eiffel Tower",
337
+
response="The Eiffel Tower is located in India."
338
+
)
339
+
```
340
+
341
+
### Legacy Metrics API
342
+
343
+
The following examples use the legacy metrics API pattern. For new projects, we recommend using the collections-based API shown above.
344
+
345
+
!!! warning "Deprecation Timeline"
346
+
This API will be deprecated in version 0.4 and removed in version 1.0. Please migrate to the collections-based API shown above.
0 commit comments