You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
docs: update NoiseSensitivity metric to collections API
- Add primary example using collections-based API from ragas.metrics.collections
- Include examples for both 'relevant' and 'irrelevant' modes with outputs
- Add synchronous usage note
- Move legacy examples to Legacy Metrics API section with deprecation warning
- Preserve 'How It's Calculated' section and credits
- Tested example code and verified it works correctly
Copy file name to clipboardExpand all lines: docs/concepts/metrics/available_metrics/noise_sensitivity.md
+92-13Lines changed: 92 additions & 13 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -10,14 +10,22 @@ $$
10
10
$$
11
11
12
12
13
-
14
-
## Example
13
+
### Example
15
14
16
15
```python
17
-
from ragas.dataset_schema import SingleTurnSample
18
-
from ragas.metrics import NoiseSensitivity
16
+
from openai import AsyncOpenAI
17
+
from ragas.llms import llm_factory
18
+
from ragas.metrics.collections import NoiseSensitivity
19
19
20
-
sample = SingleTurnSample(
20
+
# Setup LLM
21
+
client = AsyncOpenAI()
22
+
llm = llm_factory("gpt-4o-mini", client=client)
23
+
24
+
# Create metric
25
+
scorer = NoiseSensitivity(llm=llm)
26
+
27
+
# Evaluate
28
+
result =await scorer.ascore(
21
29
user_input="What is the Life Insurance Corporation of India (LIC) known for?",
22
30
response="The Life Insurance Corporation of India (LIC) is the largest insurance company in India, known for its vast portfolio of investments. LIC contributes to the financial stability of the country.",
23
31
reference="The Life Insurance Corporation of India (LIC) is the largest insurance company in India, established in 1956 through the nationalization of the insurance industry. It is known for managing a large portfolio of investments.",
@@ -28,22 +36,51 @@ sample = SingleTurnSample(
28
36
"The Indian economy is one of the fastest-growing major economies in the world, thanks to sectors like finance, technology, manufacturing etc."
29
37
]
30
38
)
31
-
32
-
scorer = NoiseSensitivity(llm=evaluator_llm)
33
-
await scorer.single_turn_ascore(sample)
39
+
print(f"Noise Sensitivity Score: {result.value}")
34
40
```
35
-
Output
41
+
42
+
Output:
43
+
36
44
```
37
-
0.3333333333333333
45
+
Noise Sensitivity Score: 0.3333333333333333
38
46
```
39
47
40
-
To calculate noise sensitivity of irrelevant context, you can set the `mode` parameter to `irrelevant`.
48
+
To calculate noise sensitivity of irrelevant context, you can set the `mode` parameter to `irrelevant`:
user_input="What is the Life Insurance Corporation of India (LIC) known for?",
54
+
response="The Life Insurance Corporation of India (LIC) is the largest insurance company in India, known for its vast portfolio of investments. LIC contributes to the financial stability of the country.",
55
+
reference="The Life Insurance Corporation of India (LIC) is the largest insurance company in India, established in 1956 through the nationalization of the insurance industry. It is known for managing a large portfolio of investments.",
56
+
retrieved_contexts=[
57
+
"The Life Insurance Corporation of India (LIC) was established in 1956 following the nationalization of the insurance industry in India.",
58
+
"LIC is the largest insurance company in India, with a vast network of policyholders and huge investments.",
59
+
"As the largest institutional investor in India, LIC manages substantial funds, contributing to the financial stability of the country.",
60
+
"The Indian economy is one of the fastest-growing major economies in the world, thanks to sectors like finance, technology, manufacturing etc."
If you prefer synchronous code, you can use the `.score()` method instead of `.ascore()`:
74
+
75
+
```python
76
+
result = scorer.score(
77
+
user_input="What is the Life Insurance Corporation of India (LIC) known for?",
78
+
response="The Life Insurance Corporation of India (LIC) is the largest insurance company in India...",
79
+
reference="The Life Insurance Corporation of India (LIC) is the largest insurance company...",
80
+
retrieved_contexts=[...]
81
+
)
82
+
```
83
+
47
84
## How It’s Calculated
48
85
49
86
!!! example
@@ -104,4 +141,46 @@ Let's examine how noise sensitivity in relevant context was calculated:
104
141
This results in a noise sensitivity score of 0.333, indicating that one out of three claims in the answer was incorrect.
105
142
106
143
144
+
## Legacy Metrics API
145
+
146
+
The following examples use the legacy metrics API pattern. For new projects, we recommend using the collections-based API shown above.
147
+
148
+
!!! warning "Deprecation Timeline"
149
+
This API will be deprecated in version 0.4 and removed in version 1.0. Please migrate to the collections-based API shown above.
150
+
151
+
### Example with SingleTurnSample
152
+
153
+
```python
154
+
from ragas.dataset_schema import SingleTurnSample
155
+
from ragas.metrics import NoiseSensitivity
156
+
157
+
sample = SingleTurnSample(
158
+
user_input="What is the Life Insurance Corporation of India (LIC) known for?",
159
+
response="The Life Insurance Corporation of India (LIC) is the largest insurance company in India, known for its vast portfolio of investments. LIC contributes to the financial stability of the country.",
160
+
reference="The Life Insurance Corporation of India (LIC) is the largest insurance company in India, established in 1956 through the nationalization of the insurance industry. It is known for managing a large portfolio of investments.",
161
+
retrieved_contexts=[
162
+
"The Life Insurance Corporation of India (LIC) was established in 1956 following the nationalization of the insurance industry in India.",
163
+
"LIC is the largest insurance company in India, with a vast network of policyholders and huge investments.",
164
+
"As the largest institutional investor in India, LIC manages substantial funds, contributing to the financial stability of the country.",
165
+
"The Indian economy is one of the fastest-growing major economies in the world, thanks to sectors like finance, technology, manufacturing etc."
166
+
]
167
+
)
168
+
169
+
scorer = NoiseSensitivity(llm=evaluator_llm)
170
+
await scorer.single_turn_ascore(sample)
171
+
```
172
+
173
+
Output:
174
+
175
+
```
176
+
0.3333333333333333
177
+
```
178
+
179
+
To calculate noise sensitivity of irrelevant context, you can set the `mode` parameter to `irrelevant`:
180
+
181
+
```python
182
+
scorer = NoiseSensitivity(mode="irrelevant")
183
+
await scorer.single_turn_ascore(sample)
184
+
```
185
+
107
186
Credits: Noise sensitivity was introduced in [RAGChecker](https://github.com/amazon-science/RAGChecker/tree/main/ragchecker)
0 commit comments