Skip to content

Commit 5179ed0

Browse files
committed
docs: update NoiseSensitivity metric to collections API
- Add primary example using collections-based API from ragas.metrics.collections - Include examples for both 'relevant' and 'irrelevant' modes with outputs - Add synchronous usage note - Move legacy examples to Legacy Metrics API section with deprecation warning - Preserve 'How It's Calculated' section and credits - Tested example code and verified it works correctly
1 parent e967181 commit 5179ed0

File tree

1 file changed

+92
-13
lines changed

1 file changed

+92
-13
lines changed

docs/concepts/metrics/available_metrics/noise_sensitivity.md

Lines changed: 92 additions & 13 deletions
Original file line numberDiff line numberDiff line change
@@ -10,14 +10,22 @@ $$
1010
$$
1111

1212

13-
14-
## Example
13+
### Example
1514

1615
```python
17-
from ragas.dataset_schema import SingleTurnSample
18-
from ragas.metrics import NoiseSensitivity
16+
from openai import AsyncOpenAI
17+
from ragas.llms import llm_factory
18+
from ragas.metrics.collections import NoiseSensitivity
1919

20-
sample = SingleTurnSample(
20+
# Setup LLM
21+
client = AsyncOpenAI()
22+
llm = llm_factory("gpt-4o-mini", client=client)
23+
24+
# Create metric
25+
scorer = NoiseSensitivity(llm=llm)
26+
27+
# Evaluate
28+
result = await scorer.ascore(
2129
user_input="What is the Life Insurance Corporation of India (LIC) known for?",
2230
response="The Life Insurance Corporation of India (LIC) is the largest insurance company in India, known for its vast portfolio of investments. LIC contributes to the financial stability of the country.",
2331
reference="The Life Insurance Corporation of India (LIC) is the largest insurance company in India, established in 1956 through the nationalization of the insurance industry. It is known for managing a large portfolio of investments.",
@@ -28,22 +36,51 @@ sample = SingleTurnSample(
2836
"The Indian economy is one of the fastest-growing major economies in the world, thanks to sectors like finance, technology, manufacturing etc."
2937
]
3038
)
31-
32-
scorer = NoiseSensitivity(llm=evaluator_llm)
33-
await scorer.single_turn_ascore(sample)
39+
print(f"Noise Sensitivity Score: {result.value}")
3440
```
35-
Output
41+
42+
Output:
43+
3644
```
37-
0.3333333333333333
45+
Noise Sensitivity Score: 0.3333333333333333
3846
```
3947

40-
To calculate noise sensitivity of irrelevant context, you can set the `mode` parameter to `irrelevant`.
48+
To calculate noise sensitivity of irrelevant context, you can set the `mode` parameter to `irrelevant`:
4149

4250
```python
43-
scorer = NoiseSensitivity(mode="irrelevant")
44-
await scorer.single_turn_ascore(sample)
51+
scorer = NoiseSensitivity(llm=llm, mode="irrelevant")
52+
result = await scorer.ascore(
53+
user_input="What is the Life Insurance Corporation of India (LIC) known for?",
54+
response="The Life Insurance Corporation of India (LIC) is the largest insurance company in India, known for its vast portfolio of investments. LIC contributes to the financial stability of the country.",
55+
reference="The Life Insurance Corporation of India (LIC) is the largest insurance company in India, established in 1956 through the nationalization of the insurance industry. It is known for managing a large portfolio of investments.",
56+
retrieved_contexts=[
57+
"The Life Insurance Corporation of India (LIC) was established in 1956 following the nationalization of the insurance industry in India.",
58+
"LIC is the largest insurance company in India, with a vast network of policyholders and huge investments.",
59+
"As the largest institutional investor in India, LIC manages substantial funds, contributing to the financial stability of the country.",
60+
"The Indian economy is one of the fastest-growing major economies in the world, thanks to sectors like finance, technology, manufacturing etc."
61+
]
62+
)
63+
print(f"Noise Sensitivity (Irrelevant) Score: {result.value}")
4564
```
4665

66+
Output:
67+
68+
```
69+
Noise Sensitivity (Irrelevant) Score: 0.0
70+
```
71+
72+
!!! note "Synchronous Usage"
73+
If you prefer synchronous code, you can use the `.score()` method instead of `.ascore()`:
74+
75+
```python
76+
result = scorer.score(
77+
user_input="What is the Life Insurance Corporation of India (LIC) known for?",
78+
response="The Life Insurance Corporation of India (LIC) is the largest insurance company in India...",
79+
reference="The Life Insurance Corporation of India (LIC) is the largest insurance company...",
80+
retrieved_contexts=[...]
81+
)
82+
```
83+
4784
## How It’s Calculated
4885

4986
!!! example
@@ -104,4 +141,46 @@ Let's examine how noise sensitivity in relevant context was calculated:
104141
This results in a noise sensitivity score of 0.333, indicating that one out of three claims in the answer was incorrect.
105142

106143

144+
## Legacy Metrics API
145+
146+
The following examples use the legacy metrics API pattern. For new projects, we recommend using the collections-based API shown above.
147+
148+
!!! warning "Deprecation Timeline"
149+
This API will be deprecated in version 0.4 and removed in version 1.0. Please migrate to the collections-based API shown above.
150+
151+
### Example with SingleTurnSample
152+
153+
```python
154+
from ragas.dataset_schema import SingleTurnSample
155+
from ragas.metrics import NoiseSensitivity
156+
157+
sample = SingleTurnSample(
158+
user_input="What is the Life Insurance Corporation of India (LIC) known for?",
159+
response="The Life Insurance Corporation of India (LIC) is the largest insurance company in India, known for its vast portfolio of investments. LIC contributes to the financial stability of the country.",
160+
reference="The Life Insurance Corporation of India (LIC) is the largest insurance company in India, established in 1956 through the nationalization of the insurance industry. It is known for managing a large portfolio of investments.",
161+
retrieved_contexts=[
162+
"The Life Insurance Corporation of India (LIC) was established in 1956 following the nationalization of the insurance industry in India.",
163+
"LIC is the largest insurance company in India, with a vast network of policyholders and huge investments.",
164+
"As the largest institutional investor in India, LIC manages substantial funds, contributing to the financial stability of the country.",
165+
"The Indian economy is one of the fastest-growing major economies in the world, thanks to sectors like finance, technology, manufacturing etc."
166+
]
167+
)
168+
169+
scorer = NoiseSensitivity(llm=evaluator_llm)
170+
await scorer.single_turn_ascore(sample)
171+
```
172+
173+
Output:
174+
175+
```
176+
0.3333333333333333
177+
```
178+
179+
To calculate noise sensitivity of irrelevant context, you can set the `mode` parameter to `irrelevant`:
180+
181+
```python
182+
scorer = NoiseSensitivity(mode="irrelevant")
183+
await scorer.single_turn_ascore(sample)
184+
```
185+
107186
Credits: Noise sensitivity was introduced in [RAGChecker](https://github.com/amazon-science/RAGChecker/tree/main/ragchecker)

0 commit comments

Comments
 (0)