Skip to content

Commit 861dcff

Browse files
authored
Update context_precision.md (#2262)
Correct misleading and wrong definition of context precision in docs, with examples ## Issue Link / Problem Description <!-- Link to related issue or describe the problem this PR solves --> - Fixes #2013 - OR describe the issue: What problem does this solve? How can it be replicated? The current definition of context precision is misleading (see discussion in linked issue). This PR is aimed at correcting the definition. ## Changes Made <!-- Describe what you changed and why --> - Corrected the definition of context precision - Added examples to illustrate the metric better - ## References <!-- Link to related issues, discussions, forums, or external resources --> - Related issues: #2013 - Documentation: - External references: --- <!-- Thank you for contributing to Ragas! Please fill out the sections above as completely as possible. The more information you provide, the faster your PR can be reviewed and merged. -->
1 parent 8d1e166 commit 861dcff

File tree

1 file changed

+47
-1
lines changed

1 file changed

+47
-1
lines changed

docs/concepts/metrics/available_metrics/context_precision.md

Lines changed: 47 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,7 @@
11
# Context Precision
2-
Context Precision is a metric that measures the proportion of relevant chunks in the `retrieved_contexts`. It is calculated as the mean of the precision@k for each chunk in the context. Precision@k is the ratio of the number of relevant chunks at rank k to the total number of chunks at rank k.
2+
Context Precision is a metric that evaluates the retriever’s ability to rank relevant chunks higher than irrelevant ones for a given query in the retrieved context. Specifically, it assesses the degree to which relevant chunks in the retrieved context are placed at the top of the ranking.
3+
4+
It is calculated as the mean of the precision@k for each chunk in the context. Precision@k is the ratio of the number of relevant chunks at rank k to the total number of chunks at rank k.
35

46
$$
57
\text{Context Precision@K} = \frac{\sum_{k=1}^{K} \left( \text{Precision@k} \times v_k \right)}{\text{Total number of relevant items in the top } K \text{ results}}
@@ -41,6 +43,50 @@ Output
4143
0.9999999999
4244
```
4345

46+
Note that even if an irrelevant chunk is present at the second position in the array, context precision remains the same.
47+
48+
```python
49+
from ragas import SingleTurnSample
50+
from ragas.metrics import LLMContextPrecisionWithoutReference
51+
52+
context_precision = LLMContextPrecisionWithoutReference(llm=evaluator_llm)
53+
54+
sample = SingleTurnSample(
55+
user_input="Where is the Eiffel Tower located?",
56+
response="The Eiffel Tower is located in Paris.",
57+
retrieved_contexts=["The Eiffel Tower is located in Paris.", "The Brandenburg Gate is located in Berlin."],
58+
)
59+
60+
61+
await context_precision.single_turn_ascore(sample)
62+
```
63+
Output
64+
```
65+
0.9999999999
66+
```
67+
68+
However, if this irrelevant chunk is placed at the first position, context precision reduces.
69+
70+
```python
71+
from ragas import SingleTurnSample
72+
from ragas.metrics import LLMContextPrecisionWithoutReference
73+
74+
context_precision = LLMContextPrecisionWithoutReference(llm=evaluator_llm)
75+
76+
sample = SingleTurnSample(
77+
user_input="Where is the Eiffel Tower located?",
78+
response="The Eiffel Tower is located in Paris.",
79+
retrieved_contexts=["The Brandenburg Gate is located in Berlin.", "The Eiffel Tower is located in Paris." ],
80+
)
81+
82+
83+
await context_precision.single_turn_ascore(sample)
84+
```
85+
Output
86+
```
87+
0.49999999995
88+
```
89+
4490
### Context Precision with reference
4591

4692
`LLMContextPrecisionWithReference` metric is can be used when you have both retrieved contexts and also reference context associated with a `user_input`. To estimate if a retrieved contexts is relevant or not this method uses the LLM to compare each of the retrieved context or chunk present in `retrieved_contexts` with `reference`.

0 commit comments

Comments
 (0)