|
1 | 1 | # Context Precision |
2 | | -Context Precision is a metric that measures the proportion of relevant chunks in the `retrieved_contexts`. It is calculated as the mean of the precision@k for each chunk in the context. Precision@k is the ratio of the number of relevant chunks at rank k to the total number of chunks at rank k. |
| 2 | +Context Precision is a metric that evaluates the retriever’s ability to rank relevant chunks higher than irrelevant ones for a given query in the retrieved context. Specifically, it assesses the degree to which relevant chunks in the retrieved context are placed at the top of the ranking. |
| 3 | + |
| 4 | +It is calculated as the mean of the precision@k for each chunk in the context. Precision@k is the ratio of the number of relevant chunks at rank k to the total number of chunks at rank k. |
3 | 5 |
|
4 | 6 | $$ |
5 | 7 | \text{Context Precision@K} = \frac{\sum_{k=1}^{K} \left( \text{Precision@k} \times v_k \right)}{\text{Total number of relevant items in the top } K \text{ results}} |
@@ -41,6 +43,50 @@ Output |
41 | 43 | 0.9999999999 |
42 | 44 | ``` |
43 | 45 |
|
| 46 | +Note that even if an irrelevant chunk is present at the second position in the array, context precision remains the same. |
| 47 | + |
| 48 | +```python |
| 49 | +from ragas import SingleTurnSample |
| 50 | +from ragas.metrics import LLMContextPrecisionWithoutReference |
| 51 | + |
| 52 | +context_precision = LLMContextPrecisionWithoutReference(llm=evaluator_llm) |
| 53 | + |
| 54 | +sample = SingleTurnSample( |
| 55 | + user_input="Where is the Eiffel Tower located?", |
| 56 | + response="The Eiffel Tower is located in Paris.", |
| 57 | + retrieved_contexts=["The Eiffel Tower is located in Paris.", "The Brandenburg Gate is located in Berlin."], |
| 58 | +) |
| 59 | + |
| 60 | + |
| 61 | +await context_precision.single_turn_ascore(sample) |
| 62 | +``` |
| 63 | +Output |
| 64 | +``` |
| 65 | +0.9999999999 |
| 66 | +``` |
| 67 | + |
| 68 | +However, if this irrelevant chunk is placed at the first position, context precision reduces. |
| 69 | + |
| 70 | +```python |
| 71 | +from ragas import SingleTurnSample |
| 72 | +from ragas.metrics import LLMContextPrecisionWithoutReference |
| 73 | + |
| 74 | +context_precision = LLMContextPrecisionWithoutReference(llm=evaluator_llm) |
| 75 | + |
| 76 | +sample = SingleTurnSample( |
| 77 | + user_input="Where is the Eiffel Tower located?", |
| 78 | + response="The Eiffel Tower is located in Paris.", |
| 79 | + retrieved_contexts=["The Brandenburg Gate is located in Berlin.", "The Eiffel Tower is located in Paris." ], |
| 80 | +) |
| 81 | + |
| 82 | + |
| 83 | +await context_precision.single_turn_ascore(sample) |
| 84 | +``` |
| 85 | +Output |
| 86 | +``` |
| 87 | +0.49999999995 |
| 88 | +``` |
| 89 | + |
44 | 90 | ### Context Precision with reference |
45 | 91 |
|
46 | 92 | `LLMContextPrecisionWithReference` metric is can be used when you have both retrieved contexts and also reference context associated with a `user_input`. To estimate if a retrieved contexts is relevant or not this method uses the LLM to compare each of the retrieved context or chunk present in `retrieved_contexts` with `reference`. |
|
0 commit comments