|
1 | 1 | # Context Precision |
2 | 2 |
|
3 | | - |
4 | | -This metric gauges the precision of the retrieved context, calculated based on both the `question` and `contexts`. The values fall within the range of (0, 1), with higher values indicating better precision. |
5 | | - |
6 | | -Ideally, the retrieved context should exclusively contain essential information to address the provided query. To compute this, we initially estimate the value of $|S|$ by identifying sentences within the retrieved context that are relevant for answering the given question. The final score is determined by the following formula: |
| 3 | +Context Precision is a metric that evaluates whether all of the ground-truth relevant items present in the `contexts` are ranked higher or not. Ideally all the relevant chunks must appear at the top ranks. This metric is computed using the `question` and the `contexts`, with values ranging between 0 and 1, where higher scores indicate better precision. |
7 | 4 |
|
8 | 5 | ```{math} |
9 | | -:label: context_precision |
10 | | -\text{context precision} = {|S| \over |\text{Total number of sentences in retrived context}|} |
11 | | -``` |
12 | | - |
13 | | -```{hint} |
14 | | -Question: What is the capital of France? |
| 6 | +\text{Context Precision@k} = {\sum {\text{precision@k}} \over \text{total number of relevant items in the top K results}} |
| 7 | +```` |
15 | 8 |
|
16 | | -High context precision: France, in Western Europe, encompasses medieval cities, alpine villages and Mediterranean beaches. Paris, its capital, is famed for its fashion houses, classical art museums including the Louvre and monuments like the Eiffel Tower. |
17 | | -
|
18 | | -Low context precision: France, in Western Europe, encompasses medieval cities, alpine villages and Mediterranean beaches. Paris, its capital, is famed for its fashion houses, classical art museums including the Louvre and monuments like the Eiffel Tower. The country is also renowned for its wines and sophisticated cuisine. Lascaux’s ancient cave drawings, Lyon’s Roman theater and the vast Palace of Versailles attest to its rich history. |
19 | | -
|
20 | | -``` |
| 9 | +```{math} |
| 10 | +\text{Precision@k} = {\text{true positives@k} \over (\text{true positives@k} + \text{false positives@k})} |
| 11 | +```` |
21 | 12 |
|
22 | 13 |
|
| 14 | +Where k is the total number of chunks in `contexts` |
23 | 15 | ## Example |
24 | 16 |
|
25 | 17 | ```{code-block} python |
26 | | -:caption: Context precision using cross-encoder/nli-deberta-v3-xsmall |
| 18 | +:caption: Context precision |
27 | 19 | from ragas.metrics import ContextPrecision |
28 | | -context_precision = ContextPrecision( |
29 | | - model_name="cross-encoder/nli-deberta-v3-xsmall" |
30 | | - ) |
| 20 | +context_precision = ContextPrecision() |
31 | 21 |
|
32 | | -# run init models to load the models used |
33 | | -context_precision.init_model() |
34 | 22 |
|
35 | 23 | # Dataset({ |
36 | 24 | # features: ['question','contexts'], |
|
0 commit comments