Skip to content

Commit c1ed36e

Browse files
authored
docs: add context_precision (#236)
1 parent 92fb854 commit c1ed36e

File tree

3 files changed

+49
-21
lines changed

3 files changed

+49
-21
lines changed

docs/concepts/metrics/context_precision.md

Lines changed: 9 additions & 21 deletions
Original file line numberDiff line numberDiff line change
@@ -1,36 +1,24 @@
11
# Context Precision
22

3-
4-
This metric gauges the precision of the retrieved context, calculated based on both the `question` and `contexts`. The values fall within the range of (0, 1), with higher values indicating better precision.
5-
6-
Ideally, the retrieved context should exclusively contain essential information to address the provided query. To compute this, we initially estimate the value of $|S|$ by identifying sentences within the retrieved context that are relevant for answering the given question. The final score is determined by the following formula:
3+
Context Precision is a metric that evaluates whether all of the ground-truth relevant items present in the `contexts` are ranked higher or not. Ideally all the relevant chunks must appear at the top ranks. This metric is computed using the `question` and the `contexts`, with values ranging between 0 and 1, where higher scores indicate better precision.
74

85
```{math}
9-
:label: context_precision
10-
\text{context precision} = {|S| \over |\text{Total number of sentences in retrived context}|}
11-
```
12-
13-
```{hint}
14-
Question: What is the capital of France?
6+
\text{Context Precision@k} = {\sum {\text{precision@k}} \over \text{total number of relevant items in the top K results}}
7+
````
158
16-
High context precision: France, in Western Europe, encompasses medieval cities, alpine villages and Mediterranean beaches. Paris, its capital, is famed for its fashion houses, classical art museums including the Louvre and monuments like the Eiffel Tower.
17-
18-
Low context precision: France, in Western Europe, encompasses medieval cities, alpine villages and Mediterranean beaches. Paris, its capital, is famed for its fashion houses, classical art museums including the Louvre and monuments like the Eiffel Tower. The country is also renowned for its wines and sophisticated cuisine. Lascaux’s ancient cave drawings, Lyon’s Roman theater and the vast Palace of Versailles attest to its rich history.
19-
20-
```
9+
```{math}
10+
\text{Precision@k} = {\text{true positives@k} \over (\text{true positives@k} + \text{false positives@k})}
11+
````
2112
2213
14+
Where k is the total number of chunks in `contexts`
2315
## Example
2416
2517
```{code-block} python
26-
:caption: Context precision using cross-encoder/nli-deberta-v3-xsmall
18+
:caption: Context precision
2719
from ragas.metrics import ContextPrecision
28-
context_precision = ContextPrecision(
29-
model_name="cross-encoder/nli-deberta-v3-xsmall"
30-
)
20+
context_precision = ContextPrecision()
3121
32-
# run init models to load the models used
33-
context_precision.init_model()
3422
3523
# Dataset({
3624
# features: ['question','contexts'],
Lines changed: 38 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,38 @@
1+
# Context Relevancy
2+
3+
4+
This metric gauges the relevancy of the retrieved context, calculated based on both the `question` and `contexts`. The values fall within the range of (0, 1), with higher values indicating better relevancy.
5+
6+
Ideally, the retrieved context should exclusively contain essential information to address the provided query. To compute this, we initially estimate the value of $|S|$ by identifying sentences within the retrieved context that are relevant for answering the given question. The final score is determined by the following formula:
7+
8+
```{math}
9+
:label: context_relevancy
10+
\text{context relevancy} = {|S| \over |\text{Total number of sentences in retrived context}|}
11+
```
12+
13+
```{hint}
14+
Question: What is the capital of France?
15+
16+
High context relevancy: France, in Western Europe, encompasses medieval cities, alpine villages and Mediterranean beaches. Paris, its capital, is famed for its fashion houses, classical art museums including the Louvre and monuments like the Eiffel Tower.
17+
18+
Low context relevancy: France, in Western Europe, encompasses medieval cities, alpine villages and Mediterranean beaches. Paris, its capital, is famed for its fashion houses, classical art museums including the Louvre and monuments like the Eiffel Tower. The country is also renowned for its wines and sophisticated cuisine. Lascaux’s ancient cave drawings, Lyon’s Roman theater and the vast Palace of Versailles attest to its rich history.
19+
20+
```
21+
22+
23+
## Example
24+
25+
```{code-block} python
26+
:caption: Context relevancy
27+
from ragas.metrics import ContextRelevance
28+
context_relevancy = ContextRelevance()
29+
30+
31+
# Dataset({
32+
# features: ['question','contexts'],
33+
# num_rows: 25
34+
# })
35+
dataset: Dataset
36+
37+
results = context_relevancy.score(dataset)
38+
```

docs/concepts/metrics/index.md

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -13,6 +13,7 @@ Just like in any machine learning system, the performance of individual componen
1313
- [Answer relevancy](answer_relevance.md)
1414
- [Context recall](context_recall.md)
1515
- [Context precision](context_precision.md)
16+
- [Context relevancy](context_relevancy.md)
1617

1718
## End-to-End Evaluation
1819

@@ -28,6 +29,7 @@ Evaluating the end-to-end performance of a pipeline is also crucial, as it direc
2829
faithfulness
2930
answer_relevance
3031
context_precision
32+
context_relevancy
3133
context_recall
3234
semantic_similarity
3335
answer_correctness

0 commit comments

Comments
 (0)