Skip to content

Commit ed479d4

Browse files
authored
feat: changing context_relevancy to context_precision (#157)
Right now this change is not a breaking change but `context_relevancy` will be deprecated in 0.1.0.
1 parent 7a12846 commit ed479d4

File tree

14 files changed

+191
-186
lines changed

14 files changed

+191
-186
lines changed

README.md

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -79,7 +79,7 @@ os.environ["OPENAI_API_KEY"] = "your-openai-key"
7979
dataset: Dataset
8080

8181
results = evaluate(dataset)
82-
# {'ragas_score': 0.860, 'context_relevancy': 0.817,
82+
# {'ragas_score': 0.860, 'context_precision': 0.817,
8383
# 'faithfulness': 0.892, 'answer_relevancy': 0.874}
8484
```
8585

@@ -93,7 +93,7 @@ Ragas measures your pipeline's performance against different dimensions:
9393

9494
1. **Faithfulness**: measures the information consistency of the generated answer against the given context. If any claims are made in the answer that cannot be deduced from context is penalized. It is calculated from `answer` and `retrieved context`.
9595

96-
2. **Context Relevancy**: measures how relevant retrieved contexts are to the question. Ideally, the context should only contain information necessary to answer the question. The presence of redundant information in the context is penalized. It is calculated from `question` and `retrieved context`.
96+
2. **Context Precision**: measures how relevant retrieved contexts are to the question. Ideally, the context should only contain information necessary to answer the question. The presence of redundant information in the context is penalized. It is calculated from `question` and `retrieved context`.
9797

9898
3. **Context Recall**: measures the recall of the retrieved context using annotated answer as ground truth. Annotated answer is taken as proxy for ground truth context. It is calculated from `ground truth` and `retrieved context`.
9999

docs/guides/quickstart-azure-openai.ipynb

Lines changed: 5 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -115,7 +115,7 @@
115115
"\n",
116116
"Ragas provides you with a few metrics to evaluate the different aspects of your RAG systems namely\n",
117117
"\n",
118-
"1. metrics to evaluate retrieval: offers `context_relevancy` and `context_recall` which give you the measure of the performance of your retrieval system. \n",
118+
"1. metrics to evaluate retrieval: offers `context_precision` and `context_recall` which give you the measure of the performance of your retrieval system. \n",
119119
"2. metrics to evaluate generation: offers `faithfulness` which measures hallucinations and `answer_relevancy` which measures how to-the-point the answers are to the question.\n",
120120
"\n",
121121
"The harmonic mean of these 4 aspects gives you the **ragas score** which is a single measure of the performance of your QA system across all the important aspects.\n",
@@ -126,7 +126,7 @@
126126
"\n",
127127
"1. **Faithfulness**: measures the information consistency of the generated answer against the given context. If any claims are made in the answer that cannot be deduced from context is penalized. It is calculated from `answer` and `retrieved context`.\n",
128128
"\n",
129-
"2. **Context Relevancy**: measures how relevant retrieved contexts are to the question. Ideally, the context should only contain information necessary to answer the question. The presence of redundant information in the context is penalized. It is calculated from `question` and `retrieved context`.\n",
129+
"2. **Context Precision**: measures how relevant retrieved contexts are to the question. Ideally, the context should only contain information necessary to answer the question. The presence of redundant information in the context is penalized. It is calculated from `question` and `retrieved context`.\n",
130130
"\n",
131131
"3. **Context Recall**: measures the recall of the retrieved context using annotated answer as ground truth. Annotated answer is taken as proxy for ground truth context. It is calculated from `ground truth` and `retrieved context`.\n",
132132
"\n",
@@ -183,7 +183,7 @@
183183
"outputs": [],
184184
"source": [
185185
"from ragas.metrics import (\n",
186-
" context_relevancy,\n",
186+
" context_precision,\n",
187187
" answer_relevancy,\n",
188188
" faithfulness,\n",
189189
" context_recall,\n",
@@ -193,9 +193,9 @@
193193
"# list of metrics we're going to use\n",
194194
"metrics = [\n",
195195
" faithfulness,\n",
196-
" answer_relevancy\n",
196+
" answer_relevancy,\n",
197197
" context_recall,\n",
198-
" context_relevancy,\n",
198+
" context_precision,\n",
199199
" harmfulness,\n",
200200
"]"
201201
]

0 commit comments

Comments
 (0)