You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: md-docs/user_guide/modules/rag_evaluation.md
+43-16Lines changed: 43 additions & 16 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -1,6 +1,6 @@
1
1
# Rag Evaluation
2
2
3
-
## What is RAG Evaluation?
3
+
## Introduction
4
4
5
5
RAG (Retrieval-Augmented Generation) is a way of building AI models that enhances their ability to generate accurate and contextually relevant responses by combining two main steps: **retrieval** and **generation**.
6
6
@@ -11,9 +11,11 @@ Evaluating RAG involves assessing how well the model does in both retrieval and
11
11
12
12
Our RAG evaluation module analyzes the three main components of a RAG framework:
13
13
14
-
-**User Input**: The query or question posed by the user.
15
-
-**Context**: The retrieved documents or information that the model uses to generate a response. A context can consist of one or more chunks of text.
16
-
-**Response**: The generated answer or output provided by the model.
The evaluation is performed through an LLM-as-a-Judge approach, where a Language Model (LM) acts as a judge to evaluate the quality of a RAG model.
31
+
The evaluation is performed through an LLM-as-a-Judge approach, where a Large Language Model (LLM) acts as a judge to evaluate the quality of a RAG model.
30
32
31
-
## What are the computed metrics?
33
+
## Computed metrics
32
34
33
-
Below are the metrics computed by the RAG evaluation module, divided into the three relationships mentioned above.
35
+
In this paragraph we describe the metrics computed by the RAG evaluation module, divided into the three relationships mentioned above. Every metrics computed is composed by a **score** and an **explanation** provided by the LLM, which explains the reasons behind the assigned score.
-**Relevance**: Measures how much the retrieved context is relevant to the user input. The score ranges from 1 to 5, with 5 being the highest relevance.
38
-
-**Usefulness**: Evaluates how useful the retrieved context is in generating the response. For example, if a context talks about the topic of the user query but it does not contain the information needed to answer the question, it is relevant but not useful. The score ranges from 1 to 5, with 5 being the highest usefulness.
39
-
-**Utilization**: Measures the percentage of the retrieved context that contains information that can be used to generate the response. A higher utilization score indicates that more of the retrieved context is useful for generating the response. The score ranges from 0 to 100.
40
-
-**Attribution**: Indicates which of the chunks of the retrieved context can be used to generate the response. It is a list of the indices of the chunks that are used, with the first chunk having index 1.
50
+
| Metric | Description | Score Range (Lowest-Highest) |
| Relevance | How much the retrieved context is relevant to the user input. | 1-5 |
53
+
| Usefulness | Evaluates how useful the retrieved context is in generating the response, that is if it contains the information to answer the user query | 1-5 |
54
+
| Utilization | Measures the percentage of the retrieved context that contains information for the response. A higher utilization score indicates that more of the retrieved context is useful for generating the response. | 0-100 |
55
+
| Attribution | Indicates which of the chunks of the retrieved context can be used to generate the response. | List of indices of the used chunks, first chunk has index 1 |
56
+
57
+
!!! example
58
+
The **combination** of the metrics provides a comprehensive evaluation of the quality of the retrieved context.
59
+
For instance, a **high relevance** score but **low usefulness** score indicates a context that talks about the topic of the user query but does not contain the information needed to answer it.
41
60
42
61
### Context - Response
43
-
-**Faithfulness**: Measures how much the response contradicts the retrieved context. A higher faithfulness score indicates that the response is more aligned with the context. The score ranges from 1 to 5, with 5 being the highest faithfulness.
62
+
63
+
| Metric | Description | Score Range (Lowest-Highest) |
| Faithfulness | Measures how much the response contradicts the retrieved context. A higher faithfulness score indicates that the response is more aligned with the context. | 1-5 |
44
66
45
67
### User Input - Response
46
-
-**Satisfaction**: Evaluates how satisfied the user would be with the generated response. The score ranges from 1 to 5, with 1 a response that does not address the user query and 5 a response that fully addresses and answers the user query.
47
68
48
-
## What is the required data?
69
+
| Metric | Description | Score Range (Lowest-Highest) |
| Satisfaction | Evaluates how satisfied the user would be with the generated response. A low score indicates a response that does not address the user quey, a high score indicates a response that fully addresses and answers the user query. | 1-5 |
72
+
73
+
## Required data
49
74
50
75
The RAG evaluation module computes the metrics based on the data availability for each sample.
51
76
If a sample lacks one of the three components (User Input, Context or Response), only the applicable metrics are computed.
52
-
For instance, if a sample is missing the response, only the **User Input - Context** metrics are computed.
77
+
For instance, if in a sample the **response is missing**, only the **User Input - Context** metrics are computed for that sample.
78
+
79
+
Regarding the metrics that cannot be computed for a specific sample, the lowest score is assigned, with the explanation mentioning the component that is missing.
53
80
54
81
If data added to a [Task] contains contexts with multiple chunks of text, a [context separator](../task.md#retrieval-augmented-generation) must be provided.
0 commit comments