You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
RAG (Retrieval-Augmented Generation) is a way of building AI models that enhances their ability to generate accurate and contextually relevant responses by combining two main steps: **retrieval** and **generation**.
6
4
7
-
1.**Retrieval**: The model first searches through a large set of documents or pieces of information to "retrieve" the most relevant ones based on the user query.
5
+
1.**Retrieval**: The model first searches through a large set of documents or pieces of information from a specific knowledge base defined by the system designer to "retrieve" the most relevant ones based on the user query.
8
6
2.**Generation**: It then uses these retrieved documents as context to generate a response, which is typically more accurate and aligned with the question than if it had generated text from scratch without specific guidance.
9
7
10
-
Evaluating RAG involves assessing how well the model does in both retrieval and generation.
8
+
Evaluating RAG involves assessing how well the model performs in both retrieval and generation. This evaluation is crucial to ensure that the model provides accurate and relevant responses to user queries.
11
9
12
-
The RAG evaluation module analyzes the three main components of a RAG framework:
| User Input | The query or question posed by the user. |
17
15
| Context | The retrieved documents or information that the model uses to generate a response. |
18
16
| Response | The generated answer or output provided by the model. |
19
17
20
-
In particular, the analysis is performed on the relationships between these components:
18
+
!!! example
19
+
This is an example of the three components of a RAG:
20
+
21
+
- **User Input**: "What is the capital of France?"
22
+
- **Context**: "Paris, the capital of France, ..."
23
+
- **Response**: "The capital of France is Paris."
24
+
25
+
## RAG Evaluation Module
26
+
The ML cube Platform RAG evaluation module is available for [RAG Tasks](../task.md#retrieval-augmented-generation) and generates an evaluation report for a given set of samples.
27
+
28
+
!!! info
29
+
It is possible to compute a RAG evaluation report both from [Web App] and [SDK]. The computed report can be viewed in the Web App and exported as an Excel file from the SDK.
<figcaption>The three evaluations performed by the RAG Evaluation Module</figcaption>
31
36
</figure>
32
37
33
-
The evaluation is performed through an LLM-as-a-Judge approach, where a Large Language Model (LLM) acts as a judge to evaluate the quality of a RAG model.
34
38
35
-
## Computed metrics
39
+
###Computed metrics
36
40
37
-
This paragraph describes the metrics computed by the RAG evaluation module, divided into the three relationships mentioned above. Every metrics computed is composed by a **score** and an **explanation** provided by the LLM, which explains the reasons behind the assigned score.
This paragraph describes the metrics computed by the RAG evaluation module, divided into the three relationships shown above. Every metrics computed is composed by a **value** and an **explanation** of the reasons behind the assigned value.
49
42
50
-
###User Input - Context
43
+
#### Retrieval Evaluation (User Input - Context)
51
44
52
-
| Metric | Description |Score Range (Lowest-Highest)|
| Relevance | How much the retrieved context is relevant to the user input. | 1-5 |
55
-
| Usefulness | How useful the retrieved context is in generating the response, that is if it contains the information to answer the user query. | 1-5 |
56
-
| Utilization | The percentage of the retrieved context that contains information for the response. A higher utilization score indicates that more of the retrieved context is useful for generating the response. | 0-100 |
47
+
| Relevance | How much the retrieved context is relevant to the user input. | 1-5 (lowest-highest)|
48
+
| Usefulness | How useful the retrieved context is in generating the response, that is if it contains the information to answer the user query. | 1-5 (lowest-highest)|
49
+
| Utilization | The percentage of the retrieved context that contains information for the response. A higher utilization score indicates that more of the retrieved context is useful for generating the response. | 0-100 (lowest-highest)|
57
50
| Attribution | Which of the chunks of the retrieved context can be used to generate the response. | List of indices of the used chunks, first chunk has index 1 |
58
51
59
-
!!! example
52
+
!!! note
60
53
The **combination** of the metrics provides a comprehensive evaluation of the quality of the retrieved context.
61
-
For instance, a **high relevance** score but **low usefulness** score indicates a context that talks about the topic of the user query but does not contain the information needed to answer it.
62
54
63
-
### Context - Response
55
+
For instance, a **high relevance** score but **low usefulness** score indicates a context that talks about the topic of the user query but does not contain the information needed to answer it:
56
+
57
+
- **User Input**: "How many ECTS does a Computer Science bachelor's degree have?"
58
+
- **Context**: "The main exams in a Bachelor's Degree in Computer Science typically cover programming, data structures, algorithms, computer architecture, operating systems, databases, and software engineering."
59
+
60
+
This example has a high relevance score because the context talks about a Computer Science bachelor's degree, but a low usefulness score because it does not contain the specific information about the number of ECTS.
| Faithfulness | How much the response contradicts the retrieved context. A higher faithfulness score indicates that the response is more aligned with the context. | 1-5 |
| Faithfulness | How much the response contradicts the retrieved context. A higher faithfulness score indicates that the response is more aligned with the context. | 1-5 (lowest-highest)|
68
67
69
-
###User Input - Response
68
+
#### Response Evaluation (User Input - Response)
70
69
71
-
| Metric | Description | Score Range (Lowest-Highest)|
| Satisfaction | How satisfied the user would be with the generated response. A low score indicates a response that does not address the user query, a high score indicates a response that fully addresses and answers the user query. | 1-5 |
| Satisfaction | How satisfied the user would be with the generated response. A low score indicates a response that does not address the user query, a high score indicates a response that fully addresses and answers the user query. | 1-5 (lowest-highest)|
74
73
75
-
## Required data
74
+
!!! example
75
+
This is an example of the three evaluations performed by the RAG Evaluation Module:
76
+
77
+
- **User Input**: "How many ECTS does a Computer Science bachelor's degree have?"
78
+
- **Context**: "The main exams in a Bachelor's Degree in Computer Science typically cover programming, data structures, algorithms, computer architecture, operating systems, databases, and software engineering."
79
+
- **Response**: "A Bachelor's Degree in Computer Science typically has 180 ECTS."
The RAG evaluation module computes the metrics based on the data availability for each sample.
78
-
If a sample lacks one of the three components (User Input, Context or Response), only the applicable metrics are computed.
103
+
The RAG evaluation module computes the metrics for each sample based on the data availability.
104
+
If a sample lacks one of the three components (User Input, Context or Response), only the applicable metrics are computed for that sample.
79
105
For instance, if in a sample the **response is missing**, only the **User Input - Context** metrics are computed for that sample.
80
106
81
107
Regarding the metrics that cannot be computed for a specific sample, the lowest score is assigned, with the explanation mentioning the component that is missing.
@@ -89,12 +115,30 @@ When requesting the evaluation, a **timestamp interval** must be provided to spe
89
115
The following code demonstrates how to compute a rag evaluation report for a given timestamp interval.
0 commit comments