You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: docs/howtos/integrations/amazon_bedrock.md
+24-36Lines changed: 24 additions & 36 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -1,7 +1,7 @@
1
1
2
-
# Create and Evaluate an Agent Integrated with Bedrock Knowledge Bases and Attached Action Group
2
+
# Create and Evaluate an Amazon Bedrock Agent Integrated with an Amazon Bedrock Knowledge Base and Action Groups
3
3
4
-
In this notebook, you will learn how to evaluate an Amazon Bedrock Agent. The agent we'll evaluate is a restaurant agent whose tasks include providing clients with information about adult and children's menus and managing the table booking system. This agent is inspired by a [features example notebooks](https://github.com/aws-samples/amazon-bedrock-samples/tree/main/agents-and-function-calling/bedrock-agents/features-examples/05-create-agent-with-knowledge-base-and-action-group) of [Amazon Bedrock Agents](https://aws.amazon.com/bedrock/agents/) with minor changes. You can learn more about the agent creation process [here](https://github.com/aws-samples/amazon-bedrock-samples/tree/main/agents-and-function-calling/bedrock-agents/features-examples/05-create-agent-with-knowledge-base-and-action-group).
4
+
In this notebook, you will learn how to evaluate an Amazon Bedrock Agent. The agent we'll evaluate is a restaurant agent that provides clients with information about adult and children's menus and manages the table booking system. This agent is inspired by a [features example notebooks](https://github.com/aws-samples/amazon-bedrock-samples/tree/main/agents-and-function-calling/bedrock-agents/features-examples/05-create-agent-with-knowledge-base-and-action-group) of [Amazon Bedrock Agents](https://aws.amazon.com/bedrock/agents/) with minor changes. You can learn more about the agent creation process [here](https://github.com/aws-samples/amazon-bedrock-samples/tree/main/agents-and-function-calling/bedrock-agents/features-examples/05-create-agent-with-knowledge-base-and-action-group).
5
5
6
6
The architecture is illustrated below:
7
7
@@ -22,7 +22,7 @@ The steps covered in this notebook include:
@@ -112,8 +112,7 @@ The steps covered in this notebook include:
112
112
```
113
113
114
114
### Upload the Dataset to Amazon S3
115
-
116
-
Now that we have created the knowledge base, let’s populate it with the restaurant menus dataset. In this example, we will use the [boto3 abstraction](https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/bedrock-agent/client/start_ingestion_job.html) of the API, via our helper classe.
115
+
Now that we have created the knowledge base, let's populate it with the restaurant menus dataset. In this example, we will use [boto3 abstraction](https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/bedrock-agent/client/start_ingestion_job.html) of the API, via our helper classes.
117
116
118
117
Let’s first upload the menu data available in the dataset folder to Amazon S3.
119
118
@@ -503,9 +502,10 @@ The steps covered in this notebook include:
503
502
time.sleep(30)
504
503
```
505
504
505
+
The `invokeAgent` function sends a user query to the Bedrock agent and returns both the agent’s response and trace data. It processes the event stream, capturing trace information for evaluation purposes.
"The item requested by the customer is not present in the menu and no recommendations were made."
592
581
),
593
582
"score0_description": (
594
-
"Either the item requested by the customer is present in the menu, or the conversation does not include any food or menu inquiry (e.g., booking, cancellation), "
595
-
"regardless of whether any recommendation was provided."
583
+
"Either the item requested by the customer is present in the menu, or the conversation does not include any food or menu inquiry (e.g., booking, cancellation). This score applies regardless of whether any recommendation was provided."
596
584
),
597
585
"score1_description": (
598
586
"The item requested by the customer is not present in the menu and a recommendation was provided."
@@ -625,7 +613,7 @@ brand_tone = AspectCritic(
625
613
626
614
## Evaluating Agent with Ragas
627
615
628
-
In order to perform evaluations using Ragas, the traces need to be converted into the format recognized by Ragas. To convert an AWS Bedrock agent trace into a format suitable for Ragas evaluation, Ragas provides the function [convert_to_ragas_messages][ragas.integrations.aws_bedrock.convert_to_ragas_messages], which can be used to transform AWS Bedrock messages into the format expected by Ragas. You can read more about it [here](../../concepts/components/eval_dataset.md).
616
+
In order to perform evaluations using Ragas, the traces need to be converted into the format recognized by Ragas. To convert an Amazon Bedrock agent trace into a format suitable for Ragas evaluation, Ragas provides the function [convert_to_ragas_messages][ragas.integrations.amazon_bedrock.convert_to_ragas_messages], which can be used to transform Amazon Bedrock messages into the format expected by Ragas. You can read more about it [here](../../concepts/components/eval_dataset.md).
629
617
630
618
631
619
```python
@@ -643,7 +631,7 @@ Your booking for 2 people at 7pm on the 5th of May 2025 has been successfully cr
643
631
```
644
632
645
633
```python
646
-
query ="Can you check if my previous booking? can you please delete the booking"
634
+
query ="Can you check my previous booking? Can you please delete the booking?"
For the Recommendation metric, the chicken wings inquiry scored 0 since the item was available, while both the chocolate truffle cake and Indian food inquiries scored 1 because the requested items were not on the menu and alternative recommendations were provided.
836
+
For the Recommendation metric, the chicken wings inquiry scored 0 since the item was available. Both the chocolate truffle cake and Indian food inquiries scored 1 because the requested items were not on the menu and alternative recommendations were provided.
849
837
850
838
To evaluate how well our agent utilizes information retrieved from the knowledge base, we use the RAG evaluation metrics provided by Ragas. You can learn more about these metrics [here]().
851
839
@@ -868,7 +856,7 @@ metrics = [
868
856
869
857
870
858
```python
871
-
from ragas.integrations.aws_bedrockimport extract_kb_trace
859
+
from ragas.integrations.amazon_bedrockimport extract_kb_trace
reference="The desserts on the adult menu are:\n1. Classic New York Cheesecake\n2. Apple Pie à la Mode\n3. Chocolate Lava Cake\4. Pecan Pie Bars\n5. Banana Pudding Parfait",
875
+
reference="The desserts on the adult menu are:\n1. Classic New York Cheesecake\n2. Apple Pie à la Mode\n3. Chocolate Lava Cake\n4. Pecan Pie Bars\n5. Banana Pudding Parfait",
Copy file name to clipboardExpand all lines: docs/howtos/integrations/index.md
+1-1Lines changed: 1 addition & 1 deletion
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -8,7 +8,7 @@ happy to look into it 🙂
8
8
9
9
## Frameworks
10
10
11
-
-[AWS Bedrock](./aws_bedrock.md) - AWS Bedrock is a managed framework for building, deploying, and scaling intelligent agents and integrated AI solutions; more information can be found [here](https://aws.amazon.com/bedrock/).
11
+
-[Amazon Bedrock](./amazon_bedrock.md) - Amazon Bedrock is a managed framework for building, deploying, and scaling intelligent agents and integrated AI solutions; more information can be found [here](https://aws.amazon.com/bedrock/).
12
12
-[Langchain](./langchain.md) - Langchain is a framework for building LLM applications, more information can be found [here](https://www.langchain.com/).
13
13
-[LlamaIndex](./_llamaindex.md) - LlamaIndex is a framework for building RAG applications, more information can be found [here](https://www.llamaindex.ai/).
14
14
-[Haystack](./haystack.md) - Haystack is a LLM orchestration framework to build customizable, production-ready LLM applications, more information can be found [here](https://haystack.deepset.ai/).
0 commit comments