Skip to content

Commit 98aafbe

Browse files
Refactor: Update AWS Bedrock Naming to Amazon Bedrock Across Docs & Repo Structure (#1982)
- Changed all instances of “AWS Bedrock” to “Amazon Bedrock” in URLs, sidebars, and text within our documentation.
1 parent a5e4015 commit 98aafbe

File tree

5 files changed

+27
-40
lines changed

5 files changed

+27
-40
lines changed

docs/howtos/integrations/aws_bedrock.md renamed to docs/howtos/integrations/amazon_bedrock.md

Lines changed: 24 additions & 36 deletions
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,7 @@
11

2-
# Create and Evaluate an Agent Integrated with Bedrock Knowledge Bases and Attached Action Group
2+
# Create and Evaluate an Amazon Bedrock Agent Integrated with an Amazon Bedrock Knowledge Base and Action Groups
33

4-
In this notebook, you will learn how to evaluate an Amazon Bedrock Agent. The agent we'll evaluate is a restaurant agent whose tasks include providing clients with information about adult and children's menus and managing the table booking system. This agent is inspired by a [features example notebooks](https://github.com/aws-samples/amazon-bedrock-samples/tree/main/agents-and-function-calling/bedrock-agents/features-examples/05-create-agent-with-knowledge-base-and-action-group) of [Amazon Bedrock Agents](https://aws.amazon.com/bedrock/agents/) with minor changes. You can learn more about the agent creation process [here](https://github.com/aws-samples/amazon-bedrock-samples/tree/main/agents-and-function-calling/bedrock-agents/features-examples/05-create-agent-with-knowledge-base-and-action-group).
4+
In this notebook, you will learn how to evaluate an Amazon Bedrock Agent. The agent we'll evaluate is a restaurant agent that provides clients with information about adult and children's menus and manages the table booking system. This agent is inspired by a [features example notebooks](https://github.com/aws-samples/amazon-bedrock-samples/tree/main/agents-and-function-calling/bedrock-agents/features-examples/05-create-agent-with-knowledge-base-and-action-group) of [Amazon Bedrock Agents](https://aws.amazon.com/bedrock/agents/) with minor changes. You can learn more about the agent creation process [here](https://github.com/aws-samples/amazon-bedrock-samples/tree/main/agents-and-function-calling/bedrock-agents/features-examples/05-create-agent-with-knowledge-base-and-action-group).
55

66
The architecture is illustrated below:
77

@@ -22,7 +22,7 @@ The steps covered in this notebook include:
2222

2323

2424
```python
25-
%pip install --upgrade -q boto3 opensearch-py botocore awscli retrying ragas
25+
%pip install --upgrade -q boto3 opensearch-py botocore awscli retrying ragas langchain-aws
2626
```
2727

2828

@@ -112,8 +112,7 @@ The steps covered in this notebook include:
112112
```
113113

114114
### Upload the Dataset to Amazon S3
115-
116-
Now that we have created the knowledge base, let’s populate it with the restaurant menus dataset. In this example, we will use the [boto3 abstraction](https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/bedrock-agent/client/start_ingestion_job.html) of the API, via our helper classe.
115+
Now that we have created the knowledge base, let's populate it with the restaurant menus dataset. In this example, we will use [boto3 abstraction](https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/bedrock-agent/client/start_ingestion_job.html) of the API, via our helper classes.
117116

118117
Let’s first upload the menu data available in the dataset folder to Amazon S3.
119118

@@ -503,9 +502,10 @@ The steps covered in this notebook include:
503502
time.sleep(30)
504503
```
505504

505+
The `invokeAgent` function sends a user query to the Bedrock agent and returns both the agent’s response and trace data. It processes the event stream, capturing trace information for evaluation purposes.
506506

507507
```python
508-
def invokeAgent(query, session_id, enable_trace=True, session_state=dict()):
508+
def invokeAgent(query, session_id, session_state=dict()):
509509
end_session: bool = False
510510

511511
# invoke the agent API
@@ -514,7 +514,7 @@ def invokeAgent(query, session_id, enable_trace=True, session_state=dict()):
514514
agentId=agent_id,
515515
agentAliasId=alias_id,
516516
sessionId=session_id,
517-
enableTrace=enable_trace,
517+
enableTrace=True,
518518
endSession=end_session,
519519
sessionState=session_state,
520520
)
@@ -560,24 +560,13 @@ Ragas includes metrics suited to such evaluations, and we will explore some of t
560560

561561

562562
```python
563-
import getpass
564-
import os
565-
566-
if "OPENAI_API_KEY" not in os.environ:
567-
os.environ["OPENAI_API_KEY"] = getpass.getpass("Enter your OpenAI API key: ")
568-
```
563+
from langchain_aws import ChatBedrock
569564

565+
model_id = "us.amazon.nova-pro-v1:0" # Choose your desired model
566+
region_name = "us-east-1" # Choose your desired AWS region
570567

571-
```python
572-
from ragas.llms import LangchainLLMWrapper
573-
from ragas.embeddings import LangchainEmbeddingsWrapper
574-
from langchain_openai import ChatOpenAI, OpenAIEmbeddings
575-
576-
577-
evaluator_llm = LangchainLLMWrapper(ChatOpenAI(model="gpt-4o-mini"))
578-
evaluator_embeddings = LangchainEmbeddingsWrapper(
579-
OpenAIEmbeddings(model="text-embedding-3-small")
580-
)
568+
bedrock_llm = ChatBedrock(model_id=model_id, region_name=region_name)
569+
evaluator_llm = LangchainLLMWrapper(bedrock_llm)
581570
```
582571

583572

@@ -591,8 +580,7 @@ rubrics = {
591580
"The item requested by the customer is not present in the menu and no recommendations were made."
592581
),
593582
"score0_description": (
594-
"Either the item requested by the customer is present in the menu, or the conversation does not include any food or menu inquiry (e.g., booking, cancellation), "
595-
"regardless of whether any recommendation was provided."
583+
"Either the item requested by the customer is present in the menu, or the conversation does not include any food or menu inquiry (e.g., booking, cancellation). This score applies regardless of whether any recommendation was provided."
596584
),
597585
"score1_description": (
598586
"The item requested by the customer is not present in the menu and a recommendation was provided."
@@ -625,7 +613,7 @@ brand_tone = AspectCritic(
625613

626614
## Evaluating Agent with Ragas
627615

628-
In order to perform evaluations using Ragas, the traces need to be converted into the format recognized by Ragas. To convert an AWS Bedrock agent trace into a format suitable for Ragas evaluation, Ragas provides the function [convert_to_ragas_messages][ragas.integrations.aws_bedrock.convert_to_ragas_messages], which can be used to transform AWS Bedrock messages into the format expected by Ragas. You can read more about it [here](../../concepts/components/eval_dataset.md).
616+
In order to perform evaluations using Ragas, the traces need to be converted into the format recognized by Ragas. To convert an Amazon Bedrock agent trace into a format suitable for Ragas evaluation, Ragas provides the function [convert_to_ragas_messages][ragas.integrations.amazon_bedrock.convert_to_ragas_messages], which can be used to transform Amazon Bedrock messages into the format expected by Ragas. You can read more about it [here](../../concepts/components/eval_dataset.md).
629617

630618

631619
```python
@@ -643,7 +631,7 @@ Your booking for 2 people at 7pm on the 5th of May 2025 has been successfully cr
643631
```
644632

645633
```python
646-
query = "Can you check if my previous booking? can you please delete the booking"
634+
query = "Can you check my previous booking? Can you please delete the booking?"
647635
agent_answer, traces_2 = invokeAgent(query, session_id)
648636

649637
print(agent_answer)
@@ -654,10 +642,10 @@ Your reservation was found and has been successfully canceled.
654642
```
655643

656644
```python
657-
from ragas.integrations.aws_bedrock import convert_to_ragas_messages
645+
from ragas.integrations.amazon_bedrock import convert_to_ragas_messages
658646

659-
# Convert AWS traces to messages accepted by RAGAS.
660-
# The convert_to_ragas_messages function transforms AWS-specific trace data
647+
# Convert Amazon Bedrock traces to messages accepted by RAGAS.
648+
# The convert_to_ragas_messages function transforms Bedrock-specific trace data
661649
# into a format that RAGAS can process as conversation messages.
662650
ragas_messages_trace_1 = convert_to_ragas_messages(traces_1)
663651
ragas_messages_trace_2 = convert_to_ragas_messages(traces_2)
@@ -782,7 +770,7 @@ I could not find Indian food on our menu. However, we offer a variety of other c
782770
```
783771

784772
```python
785-
from ragas.integrations.aws_bedrock import convert_to_ragas_messages
773+
from ragas.integrations.amazon_bedrock import convert_to_ragas_messages
786774

787775
ragas_messages_trace_3 = convert_to_ragas_messages(traces_3)
788776
ragas_messages_trace_4 = convert_to_ragas_messages(traces_4)
@@ -845,7 +833,7 @@ Evaluating: 100%|██████████| 3/3 [00:00<?, ?it/s]
845833

846834

847835

848-
For the Recommendation metric, the chicken wings inquiry scored 0 since the item was available, while both the chocolate truffle cake and Indian food inquiries scored 1 because the requested items were not on the menu and alternative recommendations were provided.
836+
For the Recommendation metric, the chicken wings inquiry scored 0 since the item was available. Both the chocolate truffle cake and Indian food inquiries scored 1 because the requested items were not on the menu and alternative recommendations were provided.
849837

850838
To evaluate how well our agent utilizes information retrieved from the knowledge base, we use the RAG evaluation metrics provided by Ragas. You can learn more about these metrics [here]().
851839

@@ -868,7 +856,7 @@ metrics = [
868856

869857

870858
```python
871-
from ragas.integrations.aws_bedrock import extract_kb_trace
859+
from ragas.integrations.amazon_bedrock import extract_kb_trace
872860

873861
kb_trace_3 = extract_kb_trace(traces_3)
874862
kb_trace_4 = extract_kb_trace(traces_4)
@@ -884,7 +872,7 @@ trace_4_single_turn_sample = SingleTurnSample(
884872
user_input=kb_trace_4[0].get("user_input"),
885873
retrieved_contexts=kb_trace_4[0].get("retrieved_contexts"),
886874
response=kb_trace_4[0].get("response"),
887-
reference="The desserts on the adult menu are:\n1. Classic New York Cheesecake\n2. Apple Pie à la Mode\n3. Chocolate Lava Cake\4. Pecan Pie Bars\n5. Banana Pudding Parfait",
875+
reference="The desserts on the adult menu are:\n1. Classic New York Cheesecake\n2. Apple Pie à la Mode\n3. Chocolate Lava Cake\n4. Pecan Pie Bars\n5. Banana Pudding Parfait",
888876
)
889877

890878
single_turn_samples = [trace_3_single_turn_sample, trace_4_single_turn_sample]
@@ -992,7 +980,7 @@ Here are the entrees available for children:
992980
```
993981

994982
```python
995-
from ragas.integrations.aws_bedrock import convert_to_ragas_messages
983+
from ragas.integrations.amazon_bedrock import convert_to_ragas_messages
996984

997985
ragas_messages_trace_6 = convert_to_ragas_messages(traces_6)
998986

@@ -1092,7 +1080,7 @@ Evaluating: 100%|██████████| 1/1 [00:00<?, ?it/s]
10921080

10931081

10941082

1095-
In both scenarios, the agent earned a score of 1 by comprehensively providing all available options—whether listing all childrens entrees.
1083+
In both scenarios, the agent earned a score of 1 by comprehensively providing all available options—specifically by listing all children's entrees.
10961084

10971085
## Clean-up
10981086
Let's delete all the associated resources created to avoid unnecessary costs.

docs/howtos/integrations/index.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -8,7 +8,7 @@ happy to look into it 🙂
88

99
## Frameworks
1010

11-
- [AWS Bedrock](./aws_bedrock.md) - AWS Bedrock is a managed framework for building, deploying, and scaling intelligent agents and integrated AI solutions; more information can be found [here](https://aws.amazon.com/bedrock/).
11+
- [Amazon Bedrock](./amazon_bedrock.md) - Amazon Bedrock is a managed framework for building, deploying, and scaling intelligent agents and integrated AI solutions; more information can be found [here](https://aws.amazon.com/bedrock/).
1212
- [Langchain](./langchain.md) - Langchain is a framework for building LLM applications, more information can be found [here](https://www.langchain.com/).
1313
- [LlamaIndex](./_llamaindex.md) - LlamaIndex is a framework for building RAG applications, more information can be found [here](https://www.llamaindex.ai/).
1414
- [Haystack](./haystack.md) - Haystack is a LLM orchestration framework to build customizable, production-ready LLM applications, more information can be found [here](https://haystack.deepset.ai/).

mkdocs.yml

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -110,7 +110,7 @@ nav:
110110
- Integrations:
111111
- howtos/integrations/index.md
112112
- Arize: howtos/integrations/_arize.md
113-
- AWS Bedrock: howtos/integrations/aws_bedrock.md
113+
- Amazon Bedrock: howtos/integrations/amazon_bedrock.md
114114
- Haystack: howtos/integrations/haystack.md
115115
- LangChain: howtos/integrations/langchain.md
116116
- LangGraph: howtos/integrations/_langgraph_agent_evaluation.md

src/ragas/embeddings/base.py

Lines changed: 1 addition & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -244,10 +244,9 @@ def __post_init__(self):
244244
self.model_name, **self.model_kwargs
245245
)
246246
else:
247-
self.model = sentence_transformers.SentenceTransformer(
247+
self.model = sentence_transformers.SentenceTransformer( # type: ignore
248248
self.model_name, cache_folder=self.cache_folder, **self.model_kwargs
249249
)
250-
251250
# ensure outputs are tensors
252251
if "convert_to_tensor" not in self.encode_kwargs:
253252
self.encode_kwargs["convert_to_tensor"] = True
File renamed without changes.

0 commit comments

Comments
 (0)