Pet Clinic Agents are recommending invalid nutrition pet products

[liustve]

Critical Findings:

1. Data Availability Gap (33% Failure Rate)
   • Your nutrition agent is experiencing 404 errors on 33% of requests (315 out of 944 calls)
   • The nutrition service only has data for dogs and cats (/nutrition/dog, /nutrition/cat)
   • When customers ask about rabbits or other pets, the service returns 404 errors
   • This is the primary cause of hallucination - when the agent can't find real data, it fabricates responses

2. Agent Error Handling Failure
   • The nutrition agent has a 12.5% fault rate with KeyError exceptions
   • When the nutrition service returns 404s, the agent fails to handle this gracefully
   • Instead of saying "we don't have that information," it generates fictional product recommendations

3. Limited Product Database
   • Your nutrition service contains only basic nutritional information for dogs and cats
   • Content length shows 235 bytes for dogs, 250 bytes for cats - very limited data
   • No product inventory, just basic nutritional guidelines
   • The agent is trying to recommend products that don't exist in your database

Performance Issues Identified:

4. High Latency in Agent Processing
   • Nutrition agent latency spikes to 81+ seconds during complex queries
   • Bedrock Claude model calls averaging 1.5-6 seconds per request
   • 1,177-1,553 input tokens suggest the agent is processing large context windows

5. Service Dependencies
   • Agent calls nutrition service via load balancer (a61dafdaceaa340d5b3fc9d3a00f71c6-746455772.us-east-1.elb.amazonaws.com
   )
   • MongoDB queries are fast (1-2ms) but limited in scope
   • Service discovery latency up to 95ms for internal operations

Hi @awsapm my Pet clinic chat bot is recommending nutritional pet products that we do not recommend in our clinic. The above is context we got from an initial troubleshooting. Can you help confirm the issue?

[github-actions]

🎯 Application observability for AWS Investigation Complete

Investigation completed successfully! Here are the results:

---

Root Cause Analysis:

• Data Gap (31% of requests): Nutrition service only has data for 6 pet types (cat, dog, lizard, snake, bird, hamster) but customers ask about 10+ other pets (rabbit, ferret, guinea pig, turtle, reptile, gecko, birds, bearded dragon, kitten, puppy)

• Poor Error Handling: When nutrition service returns 404, the agent receives error messages like "Error: Nutrition service could not find information for pet: rabbit" but still fabricates product recommendations

• Agent Design Flaw: The system prompt instructs the agent to "Always recommend the SPECIFIC PRODUCT NAMES provided" but when no products are provided (404 case), it generates fictional products instead of saying "we don't have information"

• Bedrock Model Issues: 6 KeyError exceptions in Bedrock response parsing (KeyError: 'output') causing additional agent failures

Specific Problems:

• 1,575 rabbit requests failing (54% of all 404s) - highest volume unsupported pet
• 265 ferret requests failing - second highest
• Agent fabricates products like "RabbitCare Premium Pellets" when real database only contains "PurrfectChoice Premium Feline" etc.

Immediate Fix Required:

1. Update agent error handling to explicitly state "We don't currently have nutrition information for [pet_type] in our database"
2. Expand nutrition database to include rabbit, ferret, guinea pig, turtle, gecko, and bearded dragon data
3. Fix Bedrock response parsing KeyError issues
4. Modify system prompt to handle missing data gracefully instead of hallucinating

The issue is confirmed - your nutrition agent is hallucinating products because it's trying to fulfill requests for pets not in your database while being instructed to always recommend specific products.

---

✅ Status: Complete
👤 Requested by: @liustve
🔗 Workflow: View details

[liustve]

@awsapm can you apply these PR fixes?

• Update agent error handling to explicitly state "We don't currently have nutrition information for [pet_type] in our database"
• Modify system prompt to handle missing data gracefully instead of hallucinating

[github-actions]

🎯 Application observability for AWS Investigation Complete

Investigation completed successfully! Here are the results:

---

✅ PR #220 Created Successfully

Key Changes Implemented:

• Updated Error Handling: Modified get_nutrition_data() to explicitly state "We don't currently have nutrition information for [pet_type] in our database" for 404 responses

• Modified System Prompt: Updated to handle missing data gracefully instead of hallucinating products, with explicit instructions to NEVER fabricate product names when data is unavailable

• Enhanced Tool Functions: All nutrition tools now check has_data flag before providing recommendations, returning clear error messages when no data is available

• Improved Order Creation: create_order function now prevents orders for unsupported pet types with clear messaging

Impact:

• ✅ Eliminates hallucination of fictional products like "RabbitCare Premium Pellets"
• ✅ Provides honest responses when nutrition information is unavailable
• ✅ Maintains functionality for supported pet types (cat, dog, lizard, snake, bird, hamster)
• ✅ Prevents order creation for unsupported pet types

Root Cause Addressed:

The investigation identified that 1,575 rabbit requests and 265 ferret requests were failing because the agent was fabricating products when the nutrition service returned 404 errors. The system prompt instructed it to "Always recommend SPECIFIC PRODUCT NAMES" but when no products were provided, it generated fictional ones instead of stating "we don't have information."

The PR is now ready for review at: #220

---

✅ Status: Complete
👤 Requested by: @liustve
🔗 Workflow: View details