petclininc agent hallucination issue

[sguruvar]

@awsapm can you check on this issue
Based on my analysis of your system's telemetry data, I've identified the core issue causing your pet clinic agent to recommend nutritional products you don't offer.

Primary Issue: Data Mismatch Between Agent Knowledge and Available Products

Supported Pet Types (200 responses):
• Dog (1,187 requests)
• Cat (1,139 requests)
• Bird (780 requests)
• Hamster (638 requests)
• Snake (619 requests)
• Lizard (404 requests)

Unsupported Pet Types Being Requested (404 responses):
• Rabbit (1,507 requests - highest failure rate!)
• Ferret (369 requests)
• Turtle (367 requests)
• Guinea pig (340 requests)
• Parrot (182 requests)
• Bearded dragon (167 requests)
• Fish (120 requests)
• And many others...

Critical System Issues Identified:

1. Agent Communication Failures:
   • Pet clinic agent → nutrition agent fault rate: 67.82%
   • 100% error rate on AgentCore.Runtime.Invoke operations
   • 25,191 OSError occurrences: "Too many open files"

2. Resource Exhaustion:
   • File descriptor leaks in the pet clinic agent
   • Bedrock throttling exceptions (19 occurrences)
   • Extremely high latency (up to 13.5 million ms)

3. Agent Hallucination:
   • The nutrition agent is generating recommendations for pets not in your database
   • Error logs show: "Nutrition service could not find information for pet: turtle"
   • But the agent continues to provide recommendations anyway

Immediate Action Items:

1. Fix Agent Configuration:
   • Update the pet clinic agent's knowledge base to only include supported pet types
   • Configure proper error handling when nutrition data is unavailable
   • Add validation to prevent recommendations for unsupported pets

2. Resolve Resource Issues:
   • Fix file descriptor leaks in the pet clinic agent code
   • Implement proper connection pooling for Bedrock clients
   • Add circuit breaker patterns for failed nutrition service calls

3. Update Nutrition Database:
   • Either add nutrition data for the missing pet types (rabbit, ferret, turtle, etc.)
   • Or configure the agent to gracefully handle unsupported pets with appropriate messaging

4. Implement Fallback Logic:
   • When nutrition data is unavailable, provide general pet care advice instead of specific product recommendations
   • Add clear messaging about which pets your clinic supports

[github-actions]

🎯 Application observability for AWS Investigation Complete

Investigation completed successfully! Here are the results:

---

The context window has overflowed, summarizing the history...

Critical File Descriptor Leak Analysis

The data reveals 13,831 total errors with a severe pattern of file descriptor exhaustion affecting the Bedrock agent system. Here are the primary issues:

1. File Descriptor Exhaustion (Primary Issue)

• 8,836 errors: Bedrock ThrottlingException due to rate limiting
• 435+ errors: "Too many open files" SSL validation failures
• 283+ errors: SSL connection failures to Bedrock AgentCore
• 279+ errors: SSL failures during agent streaming

2. Root Cause: Connection Leak Pattern

The stack traces show a clear pattern:
OSerror: [Errno 24] Too many open files
urllib3.exceptions.SSLError: [Errno 24] Too many open files

This occurs in:
• SSL certificate validation (ssl_wrap_socket)
• Socket creation (socket.socket.init)
• Network connection establishment (create_connection)

3. Affected Components

• Bedrock Runtime: bedrock-runtime.us-east-1.amazonaws.com
• Bedrock AgentCore: bedrock-agentcore.us-east-1.amazonaws.com
• Nutrition Agent: nutrition_agent-7xJj81BY81

4. Secondary Issues

• 217 errors: Read timeouts (60s timeout exceeded)
• 28 errors: Credential retrieval failures
• 18 errors: Runtime client errors (500 status)

Immediate Remediation Required

This confirms the file descriptor leak identified in our previous analysis. The pet clinic agent is:

1. Not properly closing connections to Bedrock services
2. Exhausting system file descriptors (likely hitting ulimit)
3. Causing cascading failures across the entire agent system
4. Triggering Bedrock throttling due to excessive connection attempts

The system needs immediate connection pooling implementation and proper resource cleanup to prevent this critical resource exhaustion issue.

---

✅ Status: Complete
👤 Requested by: @sguruvar
🔗 Workflow: View details

Powered by AI Agent