Pet Clinic AI Chatbot is recommending nutritional products not available in our clinic

[liustve]

Based on my comprehensive analysis, I've identified the root cause of your pet clinic chatbot hallucination issue. Here's what's happening:

Root Cause Analysis

The Problem: Your pet clinic chatbot is recommending nutritional products you don't offer because the nutrition agent is hallucinating responses when
the nutrition service returns 404 errors.


Key Findings:

1. High 404 Error Rate: The nutrition-service-nodejs is returning 60% 404 responses (10,816 out of 18,008 requests), indicating customers are asking 
   about pet types not in your database.
2. Agent Hallucination Pattern: When the nutrition service returns "Error: Nutrition service could not find information for pet: [pet_type]", the 
   nutrition agent is fabricating product recommendations instead of gracefully handling the missing data.
3. Service Architecture Issue: The nutrition agent is designed to provide recommendations but lacks proper fallback logic when the underlying 
   nutrition service doesn't have data for specific pet types.
4. Error Propagation: The sample error logs show the nutrition agent receiving tool results like:
     

      "Nutrition info for gecko: Error: Nutrition service could not find information for pet: gecko"
      "Nutrition info for fish: Error: Nutrition service could not find information for pet: fish"
      
     But instead of saying "we don't carry products for that pet type," it's making up recommendations.

Hi @awsapm my pet clinic chat bot is recommending nutritional pet products we do not have in our clinic. The above is context we got from an initial troubleshooting. Can you help confirm the issue?

[github-actions]

❌ Application observability for AWS Investigation Failed

The investigation could not be completed. Please check the workflow logs for more details.

👤 Requested by: @liustve
🔗 Workflow: View details

If this issue persists, please check your action configuration and try again.