Merge branch 'feature/optimize-agent-analytics-metadata' into 'develop'

Optimize analytics agent by embedding database overview in system prompt

See merge request genaiic-reusable-assets/engagement-artifacts/genaiic-idp-accelerator!324

Author: rstrahan

CHANGELOG.md

@@ -6,11 +6,12 @@ SPDX-License-Identifier: MIT-0
 ## [Unreleased]
 
 ### Added
-- **Analytics Agent 2-Phase Schema Optimization for Improved Performance**
-  - Implemented progressive schema disclosure system with efficient 2-phase approach
-  - Phase 1: `get_database_overview()` provides fast table listing and guidance (~500 tokens vs 3000+ tokens - 6x faster)
-  - Phase 2: `get_table_info(['specific_tables'])` loads detailed schemas only for tables actually needed by the query
-  - Enhanced SQL guidance with comprehensive Athena/Trino function reference and PostgreSQL operator warnings to prevent common query failures
+- **Analytics Agent Schema Optimization for Improved Performance**
+  - **Embedded Database Overview**: Complete table listing and guidance embedded directly in system prompt (no tool call needed)
+  - **On-Demand Detailed Schemas**: `get_table_info(['specific_tables'])` loads detailed column information only for tables actually needed by the query
+  - **Significant Performance Gains**: Eliminates redundant tool calls on every query while maintaining token efficiency
+  - **Enhanced SQL Guidance**: Comprehensive Athena/Trino function reference with explicit PostgreSQL operator warnings to prevent common query failures like `~` regex operator mistakes
+  - **Faster Time-to-Query**: Agent has immediate access to table overview and can proceed directly to detailed schema loading for relevant tables
 
 ### Fixed
 - Fix missing data in Glue tables when using a document class that contains a dash (-). 

lib/idp_common_pkg/idp_common/agents/analytics/agent.py

@@ -15,9 +15,9 @@
 from ..common.config import load_result_format_description
 from ..common.strands_bedrock_model import create_strands_bedrock_model
 from .config import load_python_plot_generation_examples
+from .schema_provider import get_database_overview as _get_database_overview
 from .tools import (
     CodeInterpreterTools,
-    get_database_overview,
     get_table_info,
     run_athena_query,
 )
@@ -48,14 +48,17 @@ def create_analytics_agent(
     # Load python code examples
     python_plot_generation_examples = load_python_plot_generation_examples()
 
+    # Load database overview once during agent creation for embedding in system prompt
+    database_overview = _get_database_overview()
+
     # Define the system prompt for the analytics agent
     system_prompt = f"""
     You are an AI agent that converts natural language questions into Athena queries, executes those queries, and writes python code to convert the query results into json representing either a plot, a table, or a string.
     
     # Task
     Your task is to:
     1. Understand the user's question
-    2. **EFFICIENT APPROACH**: Use get_database_overview() to get a fast overview of available tables and their purposes
+    2. **EFFICIENT APPROACH**: Review the database overview below to see available tables and their purposes
     3. Apply the Question-to-Table mapping rules below to select the correct tables for your query
     4. Use get_table_info(['table1', 'table2']) to get detailed schemas ONLY for the tables you need
     5. Generate a valid Athena query based on the targeted schema information
@@ -70,15 +73,18 @@ def create_analytics_agent(
     9. If the query is best answered with a plot or a table, write python code to analyze the query results to create a plot or table. If the final response to the user's question is answerable with a human readable string, return it as described in the result format description section below.
     10. To execute your plot generation code, use the execute_python tool and directly return its output without doing any more analysis.
 
-    # CRITICAL: Two-Step Database Information Approach
+    # Database Overview - Available Tables
+    {database_overview}
+    
+    # CRITICAL: Optimized Database Information Approach
     **For optimal performance and accuracy:**
     
-    ## Step 1: Overview (Fast)
-    - Always start with `get_database_overview()` to see available tables
+    ## Step 1: Review Database Overview (Above)
+    - The complete database overview is provided above in this prompt
     - This gives you table names, purposes, and question-to-table mapping guidance
-    - **~500 tokens vs 3000+ tokens** - much faster for simple questions
+    - No tool call needed - information is immediately available
     
-    ## Step 2: Detailed Schemas (On-Demand) 
+    ## Step 2: Get Detailed Schemas (On-Demand Only)
     - Use `get_table_info(['table1', 'table2'])` for specific tables you need
     - Only request detailed info for tables relevant to your query
     - Get complete column listings, sample queries, and aggregation rules
@@ -287,7 +293,6 @@ def run_athena_query_with_config(
         run_athena_query_with_config,
         code_interpreter_tools.write_query_results_to_code_sandbox,
         code_interpreter_tools.execute_python,
-        get_database_overview,  # Fast, lightweight table overview
         get_table_info,  # Detailed schema for specific tables
     ]