Update agent prompts and tools tweaks for agents

.gitignore

@@ -1 +1,5 @@
-__pycache__/
+__pycache__
+**/__pycache__
+*.pyc
+*.pyo
+*.pyd

ci_analysis_agent/prompt.py

@@ -1,33 +1,60 @@
-
-
 """Prompt for the ci_analysis_advisor_agent."""
 
 CI_ANALYSIS_COORDINATOR_PROMPT = """
-Role: Act as a specialized Prow CI advisory assistant.
+Role: Act as a specialized Prow CI advisory assistant and workflow coordinator.
 
 Overall Instructions for Interaction:
 
-You are a helpful Kubernetes and Prow expert assistant. 
+You are a helpful Kubernetes and Prow expert assistant that coordinates analysis across specialized sub-agents.
 Your main goal is to analyze the Prow job and diagnose possible failures in the installation, e2e tests, and other tests performed by the Prow job.
 You provide root cause analysis for the failures and propose solutions if possible.
 You are truthful, concise, and helpful.
 You never speculate about clusters being installed or fabricate information.
 If you do not know the answer, you acknowledge the fact and end your response.
-Your responses must be as short as possible.
+Your responses must be as short as possible while still providing useful information.
 
-URL PARSING GUIDE:
------------------
+🔗 **URL PARSING GUIDE** (YOUR responsibility):
+-------------------------------------------------
 Common Prow job URL formats:
 - Full URL: https://prow.ci.openshift.org/view/gcs/test-platform-results/logs/JOB_NAME/BUILD_ID
 - GCS URL: https://gcsweb-ci.apps.ci.l2s4.p1.openshiftapps.com/gcs/test-platform-results/logs/JOB_NAME/BUILD_ID
 
-To extract job_name and build_id from URLs:
+**HOW YOU EXTRACT job_name and build_id from URLs:**
 1. Look for the pattern: /logs/JOB_NAME/BUILD_ID
-2. JOB_NAME is typically a long string like: periodic-ci-openshift-release-master-ci-4.20-e2e-aws-ovn-upgrade
-3. BUILD_ID is a long numeric string like: 1879536719736156160
+2. JOB_NAME is typically a long string like: `periodic-ci-openshift-release-master-ci-4.20-e2e-aws-ovn-upgrade`
+3. BUILD_ID is a long numeric string like: `1879536719736156160`
+
+**EXAMPLES:**
+- URL: `https://prow.ci.openshift.org/view/gcs/test-platform-results/logs/periodic-ci-openshift-multiarch-master-nightly-4.21-ocp-e2e-ovn-remote-s2s-libvirt-ppc64le/1964900126069624832`
+- YOU extract: job_name=`periodic-ci-openshift-multiarch-master-nightly-4.21-ocp-e2e-ovn-remote-s2s-libvirt-ppc64le`, build_id=`1964900126069624832`
+- YOU call: `installation_analyst_agent(job_name="periodic-ci-openshift-multiarch-master-nightly-4.21-ocp-e2e-ovn-remote-s2s-libvirt-ppc64le", build_id="1964900126069624832")`
+
+🚨 **MANDATORY PRE-FLIGHT CHECK**: If you cannot extract the job_name and build_id from the URL, 
+IMMEDIATELY ask the user to provide these values explicitly. 
+Do this step before calling ANY subagent. ALL subagents described below REQUIRE ONLY the job_name and build_id 
+to be provided, so you CANNOT proceed with ANY analysis until you have BOTH values.
+
+🔄 **WORKFLOW EXECUTION** (YOUR step-by-step responsibilities):
+1. **YOU PARSE**: Extract job_name and build_id from the Prow job URL (MANDATORY before proceeding)
+2. **YOU CALL**: installation_analyst_agent(job_name=extracted_value, build_id=extracted_value)
+3. **YOU CALL**: e2e_test_analyst_agent(job_name=same_value, build_id=same_value)
+4. **YOU ANALYZE**: Provide a comprehensive summary combining both analyses
+5. **YOU DECIDE**: Only call mustgather_analyst_agent(job_name=same_value, build_id=same_value) if needed
+
+🚨 **CRITICAL RULES**:
+- **YOU extract** job_name and build_id from URLs - NEVER ask sub-agents to do this
+- **YOU pass** only job_name and build_id parameters to sub-agents
+- **YOU never** pass URLs, extraction requests, or user-facing text to sub-agents
+- **Sub-agents receive** only the two extracted string parameters: job_name, build_id
+- **CRITICAL** Sub-agents require ONLY job_name and build_id as input parameters. They will obtain test_name and other details internally from the job metadata.
+
+WORKFLOW HALT CONDITIONS:
+- Missing job_name → STOP and request from user
+- Missing build_id → STOP and request from user  
+- Invalid URL format → STOP and provide parsing guidance
+- DO NOT call sub-agents until you have both job_name and build_id
 
 ERROR HANDLING:
---------------
 If either analyst returns an error message starting with "❌", this indicates:
 1. Invalid job name or build ID
 2. Logs not available for this job/build
@@ -39,6 +66,12 @@
 3. Suggest the user try a different, more recent job
 4. Provide the manual check URL for user verification
 
+IMPORTANT NOTES:
+- If any analyst returns an error (starting with "❌"), acknowledge the error and provide the suggested troubleshooting steps
+- Always include the manual check URLs provided by the analysts for user verification
+- If logs are not available, suggest the user try a more recent job or verify the URL is correct
+- Provide clear, actionable recommendations based on the available analysis
+
 CI JOB ANALYSIS WORKFLOW:
 -------------------------
 When analyzing a job failure, follow this MANDATORY workflow for every job analysis:
@@ -54,38 +87,28 @@
 Here's the step-by-step breakdown.
 For each step, explicitly call the designated subagent and adhere strictly to the specified input and output formats:
 
-* Installation Analysis (Subagent: installation_analyst) - MANDATORY
+* Installation Analysis (Subagent: installation_analyst_agent) - MANDATORY
 
-Input: Prompt the user to provide the link to the prow job they wish to analyze. 
-Action: Parse the URL for the job_name and build_id. Call the installation_analyst subagent, passing the user-provided job_name and build_id.
-Expected Output: The installation_analyst subagent MUST return the job's job_name, build_id, test_name and a comprehensive data analysis for the installation of the cluster for the given job.
+**YOUR RESPONSIBILITY**: First, YOU extract job_name and build_id from the user-provided Prow job URL.
+**THEN**: Call the installation_analyst_agent subagent with the extracted job_name and build_id as parameters.
+**NEVER**: Ask sub-agents to extract URLs or parse job information - YOU do this step.
+Expected Output: The installation_analyst_agent subagent MUST return comprehensive installation analysis including job details and cluster installation metrics.
 
-* E2E Test Analysis (Subagent: e2e_test_analyst) - MANDATORY
+* E2E Test Analysis (Subagent: e2e_test_analyst_agent) - MANDATORY
 
-Input: The installation_analysis_output from the installation_analyst subagent.
-Action: ALWAYS call the e2e_test_analyst subagent, passing the job_name and build_id from the installation analysis. This will analyze the e2e test logs, extract openshift-tests binary commit information, identify failed tests, and provide source code links.
-Expected Output: The e2e_test_analyst subagent MUST return a comprehensive analysis of the e2e test execution, including:
+**YOUR ACTION**: Call the e2e_test_analyst_agent subagent with the same job_name and build_id you extracted in step 1.
+**PARAMETERS TO PASS**: job_name, build_id (extracted by YOU from the URL)
+**NEVER**: Ask the agent to extract or parse anything - just pass the parameters.
+Expected Output: The e2e_test_analyst_agent subagent MUST return a comprehensive analysis of the e2e test execution, including:
 - openshift-tests binary commit information and source code links
 - Failed test details with GitHub links to test source code
 - Test execution patterns and performance insights
 - Root cause analysis of test failures
 
-* Must_Gather Analysis (Subagent: mustgather_analyst) - OPTIONAL
+* Must_Gather Analysis (Subagent: mustgather_analyst_agent) - OPTIONAL
 
-Input: The installation_analysis_output from the installation_analyst subagent. Use /tmp/must-gather as the target_folder for the must-gather directory.
-Action: Only call if additional cluster-level debugging is needed. Call the mustgather_analyst subagent, passing the job_name, test_name and build_id. Download the must-gather logs: use /tmp/must-gather as the target_folder. Then analyze them by navigating the directory structure, reading files and searching for relevant information.
-Expected Output: The mustgather_analyst subagent MUST return a comprehensive data analysis for the execution of the given job.
-
-WORKFLOW EXECUTION:
-1. Parse the Prow job URL to extract job_name and build_id
-2. Call installation_analyst with job_name and build_id
-3. IMMEDIATELY call e2e_test_analyst with the same job_name and build_id
-4. Provide a comprehensive summary combining both analyses
-5. Only call mustgather_analyst if specifically requested or if deeper analysis is needed
-
-IMPORTANT NOTES:
-- If any analyst returns an error (starting with "❌"), acknowledge the error and provide the suggested troubleshooting steps
-- Always include the manual check URLs provided by the analysts for user verification
-- If logs are not available, suggest the user try a more recent job or verify the URL is correct
-- Provide clear, actionable recommendations based on the available analysis
-"""
+**YOUR ACTION**: Only call if additional cluster-level debugging is needed. Call the mustgather_analyst_agent subagent with the same job_name and build_id you extracted in step 1.
+**PARAMETERS TO PASS**: job_name, build_id (extracted by YOU from the URL)  
+**NEVER**: Ask the agent to extract or parse anything - just pass the parameters.
+Expected Output: The mustgather_analyst_agent subagent MUST return a comprehensive data analysis for the execution of the given job.
+"""

sub_agents/e2e_test_analyst/agent.py

@@ -6,7 +6,6 @@
 
 import asyncio
 import httpx
-import threading
 import concurrent.futures
 import re
 import os
@@ -139,10 +138,11 @@ def generate_source_code_links(test_name: str, commit_hash: Optional[str] = None
 async def get_e2e_test_logs_async(job_name: str, build_id: str, test_name: str) -> str:
     """Get e2e test logs from Prow."""
 
-
     # E2E test logs are typically in openshift-e2e-test directory
     if "sno" in test_name:
         e2e_test_path = f"artifacts/{test_name}/single-node-e2e-test/build-log.txt"
+    elif "libvirt" in test_name:
+        e2e_test_path = f"artifacts/{test_name}/openshift-e2e-libvirt-test/build-log.txt"
     else:
         e2e_test_path = f"artifacts/{test_name}/openshift-e2e-test/build-log.txt"
 
@@ -269,6 +269,8 @@ async def get_junit_results_async(job_name: str, build_id: str, test_name: str)
       # E2E test logs are typically in openshift-e2e-test directory
     if "sno" in test_name:
         e2e_test_path = f"artifacts/{test_name}/single-node-e2e-test"
+    elif "libvirt" in test_name:
+        e2e_test_path = f"artifacts/{test_name}/openshift-e2e-libvirt-test"
     else:
         e2e_test_path = f"artifacts/{test_name}/openshift-e2e-test"
 
@@ -316,16 +318,92 @@ def run_in_thread():
         return future.result()
 
 def get_job_metadata_tool(job_name: str, build_id: str):
-    """Get metadata and status for a specific Prow job name and build ID."""
+    """Retrieves comprehensive metadata and status information for a specific Prow CI job.
+
+    This tool fetches the prowjob.json metadata which contains essential information about
+    the CI job execution, including current status, build configuration, test targets,
+    and execution parameters. This is typically the first tool to use when analyzing a failed CI job.
+
+    Args:
+        job_name (str): The name of the Prow job (e.g., 'periodic-ci-openshift-multiarch-master-nightly-4.20-ocp-e2e-aws-ovn-sno')
+        build_id (str): The specific build ID for the job run (e.g., '1940296163760541696')
+
+    Returns:
+        dict: Job metadata including status, build_id, job_name, test_name, and error details if applicable
+    """
     return run_async_in_thread(get_job_metadata_async(job_name, build_id))
 
-def get_e2e_test_logs_tool(job_name: str, build_id: str, test_name: str):
-    """Get e2e test logs from the openshift-e2e-test directory with commit info and source code links."""
-    return run_async_in_thread(get_e2e_test_logs_async(job_name, build_id, test_name))
+def get_e2e_test_logs_tool(job_name: str, build_id: str, test_name: str, include_full_log: bool = True):
+    """Analyzes end-to-end test execution logs with source code tracing and failure analysis.
+
+    This tool retrieves and analyzes e2e test logs from the openshift-e2e-test directory,
+    extracting critical information including failed test details, openshift-tests binary version,
+    source code commit information, and direct links to test source code for debugging.
 
-def get_junit_results_tool(job_name: str, build_id: str, test_name: str):
-    """Get JUnit test results from the e2e test artifacts."""
-    return run_async_in_thread(get_junit_results_async(job_name, build_id, test_name))
+    Args:
+        job_name (str): The name of the Prow job containing e2e tests
+        build_id (str): The specific build ID for the job run  
+        test_name (str): The test component name (e.g., 'ocp-e2e-aws-ovn-sno-multi-a-a')
+        include_full_log (bool, optional): Whether to include the complete log content in response.
+                                         If False, only provides summary and key sections. Defaults to True.
+
+    Returns:
+        str: Comprehensive e2e test analysis including failed tests with source links, 
+             openshift-tests binary info, and formatted log content
+    """
+    result = run_async_in_thread(get_e2e_test_logs_async(job_name, build_id, test_name))
+
+    # If include_full_log is False, remove the full log section to reduce response size
+    if not include_full_log and isinstance(result, str):
+        lines = result.split('\n')
+        filtered_lines = []
+        skip_full_log = False
+
+        for line in lines:
+            if line.startswith('📋 FULL E2E TEST LOG:'):
+                skip_full_log = True
+                continue
+            if not skip_full_log:
+                filtered_lines.append(line)
+
+        result = '\n'.join(filtered_lines)
+
+    return result
+
+def get_junit_results_tool(job_name: str, build_id: str, test_name: str, parse_xml: bool = True):
+    """Retrieves JUnit XML test results from e2e test execution with structured failure analysis.
+
+    This tool fetches JUnit XML files generated by e2e test runs, which contain structured
+    test results including pass/fail status, execution times, error messages, and detailed
+    failure information. JUnit results are often more structured than raw logs.
+
+    Args:
+        job_name (str): The name of the Prow job containing e2e tests
+        build_id (str): The specific build ID for the job run
+        test_name (str): The test component name that generated JUnit results
+        parse_xml (bool, optional): Whether to provide structured parsing hints for XML content.
+                                   If True, includes guidance for extracting test results. Defaults to True.
+
+    Returns:
+        str: JUnit test results content with parsing guidance if requested, 
+             or error message if results are not found
+    """
+    result = run_async_in_thread(get_junit_results_async(job_name, build_id, test_name))
+
+    if parse_xml and isinstance(result, str) and not result.startswith("Could not find") and not result.startswith("Error"):
+        # Add XML parsing guidance for the LLM
+        guidance = """
+💡 JUNIT XML ANALYSIS TIPS:
+- Look for <failure> or <error> elements within <testcase> elements
+- Check 'name' and 'classname' attributes for test identification  
+- Examine failure messages in <failure message="..."> or <system-out> sections
+- Count total tests, failures, errors, and skipped from <testsuite> attributes
+- Look for patterns in failure messages to identify common root causes
+
+"""
+        result = guidance + result
+
+    return result
 
 e2e_test_analyst_agent = Agent(
     model=LiteLlm(model=MODEL),

sub_agents/e2e_test_analyst/prompt.py

@@ -2,6 +2,13 @@
 
 E2E_TEST_SPECIALIST_PROMPT = """You are an expert OpenShift E2E Test Analyst specializing in analyzing end-to-end test results from CI/CD pipelines.
 
+🚨 **CRITICAL REQUIREMENTS - READ FIRST**:
+- You will be called with ONLY job_name and build_id as input parameters
+- These are the ONLY parameters you need to start analysis
+- You will obtain test_name and other details internally from job metadata
+- If job_name or build_id are missing or invalid, IMMEDIATELY halt and report the error
+- Do NOT request additional parameters from the caller - all information comes from the job metadata
+
 Your primary responsibilities include:
 1. Analyzing e2e test logs from OpenShift CI jobs
 2. Identifying test failures, flakes, and patterns
@@ -33,20 +40,36 @@
 - Resource constraints and performance issues
 - Operator and component health checks
 
-Available tools:
-- get_job_metadata: Get basic job information and status
-- get_e2e_test_logs: Fetch e2e test logs with commit info and source code links
-- get_junit_results: Get JUnit XML test results when available
+🛠️ **AVAILABLE TOOLS**:
+- **get_job_metadata_tool**: Get basic job information and status (CALL FIRST)
+  - Input: job_name, build_id (from caller)
+  - Output: Job metadata, status, test_name, and basic information
+- **get_e2e_test_logs_tool**: Fetch e2e test logs with commit info and source code links  
+  - Input: job_name, build_id (from caller), test_name (from job metadata)
+  - Output: Test logs, openshift-tests commit info, failure details with GitHub links
+- **get_junit_results_tool**: Get JUnit XML test results when available
+  - Input: job_name, build_id (from caller), test_name (from job metadata)
+  - Output: Structured JUnit test results and statistics
+
+⚠️ **ERROR HANDLING**:
+If you receive incomplete parameters or any tool returns errors:
+1. Verify job_name and build_id are correctly provided
+2. Check if the job exists and has completed
+3. Inform the user of the specific missing requirements
+4. Do NOT attempt analysis with incomplete data
 
-When analyzing test results:
-1. Start by getting job metadata to understand the test context
-2. Fetch the e2e test logs which will automatically extract:
+📋 **ANALYSIS WORKFLOW**:
+1. **FIRST**: Call get_job_metadata_tool with the provided job_name and build_id to:
+   - Understand the test context
+   - Obtain the test_name needed for subsequent calls
+   - Get job status and basic information
+2. **SECOND**: Use get_e2e_test_logs_tool with job_name, build_id, and test_name (from step 1) to fetch the e2e test logs which will automatically extract:
    - openshift-tests binary commit information
    - Failed test names and durations
    - Source code links for each failure
-3. Look for JUnit results for additional structured test data
-4. Identify failed tests, their failure reasons, and patterns
-5. Provide actionable insights and recommendations
+3. **THIRD**: Use get_junit_results_tool with job_name, build_id, and test_name (from step 1) for additional structured test data
+4. **ANALYZE**: Identify failed tests, their failure reasons, and patterns
+5. **REPORT**: Provide actionable insights and recommendations with source code links
 
 Focus on:
 - Test failure root causes with links to source code