Merge branch 'feature/idp-cli-doc-status' into 'develop'

Feat: Extend the new idp-cli status command to check job COMPLETED/FAILURE status

See merge request genaiic-reusable-assets/engagement-artifacts/genaiic-idp-accelerator!401

Author: rstrahan

CHANGELOG.md

@@ -15,7 +15,6 @@ SPDX-License-Identifier: MIT-0
   - **Persistent Chat Memory**: DynamoDB-backed conversation history with automatic loading of last 20 turns, turn-based message grouping, and intelligent context management with sliding window optimization
   - **Real-Time Streaming**: AppSync GraphQL subscriptions enable incremental response streaming with proper async task cleanup and thinking tag removal for clean display
   - **Code Intelligence Agent**: New specialized agent for code-related assistance with DeepWiki MCP server integration, security guardrails to prevent sensitive data exposure, and user-controlled opt-in toggle (default: enabled)
-  - **Sub-Agent Streaming**: Real-time lifecycle events (start, stream, end, error) with async generator tools, structured data detection for tables/charts, and tool usage tracking
   - **Rich Chat Interface**: Modern UI with CloudScape Design System featuring real-time message streaming, multi-agent support (Analytics, Code Intelligence, Error Analyzer, General), Markdown rendering with syntax highlighting, structured data visualization (charts via Chart.js, sortable tables), expandable tool usage sections, sample prompts, and auto-scroll behavior
   - **Privacy & Security**: Explicit user consent for Code Intelligence third-party services, session isolation with unique session IDs, error boundary protection, input validation
 
@@ -29,6 +28,20 @@ SPDX-License-Identifier: MIT-0
   - **Backward Compatible**: Legacy format remains supported through automatic migration - no manual configuration updates required
   - **Comprehensive Documentation**: New migration guide with format comparison, field mapping table, and best practices
 
+- **IDP CLI Single Document Status Support with Programmatic Output**
+  - Enhanced `status` command to support checking individual document status via new `--document-id` option as alternative to `--batch-id`
+  - Added programmatic output capabilities with exit codes (0=success, 1=failure, 2=processing) for scripting and automation
+  - JSON format output (`--format json`) provides structured data for parsing in CI/CD pipelines and scripts
+  - Live monitoring support with `--wait` flag works for both batch and single document status checks
+  - Mutual exclusion validation ensures only one of `--batch-id` or `--document-id` is specified
+- **Error Analyzer CloudWatch Tool Enhancements**
+  - Enhanced CloudWatch log filtering with request ID-based filtering for more targeted error analysis
+  - Improved XRay tool tracing and logging capabilities for better diagnostic accuracy
+  - Enhanced error context correlation between CloudWatch logs and X-Ray traces
+  - Consolidated and renamed tools
+  - Provided tools access to agent
+  - Updated system prompt
+
 - **Error Analyzer CloudWatch Tool Enhancements**
   - Enhanced CloudWatch log filtering with request ID-based filtering for more targeted error analysis
   - Improved XRay tool tracing and logging capabilities for better diagnostic accuracy

VERSION

@@ -1 +1 @@
-0.4.0-rc6
+0.4.0-rc7

docs/idp-cli.md

@@ -356,27 +356,113 @@ https://github.com/user-attachments/assets/28deadbb-378b-42b7-a5e2-f929af9b0e41
 
 ### `status`
 
-Check batch processing status.
+Check status of a batch or single document.
 
 **Usage:**
 ```bash
 idp-cli status [OPTIONS]
 ```
 
+**Document Source (choose ONE):**
+- `--batch-id`: Batch identifier (check all documents in batch)
+- `--document-id`: Single document ID (check individual document)
+
 **Options:**
 - `--stack-name` (required): CloudFormation stack name
-- `--batch-id` (required): Batch identifier
 - `--wait`: Wait for all documents to complete
 - `--refresh-interval`: Seconds between status checks (default: 5)
+- `--format`: Output format - `table` (default) or `json`
 - `--region`: AWS region (optional)
 
-**Example:**
+**Examples:**
 
 ```bash
+# Check batch status
 idp-cli status \
     --stack-name my-stack \
-    --batch-id cli-batch-20251015-143000 \
+    --batch-id cli-batch-20251015-143000
+
+# Check single document status
+idp-cli status \
+    --stack-name my-stack \
+    --document-id batch-123/invoice.pdf
+
+# Monitor single document until completion
+idp-cli status \
+    --stack-name my-stack \
+    --document-id batch-123/invoice.pdf \
     --wait
+
+# Get JSON output for scripting
+idp-cli status \
+    --stack-name my-stack \
+    --document-id batch-123/invoice.pdf \
+    --format json
+```
+
+**Programmatic Use:**
+
+The command returns exit codes for scripting:
+- `0` - Document(s) completed successfully
+- `1` - Document(s) failed
+- `2` - Document(s) still processing
+
+**JSON Output Format:**
+
+```bash
+# Single document
+$ idp-cli status --stack-name my-stack --document-id batch-123/invoice.pdf --format json
+{
+  "document_id": "batch-123/invoice.pdf",
+  "status": "COMPLETED",
+  "duration": 125.4,
+  "start_time": "2025-01-01T10:30:45Z",
+  "end_time": "2025-01-01T10:32:50Z",
+  "num_sections": 2,
+  "exit_code": 0
+}
+
+# Table output includes final status summary
+$ idp-cli status --stack-name my-stack --document-id batch-123/invoice.pdf
+[status table]
+
+FINAL STATUS: COMPLETED | Duration: 125.4s | Exit Code: 0
+```
+
+**Scripting Examples:**
+
+```bash
+#!/bin/bash
+# Wait for document completion and check result
+idp-cli status --stack-name prod --document-id batch-001/invoice.pdf --wait
+exit_code=$?
+
+if [ $exit_code -eq 0 ]; then
+  echo "Document processed successfully"
+  # Proceed with downstream processing
+else
+  echo "Document processing failed"
+  exit 1
+fi
+```
+
+```bash
+#!/bin/bash
+# Poll document status in script
+while true; do
+  status=$(idp-cli status --stack-name prod --document-id batch-001/invoice.pdf --format json)
+  state=$(echo "$status" | jq -r '.status')
+  
+  if [ "$state" = "COMPLETED" ]; then
+    echo "Processing complete!"
+    break
+  elif [ "$state" = "FAILED" ]; then
+    echo "Processing failed!"
+    exit 1
+  fi
+  
+  sleep 5
+done
 ```
 
 ---

idp_cli/idp_cli/cli.py

@@ -804,49 +804,96 @@ def run_inference(
 
 @cli.command()
 @click.option("--stack-name", required=True, help="CloudFormation stack name")
-@click.option("--batch-id", required=True, help="Batch identifier")
+@click.option("--batch-id", help="Batch identifier")
+@click.option("--document-id", help="Single document ID (alternative to --batch-id)")
 @click.option("--wait", is_flag=True, help="Wait for all documents to complete")
 @click.option(
     "--refresh-interval",
     default=5,
     type=int,
     help="Seconds between status checks (default: 5)",
 )
+@click.option(
+    "--format",
+    "output_format",
+    type=click.Choice(["table", "json"]),
+    default="table",
+    help="Output format: table (default) or json",
+)
 @click.option("--region", help="AWS region (optional)")
 def status(
     stack_name: str,
-    batch_id: str,
+    batch_id: Optional[str],
+    document_id: Optional[str],
     wait: bool,
     refresh_interval: int,
+    output_format: str,
     region: Optional[str],
 ):
     """
-    Check status of a batch processing job
+    Check status of a batch or single document
+
+    Specify ONE of:
+      --batch-id: Check status of all documents in a batch
+      --document-id: Check status of a single document
 
     Examples:
 
-      # Check current status once
+      # Check batch status
       idp-cli status --stack-name my-stack --batch-id cli-batch-20250110-153045-abc12345
 
-      # Monitor until completion
-      idp-cli status --stack-name my-stack --batch-id cli-batch-20250110-153045-abc12345 --wait
+      # Check single document status
+      idp-cli status --stack-name my-stack --document-id batch-123/invoice.pdf
+
+      # Monitor single document until completion
+      idp-cli status --stack-name my-stack --document-id batch-123/invoice.pdf --wait
+
+      # Get JSON output for scripting
+      idp-cli status --stack-name my-stack --document-id batch-123/invoice.pdf --format json
     """
     try:
-        # Get batch info
-        processor = BatchProcessor(stack_name=stack_name, region=region)
-        batch_info = processor.get_batch_info(batch_id)
+        # Validate mutually exclusive options
+        if not batch_id and not document_id:
+            console.print(
+                "[red]✗ Error: Must specify either --batch-id or --document-id[/red]"
+            )
+            sys.exit(1)
 
-        if not batch_info:
-            console.print(f"[red]✗ Batch not found: {batch_id}[/red]")
+        if batch_id and document_id:
+            console.print(
+                "[red]✗ Error: Cannot specify both --batch-id and --document-id[/red]"
+            )
             sys.exit(1)
 
-        document_ids = batch_info["document_ids"]
+        # Initialize processor to get resources
+        processor = BatchProcessor(stack_name=stack_name, region=region)
+
+        # Get document IDs to monitor
+        if batch_id:
+            # Get batch info
+            batch_info = processor.get_batch_info(batch_id)
+            if not batch_info:
+                console.print(f"[red]✗ Batch not found: {batch_id}[/red]")
+                sys.exit(1)
+            document_ids = batch_info["document_ids"]
+            identifier = batch_id
+        else:
+            # Single document
+            document_ids = [document_id]
+            identifier = document_id
 
         if wait:
+            # JSON format not compatible with live monitoring
+            if output_format == "json":
+                console.print(
+                    "[yellow]Warning: --format json ignored with --wait (using table display for live monitoring)[/yellow]"
+                )
+                console.print()
+
             # Monitor until completion
             _monitor_progress(
                 stack_name=stack_name,
-                batch_id=batch_id,
+                batch_id=identifier,
                 document_ids=document_ids,
                 refresh_interval=refresh_interval,
                 region=region,
@@ -860,13 +907,32 @@ def status(
             status_data = monitor.get_batch_status(document_ids)
             stats = monitor.calculate_statistics(status_data)
 
-            console.print()
-            console.print(f"[bold blue]Batch: {batch_id}[/bold blue]")
-            display.display_status_table(status_data)
+            if output_format == "json":
+                # JSON output for programmatic use
+                json_output = display.format_status_json(status_data, stats)
+                console.print(json_output)
 
-            # Show statistics
-            console.print(display.create_statistics_panel(stats))
-            console.print()
+                # Determine exit code from JSON
+                import json as json_module
+
+                result = json_module.loads(json_output)
+                sys.exit(result.get("exit_code", 2))
+            else:
+                # Table output for human viewing
+                console.print()
+                if batch_id:
+                    console.print(f"[bold blue]Batch: {batch_id}[/bold blue]")
+                else:
+                    console.print(f"[bold blue]Document: {document_id}[/bold blue]")
+
+                display.display_status_table(status_data)
+
+                # Show statistics
+                console.print(display.create_statistics_panel(stats))
+
+                # Show final status summary and exit with appropriate code
+                exit_code = display.show_final_status_summary(status_data, stats)
+                sys.exit(exit_code)
 
     except Exception as e:
         logger.error(f"Error checking status: {e}", exc_info=True)