Merge branch 'main' into feature/security-review

Author: dylan-mccarthy

DEMO.md

@@ -0,0 +1,145 @@
+# Quick Start - Invoice Classifier Demo
+
+Get the Business Process Agents platform running in under 5 minutes!
+
+## Prerequisites
+
+- Docker and Docker Compose
+- 4GB+ RAM available
+- Linux, macOS, or Windows (WSL2)
+
+Install required tools:
+```bash
+# macOS
+brew install jq curl
+
+# Ubuntu/Debian
+sudo apt-get install -y jq curl
+```
+
+## 1-Minute Quick Start
+
+```bash
+# Clone the repository
+git clone https://github.com/dylan-mccarthy/Scalable-Process-Agent-System.git
+cd Scalable-Process-Agent-System
+
+# Run the interactive demo
+./demo-invoice-classifier.sh
+```
+
+That's it! The demo will:
+1. ✅ Start all required services
+2. ✅ Deploy the Invoice Classifier agent
+3. ✅ Show you the complete workflow
+4. ✅ Demonstrate observability features
+
+## What You'll See
+
+The demo showcases invoice classification using AI:
+
+```
+Invoice → Azure Service Bus → Node Runtime → GPT-4 Classification → API Delivery
+```
+
+**Sample Invoice:**
+```json
+{
+  "vendorName": "Office Depot",
+  "invoiceNumber": "DEMO-001",
+  "totalAmount": 542.75,
+  "currency": "USD"
+}
+```
+
+**Classification Result:**
+```json
+{
+  "vendorCategory": "Office Supplies",
+  "routingDestination": "Procurement Department",
+  "confidence": 0.98
+}
+```
+
+## Services Started
+
+| Service | URL | Purpose |
+|---------|-----|---------|
+| Control Plane API | http://localhost:8080 | REST + gRPC API |
+| PostgreSQL | localhost:5432 | Persistent storage |
+| Redis | localhost:6379 | Leases & locks |
+| NATS | localhost:4222 | Event streaming |
+| Node Runtime | (internal) | Agent execution |
+
+## Explore the API
+
+After the demo starts, try these commands:
+
+```bash
+# List agents
+curl http://localhost:8080/v1/agents | jq
+
+# List nodes
+curl http://localhost:8080/v1/nodes | jq
+
+# View agent details
+curl http://localhost:8080/v1/agents/invoice-classifier | jq
+```
+
+## View Logs
+
+```bash
+# All logs
+docker compose logs -f
+
+# Control Plane only
+docker compose logs -f control-plane
+
+# Node Runtime only
+docker compose logs -f node-runtime
+```
+
+## Cleanup
+
+```bash
+./demo-invoice-classifier.sh cleanup
+```
+
+To remove all data:
+```bash
+docker compose down -v
+```
+
+## Next Steps
+
+1. **Read the full walkthrough**: [DEMO.md](DEMO.md)
+2. **Deploy to Kubernetes**: `./infra/scripts/setup-k3d.sh`
+3. **Run E2E tests**: `dotnet test tests/E2E.Tests/`
+4. **Configure Azure AI**: [docs/AZURE_AI_FOUNDRY_INTEGRATION.md](docs/AZURE_AI_FOUNDRY_INTEGRATION.md)
+
+## Troubleshooting
+
+**Services won't start?**
+- Check Docker is running: `docker ps`
+- Ensure ports are available: 5432, 6379, 4222, 8080
+- Check available RAM: Minimum 4GB required
+
+**Control Plane not ready?**
+- Wait up to 60 seconds for first startup
+- Check logs: `docker compose logs control-plane`
+
+**Need help?**
+- See full troubleshooting guide in [DEMO.md](DEMO.md)
+
+## What's Running?
+
+The platform demonstrates:
+- ✅ **Distributed Orchestration**: Control Plane schedules work to nodes
+- ✅ **Agent Execution**: Microsoft Agent Framework + GPT-4
+- ✅ **Service Integration**: Azure Service Bus + HTTP output
+- ✅ **Observability**: OpenTelemetry metrics, traces, and logs
+- ✅ **Resilience**: DLQ, retries, lease management
+
+---
+
+**Ready for more?** See the [full demo walkthrough](DEMO.md) for detailed explanations and advanced features.

QUICKSTART.md

@@ -29,6 +29,24 @@ This is the Business Process Agents MVP project, providing a complete platform f
 
 ## Quick Start
 
+### 🎬 Interactive Demo (NEW!)
+
+Experience the complete invoice classification workflow:
+
+```bash
+./demo-invoice-classifier.sh
+```
+
+This interactive demo walks you through:
+- ✅ Deploying the Invoice Classifier agent
+- ✅ Understanding the end-to-end processing flow
+- ✅ Exploring observability features
+- ✅ Testing with sample invoices
+
+**See [DEMO.md](DEMO.md) for detailed walkthrough documentation.**
+
+**For a faster start, see [QUICKSTART.md](QUICKSTART.md).**
+
 ### Option 1: Local Kubernetes with k3d (Recommended)
 
 The fastest way to get a complete environment running locally:
@@ -52,6 +70,8 @@ This will create a local Kubernetes cluster and deploy all services:
 - Control Plane API: http://localhost:8080
 - Admin UI: http://localhost:3000
 
+> **Note**: The k3d setup does not include Azure AI Foundry. Configure Azure AI Foundry credentials for the Node Runtime to enable agent execution. See [Azure AI Foundry Configuration](#azure-ai-foundry-configuration) section below.
+
 **Cleanup:**
 ```bash
 ./infra/scripts/cleanup-k3d.sh
@@ -78,6 +98,8 @@ docker-compose down
 - Control Plane API: http://localhost:8080
 - Admin UI: http://localhost:3000
 
+> **Note**: Docker Compose does not include Azure AI Foundry. You must configure Azure AI Foundry credentials in `src/Node.Runtime/appsettings.json` or use environment variables. See [Azure AI Foundry Configuration](#azure-ai-foundry-configuration) section below.
+
 ### Option 3: Local Development (No Docker)
 
 Build and run individual services for development:
@@ -118,6 +140,23 @@ The API will be available at `http://localhost:5109`.
 
 #### Running Node Runtime
 
+Before running the Node Runtime, **configure Azure AI Foundry** (required for agent execution):
+
+```bash
+# Option 1: Use user secrets (recommended for development)
+cd src/Node.Runtime
+dotnet user-secrets set "AgentRuntime:AzureAIFoundry:Endpoint" "https://your-resource.openai.azure.com/"
+dotnet user-secrets set "AgentRuntime:AzureAIFoundry:ApiKey" "your-api-key"
+dotnet user-secrets set "AgentRuntime:AzureAIFoundry:DeploymentName" "gpt-4o-mini"
+
+# Option 2: Use environment variables
+export AgentRuntime__AzureAIFoundry__Endpoint="https://your-resource.openai.azure.com/"
+export AgentRuntime__AzureAIFoundry__ApiKey="your-api-key"
+export AgentRuntime__AzureAIFoundry__DeploymentName="gpt-4o-mini"
+```
+
+Then start the Node Runtime:
+
 ```bash
 cd src/Node.Runtime
 dotnet run
@@ -134,6 +173,134 @@ The Node Runtime will:
 - PostgreSQL 14 or later (for production use)
 - Redis 6.0 or later (for lease and lock management)
 - NATS Server 2.10+ with JetStream enabled (for event streaming)
+- **Azure AI Foundry** or **Azure OpenAI Service** (for LLM-powered agent execution)
+
+## Azure AI Foundry Configuration
+
+The platform uses **Azure AI Foundry** (or Azure OpenAI Service) to power LLM-based agent execution. You must configure Azure AI Foundry for agents to process requests using AI models like GPT-4.
+
+### Quick Setup
+
+1. **Create Azure AI Foundry Resource**:
+   ```bash
+   # Create resource group
+   az group create --name rg-bpa-agents --location eastus
+   
+   # Create Azure AI Foundry resource
+   az cognitiveservices account create \
+     --name my-ai-foundry \
+     --resource-group rg-bpa-agents \
+     --kind AIServices \
+     --sku S0 \
+     --location eastus
+   ```
+
+2. **Deploy a Model**:
+   - Navigate to your Azure AI Foundry resource in the Azure Portal
+   - Go to "Deployments" → "Create new deployment"
+   - Select model: `gpt-4o-mini` (recommended for cost-effective MVP)
+   - Name: `gpt-4o-mini`
+   - Note your endpoint: `https://your-resource.openai.azure.com/`
+
+3. **Configure Node Runtime** (edit `src/Node.Runtime/appsettings.json`):
+   ```json
+   {
+     "AgentRuntime": {
+       "DefaultModel": "gpt-4o-mini",
+       "DefaultTemperature": 0.7,
+       "MaxTokens": 4000,
+       "MaxDurationSeconds": 60,
+       "AzureAIFoundry": {
+         "Endpoint": "https://your-resource.openai.azure.com/",
+         "DeploymentName": "gpt-4o-mini",
+         "ApiKey": "your-api-key-here",
+         "UseManagedIdentity": false
+       }
+     }
+   }
+   ```
+
+   > **Security Best Practice**: Never commit API keys to source control. Use one of these approaches:
+   > - **Development**: `dotnet user-secrets set "AgentRuntime:AzureAIFoundry:ApiKey" "your-key"`
+   > - **Production**: Use Managed Identity (set `UseManagedIdentity: true`) or Azure Key Vault
+   > - **Environment Variables**: `export AgentRuntime__AzureAIFoundry__ApiKey="your-key"`
+
+### Supported Models
+
+Azure AI Foundry supports various models for different use cases:
+
+| Model Family | Model | Best For | Cost |
+|-------------|-------|----------|------|
+| **GPT-4 Optimized** | `gpt-4o` | Latest performance, multimodal | $$$ |
+| | `gpt-4o-mini` | Cost-effective, fast, recommended for MVP | $ |
+| **GPT-4** | `gpt-4` | Complex reasoning tasks | $$$$ |
+| | `gpt-4-32k` | Extended context (32K tokens) | $$$$$ |
+| **GPT-3.5** | `gpt-3.5-turbo` | Fast, cost-effective | $ |
+| | `gpt-3.5-turbo-16k` | Extended context (16K tokens) | $$ |
+
+### Configuration Options
+
+| Setting | Required | Default | Description |
+|---------|----------|---------|-------------|
+| `Endpoint` | ✓ | - | Azure AI Foundry endpoint URL |
+| `DeploymentName` | ✓ | - | Model deployment name in Azure |
+| `ApiKey` | ✓* | - | API key for authentication |
+| `UseManagedIdentity` | | `false` | Use Azure Managed Identity instead of API key |
+
+*Required if `UseManagedIdentity` is `false`
+
+### Using Managed Identity (Recommended for Production)
+
+Managed Identity eliminates the need for API keys:
+
+```json
+{
+  "AgentRuntime": {
+    "AzureAIFoundry": {
+      "Endpoint": "https://your-resource.openai.azure.com/",
+      "DeploymentName": "gpt-4o-mini",
+      "UseManagedIdentity": true
+    }
+  }
+}
+```
+
+Grant your Node Runtime's managed identity access:
+
+```bash
+# Get Node Runtime's managed identity principal ID
+PRINCIPAL_ID=$(az aks show --name my-aks --resource-group my-rg --query identityProfile.kubeletidentity.clientId -o tsv)
+
+# Get Azure AI Foundry resource ID
+AI_RESOURCE_ID=$(az cognitiveservices account show --name my-ai-foundry --resource-group rg-bpa-agents --query id -o tsv)
+
+# Assign Cognitive Services User role
+az role assignment create \
+  --assignee $PRINCIPAL_ID \
+  --role "Cognitive Services User" \
+  --scope $AI_RESOURCE_ID
+```
+
+### Budget & Cost Management
+
+Control costs by setting budget constraints in agent definitions:
+
+```json
+{
+  "agentId": "invoice-classifier",
+  "budget": {
+    "maxTokens": 2000,
+    "maxDurationSeconds": 30
+  }
+}
+```
+
+The platform automatically tracks token usage and costs for each run. Monitor in:
+- Azure Portal: Cost analysis for Azure AI Foundry
+- Application logs: Token usage per execution
+- OpenTelemetry metrics: `run_tokens`, `run_cost_usd`
+
+**For detailed Azure AI Foundry configuration, see [docs/AZURE_AI_FOUNDRY_INTEGRATION.md](docs/AZURE_AI_FOUNDRY_INTEGRATION.md).**
 
 ## Database Setup
 
@@ -1171,7 +1338,9 @@ cd agents
 
 ## Documentation
 
+- **[Demo Walkthrough (DEMO.md)](DEMO.md)** - Interactive demo guide for invoice classification (E8-T3)
 - [System Architecture Document (SAD)](sad.md) - High-level system design and architecture
+- [Architecture Diagrams](docs/ARCHITECTURE_DIAGRAMS.md) - C4 context and container diagrams
 - [Invoice Classifier Agent](docs/INVOICE_CLASSIFIER.md) - Technical documentation for the MVP Invoice Classifier agent
 - [Agent Definitions Guide](agents/README.md) - Guide to agent definitions and seeding agents
 - [Agent Versioning and Validation](docs/VERSIONING.md) - Guide to agent versioning, semantic versioning, and spec validation