[Design Proposal] Consume existing LLM Providers from an Agent

[menakaj]

Problem

Agent developers need to configure AI capabilities (LLM providers, MCP tools) for their agents across multiple environments (dev, staging, prod). Currently, the system lacks:

1. No AI Resource Catalog: Developers cannot see which LLM providers and MCP proxies are approved and available for use in their organization.

2. No Environment-Specific Configuration: Developers must configure AI resources separately for each environment, leading to configuration drift and manual overhead.

3. No Agent-Level Guardrails: Developers cannot apply agent-specific content filters or rate limits on top of organization-level policies.

4. No Unified Configuration Model: Creating LLM proxies requires understanding low-level gateway concepts instead of high-level agent configuration.

5. No Configuration Versioning: Changes to agent AI configuration are not tracked or versioned, making rollbacks and audits difficult.

6. No Deployment Automation: After configuring AI resources, developers must manually deploy to each environment's gateway.

User Stories

Agent Developer (Delia)

AI Resource Discovery:

• As Delia, I want to see a catalog of approved LLM providers (OpenAI GPT-4, Anthropic Claude, Azure OpenAI) so that I know which models I can use.
• As Delia, I want to see a catalog of approved MCP proxies (filesystem, database, web search) so that I know which tools my agent can access.
• As Delia, I want to filter the catalog by capabilities (e.g., "show me all providers that support function calling") so that I can find the right model for my use case.
• As Delia, I want to see provider details (rate limits, cost, latency SLA) so that I can make informed decisions.

Agent Model Configuration:

• As Delia, I want to configure which LLM provider my agent uses per environment (dev uses GPT-3.5, prod uses GPT-4) so that I can test with cheaper models before deploying to production.
• As Delia, I want to select a different provider for each environment with a single UI action so that I don't need to repeat configuration.
• As Delia, I want to apply agent-level guardrails (custom content filters, lower rate limits) on top of organization defaults so that I can enforce stricter policies for my specific agent.
• As Delia, I want to configure model parameters (temperature, max tokens) per environment so that I can tune behavior for different use cases.

Agent Tool Configuration:

• As Delia, I want to attach MCP proxies to my agent (filesystem tool for dev, read-only database tool for prod) so that my agent can use approved tools.
• As Delia, I want to configure which tools are available per environment so that dev agents can use debug tools while prod agents use only production tools.
• As Delia, I want to see which tools require special permissions so that I can request access from Adam.

Deployment & Testing:

• As Delia, I want to deploy my agent's AI configuration to dev first, test it, then promote to staging and prod so that I can verify behavior before production rollout.
• As Delia, I want to see deployment status per environment (pending, deployed, failed) so that I know when my agent is ready to use.
• As Delia, I want to rollback a bad configuration change so that I can quickly recover from errors.

Monitoring & Debugging:

• As Delia, I want to see my agent's LLM usage metrics (requests/day, tokens/day, cost/day) per environment so that I can optimize for performance and cost.
• As Delia, I want to see which guardrails are blocking my agent's requests so that I can debug unexpected behavior.
• As Delia, I want to test my agent's configuration in a sandbox environment before deploying so that I can catch issues early.

Existing Solutions

N/A

Proposed Solution

Overview

This proposal introduces agent-level AI configuration that allows developers to consume organization-approved LLM providers and MCP proxies through a simplified, environment-aware interface. The system abstracts away low-level gateway concepts (LLM proxies) and presents a unified "Model Configuration" entity that groups environment-specific deployments.

Key Concepts

AI Resource Catalog: Organization-level registry of approved LLM providers and MCP proxies (managed by Adam) that developers can browse and select.

Agent Model Configuration: A grouped entity representing "which LLM provider my agent uses" with per-environment selections. Translates to LLM proxies deployed to each environment's gateway.

Agent Tool Configuration: A grouped entity representing "which MCP tools my agent uses" with per-environment selections. Translates to MCP proxy references deployed to each environment's gateway.

Environment-Specific Overrides: Per-environment configuration (dev uses GPT-3.5, prod uses GPT-4; dev has debug tools, prod has read-only tools).

Agent-Level Guardrails: Additional content filters, rate limits, and policies applied on top of organization defaults.

Deployment Abstraction: Developers work with logical configurations; Agent Manager translates to LLM proxies and deploys to gateways automatically.

Design

Architecture Overview

The architecture follows a catalog-based discovery pattern with automatic proxy provisioning:

┌─────────────────────────────────────────────────────────────────────────┐
│                    AGENT MANAGER (Control Plane)                        │
│                                                                         │
│  ┌────────────────────────────────────────────────────────────────────┐ │
│  │  AI Resource Catalog (Organization-Scoped)                         │ │
│  │  - LLM Providers marked as in_catalog = TRUE                       │ │
│  │  - Agents from OpenChoreo                                          │ │
│  └────────────────────────────────────────────────────────────────────┘ │
│                         ↓ (Browse & Select)                             │
│  ┌────────────────────────────────────────────────────────────────────┐ │
│  │  Agent Model Configuration (Agent-Scoped)                          │ │
│  │  - Name: "My OpenAI Config"                                        │ │
│  │  - Type: llm | mcp | other                                         │ │
│  │  - Per-environment mappings:                                       │ │
│  │    • dev → {provider: openai-gpt35, policies: [...]}               │ │
│  │    • prod → {provider: openai-gpt4, policies: [...]}               │ │
│  └────────────────────────────────────────────────────────────────────┘ │
│                         ↓ (Auto-provisions)                             │
│  ┌────────────────────────────────────────────────────────────────────┐ │
│  │  LLM Proxy Provisioning (per environment)                          │ │
│  │  For each environment mapping:                                     │ │
│  │  1. Create LLM proxy (via LLMProxyService)                         │ │
│  │  2. Deploy to gateway (via LLMProxyDeploymentService)              │ │
│  │  3. Generate API key (via LLMProxyAPIKeyService)                   │ │
│  │  4. Resolve URL: {gateway_vhost}{proxy_context}                    │ │
│  │  5. Store secrets in OpenChoreo (TODO)                             │ │
│  │  6. Create env variable mappings:                                  │ │
│  │     - {CONFIG_NAME}_URL → secret reference                         │ │
│  │     - {CONFIG_NAME}_API_KEY → secret reference                     │ │
│  └────────────────────────────────────────────────────────────────────┘ │
└─────────────────────────────────────────────────────────────────────────┘
                              ↓
        ┌─────────────────────┴─────────────────────┐
        ↓                                           ↓
┌────────────────────┐                    ┌────────────────────┐
│  Gateway (Dev)     │                    │  Gateway (Prod)    │
│                    │                    │                    │
│  LLM Proxy:        │                    │  LLM Proxy:        │
│  my-config-dev     │                    │  my-config-prod    │
│  → GPT-3.5         │                    │  → GPT-4           │
│  → context: /llm/..│                    │  → context: /llm/..│
│                    │                    │                    │
│  Agent Runtime     │                    │  Agent Runtime     │
│  Env Vars:         │                    │  Env Vars:         │
│  MY_OPENAI_CONFIG_ │                    │  MY_OPENAI_CONFIG_ │
│    URL=https://... │                    │    URL=https://... │
│  MY_OPENAI_CONFIG_ │                    │  MY_OPENAI_CONFIG_ │
│    API_KEY=sk-...  │                    │    API_KEY=sk-...  │
└────────────────────┘                    └────────────────────┘

Design Decisions

1. Catalog Pattern: in_catalog Flag vs. Separate Table

Decision: Add in_catalog boolean column to existing artifacts table.

Rationale:

• Reuses existing artifact infrastructure (handle, name, version, kind)
• Minimal schema changes
• Consistent with existing patterns in codebase
• Avoids data duplication

Implementation:

ALTER TABLE artifacts ADD COLUMN in_catalog BOOLEAN DEFAULT FALSE;
CREATE INDEX idx_artifacts_in_catalog ON artifacts(in_catalog, organization_uuid)
  WHERE in_catalog = TRUE;

Catalog Query Pattern:

• LLM Providers: SELECT * FROM artifacts WHERE kind = 'llmProvider' AND in_catalog = TRUE
• Agents: Fetched from OpenChoreo (always in catalog)
• MCP Proxies: Fetched from OpenChoreo (always in catalog)

2. Environment Resolution: OpenChoreo vs. Local Storage

Decision: Environments are resolved from OpenChoreo, NOT stored in local database.

Rationale:

• OpenChoreo is the source of truth for infrastructure resources
• Avoids data synchronization issues
• Follows existing pattern (see gateway_environment_mappings)

Implementation:

• NO foreign key constraint on environment_uuid columns
• Runtime validation against OpenChoreo via InfraResourceManager.ListOrgEnvironments()
• Store environment UUID as reference, not with FK constraint

Database Pattern:

CREATE TABLE env_agent_model_mapping (
    environment_uuid UUID NOT NULL,  -- NO FK constraint
    ...
);
COMMENT ON COLUMN env_agent_model_mapping.environment_uuid IS
  'Environment UUID from OpenChoreo (validated at runtime, no FK constraint)';

3. Configuration Scope: Per-Agent vs. Per-Project

Decision: One configuration per agent (scoped to agent).

Rationale:

• Each agent has unique AI requirements (model choice, parameters, policies)
• Agent-level scoping enables fine-grained control
• Simplifies agent runtime (one config to load)

Implementation:

CONSTRAINT uq_agent_config_name UNIQUE(agent_id, name, organization_uuid)

4. Secret Management: Direct Storage vs. Secret References

Decision: Store secret references (e.g., choreo:///default/secret/my-config-url), NOT actual secret values.

Rationale:

• Secrets are managed by OpenChoreo (centralized, auditable)
• Secret references enable rotation without code changes
• Follows zero-trust security model

Implementation (TODO - pending OpenChoreo API):

Variable Naming: {CONFIG_NAME}_URL, {CONFIG_NAME}_API_KEY
  where CONFIG_NAME = uppercase(replace(config.name, /[- ]/, '_'))

Secret Naming: {config-name}-{env-name}-url, {config-name}-{env-name}-apikey

Secret Reference: choreo:///default/secret/{secret-name}

Example:

Configuration: "my-openai-config"
Environment: "production"

Variables:
- MY_OPENAI_CONFIG_URL
- MY_OPENAI_CONFIG_API_KEY

Secrets in OpenChoreo:
- my-openai-config-production-url
- my-openai-config-production-apikey

Secret References (stored in DB):
- choreo:///default/secret/my-openai-config-production-url
- choreo:///default/secret/my-openai-config-production-apikey

5. Gateway Selection: First-Match vs. Priority-Based

Decision: Priority-based gateway selection with AI-first preference.

Rationale:

• Multiple gateways can serve same environment
• AI workloads benefit from AI-specific gateways (optimized routing, caching)
• Fallback ensures resilience

Implementation:

SELECT g.*
FROM gateways g
JOIN gateway_environment_mappings gem ON g.uuid = gem.gateway_uuid
WHERE gem.environment_uuid = ? AND g.is_active = true
ORDER BY
  CASE
    WHEN g.gateway_functionality_type = 'ai' THEN 1
    WHEN g.gateway_functionality_type = 'api' THEN 2
    ELSE 3
  END,
  g.created_at DESC
LIMIT 1;

Error Handling:

• No gateway found → 500 Internal Server Error
• Gateway has no vhost → 500 Internal Server Error

6. Transaction Boundaries: Single-System vs. Distributed

Decision: Database transaction + best-effort cleanup for external systems.

Rationale:

• No distributed transaction support across DB + Gateways + OpenChoreo
• Database transaction ensures data consistency
• External system failures trigger rollback + cleanup

Implementation:

BEGIN TRANSACTION
  1. Create agent_configurations record
  2. For each environment:
     - Create LLM proxy (track UUID for rollback)
     - Deploy to gateway (track deployment ID)
     - Generate API key
     - Store secret references in DB
     - Create env_agent_model_mapping record

ON ERROR:
  ROLLBACK TRANSACTION
  Best-effort cleanup:
    - Undeploy proxies from gateways
    - Delete created proxies
    - Delete secrets from OpenChoreo (TODO)
COMMIT TRANSACTION

7. API Response Design: Full Model vs. Subset

Decision: Return only necessary fields in API responses, hide internal details.

Rationale:

• Security: Don't expose secret references
• Performance: Smaller payloads
• Versioning: Easier to evolve API

Implementation:

// Internal Model
type AgentEnvVariable struct {
    SecretReference string `gorm:"column:secret_reference" json:"-"` // Hidden
}

// API Response
type EnvironmentVariableConfig struct {
    Name string `json:"name"` // Only expose variable name
}

// LLM Proxy Info (subset)
type LLMProxyInfo struct {
    ProxyUUID string `json:"proxyUuid"`
    ProxyName string `json:"proxyName"`
    Context   string `json:"context"`
    Status    string `json:"status"`
    // NOT exposing: ProviderUUID, internal config, etc.
}

API Surface

1. Catalog Endpoints

List Catalog Resources:

GET /orgs/{orgName}/catalog

Query Parameters:
- kind (optional): llmProvider | agent | mcp
- limit (optional): default 20
- offset (optional): default 0

Response:
{
  "entries": [
    {
      "uuid": "...",
      "handle": "openai-gpt4",
      "name": "OpenAI GPT-4",
      "version": "1.0.0",
      "kind": "llmProvider",
      "inCatalog": true,
      "createdAt": "..."
    }
  ],
  "total": 10,
  "limit": 20,
  "offset": 0
}

Implementation:

• kind=llmProvider: Query artifacts WHERE kind='llmProvider' AND in_catalog=TRUE
• kind=agent: Fetch from OpenChoreo via InfraResourceManager.ListComponents(orgName, projectName)
• kind=mcp: Fetch MCP proxies from OpenChoreo

2. Agent Model Configuration Endpoints

Create Configuration:

POST /orgs/{orgName}/projects/{projectName}/agents/{agentName}/model-configs

Body:
{
  "name": "My OpenAI Config",
  "description": "GPT-3.5 for dev, GPT-4 for prod",
  "type": "llm",
  "envMappings": {
    "dev-env-uuid": {
      "providerUuid": "provider-uuid-1",
      "configuration": {
        "policies": [
          { "type": "rate-limit", "requestsPerMinute": 100 }
        ]
      }
    },
    "prod-env-uuid": {
      "providerUuid": "provider-uuid-2",
      "configuration": {
        "policies": [
          { "type": "rate-limit", "requestsPerMinute": 1000 }
        ]
      }
    }
  }
}

Response:
{
  "uuid": "config-uuid",
  "name": "My OpenAI Config",
  "agentId": "my-agent",
  "type": "llm",
  "projectName": "my-project",
  "envModelConfig": {
    "dev-env-uuid": {
      "environmentUuid": "dev-env-uuid",
      "environmentName": "Development",
      "llmProxy": {
        "proxyUuid": "proxy-uuid-1",
        "proxyName": "my-openai-config-dev",
        "context": "/llm/openai/v1",
        "status": "deployed"
      }
    },
    "prod-env-uuid": {
      "environmentUuid": "prod-env-uuid",
      "environmentName": "Production",
      "llmProxy": {
        "proxyUuid": "proxy-uuid-2",
        "proxyName": "my-openai-config-prod",
        "context": "/llm/openai/v1",
        "status": "deployed"
      }
    }
  },
  "environmentVariables": [
    { "name": "MY_OPENAI_CONFIG_URL" },
    { "name": "MY_OPENAI_CONFIG_API_KEY" }
  ],
  "createdAt": "...",
  "updatedAt": "..."
}

List Configurations:

GET /orgs/{orgName}/projects/{projectName}/agents/{agentName}/model-configs

Query Parameters:
- limit (optional): default 20
- offset (optional): default 0

Response:
{
  "configs": [
    {
      "uuid": "...",
      "name": "My OpenAI Config",
      "agentId": "my-agent",
      "description": "...",
      "type": "llm",
      "projectName": "my-project",
      "createdAt": "..."
    }
  ],
  "pagination": {
    "count": 5,
    "offset": 0,
    "limit": 20
  }
}

Get Configuration:

GET /orgs/{orgName}/projects/{projectName}/agents/{agentName}/model-configs/{configUuid}

Response: Same as Create response

Update Configuration:

PUT /orgs/{orgName}/projects/{projectName}/agents/{agentName}/model-configs/{configUuid}

Body:
{
  "name": "Updated Config Name",
  "description": "Updated description",
  "envMappings": {
    "dev-env-uuid": {
      "providerUuid": "new-provider-uuid",
      "configuration": { ... }
    }
  }
}

Response: Updated configuration object

Delete Configuration:

DELETE /orgs/{orgName}/projects/{projectName}/agents/{agentName}/model-configs/{configUuid}

Response: 204 No Content

Data Model Changes

New Tables

1. Agent Configurations:

CREATE TABLE agent_configurations (
    uuid UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    name VARCHAR(255) NOT NULL,
    description TEXT,
    agent_id VARCHAR(255) NOT NULL,
    type VARCHAR(50) NOT NULL DEFAULT 'llm',
    organization_uuid UUID NOT NULL,
    project_name VARCHAR(255) NOT NULL,
    created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
    updated_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,

    CONSTRAINT fk_agent_config_organization FOREIGN KEY (organization_uuid)
        REFERENCES organizations(uuid) ON DELETE CASCADE,
    CONSTRAINT chk_agent_config_type CHECK (type IN ('llm', 'mcp', 'other')),
    CONSTRAINT uq_agent_config_name UNIQUE(agent_id, name, organization_uuid)
);

CREATE INDEX idx_agent_config_agent ON agent_configurations(agent_id);
CREATE INDEX idx_agent_config_org_project ON agent_configurations(organization_uuid, project_name);

2. Environment Agent Model Mapping:

CREATE TABLE env_agent_model_mapping (
    id SERIAL PRIMARY KEY,
    config_uuid UUID NOT NULL,
    environment_uuid UUID NOT NULL,  -- From OpenChoreo, no FK
    llm_proxy_uuid UUID NOT NULL,
    created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,

    CONSTRAINT fk_env_mapping_config FOREIGN KEY (config_uuid)
        REFERENCES agent_configurations(uuid) ON DELETE CASCADE,
    CONSTRAINT fk_env_mapping_proxy FOREIGN KEY (llm_proxy_uuid)
        REFERENCES llm_proxies(uuid) ON DELETE CASCADE,
    CONSTRAINT uq_env_mapping UNIQUE(config_uuid, environment_uuid)
);

CREATE INDEX idx_env_mapping_config ON env_agent_model_mapping(config_uuid);
CREATE INDEX idx_env_mapping_environment ON env_agent_model_mapping(environment_uuid);
CREATE INDEX idx_env_mapping_proxy ON env_agent_model_mapping(llm_proxy_uuid);

3. Agent Environment Config Variables:

CREATE TABLE agent_env_config_variables_mapping (
    id SERIAL PRIMARY KEY,
    config_uuid UUID NOT NULL,
    environment_uuid UUID NOT NULL,  -- From OpenChoreo, no FK
    variable_name VARCHAR(255) NOT NULL,
    secret_reference TEXT NOT NULL,  -- NOT exposed in API
    created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,

    CONSTRAINT fk_env_var_config FOREIGN KEY (config_uuid)
        REFERENCES agent_configurations(uuid) ON DELETE CASCADE,
    CONSTRAINT uq_env_var UNIQUE(config_uuid, environment_uuid, variable_name)
);

CREATE INDEX idx_env_var_config ON agent_env_config_variables_mapping(config_uuid);
CREATE INDEX idx_env_var_environment ON agent_env_config_variables_mapping(environment_uuid);

Modified Tables

Artifacts Table:

ALTER TABLE artifacts ADD COLUMN in_catalog BOOLEAN DEFAULT FALSE;
CREATE INDEX idx_artifacts_in_catalog ON artifacts(in_catalog, organization_uuid)
  WHERE in_catalog = TRUE;

Component Interactions

┌─────────────┐
│   Console   │
│  (React UI) │
└──────┬──────┘
       │ 1. Create Model Config
       ↓
┌──────────────────────────────────────────────────────────────────┐
│            Agent Configuration Controller                        │
│  - Validate request                                              │
│  - Authenticate user                                             │
└──────┬───────────────────────────────────────────────────────────┘
       │ 2. Call service
       ↓
┌──────────────────────────────────────────────────────────────────┐
│         Agent Configuration Service                              │
│  - Validate agent exists (via InfraResourceManager)              │
│  - Validate environments exist (via InfraResourceManager)        │
│  - Validate providers in catalog (via LLMProviderRepository)     │
│  - Begin transaction                                             │
└──────┬───────────────────────────────────────────────────────────┘
       │ 3. For each environment
       ↓
┌──────────────────────────────────────────────────────────────────┐
│              LLM Proxy Service                                   │
│  - Create proxy configuration                                    │
│  - Return proxy UUID                                             │
└──────┬───────────────────────────────────────────────────────────┘
       │ 4. Deploy proxy
       ↓
┌──────────────────────────────────────────────────────────────────┐
│         LLM Proxy Deployment Service                             │
│  - Resolve gateway for environment                               │
│  - Deploy proxy to gateway                                       │
│  - Return deployment status                                      │
└──────┬───────────────────────────────────────────────────────────┘
       │ 5. Generate API key
       ↓
┌──────────────────────────────────────────────────────────────────┐
│           LLM Proxy API Key Service                              │
│  - Generate API key for proxy                                    │
│  - Return API key                                                │
└──────┬───────────────────────────────────────────────────────────┘
       │ 6. Store secrets (TODO)
       ↓
┌──────────────────────────────────────────────────────────────────┐
│             OpenChoreo Client (TODO)                             │
│  - CreateSecret(url-secret)                                      │
│  - CreateSecret(apikey-secret)                                   │
│  - Return secret references                                      │
└──────┬───────────────────────────────────────────────────────────┘
       │ 7. Save to database
       ↓
┌──────────────────────────────────────────────────────────────────┐
│      Agent Configuration Repository                              │
│  - Insert agent_configurations                                   │
│  - Insert env_agent_model_mapping                                │
│  - Insert agent_env_config_variables_mapping                     │
│  - Commit transaction                                            │
└──────────────────────────────────────────────────────────────────┘

Error Handling Flow:

IF error at any step:
  1. Rollback database transaction
  2. Cleanup created resources:
     - Undeploy proxies (LLMProxyDeploymentService.Undeploy)
     - Delete proxies (LLMProxyService.Delete)
     - Delete secrets (OpenChoreoClient.DeleteSecret - TODO)
  3. Return error to client

UX

Out of Scope

• Organization-level provider management: Managed by admins (separate proposal)
• MCP tool configuration: Deferred to Phase 2 (this proposal focuses on LLM configuration)
• Agent lifecycle management: Deployment, scaling, monitoring
• Configuration versioning: Rollback based on database history (not Git-based)
• Multi-region deployments: Single region only in Phase 1
• Cost tracking/billing: Usage metrics only
• Custom policy authoring: Use pre-defined policies only

Alternatives Considered

No response

Open Questions

No response

Milestones

Phase	Scope	Milestone
Phase 1: Catalog API	Database & Backend for Catalog Discovery Add in_catalog column to artifacts table via Migration 008. Implement catalog repository with query methods supporting filtering by kind (llmProvider/agent/mcp) and pagination. Create catalog service and REST endpoint GET /orgs/{orgName}/catalog
Phase 2: Catalog UI	Frontend for Browsing LLM Providers Build React components for catalog browser with filtering capabilities by kind and search. Implement provider detail view showing capabilities, rate limits, and cost estimates. Add pagination controls and loading states with proper error handling.
Phase 3: Agent Config Backend & API	Complete Backend Implementation for Configuration Management Create database schema with three tables (agent_configurations, env_agent_model_mapping, agent_env_config_variables_mapping). Implement Go models with corrected GORM tags and repository layer. Build AgentConfigurationService with full CRUD operations: Create with proxy provisioning loop (create proxy, deploy to gateway with AI-first selection, generate API key, resolve URL), Update with environment diff calculation and smart redeployment, Delete with full cleanup. Implement all REST endpoints (POST/GET/PUT/DELETE /model-configs) with transaction handling, rollback on errors, and security-enforced response DTOs.
Phase 4: Agent Config Frontend	UI for Configuration Management Build configuration form with environment selector and provider picker integrated with catalog. Create list view, detail view with environment variable display, and validation error handling. Add loading states for async operations.
Phase 5: Secret Integration	OpenChoreo Secret Management (Blocked on API availability) Integrate with openchoreo secret management implementation for storing configuration