Comprehensive Provider Abstraction Layer Needed

[ebowwa]

Context

Following the resolution of #32, which revealed deeper architectural needs beyond the immediate system_instruction fix, this issue tracks the comprehensive work needed to achieve true provider abstraction in ai-proxy-core.

Problem Statement

While fixing #32, we discovered that parameter handling is inconsistent across providers. The library currently requires developers to understand provider-specific differences, defeating the purpose of an abstraction layer.

Current Gaps

1. Parameter Normalization

• Temperature: Different ranges (0-1 vs 0-2)
• Token limits: max_tokens vs max_output_tokens vs max_new_tokens
• Response formats: JSON mode implementations vary
• Stop sequences: Different parameter names and behaviors
• Sampling parameters: top_p, top_k availability varies

2. Feature Parity Issues

• Tool/Function calling: Completely different implementations
• Streaming: Inconsistent response formats
• Vision capabilities: Not all providers support images
• Embeddings: Different dimensionality and models
• Safety settings: Provider-specific implementations

3. Error Handling

• Different error formats and codes
• Inconsistent rate limiting responses
• No unified retry strategy
• Missing automatic failover

4. Missing Abstractions

# What we have:
system_instruction="Be helpful"  # Works after v0.4.3

# What we need:
response = await client.create_completion(
    messages=messages,
    model="auto",  # Provider selection based on requirements
    temperature=0.7,  # Normalized across all providers
    json_mode=True,  # Unified JSON response handling
    safety="medium",  # Abstracted safety levels
    retry_strategy="exponential",
    fallback_providers=["openai", "gemini", "ollama"],
    capabilities_required=["streaming", "tools", "vision"]
)

Proposed Solution

Phase 1: Discovery & Documentation

• Audit ALL parameters across providers
• Create compatibility matrix
• Document provider capabilities
• Identify normalization strategies

Phase 2: Core Abstractions

• Implement parameter normalization layer
• Create unified error handling
• Build capability detection system
• Add provider selection logic

Phase 3: Advanced Features

• Automatic failover between providers
• Cost optimization routing
• Performance-based selection
• Feature fallback strategies

Phase 4: Testing & Validation

• Cross-provider integration tests
• Performance benchmarks
• Error scenario testing
• Real-world usage validation

Success Criteria

1. Zero provider-specific code in applications using the library
2. Automatic handling of provider differences
3. Seamless failover when providers are unavailable
4. Consistent behavior across all providers
5. Clear capability reporting for feature availability

Technical Design Considerations

1. Backwards compatibility: Existing code must continue working
2. Progressive enhancement: Advanced features optional
3. Performance: Abstraction shouldn't add significant overhead
4. Extensibility: Easy to add new providers
5. Debugging: Clear visibility into provider selection and transformations

Related Issues

• OpenAI provider fails with 'unexpected keyword argument system_instruction' in v0.4.1 #32 - Original issue that exposed these needs

Questions for Discussion

1. Should provider-specific features be exposed or hidden?
2. How do we handle features that only some providers support?
3. What's the right balance between abstraction and control?
4. Should we support provider-specific escape hatches?

---

This is a significant architectural evolution that will make ai-proxy-core truly provider-agnostic. Looking for community input on priorities and approach.

[ebowwa]

📊 Provider Abstraction Analysis Complete

I've conducted a comprehensive analysis of the codebase to understand the full scope of abstraction work needed. The findings are documented in PROVIDER_ANALYSIS.md.

Key Findings

🔴 Critical Issues

1. Parameter Chaos: max_tokens vs max_output_tokens vs num_predict
2. Temperature Ranges: OpenAI uses 0-2, others use 0-1
3. JSON Mode: Three different implementations
4. Error Formats: Every provider throws different error structures

🟡 Missing Abstractions

• No automatic provider selection
• No feature detection ("Can this provider do X?")
• No failover when providers fail
• No cost optimization routing
• No capability matching

🟢 What's Working

• Basic message format (after v0.4.3)
• System instructions (after v0.4.3)
• Core text generation

Discovered Parameter Differences

# Current reality - developers need to know:
if provider == "openai":
    params["max_tokens"] = 1000
    params["temperature"] = 1.4  # 0-2 range
elif provider == "gemini":
    params["max_output_tokens"] = 1000
    params["temperature"] = 0.7  # 0-1 range
elif provider == "ollama":
    params["options"]["num_predict"] = 1000
    params["options"]["temperature"] = 0.7  # 0-1 range

Proposed Solution Architecture

# Future vision - true abstraction:
response = await client.create_completion(
    messages=messages,
    model="auto",  # Automatically select best provider
    temperature=0.7,  # Normalized across all providers
    max_tokens=1000,  # Abstracted parameter name
    require_features=["json_mode", "tools"],  # Capability requirements
    fallback_providers=["openai", "gemini"],  # Automatic failover
    optimize_for="cost"  # or "speed" or "quality"
)

Recommended Roadmap

Phase 1: Parameter Normalization (v0.4.4)

• Normalize all common parameters
• Handle range differences
• Maintain backward compatibility

Phase 2: Error Abstraction (v0.4.5)

• Unified error types
• Consistent retry logic
• Better error messages

Phase 3: Feature Detection (v0.5.0)

• Provider capability API
• Runtime feature checking
• Graceful degradation

Phase 4: Smart Routing (v0.6.0)

• Automatic provider selection
• Cost optimization
• Performance-based routing

Phase 5: Complete Abstraction (v1.0.0)

• Full feature parity handling
• Plugin architecture
• Production-ready abstraction

Next Steps

The full analysis in PROVIDER_ANALYSIS.md includes:

• Complete parameter mapping table
• Feature parity matrix
• Error handling analysis
• Testing requirements
• Migration strategies

This is a significant undertaking but will truly fulfill the library's vision of provider-agnostic AI integration.

View the full report: PROVIDER_ANALYSIS.md

[ebowwa]

Priority Implementation Roadmap

Based on the comprehensive analysis, here's my proposed prioritization for addressing the abstraction gaps:

1️⃣ JSON Mode Abstraction (Highest Priority)

• Implement unified JSON response handling across all providers
• Normalize the different approaches:
  • OpenAI: response_format: {"type": "json_object"}
  • Gemini: response_mime_type: "application/json"
  • Ollama: Currently unsupported - need fallback strategy
• Rationale: JSON mode is fundamental for structured data exchange and API integrations. Many production use cases depend on reliable JSON responses.

2️⃣ Parameter Normalization (Critical Foundation)

• Create parameter mapping layer for common inconsistencies
• Handle max_tokens vs max_output_tokens vs num_predict
• Normalize temperature ranges (0-2 vs 0-1)
• Rationale: This is low-hanging fruit that significantly improves developer experience without breaking changes.

3️⃣ Feature Detection Enhancement (Build on Existing)

• Leverage existing model detection capabilities we already have
• Extend to include feature availability per provider/model
• Create runtime capability queries
• Rationale: We already have some infrastructure here - let's expand it rather than rebuild from scratch.

4️⃣ Tools/Function Calling (Important but Complex)

• Abstract the different tool/function formats between providers
• Create unified tool definition schema
• Handle providers that don't support tools gracefully
• Rationale: While critical for advanced use cases, this is more complex and can follow after we nail the fundamentals.

5️⃣ Smart Routing (Future Enhancement)

• Defer automatic provider selection for now
• Focus on explicit provider control with better abstractions
• Rationale: Let's perfect the abstraction layer before adding automatic routing complexity. Users can still explicitly choose providers.

Implementation Philosophy

Rather than trying to solve everything at once, let's take an iterative approach:

1. Start with the most common pain points (JSON mode, parameters)
2. Build on existing code where possible (feature detection)
3. Defer complex features until the foundation is solid (smart routing)

This pragmatic approach gets us immediate value while building toward the complete vision.

---

Updated prioritization based on practical implementation considerations and immediate user needs.

[ebowwa]

🎉 Great timing with PRs #28 and #29!

These merged PRs actually demonstrate both the progress and challenges we're discussing:

PR #29: Client Context & Telemetry

This adds client identification and telemetry tagging - exactly the kind of cross-cutting concern that should be abstracted across all providers. Currently, each provider has to handle telemetry individually. In a properly abstracted system, this would be handled once at the abstraction layer.

What this reveals:

• ✅ Shows we're already thinking about provider-agnostic features
• ⚠️ But implementation is duplicated across providers
• 💡 Opportunity: Abstract telemetry/context handling in the base layer

PR #28: Content Type Support for Gemini

Perfect example of provider-specific capabilities that need abstraction:

• Gemini now supports PDF, video, documents natively
• OpenAI handles these differently (or not at all)
• Ollama has its own approach

What this reveals:

• ✅ Gemini's advanced content handling is now available
• ⚠️ But other providers can't handle these content types the same way
• 💡 Opportunity: Create content type abstraction layer that:
  • Routes to native handling when available (Gemini)
  • Falls back to extraction/conversion for others
  • Gracefully degrades when not supported

How These PRs Inform Our Abstraction Strategy

1. Client Context (Add client_context + IP fallback for REST and WebSocket; telemetry tagging and minor logging cleanup #29) → Should be handled once in CompletionClient, not in each provider
2. Content Types (feat: Add comprehensive content type support for PDF, video, and document formats #28) → Need capability detection: "Can this provider handle PDFs natively?"
3. Both PRs → Show the pattern: features get added per-provider, creating inconsistency

Updated Priority Considerations

Given these PRs, we should also prioritize:

• Content Type Abstraction - Normalize how different content types are handled
• Telemetry/Context Layer - Single implementation for cross-cutting concerns
• Capability Advertisement - Let providers declare what content types they support

These recent changes make the case for abstraction even stronger - we're adding features that would benefit from being implemented once at the abstraction layer rather than repeatedly in each provider.