feat: Add universal LLM provider abstraction layer#344
Open
SolariSystems wants to merge 3 commits intoSpectral-Finance:mainfrom
Open
feat: Add universal LLM provider abstraction layer#344SolariSystems wants to merge 3 commits intoSpectral-Finance:mainfrom
SolariSystems wants to merge 3 commits intoSpectral-Finance:mainfrom
Conversation
Fixes Spectral-Finance#96 Implements complete Ollama integration enabling self-hosted LLM capabilities: ## Features - Full Lux.LLM behaviour implementation with call/3 - Model management: list, pull, delete, show, ensure_model - Health check for connection verification - Tool calling support with Beams, Prisms, and Lenses - JSON response formatting ## Configuration - Configurable endpoint (default: http://localhost:11434) - Model presets: default, smartest, fastest, coding - Environment variable overrides (OLLAMA_HOST, OLLAMA_MODEL) - Adjustable timeouts and connection pooling ## Files - lib/lux/llm/ollama.ex - Main provider implementation - config/ollama.exs - Configuration reference - test/integration/ollama_test.exs - Integration tests - config/config.exs - Default model presets Generated by Solari Bounty System https://github.com/SolariSystems Co-Authored-By: Solari Systems <solarisys2025@gmail.com>
Fixes Spectral-Finance#99 Implements a comprehensive provider management system for LLM integrations: ## Features - **ProviderManager GenServer** (`lib/lux/llm/provider_manager.ex`) - Universal interface to call any registered LLM provider - Automatic provider and model selection based on task type - Intelligent fallback handling when providers fail - Cost tracking and estimation per provider - Performance monitoring (latency, success/failure rates) - Statistics collection and reporting - **Provider Configuration** (`config/providers.exs`) - Default provider and fallback chain settings - Cost-per-token configuration for budgeting - Task-to-provider mappings (coding, creative, analysis, etc.) - Environment variable support for runtime configuration - **Unit Tests** (`test/lux/llm/provider_manager_test.exs`) - Tests for provider listing and status checking - Task recommendation tests - Cost estimation tests - Statistics tracking tests ## Usage ```elixir # Simple call with default provider ProviderManager.call("What is 2+2?") # Task-based automatic selection ProviderManager.call("Generate code", task: :coding) # With fallback chain ProviderManager.call("Complex task", fallback: [:openai, :anthropic, :ollama]) # Get statistics ProviderManager.get_stats() ``` ## Supported Providers - OpenAI (GPT-4o, GPT-4o-mini) - Anthropic (Claude 3) - Ollama (local models) - TogetherAI - Mira Generated by Solari Bounty System https://github.com/SolariSystems Co-Authored-By: Solari Systems <solarisys2025@gmail.com>
7 tasks
6 tasks
Enhanced provider manager with: - Circuit breaker pattern: Automatic failover when providers fail (threshold: 5 failures, 1min recovery) - LRU caching: Response caching with TTL and size limits (1000 max) - Rate limiting: Per-provider token bucket (60/min cloud, 1000/min local) - Health checks: Periodic provider availability monitoring - Latency percentiles: p50/p95/p99 tracking for SLA monitoring - Model-specific pricing: Accurate cost tracking per model variant - Token extraction: Real usage tracking from API responses Tests expanded from 10 to 40+ covering all new features.
SolariSystems
force-pushed
the
feat/provider-abstraction-layer
branch
from
e7b02fa to
ff1d294
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
/claim #99
Summary
Fixes #99
Implements a comprehensive LLM provider management system that enables:
Features
ProviderManager GenServer (
lib/lux/llm/provider_manager.ex)call/3- Universal LLM call with automatic provider selectionlist_providers/0- List all providers and their statusrecommend_provider/1- Get best provider for a task typeestimate_cost/3- Estimate API cost before callingget_stats/0- Get performance and cost statisticsProvider Configuration (
config/providers.exs)Unit Tests (
test/lux/llm/provider_manager_test.exs)Usage
Supported Providers
Test Plan