feat: add per-model inference provider selection #2092
Closed
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Allow users to select their preferred inference provider per model with options including: - Auto (respects user's HuggingFace preference order) - Fastest (highest throughput provider) - Cheapest (lowest cost per output token) - Specific providers (together, sambanova, etc.) The provider preference is appended to the model ID when making inference requests (e.g., "model:fastest") following HuggingFace's documented API. Router models (Omni) are excluded from this feature as they handle their own intelligent routing. Changes: - Add providerOverrides setting (Record<string, string>) to user settings - Add provider dropdown UI on model settings page (hidden for router models) - Pass provider preference through text generation pipeline - Support provider suffix in both regular endpoint and MCP tool flow https://claude.ai/code/session_016q6bhcJrusLKVMLq6x739B
- Add isHuggingChat check to UI dropdown visibility - Add isHuggingChat check to server-side provider usage https://claude.ai/code/session_016q6bhcJrusLKVMLq6x739B
- Use optgroups to separate selection modes (Auto/Fastest/Cheapest) from specific providers - Remove redundant "Available:" badges below the dropdown - Remove unused PROVIDERS_HUB_ORGS import
The providerOverrides field was missing from: - GET /user/settings response - POST /user/settings validation schema - POST /settings validation schema This prevented the provider selection from being persisted to the database.
Display a small badge next to the model name when a provider override is set: - ⚡ for "fastest" - 💰 for "cheapest" - Provider name for specific providers Hidden for router models and when set to "auto".
- Add IconCheap and IconFast from moon-landing - Use green theme for fastest, blue for cheapest - Show provider avatar for specific providers - Only display icons, no text
- Add provider indicator badge in the model navigation sidebar - Shows fastest (green), cheapest (blue), or provider avatar icons - Minor styling tweaks to provider select and chat window icon
Replace native select element with bits-ui Select component to display provider icons in the dropdown options. Includes styled icon containers for auto (magic wand), fastest (bolt), cheapest (coin), and provider avatars matching the model nav list styling.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary
Changes
IconFastandIconCheapicons for selection modesproviderOverridesin user settingsTest plan