feat: add capability flags to custom provider models#1387
Open
brunobuddy wants to merge 1 commit intomainfrom
Open
feat: add capability flags to custom provider models#1387brunobuddy wants to merge 1 commit intomainfrom
brunobuddy wants to merge 1 commit intomainfrom
Conversation
… models Custom provider models had capability flags hardcoded to false, capping quality scores at 2 and preventing them from being assigned to the reasoning tier. This adds optional capability_reasoning and capability_code boolean fields to the custom provider model schema, enabling proper quality scoring and tier assignment for models served through custom providers (e.g., gateway proxies). Closes #1047
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #1387 +/- ##
=======================================
Coverage 98.67% 98.67%
=======================================
Files 117 117
Lines 7302 7302
Branches 2829 2829
=======================================
Hits 7205 7205
Misses 88 88
Partials 9 9
Flags with carried forward coverage won't be shown. Click here to find out more. 🚀 New features to boost your workflow:
|
![cubic-dev-ai[bot]](https://avatars.githubusercontent.com/in/1082092?s=60&v=4){kind=small}
There was a problem hiding this comment.
1 issue found across 10 files
Prompt for AI agents (unresolved issues)
Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.
<file name="packages/backend/src/model-discovery/model-discovery.service.ts">
<violation number="1" location="packages/backend/src/model-discovery/model-discovery.service.ts:239">
P2: Pass the raw custom model name into computeQualityScore; using the prefixed `modelKey` adds an extra slash segment that prevents QUALITY_OVERRIDES from matching vendor-prefixed names (e.g., `anthropic/claude-sonnet-4-20250514`), so those models get the wrong quality tier.</violation>
</file>
Reply with feedback, questions, or to request a fix. Tag @cubic-dev-ai to re-run a review.
| capabilityReasoning: capReasoning, | ||
| capabilityCode: capCode, | ||
| qualityScore: computeQualityScore({ | ||
| model_name: modelKey, |
There was a problem hiding this comment.
P2: Pass the raw custom model name into computeQualityScore; using the prefixed modelKey adds an extra slash segment that prevents QUALITY_OVERRIDES from matching vendor-prefixed names (e.g., anthropic/claude-sonnet-4-20250514), so those models get the wrong quality tier.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At packages/backend/src/model-discovery/model-discovery.service.ts, line 239:
<comment>Pass the raw custom model name into computeQualityScore; using the prefixed `modelKey` adds an extra slash segment that prevents QUALITY_OVERRIDES from matching vendor-prefixed names (e.g., `anthropic/claude-sonnet-4-20250514`), so those models get the wrong quality tier.</comment>
<file context>
@@ -224,16 +224,25 @@ export class ModelDiscoveryService {
+ capabilityReasoning: capReasoning,
+ capabilityCode: capCode,
+ qualityScore: computeQualityScore({
+ model_name: modelKey,
+ input_price_per_token: inputPerToken,
+ output_price_per_token: outputPerToken,
</file context>
Suggested change
| model_name: modelKey, | |
| model_name: m.model_name, |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
capability_reasoningandcapability_codeboolean fields to the custom provider model schema (entity, DTO, service, model discovery)computeQualityScore()instead of being hardcoded to score 2capability_reasoning: trueare now eligible for the reasoning tierThis unblocks the use case described in #1047 — users routing built-in providers through a gateway proxy can now set capability flags on custom provider models so they receive proper quality scoring and tier assignment.
Backward-compatible: existing custom providers without these fields default to
false(no behavior change). No database migration needed (JSON column).Test plan
Closes #1047
Summary by cubic
Adds optional
capability_reasoningandcapability_codeto custom provider models and switches them to dynamic quality scoring. This lets gateway-proxied models get correct scores and reasoning-tier eligibility (closes #1047).capability_reasoningandcapability_codeacross entity/DTO/service, surfaced via model discovery and frontend API types.computeQualityScore(); remove fixed score of 2 for custom models.Written for commit 41d5871. Summary will update on new commits.