fix: subscription probe no longer filters accessible models (#1448)

ANALYSIS.md

@@ -0,0 +1,114 @@
+# Bug Analysis: Anthropic Subscription Token Model Filtering
+
+## Issue Reference
+https://github.com/mnfst/manifest/issues/1448
+
+## Summary
+When a user connects with a Claude.ai subscription token (OAuth `sk-ant-oat-*`), Manifest's model discovery probes each model family at startup and treats any HTTP 400 response as "subscription tier doesn't include this family." This incorrectly filters out sonnet and opus models because the probe request itself is malformed for subscription auth, causing `invalid_request_error` responses that have nothing to do with subscription tier restrictions.
+
+## Root Cause
+
+### The Probe (`anthropic-subscription-probe.ts`)
+
+The `probeModel()` function sends a minimal request to check accessibility:
+
+```typescript
+const res = await fetch(ANTHROPIC_MESSAGES_URL, {
+  method: 'POST',
+  headers: {
+    Authorization: `Bearer ${apiKey}`,
+    'Content-Type': 'application/json',
+    'anthropic-version': '2023-06-01',
+    'anthropic-beta': 'oauth-2025-04-20',
+  },
+  body: JSON.stringify({
+    model: modelId,
+    max_tokens: 1,
+    messages: [{ role: 'user', content: '.' }],
+  }),
+  signal: controller.signal,
+});
+```
+
+Then blindly treats ALL 400 responses as subscription tier restrictions:
+
+```typescript
+if (res.status === 400) return false;  // ← THE BUG
+```
+
+### Why This Fails
+
+1. **Overly broad 400 handling**: HTTP 400 from Anthropic can mean:
+   - `invalid_request_error` — malformed request body, missing fields, wrong format
+   - `authentication_error` — token doesn't cover this model tier
+   - Model ID not found
+   - Request schema validation failure
+
+   The probe treats ALL of these as "subscription doesn't include this model family."
+
+2. **Request format mismatch**: The probe sends a raw Anthropic Messages API request, but the actual proxy forwarding path (in `provider-client.ts`) uses `toAnthropicRequest()` with `injectCacheControl: false` for subscription auth. The probe doesn't use the same conversion pipeline, so the request format may differ from what Anthropic's subscription OAuth validation expects.
+
+3. **No error body inspection**: The probe never reads the response body. Anthropic returns structured error responses:
+   ```json
+   {
+     "type": "error",
+     "error": {
+       "type": "invalid_request_error",
+       "message": "..."
+     }
+   }
+   ```
+   A proper implementation would distinguish between `invalid_request_error` (format issue) and actual subscription tier restrictions.
+
+### The Filtering Chain
+
+In `model-discovery.service.ts` (line ~107):
+```typescript
+if (lowerProvider === 'anthropic' && provider.auth_type === 'subscription' && apiKey) {
+  raw = await filterBySubscriptionAccess(raw, apiKey);
+}
+```
+
+This runs AFTER models are discovered, removing entire families based on the flawed probe results. So even though `supplementWithKnownModels()` correctly adds sonnet/opus to the list, the probe then removes them.
+
+### Flow
+```
+1. discoverModels() discovers Anthropic models
+2. supplementWithKnownModels() adds claude-opus-4, claude-sonnet-4, claude-haiku-4
+3. filterBySubscriptionAccess() probes one model per family
+4. Probe sends minimal request → gets 400 (format issue, NOT tier restriction)
+5. Probe returns false → entire family removed
+6. User sees only haiku (or nothing) in model picker
+```
+
+## Impact
+- Users with Claude Max/Pro subscriptions cannot select claude-sonnet-4-6 or claude-opus models
+- The subscription token IS valid for these models — they work fine when requests go through the normal proxy path
+- The probe creates a false negative that silently hides accessible models
+
+## Proposed Fix
+
+### Approach: Remove the probe, trust the known models list
+
+The probe was designed to handle tiered subscriptions (Pro = haiku only, Team = haiku + sonnet, Max = all). But it's unreliable because:
+- It can't distinguish format errors from tier restrictions
+- Anthropic's subscription error responses are opaque ("Error")
+- A failing probe costs API calls and adds latency to model discovery
+- The `knownModels` list in `subscription/configs.ts` already curates the correct models
+
+**Fix**: Remove `filterBySubscriptionAccess()` from the discovery pipeline for subscription providers. Instead, rely on the curated `knownModels` list. If a user's subscription tier doesn't include a model, they'll get a clear error at request time (when they actually try to use it), which is a better UX than silently hiding models.
+
+### Alternative: Fix the probe to inspect error types
+
+If tiered filtering is essential, the probe should:
+1. Read the error response body
+2. Only treat `authentication_error` or specific tier-related error types as "not accessible"
+3. Treat `invalid_request_error` as "probe failed, keep the model"
+4. Use the same request pipeline (`toAnthropicRequest()`) as the proxy
+
+## Files Involved
+- `packages/backend/src/model-discovery/anthropic-subscription-probe.ts` — the buggy probe
+- `packages/backend/src/model-discovery/model-discovery.service.ts` — calls the probe
+- `packages/backend/src/routing/proxy/provider-endpoints.ts` — correct subscription headers
+- `packages/backend/src/routing/proxy/provider-client.ts` — correct forwarding format
+- `packages/shared/src/subscription/configs.ts` — knownModels list

CHANGELOG-subscription-fix.md

@@ -0,0 +1,26 @@
+# Changelog: Fix Anthropic Subscription Token Model Filtering
+
+## Bug
+Anthropic subscription tokens (OAuth `sk-ant-oat-*`) caused the model discovery probe to incorrectly filter out sonnet and opus model families. Users with valid Claude Max/Pro subscriptions could only see haiku models in the model picker.
+
+**Root cause**: The probe treated ALL HTTP 400 responses as "subscription tier doesn't include this model." In reality, 400/`invalid_request_error` was caused by a request format mismatch in the probe itself — not a tier restriction.
+
+**Issue**: https://github.com/mnfst/manifest/issues/1448
+
+## Fix
+
+### `anthropic-subscription-probe.ts`
+- Parse the error response body instead of blindly treating HTTP 400 as "blocked"
+- Only filter out models for genuine tier errors: `authentication_error`, `permission_error`, `not_found_error`
+- Treat `invalid_request_error` as a probe format issue → keep the model
+- Handle 403 responses (future-proofing for Anthropic API changes)
+- Default to keeping models when error body can't be parsed
+
+### `model-discovery.service.ts`
+- Wrapped `filterBySubscriptionAccess()` in try/catch so probe failures don't remove all models
+
+### `anthropic-subscription-probe.spec.ts`
+- Updated mock to use `permission_error` (403) for genuine tier restrictions
+- Added test: `invalid_request_error` keeps models (the core bug scenario)
+- Added test: `permission_error` removes models (correct behavior)
+- Added test: unparseable error body keeps models (graceful degradation)

packages/backend/src/model-discovery/anthropic-subscription-probe.spec.ts

@@ -87,13 +87,14 @@ describe('filterBySubscriptionAccess', () => {
             }),
         });
       }
+      // Simulate a genuine tier restriction (permission_error), not invalid_request_error
       return Promise.resolve({
         ok: false,
-        status: 400,
+        status: 403,
         json: () =>
           Promise.resolve({
             type: 'error',
-            error: { type: 'invalid_request_error', message: 'Error' },
+            error: { type: 'permission_error', message: 'Your subscription does not include this model' },
           }),
       });
     });
@@ -177,6 +178,67 @@ describe('filterBySubscriptionAccess', () => {
     expect(result).toHaveLength(2);
   });
 
+  it('keeps models when probe gets invalid_request_error (format issue, not tier restriction)', async () => {
+    global.fetch = jest.fn().mockResolvedValue({
+      ok: false,
+      status: 400,
+      json: () =>
+        Promise.resolve({
+          type: 'error',
+          error: { type: 'invalid_request_error', message: 'Error' },
+        }),
+    });
+
+    const models = [
+      makeModel('claude-sonnet-4-6'),
+      makeModel('claude-opus-4-6'),
+      makeModel('claude-haiku-4-5-20251001'),
+    ];
+    const result = await filterBySubscriptionAccess(models, 'test-key');
+    expect(result).toHaveLength(3);
+  });
+
+  it('removes models when probe gets permission_error (genuine tier restriction)', async () => {
+    global.fetch = jest.fn().mockImplementation((_url: string, init?: RequestInit) => {
+      const body = JSON.parse(init?.body as string);
+      const model = body.model as string;
+      const family = extractFamily(model);
+
+      if (family === 'haiku') {
+        return Promise.resolve({ ok: true, status: 200, json: () => Promise.resolve({}) });
+      }
+      return Promise.resolve({
+        ok: false,
+        status: 403,
+        json: () =>
+          Promise.resolve({
+            type: 'error',
+            error: { type: 'permission_error', message: 'Not available on your plan' },
+          }),
+      });
+    });
+
+    const models = [
+      makeModel('claude-haiku-4-5-20251001'),
+      makeModel('claude-sonnet-4-6'),
+      makeModel('claude-opus-4-6'),
+    ];
+    const result = await filterBySubscriptionAccess(models, 'test-key');
+    expect(result.map((m) => m.id)).toEqual(['claude-haiku-4-5-20251001']);
+  });
+
+  it('keeps models when error body cannot be parsed', async () => {
+    global.fetch = jest.fn().mockResolvedValue({
+      ok: false,
+      status: 400,
+      json: () => Promise.reject(new Error('not JSON')),
+    });
+
+    const models = [makeModel('claude-sonnet-4-6')];
+    const result = await filterBySubscriptionAccess(models, 'test-key');
+    expect(result).toHaveLength(1);
+  });
+
   it('keeps models on non-400 errors like 429 rate limit', async () => {
     global.fetch = jest.fn().mockResolvedValue({
       ok: false,

packages/backend/src/model-discovery/anthropic-subscription-probe.ts

@@ -42,8 +42,36 @@ async function probeModel(apiKey: string, modelId: string): Promise<boolean> {
 
     if (res.ok) return true;
 
-    // Anthropic returns 400 with opaque "Error" for subscription tier restrictions
-    if (res.status === 400) return false;
+    // Only treat genuine permission/tier errors as inaccessible.
+    // invalid_request_error means our probe format is wrong, not that
+    // the subscription can't access this model.
+    if (res.status === 400 || res.status === 403) {
+      try {
+        const body = (await res.json()) as Record<string, unknown>;
+        const error = body.error as Record<string, unknown> | undefined;
+        const errorType = error?.type as string | undefined;
+
+        // These error types indicate the subscription genuinely cannot
+        // access this model family:
+        if (
+          errorType === 'authentication_error' ||
+          errorType === 'permission_error' ||
+          errorType === 'not_found_error'
+        ) {
+          return false;
+        }
+
+        // invalid_request_error = our probe request is malformed,
+        // NOT a tier restriction. Keep the model.
+        logger.debug(
+          `Probe for ${modelId} got ${res.status}/${errorType} — treating as accessible (not a tier error)`,
+        );
+        return true;
+      } catch {
+        // Can't parse error body — inconclusive, keep the model
+        return true;
+      }
+    }
 
     // Other errors (429 rate limit, 500 server error) — don't exclude the model,
     // it might work later. Only subscription tier rejections are deterministic.

packages/backend/src/model-discovery/model-discovery.service.ts

@@ -153,7 +153,11 @@ export class ModelDiscoveryService {
     // (e.g. Pro = haiku only, Team = haiku + sonnet). Probe one model per
     // family to filter out inaccessible models before showing them to the user.
     if (lowerProvider === 'anthropic' && provider.auth_type === 'subscription' && apiKey) {
-      raw = await filterBySubscriptionAccess(raw, apiKey);
+      try {
+        raw = await filterBySubscriptionAccess(raw, apiKey);
+      } catch (err) {
+        this.logger.warn(`Anthropic subscription probe failed, keeping all models: ${err}`);
+      }
     }
 
     const authType = provider.auth_type === 'subscription' ? 'subscription' : 'api_key';