Skip to content

Commit a1e3219

Browse files
committed
docs: define model fallback explanation contract (br-36d)
1 parent e4a85d1 commit a1e3219

File tree

4 files changed

+104
-6
lines changed

4 files changed

+104
-6
lines changed

CHANGELOG.md

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -36,6 +36,7 @@ All notable changes to this project are documented in this file.
3636
- Added recovery workflow planning in `scripts/context_resilience.py` with resume hints, safe fallback steps, and diagnostics payloads.
3737
- Added `scripts/context_resilience_command.py` with `/resilience status` and `/resilience doctor` stress diagnostics.
3838
- Added `instructions/context_resilience_tuning.md` with practical tuning guidance and operating playbook.
39+
- Added `instructions/model_fallback_explanation_model.md` defining provider/model fallback trace structure, output levels, and redaction rules for Epic 12 Task 12.1.
3940

4041
### Changes
4142
- Documented extension evaluation outcomes and when each tool is the better fit.

IMPLEMENTATION_ROADMAP.md

Lines changed: 7 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -48,7 +48,7 @@ Use this map to avoid overlapping implementations.
4848
| E9 | Conditional Rules Injector | done | High | E1 | bd-1q8, bd-3rj, bd-fo8, bd-2ik | Enforce project conventions with scoped rules |
4949
| E10 | Auto Slash Command Detector | paused | Medium | E1, E8 | TBD | Resume only if intent precision stays high in prototypes |
5050
| E11 | Context-Window Resilience Toolkit | done | High | E4 | bd-2tj, bd-n9y, bd-2t0, bd-18e | Improve long-session stability and recovery |
51-
| E12 | Provider/Model Fallback Visibility | planned | Medium | E5 | TBD | Explain why model routing decisions happen |
51+
| E12 | Provider/Model Fallback Visibility | in_progress | Medium | E5 | bd-1jq | Explain why model routing decisions happen |
5252
| E13 | Browser Automation Profile Switching | planned | Medium | E1 | TBD | Toggle Playwright/agent-browser with checks |
5353
| E14 | Plan-to-Execution Bridge Command | planned | Medium | E2, E3 | TBD | Execute validated plans with progress tracking |
5454
| E15 | Todo Enforcer and Plan Compliance | planned | High | E14 | TBD | Keep execution aligned with approved checklists |
@@ -499,15 +499,16 @@ Every command-oriented epic must ship all of the following:
499499

500500
## Epic 12 - Provider/Model Fallback Visibility
501501

502-
**Status:** `planned`
502+
**Status:** `in_progress`
503503
**Priority:** Medium
504504
**Goal:** Make model routing and provider fallback decisions observable and explainable.
505505
**Depends on:** Epic 5
506506

507-
- [ ] Task 12.1: Define explanation model
508-
- [ ] Subtask 12.1.1: Define resolution trace format (requested -> attempted -> selected)
509-
- [ ] Subtask 12.1.2: Define compact vs verbose output levels
510-
- [ ] Subtask 12.1.3: Define redaction rules for sensitive provider details
507+
- [x] Task 12.1: Define explanation model
508+
- [x] Subtask 12.1.1: Define resolution trace format (requested -> attempted -> selected)
509+
- [x] Subtask 12.1.2: Define compact vs verbose output levels
510+
- [x] Subtask 12.1.3: Define redaction rules for sensitive provider details
511+
- [x] Notes: Added `instructions/model_fallback_explanation_model.md` defining fallback trace shape, output levels, redaction policy, and deterministic reason-code requirements.
511512
- [ ] Task 12.2: Implement resolution tracing
512513
- [ ] Subtask 12.2.1: Capture fallback chain attempts in runtime
513514
- [ ] Subtask 12.2.2: Store latest trace per command/session

README.md

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -438,6 +438,12 @@ Fallback behavior is deterministic:
438438
- unknown category -> `default_category`
439439
- unavailable model -> `default_category`
440440

441+
Fallback explanation contract (Epic 12 Task 12.1):
442+
- `instructions/model_fallback_explanation_model.md`
443+
- trace stages: `requested -> attempted -> selected`
444+
- output levels: `compact` and `verbose`
445+
- redaction policy for sensitive provider details
446+
441447
Resolution precedence (Task 5.2):
442448
1. `system_defaults`
443449
2. selected category defaults
Lines changed: 90 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,90 @@
1+
# Provider/Model Fallback Explanation Model
2+
3+
Epic 12 Task 12.1 defines the trace contract for explaining model and provider fallback decisions.
4+
5+
## Goals
6+
7+
- make every routing decision explainable in deterministic order
8+
- keep default output readable while preserving deep diagnostics for debugging
9+
- avoid exposing secrets or sensitive provider identifiers in normal traces
10+
11+
## Trace structure
12+
13+
A trace represents one routing decision in three stages:
14+
15+
1. `requested`: what the caller asked for
16+
2. `attempted`: ordered candidates that were evaluated
17+
3. `selected`: final model/provider outcome
18+
19+
Reference shape:
20+
21+
```json
22+
{
23+
"requested": {
24+
"category": "deep",
25+
"model": "openai/gpt-5.3-codex",
26+
"source": "user_override"
27+
},
28+
"attempted": [
29+
{
30+
"rank": 1,
31+
"model": "openai/gpt-5.3-codex",
32+
"provider": "openai",
33+
"result": "unavailable",
34+
"reason": "model_not_in_available_set"
35+
},
36+
{
37+
"rank": 2,
38+
"model": "openai/gpt-5-mini",
39+
"provider": "openai",
40+
"result": "accepted",
41+
"reason": "category_default_fallback"
42+
}
43+
],
44+
"selected": {
45+
"model": "openai/gpt-5-mini",
46+
"provider": "openai",
47+
"reason": "fallback_unavailable_model_to_category"
48+
}
49+
}
50+
```
51+
52+
## Output levels
53+
54+
### Compact (default)
55+
56+
- include only `requested.model/category`, final `selected`, and 1-line fallback reason
57+
- include attempted count but not full per-attempt details
58+
- optimized for `/routing status` and routine debugging
59+
60+
### Verbose
61+
62+
- include full `attempted` chain with rank and rejection reason per candidate
63+
- include resolution source metadata (`system_default`, `category_default`, `user_override`)
64+
- include deterministic timestamps or sequence ids when available
65+
66+
## Redaction rules
67+
68+
Always redact in both compact and verbose modes:
69+
70+
- API keys, tokens, bearer strings, authorization headers
71+
- full endpoint query strings containing credentials
72+
- account-scoped identifiers that can reveal tenant internals
73+
74+
Redaction behavior:
75+
76+
- preserve structural placeholders (`***redacted***`) so traces remain parseable
77+
- keep provider class labels (`openai`, `anthropic`, etc.) unless explicitly marked sensitive
78+
- if a field is fully sensitive, replace value and attach a redaction reason code
79+
80+
## Determinism requirements
81+
82+
- attempted candidates must be listed in exact evaluation order
83+
- fallback reason codes must use stable identifiers (no free-form prose)
84+
- identical inputs and availability state must yield identical trace output
85+
86+
## Integration targets
87+
88+
Task 12.2 should emit this trace model from runtime routing.
89+
Task 12.3 should expose compact and verbose views via user-facing commands.
90+
Task 12.4 should verify deterministic traces and redaction safety.

0 commit comments

Comments
 (0)