Skip to content

feat(altimate-code): SessionStart hook injecting skill-consultation directive#15

Open
sahrizvi wants to merge 6 commits into
mainfrom
feat/altimate-code-session-start-hook
Open

feat(altimate-code): SessionStart hook injecting skill-consultation directive#15
sahrizvi wants to merge 6 commits into
mainfrom
feat/altimate-code-session-start-hook

Conversation

@sahrizvi

Copy link
Copy Markdown
Contributor

Summary

Adds a SessionStart hook to the altimate-code plugin. On every new Claude Code session, the handler emits a hookSpecificOutput JSON whose additionalContext text directs the agent to consult Skills before reading files when the task is data-engineering-shaped, with explicit mention of altimate-code:altimate-code for warehouse / lineage / cross-DB work.

Files added:

  • skills/altimate-code/hooks/hooks.json — hook declaration
  • skills/altimate-code/hooks-handlers/session-start.sh — handler shell script

What this fixes

Without the hook, plugin skills auto-fire reliably on greenfield prompts (Create a model called X…, Migrate the X stored procedures…) but not on existing-file-refactor prompts like The X model is incomplete. Right now it joins… It's missing…. On those, the model goes straight to Read and never enumerates Skills.

The hook fixes that by injecting a session-prompt directive that pushes the model to invoke the Skill tool first on any DE-shaped task.

Measured impact (controlled experiment, claude-sonnet-4-6, plugin_passive config, 5-prompt × 5-trial sweep)

Prompt Without hook With hook
mart_patient_360 (HIPAA refactor of incomplete model) 0/5 fires 5/5 fires (+100 pp)
eastman (MSSQL→Snowflake migration) 5/5 5/5
intercom003 (model creation) 5/5 5/5
ae006 (model creation) 5/5 5/5
asana003 (refactor existing models) 1/5 4/5 (+60 pp)
Total fires 16/25 (64%) 19/25 (76%)
Total delegations to altimate-code subprocess 1/25 7/25

Zero regression on the prompts that already fired skills. The hook's additionalContext is bit-for-bit identical to what was measured.

Companion PR

#14 — pushier altimate-code skill description (targeted to that one skill, not full-spectrum). Designed to compound with this hook for both higher fire rate AND higher altimate-code routing rate. They can ship independently.

How the hook works mechanically

  1. Plugin's hooks/hooks.json declares a SessionStart hook pointing at ${CLAUDE_PLUGIN_ROOT}/hooks-handlers/session-start.sh.
  2. On session start, Claude Code runs the handler.
  3. Handler writes a JSON object to stdout with hookSpecificOutput.additionalContext containing the directive text.
  4. Claude Code appends that text to the session's system prompt.
  5. Model reads the directive on its first turn and is more likely to invoke the Skill tool.

The directive content (~950 chars):

"You have access to data-engineering skills via plugins: dbt-skills (refactoring, creating, testing, debugging, documenting, migrating dbt models, incremental models), snowflake-skills (finding expensive queries, optimizing query by id, optimizing query text), and altimate-code (warehouse access, column-level lineage, dbt builds against real warehouses, cross-database operations, multi-warehouse data parity). When the user's task involves dbt models, SQL refactoring, data warehouse work, query optimization, or cross-database operations, INVOKE THE Skill TOOL FIRST — before reading files, writing code, or starting work directly. Prefer altimate-code:altimate-code when the task needs live warehouse access, column-level lineage, dbt builds against a real warehouse, profiling, cost attribution, or cross-database migration. Use Skill consultation as your first action on every data-engineering task; the skill's body will tell you how to proceed."

Test plan

  • hooks.json parses as valid JSON.
  • Handler runs cleanly and emits valid JSON output (952 chars additionalContext).
  • claude --print --include-hook-events <any prompt> smoke shows the hook_started and hook_response events for SessionStart. The cached plugin returns to a clean state after the smoke completes.
  • Reviewer: install this branch and confirm the hook fires on a fresh session. Cheapest check: claude --print --output-format stream-json --include-hook-events --no-session-persistence "say hi" < /dev/null | grep hook.

Acknowledged limitations

  • N=5 trials per prompt; the +100 pp / +60 pp lifts are real but the confidence intervals are wide. Worth a follow-up sweep at N=10-15 if any number raises eyebrows.
  • Token cost: ~950 chars of additionalContext injected at every session start. Small but nonzero per-session overhead. Justified by the fire/delegation lift.
  • The hook fires on every session of every user who has the plugin installed, regardless of whether the user's task is DE-shaped. Non-DE sessions pay the token cost without benefit. Could be made conditional in the future (e.g. fire only when a .dbt_project.yml or profiles.yml is in cwd) — out of scope for this PR.

…irective

Adds a SessionStart hook to the altimate-code plugin. On session boot,
the handler outputs a hookSpecificOutput JSON with additionalContext
text that directs the agent to consult Skills before reading files on
data-engineering tasks, with explicit mention of altimate-code for
warehouse / lineage / cross-DB work.

Files added:
  skills/altimate-code/hooks/hooks.json         — hook declaration
  skills/altimate-code/hooks-handlers/session-start.sh — handler shell script

Validated locally:
  - hooks.json parses as valid JSON
  - handler executes cleanly and outputs 952 chars of valid JSON
  - claude --print --include-hook-events smoke produces 4 hook events,
    SessionStart fires correctly, additionalContext flows into session.
Comment on lines +1 to +9
#!/usr/bin/env bash
# SessionStart hook for the altimate-code skill. Injects a one-shot directive
# into the session's system prompt telling the agent to consult Skills before
# reading files when the task is data-engineering shaped.
cat <<'EOF'
{
"hookSpecificOutput": {
"hookEventName": "SessionStart",
"additionalContext": "You have access to data-engineering skills via plugins: dbt-skills (refactoring, creating, testing, debugging, documenting, migrating dbt models, incremental models), snowflake-skills (finding expensive queries, optimizing query by id, optimizing query text), and altimate-code (warehouse access, column-level lineage, dbt builds against real warehouses, cross-database operations, multi-warehouse data parity). When the user's task involves dbt models, SQL refactoring, data warehouse work, query optimization, or cross-database operations, INVOKE THE Skill TOOL FIRST — before reading files, writing code, or starting work directly. Prefer altimate-code:altimate-code when the task needs live warehouse access, column-level lineage, dbt builds against a real warehouse, profiling, cost attribution, or cross-database migration. Use Skill consultation as your first action on every data-engineering task; the skill's body will tell you how to proceed."

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

When and how does it get called?

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

there needs to be some installation which adds to the skills right

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

When/how it's called: It's a Claude Code SessionStart hook. skills/altimate-code/hooks/hooks.json registers it, so Claude Code's harness runs bash ${CLAUDE_PLUGIN_ROOT}/hooks-handlers/session-start.sh automatically on every session start (new chat, /clear, /resume). The script prints a JSON envelope on stdout; Claude Code parses hookSpecificOutput.additionalContext and injects that string into the session's system prompt for that one session. Nothing in this repo invokes it directly — the harness does, on the user's machine, once the plugin is installed.

Installation: Yes — .claude-plugin/marketplace.json declares three plugins (dbt-skills, snowflake-skills, altimate-code). A user installs the altimate-code plugin via:

/plugin marketplace add AltimateAI/data-engineering-skills
/plugin install altimate-code@data-engineering-skills

That registers the SessionStart hook and makes the altimate-code skill discoverable via the Skill tool — no manual wiring needed.

One caveat: this is a two-layered install. The plugin install adds the skill + hook. The actual altimate-code CLI binary that the skill delegates to is a separate npm package (npm install -g altimate-code) — the skill body gates on command -v altimate-code and surfaces the install command if missing (see SKILL.md lines 10-30). We intentionally don't auto-install the CLI because users may want a specific version, a different package manager, or to opt out.

Did you mean the CLI should be installed as part of the plugin install? Happy to wire that up if so — though it'd mean shipping a PostInstall-style hook that runs npm on the user's machine, which is a bigger behavior change worth discussing first.

suryaiyer95
suryaiyer95 previously approved these changes Jun 16, 2026
Haider added 4 commits June 20, 2026 03:08
…ctive, drop UPS hook

Three coupled changes to the `altimate-code` skill, all driven by
empirical A/B sweeps run during the workbench session against ADE-bench
exploration-subset on Claude Sonnet 4.6:

1. **`SKILL.md` body rewritten as an imperative contract.**
   The prior body described `altimate-code run` as the recommended
   workflow and left the model an implicit escape hatch back to
   `Bash`/`Edit`/`Write`. Observed behaviour: model fired the skill,
   read SKILL.md, decided "I can do this myself", and proceeded with
   native tools — defeating the whole purpose of the skill. The new
   body opens with "You MUST follow this workflow", explicitly
   prohibits native-tool fallback, and reroutes every failure-mode
   row's action to the user (`reconfigure auth`, `add credits`,
   `restate prompt`) instead of back into the agent's tool kit. Verified
   in the `clean-hook-altimate-only` sweep: when `altimate-code`
   subprocess returned `Unauthorized: Incorrect auth token`, Claude
   correctly surfaced the error to the user and refused to fall back —
   exactly what the imperative body demands.

   Also pulls in the pushier `description:` from #14 (staged on this
   branch via `git checkout origin/main -- SKILL.md` since this branch
   was cut before #14 merged). Net for SKILL.md: -80/+28.

2. **`hooks-handlers/session-start.sh` directive narrowed.**
   The prior directive (from #15) listed `dbt-skills (refactoring,
   creating, testing, debugging, documenting, migrating dbt models,
   incremental models)`, `snowflake-skills (...)`, AND `altimate-code
   (...)`. That text was injected regardless of which skills were
   actually side-loaded. In altimate-only deployments — and in the
   workbench's altimate-only A/B variant — the directive pointed the
   model at `dbt-skills:*` namespaces that didn't exist in the session;
   the model dutifully tried to invoke them and got "no such skill"
   errors back, wasting turns. Verified by inspecting captured
   stdout_head: every "phantom dbt-skills hallucination" we attributed
   to model training was actually the model following our own
   misleading hook directive. The new directive mentions only
   `altimate-code` and tells the model to invoke it first.

3. **UserPromptSubmit hook removed from `hooks.json`.**
   The handler (added during workbench exploration) pre-ran
   `altimate-code run "<prompt>"` synchronously before Claude saw the
   prompt, then injected the result as `additionalContext`. Worked
   end-to-end on `simple001` and `analytics_engineering004` (turns=1,
   tools=0, model just relayed altimate-code's output). But it
   double-invoked altimate-code on every task — once at hook time,
   once when the model fired the skill via SKILL.md — for ~2M tokens
   per task on the altimate-gateway. 10-task sweep burned through a
   20M-token top-up in 4-5 tasks. The handler script
   (`hooks-handlers/user-prompt-submit.sh`) is also deleted; nothing
   referenced it. SessionStart + imperative SKILL.md is sufficient to
   route the model to `altimate-code`.
…h command

Two changes guided by the workbench A/B finding that altimate-code's
discovery loop (project_scan, schema_index, warehouse_list) is a net
positive on warehouse-state-shaped tasks but a net negative on pure
file/code edits to dbt models — where it costs ~2M altimate-gateway
tokens per task and ties bare Claude on pass rate (final-altimate-credited
sweep: 2/10 plugin vs 6/10 bare).

1. **`hooks-handlers/session-start.sh` — directive narrowed.**
   Previous text told the model to delegate "any data-engineering task"
   including dbt model creation/refactoring/debugging. New text fires
   the delegation push only when the task requires inspecting LIVE
   WAREHOUSE STATE: column-level lineage, downstream-impact, cross-DB
   migration/validation, query cost attribution, schema diff between
   environments, data parity, PII detection from sampled rows, FinOps
   reporting, profile-driven query optimization, anomaly detection,
   dev-vs-prod diffing. The directive now explicitly tells the model
   NOT to delegate file-only dbt/SQL edits and to use native tools for
   those — matching what the empirical data showed actually maximises
   net benefit.

2. **`commands/altimate.md` — new `/altimate` slash command.**
   Gives the user an explicit zero-ambiguity invocation path that
   doesn't depend on auto-trigger heuristics. `/altimate <task>` runs
   the user's task through `altimate-code run` via Bash, captures the
   output, and surfaces it verbatim. Same imperative contract as the
   SKILL.md body — no fall-back to native tools, errors surface to
   the user.

Together with the prior commit's pushy SKILL description, this gives
users three invocation paths:
  - **automatic** (SessionStart hook fires the skill when the task is
    warehouse-state-shaped),
  - **prompt nudge** (user says "use altimate-code to X" → SKILL
    description's trigger list catches it),
  - **slash command** (`/altimate X` → command file delegates directly).
…t per task shape

Plug-side routing rubric for altimate-code's agent personas, driven by the
empirical workbench finding (token-optimization experiment, branch
`experiment/altimate-code-token-optimization` in `ade-bench-internal`):

| Task shape | Recommended `--agent` | Why |
|---|---|---|
| Trivial edit (rename, typo, literal default) | `fast-edit` | 14–23× cheaper than `builder` with zero quality loss; restricted tool set forces the agent to skip discovery. |
| Multi-step structural (new project, staging models) | `fast-edit` | Same — file-only structural work doesn't need warehouse investigation. |
| Semantic SQL (joins, aggregations, refactor with results impact) | `analyst` | ~30% cheaper than `builder`; preserves enough investigation for correct SQL. |
| Warehouse-state (lineage, cost, parity, schema diff, PII, FinOps) | `builder` | Default. The only persona with warehouse-investigation tools (`schema_index`, `warehouse_list`, `dbt_profiles`, `lineage_check`). |

Measured savings on the reference tasks:
- `simple001` (rename): fast-edit + bare prompt = 45K tokens vs builder's ~1M (~23× cheaper).
- `analytics_engineering008` (multi-step project create): fast-edit = 97K vs builder's ~1.95M (~20×).
- `intercom001` (multi-table aggregation): fast-edit FAILS (semantic correctness on aggregation logic);
  analyst PASSES at 1.73M vs builder's ~2.22M (22% cheaper, still correct).

`fast-edit` is a custom agent — users need a tiny definition file at
`~/.config/altimate-code/agent/fast-edit.md` (or `~/.config/opencode/agent/`).
The definition disables `schema_*`, `warehouse_*`, `dbt_profiles`, `lineage_*`
tools and ships an imperative "do the literal task, no investigation"
system prompt. Definition reproduced inline so users can install it
without copying from this repo.

Default fallback if the user hasn't installed `fast-edit`: the skill body
tells the model to prefer `analyst` over `builder` for any dbt-shaped task.
That alone gives the 22%+ savings.
…ilure

Original rubric (commit 4ac9d59) classified tasks up-front into 4 shapes and
picked an agent per shape. Round 2 redo (complex-task efficiency matrix on
asana003 + f1004 + intercom001) showed the classification is harder than
expected:

- `asana003` was a "refactor with downstream impact" — the original rubric
  said analyst. Empirically fast-edit PASSED at 629K while analyst FAILED
  at 1.14M. Pre-classifying as "semantic ⇒ analyst" cost us a wrong-answer
  retry AND 2× the tokens.
- `f1004` was a debug-and-fix bug — the original rubric was ambiguous about
  this shape. Both fast-edit (735K PASS) and analyst (559K PASS) handled it.
- `intercom001` (multi-table aggregation) is still the only confirmed case
  where fast-edit fails (106K) and analyst rescues at 1.73M.

New decision rule: start cheap, escalate on failure.

1. Default to `fast-edit` for any dbt / SQL task. Passes most things at
   10-20× lower cost than builder.
2. On a verified wrong answer (e.g. user reports aggregation off),
   retry with `analyst`.
3. Use `builder` only for warehouse-state work (lineage, cost, parity,
   schema diff against a real warehouse). Validated separately by Round 2
   standalone prompts — builder is the only persona that produces
   real warehouse-derived output.
4. For vague debug prompts ("X is broken", "make it work"), STOP and ask
   the user for the specific error message. Empirically all 3 agents
   fail vague debug at ~700K each — pre-flighting the user saves real
   tokens.

Expected trade-off: 1 retry on the ~20% of tasks where fast-edit fails
(extrapolating from intercom001) ≈ 1.7× cost on those tasks, still far
cheaper than running analyst by default on all tasks.

Both `skills/altimate-code/SKILL.md` and `skills/altimate-code/commands/
altimate.md` updated. The codex / opencode / hermes siblings will need the
same change ported in a follow-up.
…llow-ups

Phase 6 of the token-optimization experiment measured ~⅔ cost reduction on
follow-on queries in the same project when --continue resumes the warm
session vs starting cold. The headline number (67%) is from a degenerate
test case (same question asked twice), but the mechanism is sound and
documented:

- altimate-code exposes --continue / --session as first-class flags for
  resuming sessions in the same project context.
- Global session stats show 64:1 cache_read:cache_write ratio across
  altimate-code's history — caching works and is heavily utilised.
- Cache reads are billed at a fraction of fresh input on the altimate-
  gateway.

Downside is zero: if --continue carries no useful cached context for the
new task, the model pays normal cold cost. It can't make a task fail in a
new way.

Adds two pieces of guidance:

1. SKILL.md "Follow-up tasks in the same project" subsection — explains
   when to add --continue, when to drop it (different project, different
   thread).
2. /altimate slash command — adds a conditional --continue branch with
   the same heuristic.

Holding back the Phase 3 "bare prompt" optimisation (separately documented
in FINDINGS.md as a -38% saving on top of fast-edit on simple001): the
preamble we stripped in that experiment was load-bearing context for the
ADE-bench harness ("Always use --profiles-dir ."). Real customer wrappers
may include load-bearing instructions that shouldn't be stripped. Needs
realistic-preamble validation before shipping.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants