Skip to content

PROMPT-IMPROVER-VALIDATION.md

Latest commit

 

History

History
93 lines (70 loc) · 4.8 KB

PROMPT-IMPROVER-VALIDATION.md

File metadata and controls

93 lines (70 loc) · 4.8 KB

Prompt Improver Rewrite Behavior

Current Runtime Shape

  • Hook name: prompt-improver
  • Registered in session hooks via createSessionHooks()
  • Exposed handler: chat.message only
  • Config surface: src/config/schema/hooks.ts includes "prompt-improver" in HookNameSchema

The current runtime does not intercept delegate_task and does not register a tool.execute.before handler. It rewrites the first text part of a user chat message before the active model handles it.

Verified chat.message Flow

Source of truth:

  • src/hooks/prompt-improver/index.ts
  • src/plugin/hooks/create-session-hooks.ts
  • src/config/schema/hooks.ts

Verified flow:

  1. createSessionHooks() registers createPromptImproverHook(ctx) when the hook is enabled.
  2. The hook runs on chat.message.
  3. It exits early when there is no active model, no text part, only whitespace, or an unrecognized provider family.
  4. It also skips slash commands and messages wrapped in the auto-slash-command guard tags.
  5. It detects provider family from the active session model string, not from a delegate_task category or subagent selection.
  6. It calls improvePromptWithLLM() and replaces the text part only when the rewrite differs from the original.

Rewrite Scope

The implementation is narrower than the earlier documentation claimed.

  • It rewrites top-level chat input, not downstream delegated task payloads.
  • It operates on the current session model, including model names coming from alternate provider prefixes such as github-copilot/... when the model name still matches Anthropic, OpenAI, or Google families.
  • It leaves non-text parts untouched.
  • It preserves the original prompt only when rewrite infrastructure is unavailable or normalization produces an empty result.

LLM Rewrite Rules

improvePromptWithLLM() and rewritePromptViaSession() currently enforce these behaviors:

  • Internal rewrite sessions disable task, delegate_task, call_omo_agent, and question tools.
  • Rewrite sessions set thinking: { type: "disabled" }.
  • Rewrite sessions strip -thinking, -max, and -high suffixes from the model ID before prompting.
  • Assistant output is extracted only from text parts, including nested text content on text parts.
  • Rewrites are normalized before use, including stripping the internal marker and unwrapping a single fenced block.
  • Empty normalized results fall back to the original prompt.

Current OpenAI Guidance

The verified OpenAI branch is now execution-ready and avoids visible reasoning language.

  • Target structure remains markdown with # Task and ## Output Contract as the fixed base.
  • The rewrite must preserve user intent and constraints.
  • The output contract now requires Deliverable, Scope, Verification, and Final Response bullets.
  • Conditional sections are added only when the original prompt shape warrants them:
    • ## Execution Rules for multi-step or tool-driven prompts
    • ## Research Rules for investigation/evidence-driven prompts
    • ## Completion Rules for batch or coverage-sensitive prompts
  • The rewrite must not ask for visible reasoning, progress narration, or chain-of-thought output.

Verified Coverage

Current coverage is focused and file-backed, not broad integration marketing.

  • src/hooks/prompt-improver/index.test.ts
    • verifies the hook exposes chat.message and does not expose tool.execute.before
    • covers early exits for missing model, unknown provider, empty input, missing text parts, slash commands, and auto-slash-command tags
    • covers provider detection from active model strings and OpenAI rewrites without schema validation
  • src/hooks/prompt-improver/llm-improver.test.ts
    • covers passthrough OpenAI rewrites, marker stripping, fenced-output normalization, and empty-normalization fallback
    • covers non-OpenAI fence normalization behavior
  • src/hooks/prompt-improver/session-manager.test.ts
    • covers disabled thinking, reasoning suffix stripping, and assistant text extraction rules
  • src/hooks/prompt-improver/prompt-builder.test.ts
    • covers OpenAI execution-ready instructions and Gemini guide injection behavior
  • src/hooks/prompt-improver/codex-guide.test.ts
    • covers the no-visible-reasoning Codex guidance block
  • src/hooks/prompt-improver/transformer.test.ts
    • covers provider-family detection across Anthropic, OpenAI, Google, unknown, and mixed provider-prefix cases

What This Validation Does Not Claim

  • No claim of automatic delegate_task interception
  • No claim of tool.execute.before integration
  • No claim of category-based or subagent-based routing inside this hook
  • No claim of broad end-to-end production coverage beyond the focused hook test suite and typecheck verification

Verification Targets

Required verification for this documentation pass:

bun test src/hooks/prompt-improver && bun run typecheck