Skip to content

feat: natural language parsing and multi-instance for /consult #223

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
avifenesh merged 3 commits into from
Feb 17, 2026
Merged

feat: natural language parsing and multi-instance for /consult #223

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
a0b7d0b
feat: add natural language parsing and multi-instance support to /con…
avifenesh Feb 17, 2026
3028a69
fix: use placeholder timestamp in multi-session state example
avifenesh Feb 17, 2026
e3be073
fix: address Copilot review feedback
avifenesh Feb 17, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
18 changes: 9 additions & 9 deletions __tests__/consult-command.test.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -289,7 +289,7 @@ describe('security constraints', () => {
});

test('command validates --tool against allow-list', () => {
expect(commandContent).toMatch(/validate.*--tool.*allow-list|MUST be one of.*gemini.*codex.*claude/i);
expect(commandContent).toMatch(/validate.*tool.*against.*allow-list|allow-list.*gemini.*codex.*claude|one of.*gemini.*codex.*claude/i);
});

test('skill validates --context=file=PATH is within project', () => {
Expand Down Expand Up @@ -358,11 +358,11 @@ describe('interactive selection completeness', () => {
expect(commandContent).toMatch(/AskUserQuestion[\s\S]*?header:\s*"Effort"/);
});

test('Phase 2b has MUST enforcement language for effort', () => {
// Effort picker is in Step 2b (batch selection), model picker in Step 2c
expect(commandContent).toMatch(/Do NOT silently default --effort/);
expect(commandContent).toMatch(/Do NOT skip this step/);
expect(commandContent).toMatch(/Do NOT skip any missing parameter/);
test('Phase 2 has MUST enforcement language for parameters', () => {
// Check that the command enforces interactive resolution for missing params
expect(commandContent).toMatch(/Do NOT silently default any parameter|Do NOT silently use a default model/);
expect(commandContent).toMatch(/Do NOT skip model selection|Do NOT skip this/i);
expect(commandContent).toMatch(/MUST resolve ALL missing parameters/);
});

test('Phase 2 header has MUST language for all missing parameters', () => {
Expand Down Expand Up @@ -560,7 +560,7 @@ describe('codex AskUserQuestion transform compatibility', () => {
// The picker should have structured options, not just plain text
// Check that each tool option line has both label and description
const toolPickerSection = commandContent.match(
/Step 2b[\s\S]*?Step 2c/
/Step 2c[\s\S]*?Step 2d/
);
expect(toolPickerSection).not.toBeNull();

Expand All @@ -573,9 +573,9 @@ describe('codex AskUserQuestion transform compatibility', () => {
});

test('effort selection options have label+description format', () => {
// Effort picker is in Step 2b (batch selection), not Step 2c (model selection)
// Effort picker is in Step 2c (batch selection), model picker in Step 2d
const effortPickerSection = commandContent.match(
/Step 2b[\s\S]*?Step 2c/
/Step 2c[\s\S]*?Step 2d/
);
expect(effortPickerSection).not.toBeNull();

Expand Down
184 changes: 130 additions & 54 deletions adapters/codex/skills/consult/SKILL.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,59 +1,86 @@
<!-- AUTO-GENERATED by scripts/gen-adapters.js - DO NOT EDIT -->
---
name: consult
description: "Use when user asks to \"consult gemini\", \"ask codex\", \"get second opinion\", \"cross-check with claude\", \"consult another AI\", \"ask opencode\", \"copilot opinion\". Queries another AI CLI tool and returns the response."
description: "Use when user asks to \"consult gemini\", \"ask codex\", \"get second opinion\", \"cross-check with claude\", \"consult another AI\", \"ask opencode\", \"copilot opinion\", \"ask 3 codex\", \"multi-consult\". Queries another AI CLI tool and returns the response."
---

# /consult - Cross-Tool AI Consultation

You are executing the /consult command. Your job is to consult another AI CLI tool, get its response, and present the results to the user.
You are executing the /consult command. Your job is to parse the user's request (natural language or flags), resolve missing parameters interactively, and execute the consultation.

## Constraints

- NEVER expose API keys in commands or output
- NEVER run with permission-bypassing flags (`--dangerously-skip-permissions`, `bypassPermissions`)
- MUST use safe-mode defaults (`-a suggest` for Codex, `--allowedTools "Read,Glob,Grep"` for Claude)
- MUST enforce 120s timeout on all tool executions
- MUST validate `--tool` against allow-list: gemini, codex, claude, opencode, copilot (reject all others)
- MUST validate tool names against allow-list: gemini, codex, claude, opencode, copilot (reject all others)
- MUST validate `--context=file=PATH` is within the project directory (reject absolute paths outside cwd)
- MUST quote all user-provided values in shell commands to prevent injection
- NEVER execute tools the user has not explicitly requested

## Arguments
## Execution

Parse from $ARGUMENTS:
### Phase 1: Parse Input (Flags + Natural Language)

- **question**: What to ask the consulted tool (required unless --continue)
- **--tool**: Target tool: `gemini`, `codex`, `claude`, `opencode`, `copilot` (interactive picker if omitted)
- **--effort**: Thinking effort: `low`, `medium`, `high`, `max` (interactive picker if omitted)
- **--model**: Specific model name (interactive picker if omitted). Free text.
- **--context**: Auto-include context: `diff` (git diff), `file=PATH` (attach specific file), `none` (default)
- **--continue**: Continue last consultation session, or `--continue=SESSION_ID` for specific session
Parse `$ARGUMENTS` using both explicit flags and natural language extraction. Flags always take priority over NLP when both provide the same parameter.

## Execution
#### Step 1a: Extract explicit flags

Look for and remove these flags from `$ARGUMENTS`:

1. `--tool=VALUE` or `--tool VALUE` where VALUE is one of: gemini, codex, claude, opencode, copilot
2. `--effort=VALUE` or `--effort VALUE` where VALUE is one of: low, medium, high, max
3. `--model=VALUE` or `--model VALUE` (any string, including quoted)
4. `--context=VALUE` where VALUE is: diff, file=PATH, or none
5. `--continue` (optionally `--continue=SESSION_ID`)
6. `--count=N` where N is 1-5
Copy link

Copilot AI Feb 17, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The count parameter validation is mentioned in Step 1a but there's no actual validation logic in the parsing phase. The command should validate that N is between 1-5 when extracting the --count flag or parsing NLP count patterns, and immediately reject out-of-range values with the error message from the error table. Currently, the validation would only happen later when the error is displayed, but it's unclear where the actual check occurs.

Copilot uses AI. Check for mistakes.

Remove all matched flags and their values from `$ARGUMENTS`.

#### Step 1b: Natural language extraction (on remaining text)

After removing flags, parse the remaining text for these patterns:

### Phase 1: Parse Arguments
**Tool extraction** (case-insensitive):
- "with {tool}" (e.g., "with codex") -> tool
- "ask {tool}" (e.g., "ask gemini") -> tool
- "consult {tool}" -> tool
- "{tool} about" (e.g., "codex about") -> tool
- Tool names: claude, gemini, codex, opencode, copilot

Extract these values from `$ARGUMENTS`:
**Count extraction**:
- "ask {N} {tool}" (e.g., "ask 3 codex") -> count=N, tool
- "{N} {tool}" (e.g., "3 codex") -> count=N, tool
- "{N} instances" -> count=N
- "few instances" / "multiple" / "several" -> count=ambiguous (ask user in Phase 2)
Copy link

Copilot AI Feb 17, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Similar issue - NLP count extraction patterns can parse any number N, but there's no validation at the point of extraction to ensure 1 ≤ N ≤ 5. Add validation instructions after the count extraction patterns to reject invalid counts immediately.

Suggested change
- "few instances" / "multiple" / "several" -> count=ambiguous (ask user in Phase 2)
- "few instances" / "multiple" / "several" -> count=ambiguous (ask user in Phase 2)
- After extracting any numeric `{N}` above, immediately validate that `N` is an integer between 1 and 5 (inclusive).
- If `N < 1` or `N > 5`, do **not** treat it as a valid count: clear the count value, keep any other extracted parameters (like tool and question), and in Phase 2 ask the user to choose a count between 1 and 5 (or apply a safe default like 3 if the user declines to choose).

Copilot uses AI. Check for mistakes.

1. Look for `--tool=VALUE` or `--tool VALUE` where VALUE MUST be one of: gemini, codex, claude, opencode, copilot (reject others)
2. Look for `--effort=VALUE` or `--effort VALUE` where VALUE MUST be one of: low, medium, high, max
3. Look for `--model=VALUE` or `--model VALUE` (any string, including quoted strings like `"my model"`)
4. Look for `--context=VALUE` where VALUE is: diff, file=PATH, or none
5. Look for `--continue` (optionally `--continue=SESSION_ID`)
6. Remove all matched flags (including their values) from `$ARGUMENTS`. Handle quoted flag values (e.g., `--model "gpt 4"`) by removing the entire quoted string. Everything remaining is the **question**.
**Count validation**: After extracting count (from flags or NLP), validate: 1 <= count <= 5. If count < 1 or count > 5, show `[ERROR] Instance count must be 1-5. Got: {count}` and stop.

If no question text and no `--continue` flag found, show:
**Effort extraction**:
- "quick" / "fast" / "brief" -> effort=low
- "thorough" / "deep" / "carefully" / "detailed" -> effort=high
- "maximum" / "max effort" / "exhaustive" -> effort=max

**Question extraction**:
- Text after "about" is the question (e.g., "with codex about my auth approach" -> question="my auth approach")
- If no "about" pattern, everything remaining after removing tool/count/effort markers is the question

**Precedence rule**: Flags from Step 1a always override NLP from Step 1b.

If no question text and no `--continue` flag found after both steps:
```
[ERROR] Usage: /consult "your question" [--tool=gemini|codex|claude|opencode|copilot] [--effort=low|medium|high|max]
[ERROR] Usage: /consult "your question" or /consult with gemini about your question
```

### Phase 2: Interactive Parameter Selection
### Phase 2: Interactive Parameter Resolution

MUST resolve ALL missing parameters interactively before Phase 3. ONLY skip this phase if ALL of --tool, --effort, AND --model are explicitly provided by the user in $ARGUMENTS. Do NOT silently default any parameter.
MUST resolve ALL missing parameters interactively. ONLY skip this phase if ALL required params (tool, effort, model) are resolved AND either a question exists or --continue is present. Do NOT silently default any parameter.

#### Step 2a: Handle --continue

**Note:** `--continue` and `--count > 1` are mutually exclusive. Session resume applies to a single tool session. If both are present, show `[ERROR] Cannot use --continue with --count > 1. Use --continue for single session resume.` and stop.

If `--continue` is present:
1. Read the session file at `{AI_STATE_DIR}/consult/last-session.json`

Expand All @@ -64,9 +91,9 @@ If `--continue` is present:
2. If the file exists, restore the saved tool, session_id, and model from it
3. If the file does not exist, show `[WARN] No previous session found` and proceed as a fresh consultation

#### Step 2b: Batch Selection (tool + effort)
#### Step 2b: Detect installed tools

First, detect which tools are installed by running all 5 checks **in parallel** via Bash:
Run all 5 checks **in parallel** via Bash:

- `where.exe <tool> 2>nul && echo FOUND || echo NOTFOUND` (Windows)
- `which <tool> 2>/dev/null && echo FOUND || echo NOTFOUND` (Unix)
Expand All @@ -75,13 +102,15 @@ Check for: claude, gemini, codex, opencode, copilot.

If zero tools are installed: `[ERROR] No AI CLI tools found. Install at least one: npm i -g @anthropic-ai/claude-code, npm i -g @openai/codex, npm i -g opencode-ai`

Then use a SINGLE request_user_input call to ask all missing parameters at once. Include only the questions for parameters NOT already provided in $ARGUMENTS:
#### Step 2c: Batch selection for missing params

Use a SINGLE request_user_input call to ask all missing parameters at once. Include ONLY questions for parameters NOT already resolved from Phase 1:

```
request_user_input:
> **Codex**: Each question MUST include a unique `id` field (e.g., `id: "q1"`).
questions:
- header: "AI Tool" # SKIP if --tool provided
- header: "AI Tool" # SKIP if tool resolved
question: "Which AI tool should I consult?"
options (only if installed):
- label: "Claude" description: "Deep code reasoning"
Expand All @@ -90,23 +119,35 @@ request_user_input:
- label: "OpenCode" description: "Flexible model choice"
- label: "Copilot" description: "GitHub-integrated AI"

- header: "Effort" # SKIP if --effort provided
- header: "Effort" # SKIP if effort resolved
question: "What thinking effort level?"
options:
- label: "Medium (Recommended)" description: "Balanced speed and quality"
- label: "Low" description: "Fast, minimal reasoning"
- label: "High" description: "Thorough analysis"
- label: "Max" description: "Maximum reasoning depth"

- header: "Instances" # SKIP if count resolved or not hinted
question: "How many parallel consultations?"
options:
- label: "1 (Single)" description: "Standard single consultation"
- label: "2 (Compare)" description: "Two responses to compare"
- label: "3 (Panel)" description: "Three perspectives"
Comment on lines +130 to +135
Copy link

Copilot AI Feb 17, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The Instances picker only offers options 1, 2, and 3, but the specification allows counts from 1-5. Users who want 4 or 5 parallel consultations would have no way to select these values through the interactive picker. Either add options for "4 (Extensive)" and "5 (Maximum)" to the picker, or add an "Other" option that allows custom input (with validation).

Copilot uses AI. Check for mistakes.
- label: "5 (Full spread)" description: "Five diverse perspectives"
```
Comment on lines 110 to 137
Copy link

Copilot AI Feb 17, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The Codex adapter requires unique id fields for each question in request_user_input (as noted in the comment on line 107), but the example doesn't actually show the id fields. For Codex compatibility, each question block should include an explicit id field like id: "tool_select", id: "effort_select", and id: "instances_select". Without these, the Codex platform may reject or mishandle the user input request.

Copilot uses AI. Check for mistakes.
Comment on lines 111 to 137
Copy link

Copilot AI Feb 17, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The Codex adapter documentation notes that "Each question MUST include a unique id field (e.g., id: \"q1\")," but the example questions below don't include any id fields. Add id fields to each question in the request_user_input examples (e.g., "id: "tool"", "id: "effort"", "id: "instances"") to match the stated requirement.

Copilot uses AI. Check for mistakes.

Map tool choice to lowercase: "Claude" -> "claude", "Codex" -> "codex", etc.
Map effort choice: "Medium (Recommended)" -> "medium", "Low" -> "low", "High" -> "high", "Max" -> "max".
ONLY show the Instances question if:
- The user explicitly mentioned multiple instances (e.g., "few", "multiple", "several")
- The count was set but ambiguous
Do NOT show Instances question for simple single-tool requests. Default count=1 silently when no multi-instance intent detected.

IMPORTANT: Do NOT skip any missing parameter. Do NOT silently default --effort to "medium" or --tool to any value. Present pickers for ALL unresolved parameters.
Map tool choice to lowercase: "Claude" -> "claude", "Codex" -> "codex", etc.
Map effort choice: "Medium (Recommended)" -> "medium", "Low" -> "low", etc.
Map count choice: "1 (Single)" -> 1, "2 (Compare)" -> 2, "3 (Panel)" -> 3.

#### Step 2c: Model Selection (MUST ask if no --model)
#### Step 2d: Model selection (MUST ask if no --model)

After tool is resolved (from Step 2b or $ARGUMENTS), present a model picker with options specific to the selected tool. The user can always type a custom model name via the "Other" option.
After tool is resolved, present a model picker with options specific to the selected tool. The user can always type a custom model name via the "Other" option.

**For Claude:**
```
Expand Down Expand Up @@ -173,17 +214,21 @@ request_user_input:
options:
- label: "claude-sonnet-4-5" description: "Default Copilot model"
- label: "claude-opus-4-6" description: "Most capable Claude model"
- label: "gpt-5.3-codex" description: "OpenAI GPT-5.3 Codex"
- label: "gpt-5.3-codex" description: "OpenAI GPT-5.3 Codex"
- label: "gemini-3-pro" description: "Google Gemini 3 Pro"
```

Map the user's choice to the model string (strip " (Recommended)" suffix if present). Pass the selected model to the skill via `--model`.
Map the user's choice to the model string (strip " (Recommended)" suffix if present).

IMPORTANT: Do NOT skip this step. Do NOT silently use a default model. If --model was not explicitly provided in $ARGUMENTS, you MUST present this picker. The model lists above are current as of Feb 2026 - the user may type any model name supported by their tool via the "Other" option.
IMPORTANT: Do NOT skip model selection. Do NOT silently use a default model. If --model was not explicitly provided, you MUST present this picker.

### Phase 3: Invoke Consult Skill
### Phase 3: Execute Consultation

With all parameters resolved (tool, effort, model, question, and optionally context, continue), invoke the `consult` skill using the Skill tool:
With all parameters resolved (tool, effort, model, question, count, and optionally context/continue):

#### Single instance (count=1, the default)

Invoke the `consult` skill directly using the Skill tool:

```
Skill: consult
Expand All @@ -192,15 +237,35 @@ Args: "[question]" --tool=[tool] --effort=[effort] --model=[model] [--context=[c
Example: "Is this the right approach?" --tool=gemini --effort=high --model=gemini-3-pro
```

The skill handles the full consultation lifecycle: it resolves the model from the effort level, builds the CLI command, packages any context, executes the command via Bash with a 120-second timeout, and returns a plain JSON result.
The skill handles the full consultation lifecycle: model resolution, command building, context packaging, execution with 120s timeout, and returns a plain JSON result.

#### Multi-instance (count > 1)

### Phase 4: Parse Skill Output
Spawn the `consult:consult-agent` via the Task tool with all resolved parameters:

The skill returns a plain JSON object containing: `tool`, `model`, `effort`, `duration_ms`, `response`, `session_id`, and `continuable`. Parse the JSON directly from the skill output.
```
Task:
subagent_type: "consult:consult-agent"
prompt: |
Execute a multi-instance consultation with these pre-resolved parameters:
- tool: [tool]
- model: [model]
- effort: [effort]
- question: [question]
- count: [count]
- context: [context or none]

Run [count] parallel consultations with the same tool and parameters.
Return all responses formatted with numbered headers and a brief synthesis.
```

The agent handles parallel execution, temp file management, result parsing, and synthesis.

### Phase 5: Present Results
### Phase 4: Present Results

After parsing the JSON, format and display the result as human-friendly text:
#### Single instance

Parse the skill's plain JSON output and display:

```
Tool: {tool}, Model: {model}, Effort: {effort}, Duration: {duration_ms}ms.
Expand All @@ -211,35 +276,46 @@ The results of the consultation are:

For continuable tools (Claude, Gemini, Codex, OpenCode), display: `Session: {session_id} - use /consult --continue to resume`

Save session state for continuable tools (Claude, Gemini, Codex, OpenCode) to `{AI_STATE_DIR}/consult/last-session.json`.
Save session state to `{AI_STATE_DIR}/consult/last-session.json`.

#### Multi-instance

Platform state directory:
- Claude Code: `.claude/`
- OpenCode: `.opencode/`
- Codex CLI: `.codex/`
Display the agent's formatted output directly. The agent returns numbered responses with a synthesis section.

On failure: `[ERROR] Consultation Failed: {specific error message}`

## Error Handling

| Error | Output |
|-------|--------|
| No question provided | `[ERROR] Usage: /consult "your question" [--tool=gemini\|codex\|claude\|opencode\|copilot] [--effort=low\|medium\|high\|max]` |
| No question provided | `[ERROR] Usage: /consult "your question" or /consult with gemini about your question` |
| Tool not installed | `[ERROR] {tool} is not installed. Install with: {install command from skill}` |
| Tool execution fails | `[ERROR] {tool} failed: {error}. Try a different tool with --tool=[other]` |
| Timeout (>120s) | `[ERROR] {tool} timed out after 120s. Try --effort=low for faster response` |
| No tools available | `[ERROR] No AI CLI tools found. Install: npm i -g @anthropic-ai/claude-code` |
| Session not found | `[WARN] No previous session found. Starting fresh consultation.` |
| API key missing | `[ERROR] {tool} requires API key. Set {env var} (see skill for details)` |
| Count out of range | `[ERROR] Instance count must be 1-5. Got: {count}` |
| Multi-instance partial failure | Show successful responses, note failures |

## Example Usage

```bash
/consult "Is this the right approach for error handling?" --tool=gemini --effort=high
/consult "Review this function for performance issues" --tool=codex
/consult "What alternative patterns would you suggest?" --tool=claude --effort=max
# Natural language (NLP parsing)
/consult with codex about my auth approach
/consult ask 3 codex about this design
/consult gemini should I use redis or postgres
/consult thoroughly ask claude about error handling
/consult codex few instances about performance

# Explicit flags (backward compatible)
/consult "Is this the right approach?" --tool=gemini --effort=high
/consult "Review this function" --tool=codex --count=3
/consult "Suggest improvements" --tool=opencode --model=github-copilot/claude-opus-4-6
/consult "Continue from where we left off" --continue
/consult --continue
/consult "Explain this error" --context=diff --tool=gemini
/consult "Review this file" --context=file=src/index.js --tool=claude

# Mixed (flags + natural language)
/consult with gemini --effort=max about database schema design
```
Loading
Loading