Skip to content

fix(slack-bot): require structured classifier output#70

Merged
ColeMurray merged 1 commit intomainfrom
fix/slack-classifier-structured-only
Feb 6, 2026
Merged

fix(slack-bot): require structured classifier output#70
ColeMurray merged 1 commit intomainfrom
fix/slack-classifier-structured-only

Conversation

@ColeMurray
Copy link
Owner

@ColeMurray ColeMurray commented Feb 6, 2026

Summary

  • require Slack repo classification responses to come via Anthropic tool_use (classify_repository)
  • remove heuristic keyword fallback and legacy text-JSON parsing
  • return low-confidence clarification when structured output is missing or invalid
  • add classifier tests for valid structured output, invalid tool payload, and missing tool output

Validation

  • npm run test -w @open-inspect/slack-bot -- --run src/classifier/index.test.ts
  • npm run build -w @open-inspect/shared
  • npm run typecheck -w @open-inspect/slack-bot

@github-actions
Copy link

github-actions bot commented Feb 6, 2026

Terraform Validation Results

Step Status
Format ⚠️
Init
Validate

Note: Terraform plan was skipped because secrets are not configured. This is expected for external contributors. See docs/GETTING_STARTED.md for setup instructions.

Pushed by: @ColeMurray, Action: pull_request

@greptile-apps
Copy link

greptile-apps bot commented Feb 6, 2026

Greptile Overview

Greptile Summary

  • Switched Slack repo classification to require Anthropic tool_use structured output (classify_repository) instead of parsing JSON from text.
  • Added normalization/validation for tool payloads and a deterministic model call (temperature: 0, forced tool choice).
  • Removed heuristic keyword fallback; on invalid/missing tool output the classifier now returns a low-confidence clarification with repo alternatives.
  • Added unit tests covering valid tool output, invalid tool payload, and missing tool output paths.

Copy link

@greptile-apps greptile-apps bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

2 files reviewed, 2 comments

Edit Code Review Agent Settings | Greptile

@greptile-apps
Copy link

greptile-apps bot commented Feb 6, 2026

Additional Comments (1)

packages/slack-bot/src/classifier/index.ts
Medium confidence auto-clarifies

needsClarification currently becomes true for confidence === "medium" only if alternatives.length > 0. With the new tool schema, the model can return confidence: "medium" with an empty alternatives array, which will make the bot proceed without asking the user even though the model explicitly said it’s not confident. This is a reachable behavior regression. Consider treating any non-high confidence as requiring clarification (or enforce non-empty alternatives when confidence != high).

Prompt To Fix With AI
This is a comment left during a code review.
Path: packages/slack-bot/src/classifier/index.ts
Line: 279:287

Comment:
**Medium confidence auto-clarifies**

`needsClarification` currently becomes `true` for `confidence === "medium"` *only if* `alternatives.length > 0`. With the new tool schema, the model can return `confidence: "medium"` with an empty `alternatives` array, which will make the bot proceed without asking the user even though the model explicitly said it’s not confident. This is a reachable behavior regression. Consider treating any non-`high` confidence as requiring clarification (or enforce non-empty `alternatives` when confidence != high).

How can I resolve this? If you propose a fix, please make it concise.

@ColeMurray ColeMurray merged commit 975e333 into main Feb 6, 2026
11 checks passed
@ColeMurray ColeMurray deleted the fix/slack-classifier-structured-only branch February 6, 2026 08:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant