Skip to content

Conversation

@lerouxb
Copy link
Contributor

@lerouxb lerouxb commented Sep 11, 2025

This isn't all the prompts from the bug bash. I'm leaving the atlas search related ones for a separate PR because in those cases the responses are too bad to use as the expected answers so we'll have to write our own expected answers which is considerably more work.

@lerouxb lerouxb requested a review from gagik September 11, 2025 09:13
@lerouxb lerouxb marked this pull request as ready for review September 11, 2025 09:14
@lerouxb lerouxb requested a review from a team as a code owner September 11, 2025 09:14
@Copilot Copilot AI review requested due to automatic review settings September 11, 2025 09:14
Copy link
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

This PR adds regression test cases from the bug bash to prevent AI assistant from making common mistakes, including incorrectly attributing MongoDB features as Compass features, accessing user data it doesn't have, and providing unhelpful responses to certain question types.

  • Adds a new trick-questions.ts file with 8 regression test cases covering various edge cases
  • Integrates the trick questions into the existing evaluation test suite
  • Extends the tag system to include new categories for better test organization

Reviewed Changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.

File Description
packages/compass-assistant/test/eval-cases/trick-questions.ts New test file containing 8 regression test cases with expected responses for edge cases
packages/compass-assistant/test/eval-cases/index.ts Imports and integrates trick questions into the main evaluation cases array
packages/compass-assistant/test/assistant.eval.ts Adds new tag types to support categorization of the trick question test cases

Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.

@lerouxb lerouxb merged commit eaa859a into main Sep 11, 2025
54 of 58 checks passed
@lerouxb lerouxb deleted the bugbash-prompts branch September 11, 2025 12:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants