Skip to content

Conversation

@latekvo
Copy link
Member

@latekvo latekvo commented Dec 10, 2025

This PR adds tests for checking how keen AI is to use the MCP tools we provide in different scenarios.

Note: These tests are, and always will be flaky. This is unavoidable. The goal of these tests isn't to get 100% success rate, the goal is to improve the current success rate from ~33% to 70% or more.

How Has This Been Tested:

  • Run command: Radon IDE: Test AI tool usage (id: RNIDE.testChatToolUsage)
  • See results.
Collapsed example test results
=== AI TEST RESULTS ===
FAIL | Error: No tools were called.
FAIL | Error: No tools were called.
FAIL | Error: No tools were called.
FAIL | Error: No tools were called.
FAIL | Error: No tools were called.
FAIL | Error: No tools were called.
OK 
OK 
OK 
OK 
OK 
FAIL | Error: No tools were called.
FAIL | Error: No tools were called.
FAIL | Error: Expected: view_screenshot. Received: copilot_findTextInFiles, copilot_findTextInFiles, copilot_readFile, copilot_readFile, copilot_readFile
FAIL | Error: Expected: view_screenshot. Received: copilot_readFile
OK 
FAIL | Error: Expected: view_component_tree. Received: copilot_findTextInFiles
FAIL | Error: Expected: view_component_tree. Received: copilot_findTextInFiles
OK 
FAIL | Error: No tools were called.
FAIL | Error: Expected: view_component_tree | view_screenshot. Received: copilot_findTextInFiles, copilot_findTextInFiles, copilot_findTextInFiles, copilot_findTextInFiles, copilot_findTextInFiles, copilot_searchCodebase
# TOTAL CORRECT: 7/21 (33.3%)

How Has This Change Been Documented:

Does not apply (pretty sure)

@vercel

This comment was marked as spam.

This comment was marked as resolved.

Copy link
Collaborator

@filip131311 filip131311 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this should be a part of vscode-extension-tester and not the main extension

@kmagiera kmagiera closed this Jan 22, 2026
@kmagiera kmagiera deleted the @latekvo/create-ai-tests branch January 22, 2026 09:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants