Skip to content

Fix MCP test independence #5046

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 2 commits into from
+171 −8
Closed

Fix MCP test independence #5046

Changes from 1 commit
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
5cb60d9
fix: ensure MCP test independence by adding config setup to all tests
daniel-lxs Jun 23, 2025
e52478f
feat: add mcpServersInitialized event to API and ClineProvider, emit …
daniel-lxs Jun 25, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
134 changes: 133 additions & 1 deletion apps/vscode-e2e/src/suite/tools/use-mcp-tool.test.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -353,7 +353,40 @@ suite("Roo Code use_mcp_tool Tool", function () {
}
}
api.on("taskCompleted", taskCompletedHandler)
await sleep(2000) // Wait for Roo Code to fully initialize

// Trigger MCP server detection by opening and modifying the file
console.log("Triggering MCP server detection by modifying the config file...")
try {
const mcpConfigUri = vscode.Uri.file(testFiles.mcpConfig)
const document = await vscode.workspace.openTextDocument(mcpConfigUri)
const editor = await vscode.window.showTextDocument(document)

// Make a small modification to trigger the save event, without this Roo Code won't load the MCP server
const edit = new vscode.WorkspaceEdit()
const currentContent = document.getText()
const modifiedContent = currentContent.replace(
'"alwaysAllow": []',
'"alwaysAllow": ["read_file", "read_multiple_files", "write_file", "edit_file", "create_directory", "list_directory", "directory_tree", "move_file", "search_files", "get_file_info", "list_allowed_directories"]',
)

const fullRange = new vscode.Range(document.positionAt(0), document.positionAt(document.getText().length))

edit.replace(mcpConfigUri, fullRange, modifiedContent)
await vscode.workspace.applyEdit(edit)

// Save the document to trigger MCP server detection
await editor.document.save()

// Close the editor
await vscode.commands.executeCommand("workbench.action.closeActiveEditor")

console.log("MCP config file modified and saved successfully")
} catch (error) {
console.error("Failed to modify/save MCP config file:", error)
}

await sleep(5000) // Wait for MCP servers to initialize
let taskId: string
try {
// Start task requesting to use MCP filesystem write_file tool
Expand Down Expand Up @@ -481,7 +514,40 @@ suite("Roo Code use_mcp_tool Tool", function () {
}
}
api.on("taskCompleted", taskCompletedHandler)
await sleep(2000) // Wait for Roo Code to fully initialize

// Trigger MCP server detection by opening and modifying the file
console.log("Triggering MCP server detection by modifying the config file...")
try {
const mcpConfigUri = vscode.Uri.file(testFiles.mcpConfig)
const document = await vscode.workspace.openTextDocument(mcpConfigUri)
const editor = await vscode.window.showTextDocument(document)

// Make a small modification to trigger the save event, without this Roo Code won't load the MCP server
const edit = new vscode.WorkspaceEdit()
const currentContent = document.getText()
const modifiedContent = currentContent.replace(
'"alwaysAllow": []',
'"alwaysAllow": ["read_file", "read_multiple_files", "write_file", "edit_file", "create_directory", "list_directory", "directory_tree", "move_file", "search_files", "get_file_info", "list_allowed_directories"]',
)

const fullRange = new vscode.Range(document.positionAt(0), document.positionAt(document.getText().length))

edit.replace(mcpConfigUri, fullRange, modifiedContent)
await vscode.workspace.applyEdit(edit)

// Save the document to trigger MCP server detection
await editor.document.save()

// Close the editor
await vscode.commands.executeCommand("workbench.action.closeActiveEditor")

console.log("MCP config file modified and saved successfully")
} catch (error) {
console.error("Failed to modify/save MCP config file:", error)
}

await sleep(5000) // Wait for MCP servers to initialize
let taskId: string
try {
// Start task requesting MCP filesystem list_directory tool
Expand Down Expand Up @@ -620,7 +686,40 @@ suite("Roo Code use_mcp_tool Tool", function () {
}
}
api.on("taskCompleted", taskCompletedHandler)
await sleep(2000) // Wait for Roo Code to fully initialize

// Trigger MCP server detection by opening and modifying the file
console.log("Triggering MCP server detection by modifying the config file...")
try {
const mcpConfigUri = vscode.Uri.file(testFiles.mcpConfig)
const document = await vscode.workspace.openTextDocument(mcpConfigUri)
const editor = await vscode.window.showTextDocument(document)

// Make a small modification to trigger the save event, without this Roo Code won't load the MCP server
const edit = new vscode.WorkspaceEdit()
const currentContent = document.getText()
const modifiedContent = currentContent.replace(
'"alwaysAllow": []',
'"alwaysAllow": ["read_file", "read_multiple_files", "write_file", "edit_file", "create_directory", "list_directory", "directory_tree", "move_file", "search_files", "get_file_info", "list_allowed_directories"]',
)

const fullRange = new vscode.Range(document.positionAt(0), document.positionAt(document.getText().length))

edit.replace(mcpConfigUri, fullRange, modifiedContent)
await vscode.workspace.applyEdit(edit)

// Save the document to trigger MCP server detection
await editor.document.save()

// Close the editor
await vscode.commands.executeCommand("workbench.action.closeActiveEditor")

console.log("MCP config file modified and saved successfully")
} catch (error) {
console.error("Failed to modify/save MCP config file:", error)
}

await sleep(5000) // Wait for MCP servers to initialize
let taskId: string
try {
// Start task requesting MCP filesystem directory_tree tool
Expand Down Expand Up @@ -841,7 +940,40 @@ suite("Roo Code use_mcp_tool Tool", function () {
}
}
api.on("taskCompleted", taskCompletedHandler)
await sleep(2000) // Wait for Roo Code to fully initialize
Copy link
Collaborator

@cte cte Jun 23, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could we hook into our event emitter to determine if we're initialized instead of sleeping?

Copy link
Member Author

@daniel-lxs daniel-lxs Jun 23, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It seems like taskStarted will not wait for MCP servers to be initialized causing the model to not see them.

Do you know of any event I can use to wait for the MCP servers to initialize?

interface RooCodeAPIEvents {
    message: [data: {
        taskId: string;
        action: "created" | "updated";
        message: ClineMessage;
    }];
    taskCreated: [taskId: string];
    taskStarted: [taskId: string];
    taskModeSwitched: [taskId: string, mode: string];
    taskPaused: [taskId: string];
    taskUnpaused: [taskId: string];
    taskAskResponded: [taskId: string];
    taskAborted: [taskId: string];
    taskSpawned: [parentTaskId: string, childTaskId: string];
    taskCompleted: [taskId: string, tokenUsage: TokenUsage, toolUsage: ToolUsage, isSubtask: IsSubtask];
    taskTokenUsageUpdated: [taskId: string, tokenUsage: TokenUsage];
    taskToolFailed: [taskId: string, toolName: ToolName, error: string];
}

Copy link
Collaborator

@cte cte Jun 23, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe we should add one?


// Trigger MCP server detection by opening and modifying the file
console.log("Triggering MCP server detection by modifying the config file...")
try {
const mcpConfigUri = vscode.Uri.file(testFiles.mcpConfig)
const document = await vscode.workspace.openTextDocument(mcpConfigUri)
const editor = await vscode.window.showTextDocument(document)

// Make a small modification to trigger the save event, without this Roo Code won't load the MCP server
const edit = new vscode.WorkspaceEdit()
const currentContent = document.getText()
const modifiedContent = currentContent.replace(
'"alwaysAllow": []',
'"alwaysAllow": ["read_file", "read_multiple_files", "write_file", "edit_file", "create_directory", "list_directory", "directory_tree", "move_file", "search_files", "get_file_info", "list_allowed_directories"]',
)

const fullRange = new vscode.Range(document.positionAt(0), document.positionAt(document.getText().length))

edit.replace(mcpConfigUri, fullRange, modifiedContent)
await vscode.workspace.applyEdit(edit)

// Save the document to trigger MCP server detection
await editor.document.save()

// Close the editor
await vscode.commands.executeCommand("workbench.action.closeActiveEditor")

console.log("MCP config file modified and saved successfully")
} catch (error) {
console.error("Failed to modify/save MCP config file:", error)
}

await sleep(5000) // Wait for MCP servers to initialize
let taskId: string
try {
// Start task requesting MCP filesystem get_file_info tool
Expand All @@ -857,7 +989,7 @@ suite("Roo Code use_mcp_tool Tool", function () {
})

// Wait for attempt_completion to be called (indicating task finished)
await waitFor(() => attemptCompletionCalled, { timeout: 45_000 })
await waitFor(() => attemptCompletionCalled, { timeout: 50_000 })

// Verify the MCP tool was requested with valid format
assert.ok(mcpToolRequested, "The use_mcp_tool should have been requested")
Expand Down
Loading