How are you testing MCP Servers?

[pulkitsharma07]

Pre-submission Checklist

• I have verified that this discussion would not be more appropriate as an issue in a specific repository
• I have searched existing discussions to avoid duplicates

Discussion Topic

Here's my current flow.

• Update MCP Server codebase
• There is a tsc --watch process which rebuilds the server on any change.
• Go to MCP Inspector -> Click on Connect to reconnect to the MCP Server to get new changes.
• List tools -> Select Tool
• Enter the arguments (can be cumbersome if you have many args) manually
• Check the output (manually)
• Fix something in MCP Server -> Repeat from beginning.

This entire loop necessitates requirement of a testing framework for MCPs.
There can be some tests present on the MCP Server end, but for end-to-end integration testing, there should be a way to define test cases and replay those to check expectation.

Maybe something like this should be built into the Inspector itself.

[strowk]

I made a utility mcp-autotest for MCP specific tests that can be used regardless of language - https://github.com/strowk/mcp-autotest

I use its underlying golang package to test this server - https://github.com/strowk/mcp-k8s-go/
Test cases look like this - https://github.com/strowk/mcp-k8s-go/blob/main/testdata/with_k3d/list_k8s_nodes_test.yaml
or this - https://github.com/strowk/mcp-k8s-go/blob/main/testdata/with_k3d/list_k8s_contexts_test.yaml

From that server specifically some infra had to be created that is automated within go testing framework. I imagine that other language-based servers can either wrap mcp-autotest command into their own testing frameworks or port the whole library completely and implement similarly structured test definitions relatively easily too.

I think having some unified format that would by itself be language agnostic would be convenient for community at large. This format that mcp-autotest implements I have documented here - https://mcp-cases.str4.io/ .

I have also checked this testing approach against python based official example SQLite server and made a PR to servers, you can check some actually working tests here - https://github.com/modelcontextprotocol/servers/pull/1173/files#diff-4a80954a8ce52167ff3cb708f1f34956e2ec3260c626aefcb0a22a57e1067420R48-R96