Simple test automation for MCP server implementations

[steviec]

Pre-submission Checklist

• This post follows the MCP community guidelines

What would you like to share?

I built a few MCP servers that worked fine when when calling the tools directly, but would fail in a bunch of different ways when LLMs tried to use them. I didn't want to use a language-specific test library since I have MCP servers in a variety of languages, so I built a language-agnostic tool called mcp-server-tester.

It's a CLI tool you can run with npx mcp-server-tester like the Inspector, and you define the tests in a pretty simple YAML format. I'm using it all the time for my own MCP server development now.

Here's example output from one of my MCP server test suites:

> npx mcp-server-tester eval filesystem-eval-tests.yaml --server-config filesystem-server-config.json

🤖 Testing with claude-3-5-haiku-latest
✅ LLM can create and read files
✅ File manipulation workflow
✅ Error handling test
❌ Documentation query task: FAILED
    Prompt: "How do I search for configuration files in the project?"
    • Required tool 'search_docs' was not called (actual calls: list_files)
❌ Lists all available tools: FAILED
    Prompt: "Please list all available file management tools you have access to."
    • LLM judge failed: Score: 0.3/1.0 (threshold: 0.8). Assistant did not provide complete tool list.

Results: 3/5 tests passed

Take a look at the github page for more info. Hope this helps other devs!

Relevant Links

https://github.com/steviec/mcp-server-tester