evals - RuntimeError: Attempted to exit cancel scope in a different task than it was entered in

[g-eoj]

Initial Checks

• I confirm that I'm using the latest version of Pydantic AI
• I confirm that I searched for my issue in https://github.com/pydantic/pydantic-ai/issues before opening this issue

Description

When setting max_concurrency to anything other than one, eventually the RuntimeError: Attempted to exit cancel scope in a different task than it was entered in error will pop up for evals with ~10 or more cases. It appears to be connected to using toolsets created from MCP servers.

I followed https://ai.pydantic.dev/api/mcp/#pydantic_ai.mcp.MCPServerStreamableHTTP - tried using this pattern everywhere an agent is used:

import some_agent

...

async with some_agent:
    run = await some_agent.run(...)

...

Example Code

Here is a simple reproduction - this should work right?

import asyncio

from pydantic_ai.mcp import MCPServerStdio
from pydantic_evals import Case, Dataset
from pydantic_evals.evaluators import Evaluator, EvaluatorContext


class MyEvaluator(Evaluator[str, bool]):
    def evaluate(self, ctx: EvaluatorContext[str, bool]) -> bool:
        return True


mcp_server = MCPServerStdio('python', ['-m', 'tests.mcp_server'])


async def run(case: str) -> bool:
    async with mcp_server:
        await mcp_server.list_tools()
    return True


async def main():
    cases: list[Case[str, bool, dict[str, str]]] = []
    for _ in range(10):
        case = Case(
            name=str(_),
            inputs='',
            expected_output=True,
            metadata={'difficulty': 'easy'},
        )
        cases.append(case)

    dataset = Dataset(
        cases=cases,
        evaluators=[MyEvaluator()],
    )

    report = await dataset.evaluate(run, max_concurrency=4)
    report.print()


if __name__ == '__main__':
    asyncio.run(main())

Python, Pydantic AI & LLM client version

Pydantic AI 1.0.1
Python 3.12.7

[DouweM]

@g-eoj For context, the way async with server works is that each MCPServer object maintains a single connection to the server, which is opened when it is async with-entered for the first time, and closed when it's exited for the last time. That's implemented by having __aenter__ and __aexit__ keep track of a running_count which is incremented on each enter and decremented on each exit, with the actual server only being started when entered while running_count==0, and closed when exited while running_count==0 (after decrement).

When running async with in parallel in evals cases, this makes it very likely that the connection will be opened and closed in different tasks, as the first case to start (first async with to be entered) is almost never going to be the same case to finish last (async with to be exited last).

I'm not quite sure yet what the right solution is here, but as a workaround, you can make sure the server (or agent) is entered explicitly around the entire context where it's used in parallel, i.e.:

async with mcp_server:
    report = await dataset.evaluate(run, max_concurrency=4)

That way it's guaranteed to be entered first and exited last in the same place.

[MasterOdin]

We're running into a similar issue using agent.run in general. I was able to make a small reproducible case with this:

# mcp_server.py
from mcp.server.fastmcp import FastMCP

mcp = FastMCP('Pydantic AI MCP Server')
log_level = 'unset'


@mcp.tool()
async def echo_num(num: int) -> str:
    """Echo a number."""
    return f"Echoing number: {num}"

if __name__ == '__main__':
    mcp.run()

# agent.py
import asyncio

from pydantic_ai import Agent, RunContext
from pydantic_ai.mcp import CallToolFunc, MCPServerStdio, ToolResult
from pydantic_ai.models.test import TestModel

server = MCPServerStdio('python', args=['mcp_server.py'])
agent = Agent(
    model=TestModel(call_tools=['echo_num']),
    toolsets=[server]
)


async def run_agent():
    async with agent:
        result = await agent.run('Echo the number 42')
    return result.output


async def main():
    tasks = [
        asyncio.create_task(run_agent())
        for _ in range(5)
    ]
    results = await asyncio.gather(*tasks)

    for result in results:
        print(result)

asyncio.run(main())

where on running agent.py I get the same RuntimeError: Attempted to exit cancel scope in a different task than it was entered in error as above.

Similar to above, wrapping main with async with server: fixes the issue, but definitely doesn't feel great, especially for our use-case, where we have a slack bot that it using pydantic-ai to respond to user's, and where the main agent has some MCP servers and then it also has sub-agents (as tools) which also have MCP servers, and it's not great feeling trying to register them all up front. We're looking to move away from global singleton servers into ones created as part of agent.run instead.

[DouweM]

@Kludex Any ideas here?

[MasterOdin]

Our working solution was something like this:

agent = Agent(
    model=TestModel(call_tools=['echo_num']),
)

def get_server():
    return MCPServerStdio('python', args=['mcp_server.py'])


async def run_agent():
    async with agent:
        result = await agent.run('Call echo_num with 42', toolsets=[get_server()])
    return result.output

where we create a fresh MCPServer instance per agent run which gives async context isolation and avoids the issue, but does add the overhead of having to instantiate a fresh MCP server instance per agent call.

[Sjoeborg]

This also frequently occurs for me when using MCPServerStreamableHTTP. Let me know if I can provide more context/help in any way

[pinkynrg]

Same here: using MCPServerStreamableHTTP and having the same issue

[leandregagnonlewis]

I also have the issue when making multiple simultaneous agent run calls with MCPServerStreamableHTTP

[leandregagnonlewis]

This open issue modelcontextprotocol/python-sdk#577 (comment) documents that MCP connections must be cleaned up in LIFO order. Is this implemented in CombinedToolset ?

[DouweM]

@leandregagnonlewis AsyncExitStack supposedly already cleans up in LIFO order. I've commented there: modelcontextprotocol/python-sdk#577 (comment).

Do you have a code sample that triggers the issue?