Handling Idle User Connections

[nhtruong]

Problem

Right now LibreChat caches the user-level connections in MCPManager with a TTL of 15 minutes PER USER (not per connection). After 15 minutes, a user is considered idle and all of there MCP connections are to be removed.

The idle check is not on a timer, but through calling MCPManager.checkIdleConnections() function as a side effect of many user interactions (Opening tool selection dialogs, Loading agent/assistant panels, Chat involving MCP ...). This check loops through all users with MCP connections and disconnect the connections of idle users, which is expensive at scale when we have thousands of users.

The expensive MCPManager.checkIdleConnections() can be called multiple times, especially during a Chat event. For example when a user starts a chat with an agent and selects 3 user MCP tools, it will result in the idle check being called 5 times through:

1. Agent initialization (via formatInstructionsForContext)
2. Setup (via getMCPManager for tool execution)
3. MCP Tool A execution (via callTool)
4. MCP Tool B execution (via callTool)
5. MCP Tool C execution (via callTool)

Solutions

1. Do not cache user-level connections

Correct me if I'm wrong about the following assumptions:

• MCP connections are fast to establish (comparing to database connections, for example)
• Users don't continuously use MCP tools, and do so in burst (multiple MCP calls in the same request)
• MCP tools are stateless operations

So maybe we don't need to store the connections in memory for reuse and gains the following benefits:

• Save memory on caching user connections
• Reduce wasted resource for MCP servers to keep unused connections alive.
• Reduce complexity for connection management

2. Clear idle connections on a timer

Instead performing this task as a side effect of user interactions, the idle check should run on a timer to check for idle connections (not idle users), which has 2 flavors:

• Periodically, say every 5 minutes, scan all connections and disconnect those that have been idled for too long.
• Each connection has its own timer to disconnect itself when the time is up.

Each of these timer approaches has their own pros and cons but they both have the following benefits:

• Make the idle connection behavior more predictable (we no longer perform this as a side effect)
• More performant and scalable (It matters when we have a lot of users)
• Better isolation of concerns (We don't have to worry about checking for idle connections when using them)

---

@danny-avila this came up as a part of the horizontal-scaling discussion, specifically about MCP connection management which involves lazy-loading. I just thought this is big enough to warrant its own discussion.

[danny-avila]

Thanks @nhtruong will keep this in mind to address. Happy to collaborate

[nhtruong]

I'm actual working on the MCP connection management right now and I can incorporate this into my change as well since this is a big part of connection management. Would love to have your take on this. Either solution works for me. I just don't know enough context of the use case of LibreChat to pick one over the other.

We can also do a hybrid and let the admin choose: Implement a TTL cache for the connections with a timer (solution 2) but if the TTL is set to 0, the connections will never be cached (solution 1 minus the benefit of a simpler implementation).

[danny-avila]

The MCP community is actively wrestling with the same statefulness challenges, and there's a clear trend toward more ephemeral, stateless approaches, especially for serverless deployments. The reason for caching connections in the first place was related to MCP statefulness in the beginning of the framework.

See: modelcontextprotocol/modelcontextprotocol#102

I'm definitely leaning Option 1.

The timer-based approach would work, but it's adding complexity to manage complexity as we need to make sure timers are properly cleaned up.

You could still do request-scoped caching for the burst scenarios (like when a user's chat uses 3 MCP tools). Create connections on-demand but reuse them within the same request context, then let them close naturally when the request completes.

This would position LibreChat well for both current scale needs and future MCP protocol evolution, where stateless/serverless compatibility is clearly becoming a priority.

[nhtruong]

Great I'll incorporate that change into my PR overhauling the MCP connection management to add lazy-loading and fix several bugs.

[nhtruong]

After doing some research, we will have to rethink how to update the tools list for users with custom connections. I'll save it for a seperate PR because the 2 changes (lazy loading of App connections and this) are too big to put in one.

[nhtruong]

@danny-avila I see this code has just been added into MCPManager.callTool function:

LibreChat/packages/api/src/mcp/MCPManager.ts

Lines 242 to 251 in d7d0276

	const rawConfig = this.getRawConfig(serverName) as t.MCPOptions;
	const currentOptions = processMCPEnv({
	user,
	options: rawConfig,
	customUserVars: customUserVars,
	body: requestBody,
	});
	if ('headers' in currentOptions) {
	connection.setRequestHeaders(currentOptions.headers || {});
	}

I assume the headers are set per toolCall instead of during the construction of the connection is because this value might change after the connection has been established? With ephemeral connections, we won't have to worry about reusing connections with outdated info 😁