Tools management - how VS Code GitHub Copilot Agent will react to 700+ tools

[Adam-it]

How will we manage MCP Tools?
Will we expose only a few general tools like "run CLI for Microsoft 365 command" which will allow to run any CLI for Microsoft 365 command by the GitHub Agent and we would need to first provide context to the agent allowing it to decide which command should be used
OR
we expose each command as a separate tool and in this case we will end up with 700+ tools. One per command. How will GitHub agent behave in this case, and won't we just reach some limit 🤔

[Adam-it]

I found this
https://github.com/orgs/modelcontextprotocol/discussions/184
were it is stated that handling more than 70 tools becomes problematic and it seems manageable to handle 40-50 tools max and in case someone has more for each prompt we should disable the unneeded MCP's

I also found this https://forum.cursor.com/t/increase-the-mcp-tool/69194
that cursor has an internal limit of max 40 tools.

So I guess this means we should not go with the approach that each CLI for Microsoft 365 is a separate tool but rather look for an alternative approach. Maybe tools like manage-sharepoint that will allow to run any spo command and it will be on LLM side to figure out which command should be used from spo area, But still this is a loooot of context to pass for LLM to pass which command should be used even for only spo

[Adam-it]

I also found this issue which is kinda connected to our current problem modelcontextprotocol/servers#1125 with a follow up here github/github-mcp-server#120

I noticed GitHub MCP server already has 30 tools and they have plans to add more tools like support for projects etc.. How GH is solving this is by providing a feature in their MCP server will allow to pass as env or arguments which tools should be available and present on the list
github/github-mcp-server#189 with PR to this github/github-mcp-server#188

[Adam-it]

maybe we could experiment by adding 100 tools and how it will work 🤔 I think a simple POC with different tool names that use the same logic wont be a lot of effort

[rajbos]

The hard part is two fold in my opinion:

1: the client needs to figure out which tool might be helpful to call on any given prompt. With 100's of tools, this will start to slow down things. Let alone the compute impact: why should an evaluation run for ALL tools on every single prompt? If the client already knows I don't have feature X available, why shouldn't even be enabled. Currently we put that decision onto the the user: they have to turn the tools off manually. Even worse, this choice can be different for every repo that I use for example.

2: we depend on what the end user understands of MCP to decide on what should be installed/configured or not. With all the security issues around MCP at the moment, it is hard to even make these choices at all. I appreciate the way some VS Code extensions handle this currently, like the @Azure agent from Microsoft: by installing the extension from the vendor (implicates trust), they also enable MCP tools that can be used by e.g. GitHub Copilot. Great way to expose the tools, still lacks understanding that these are MCP tools (things just light up), or what they do for you/in your name.

So this is mostly an issue that needs to be managed on the tools discovery side at the client, together with improvements in the definition of the tools.

In my mind we need a way to group tools in MCP server side categories, where having a feature enabled in the correct workspace would en/disable that categorie of tools for example. This kind of intelligence is something that should be discoverable from the tools, and if the description can indicate how this works, I could even foresee a specific tool that can be called to turn these options on/off on either the tool or category level.

</end of braindump 😅>