Toolgroups API

[ehhuang]

Before OpenAI Responses, we had introduced the concept of Toolgroups (https://llamastack.github.io/docs/building_applications/tools) which Stack admins can register to be used by developers. For example, MCP tools can be registered as a tool group, which can then be referenced by the (now alpha) Agents API (https://llamastack.github.io/docs/building_applications/tools#using-remote-mcp-servers). This has the potential of allowing the admin to register certain auth-required MCPs once for all developers to use easily.

However we're not sure if this is applicable in real scenarios. Toolgroups are also not integrated with OpenAI Responses API. We're inclined to remove this concept entirely, but would like to first hear the community's thoughts on whether this is useful.

[ashwinb]

cc @derekhiggins @grs @bbrowning

[grs]

The Responses API now has a concept of connectors, allowing admins to register tool servers that can then be used by name without requiring the url. To my mind this serves a similar purpose, so I think the toolgroups api could be removed in favour of support for configuring connectors in some way.

[ashwinb]

This is the OpenAI Connector blurb:

---

[zanetworker]

Could the tool-group APIs help in the implementation of connectors (we already have an abstraction), i..e, connectors -> tool-groups (non user-facing API as an initial utility for implementing connectors), or would handling that from scratch be easier?

[grs]

In my opinion, handling it from scratch will be easier. The connector configuration is really just a way to associate a name with a url.

[zanetworker]

sounds good, it feels like there might be a need to make the "backend" implementation for connectors pluggable or at least follow a common standard.

[ehhuang]

From https://platform.openai.com/docs/guides/tools-connectors-mcp#connectors

We prioritized services that don't have official remote MCP servers. GitHub, for instance, has an official MCP server you can connect to by passing https://api.githubcopilot.com/mcp/ to the server_url field in the MCP tool.

It seems that connectors are only for services that don't have remote MCP servers, which, when released, I suspect OpenAI would just deprecate the corresponding connector?

The connector configuration is really just a way to associate a name with a url.

Do we really need this for Stack then?

[grs]

The value is that it frees applications from needing to care about the url. They just care about the logical name of the server they want to use.

[bbrowning]

I think this also starts to tie into our concept of built-in tools a bit, where we may need to help users understand what the difference between built-in tools, connectors, and MCP servers are and when stack admins want to expose different things as each how they do that and what the tradeoffs are.

[ashwinb]

I think there is also value in not just the URL shortening but also from an access control perspective. What MCPs should be allowed for server side execution is a critical issue.

[bbrowning]

A user can bring their own MCP server in a Responses API request. But, it sounds like if we want to support a scenario where a stack admin has configured a set of MCP servers for the stack and wants to ensure only those are used in server-side tool calling, we'll also need some mechanism to allow them to allow or deny custom MCP server urls in inference requests.

[bbrowning]

I do think that logically, connectors is the closest to what we had as toolgroups. The difference is we don't have any explicit API to register and expose connectors. But, the purpose connectors and toolgroups serve is a way for a stack admin to preconfigure a set of tools available in the stack.

If it's helpful, see some similar discussion at vllm-project/vllm#26703 about how some of this is being thought about in the vLLM Responses API implementation and some of the tradeoffs for different ways of exposing tools.

[ashwinb]

Yes, I think we should rebrand toolgroups CRUD (register / unregister) API as Connectors.

[jaideepr97]

@ashwinb would that approach allow support for MCP registries? that is something closely tied to the idea of connectors and probably something a lot of enterprises will want to leverage