Tool reduction

[MackinnonBuck]

Summary

This draft introduces the tool reduction feature set:

• New abstractions (IToolReductionStrategy) for selecting a subset of tools per request.
• Middleware (ToolReducingChatClient) that applies a configured strategy automatically.
• An initial embedding-based implementation (EmbeddingToolReductionStrategy) as a first strategy.
• Supporting builder extensions and tests.

All components are marked experimental and may evolve. Additional strategies can be added in follow-up work.

Next Steps

• Additional tool reduction strategy implementations.
• Refinements to abstractions as necessary.

Feedback welcome before expanding scope.

Fixes #6670

Microsoft Reviewers: Open in CodeFlow

[stephentoub]

We have a helper that ChatMessage and ChatResponse use to do this a bit more efficiently. Maybe we should look at exposing that in a way this could use as well.

That said, I wonder if we'd want the default implementation here also including reasoning content, which this implementation doesn't currently, since ChatMessage.Text only includes TextContent and not TextReasoningContent.

[stephentoub]

I wonder how we'd implement a "virtual tools" strategy ala what VS Code uses. Tools come in, they're grouped by an LLM into buckets, and then fake tools are created for each bucket; those tools are passed along to the LLM, which can then invoke one of those tools in order to expand it out into the full set from that bucket. Probably requires some prototyping. It might be that we leverage AIFunctionDeclaration so that the FICC lets a virtual tool pass through back to the TRCC. Or maybe it's another feature in FICC that enables a tool to modify the options that'll be sent back on a subsequent request. Not sure.

[PederHP]

When I've used the "virtual tools" strategy I've had success with having a single tool that allows expansion. There's no need to have a tool per bucket. It can just be an argument to the expand and collapse tools, with the bucket descriptions passed in the meta-tool description or injected into system. This is greatly more token efficient and scales better in terms of tool count. This also makes it a well-known tool with some dynamic context rather than an entirely dynamic tool.

[MackinnonBuck]

@PederHP, that sounds like a promising approach. Just to clarify: is the LLM responsible for both expanding and collapsing tool groups, or does it only expand them, with collapsing happening automatically later (e.g. after a response completes)? In other words, how long does an expanded group remain open?

I also wonder if tool grouping might be more naturally handled through agents. Each agent has a limited toolset tailored to its purpose, so as work is handed off between agents, the available tools change without needing stateful expand/collapse logic in a single chat client. A lightweight "tool reduction" middleware in the chat client (such as what's prototyped in this PR) could still help narrow focus further, but in a more stateless way.

If we do want multiple tool groups in a single client, one option could be allowing only one group to be expanded at a time. This would mimic the agent handoff scenario I described above. All groups are collapsed at the start of a response, and the LLM can call a well-known, non-invocable "expand" function with a tool group name. A middleware earlier in the pipeline can then generate another response with the expanded set of tools. Further expansions close the already-expanded tool group. This approach reduces the responsibility of the LLM to decide when to collapse a tool group, and it allows tool reduction middleware to run after each expansion, if needed. That type of composition might be harder to achieve if functions can directly mutate the tools list inside FunctionInvokingChatClient iterations.

[MackinnonBuck]

Implemented a prototype in #6865

[ericstj]

I wonder if there is a hybrid that can reduce tools, but then also expose a "get more tools" function where the model could describe the sort of tool it wanted and we could use that to give it back more tools that have embedding similarity to its request? Don't want to feature creep this, just trying to think of alternatives.

I like that this one is hands free but but I can imagine that it can hit situations where it never gives the model the chance to call what might be the best tool.

I like that the grouping ensures that all tools are made available, but it requires explicit grouping by the caller.

[MackinnonBuck]

Maybe what we could do is change MessagesEmbeddingTextSelector to be a Func<IEnumerable<ChatMessage>, Task<string>>. Then if you wanted to have the model separately describe the tool it needs, you could asynchronously have a chat client analyze the conversation and generate a short string describing the type of tool that would be helpful. Then that string is what gets used when searching for "similar" tools.

[MackinnonBuck]

Just implemented the above ^

[Copilot]

Pull Request Overview

This PR introduces a tool reduction feature set for managing large tool catalogs in AI chat applications. The feature allows for automatically selecting a subset of tools per request using configurable strategies.

Key changes:

• New abstractions for tool reduction strategies with an embedding-based implementation
• Middleware that automatically applies tool reduction to chat requests
• Comprehensive test coverage including integration tests across different scenarios

Reviewed Changes

Copilot reviewed 10 out of 10 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
src/Libraries/Microsoft.Extensions.AI.Abstractions/ToolReduction/IToolReductionStrategy.cs	Defines the core abstraction interface for tool reduction strategies
src/Libraries/Microsoft.Extensions.AI/ToolReduction/EmbeddingToolReductionStrategy.cs	Implements embedding-based tool selection using similarity scoring
src/Libraries/Microsoft.Extensions.AI/ToolReduction/ToolReducingChatClient.cs	Middleware client that applies tool reduction before delegating to inner client
src/Libraries/Microsoft.Extensions.AI/ToolReduction/ChatClientBuilderToolReductionExtensions.cs	Extension methods for integrating tool reduction into chat client pipelines
test/Libraries/Microsoft.Extensions.AI.Integration.Tests/ToolReductionTests.cs	Comprehensive unit and integration tests for tool reduction functionality
test/Libraries/Microsoft.Extensions.AI.Integration.Tests/ChatClientIntegrationTests.cs	Integration tests demonstrating real-world tool reduction scenarios
test/Libraries/Microsoft.Extensions.AI.OpenAI.Tests/OpenAIChatClientIntegrationTests.cs	Adds embedding generator support for OpenAI integration tests
src/Libraries/Microsoft.Extensions.AI/Microsoft.Extensions.AI.csproj	Adds System.Numerics.Tensors dependency for embedding similarity calculations
eng/packages/General.props	Moves System.Numerics.Tensors to general packages from test-only
eng/packages/TestOnly.props	Removes System.Numerics.Tensors from test-only packages

Comments suppressed due to low confidence (2)

src/Libraries/Microsoft.Extensions.AI/ToolReduction/EmbeddingToolReductionStrategy.cs:1

• This method exceeds 120 characters per line. Consider breaking the parameter list across multiple lines for better readability.

// Licensed to the .NET Foundation under one or more agreements.

src/Libraries/Microsoft.Extensions.AI/ToolReduction/EmbeddingToolReductionStrategy.cs:1

• This method exceeds 120 characters per line. Consider breaking the parameter list across multiple lines for better readability.

// Licensed to the .NET Foundation under one or more agreements.