Built-in toolsets for default tools #1139
Draft
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
… if default) and set exp to true
|
Questions/Action items remaining:
|
24anisha
commented
Oct 7, 2025
| // Enable if we could potentially trigger built-in grouping (when GPT model is used) | ||
| const defaultToolGroupingEnabled = this._configurationService.getExperimentBasedConfig(ConfigKey.Internal.DefaultToolsGrouped, this._experimentationService); | ||
| const couldTriggerBuiltInGrouping = this._tools.length > Constant.START_BUILTIN_GROUPING_AFTER_TOOL_COUNT && defaultToolGroupingEnabled; | ||
|
|
There was a problem hiding this comment.
This doesn't check everything we need. We also need to confirm that the model is gpt-4.1 or gpt-5, but this is supposed to be a lightweight, quick function to call.
Options:
- pass the endpoint into this function so we can check to make sure it's the correct model before going to virtual tool grouping
- let it go to virtual tool grouping every time (since START_BUILTIN_GROUPING_AFTER_TOOL_COUNT is 20) and get stopped from grouping there by the endpoint check
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Currently, we have a default toolset of 46 tools. Many of these tools are hyper-specified and often unnecessary for the task at hand. Their existence in the toolset ends up yielding worse behavior for many models, such as Sonnet 3.7 and GPT 4.1. As such, we're running an experiment to leverage the existing logic for MCP server toolsets to create built in toolsets from the default tools and only exposing a handful of tools to the user directly.
GPT 4.1 Pass Rate
GPT 5 Pass Rate
Sonnet 3.7 Success = True for Both