feat(code-template): Addition of Code Template And Setting Up Foundation For AI Tool Calling Support

[thecodacus]

Addition of Code Template And Setting Up Foundation For AI Tool Calling Support

Overview

Adds an AI tool calling system that enables template selection, tool execution, and interactive user confirmations. This system integrates with existing chat functionality to provide a seamless experience for template-based project initialization and AI-driven operations.

Key Changes

Tool System

• New ToolStore for managing AI operations
• Template selection with GitHub repo integration
• User confirmation dialogs
• Tool call streaming support
• Integrated with workbench architecture

Message System

• Enhanced to support tool calls/results
• Added visibility controls via message annotations
• New ToolMessage component for execution status
• Visual indicators for tool states

Template Management

• GitHub-based template fetching
• Recursive directory creation
• Automatic file content handling
• Webcontainer filesystem integration

Tool Calling Switch

• Added a toggle for tool calling
• Tool calling is disabled by default (not all model supports tool calling)

Testing Status

✅ Verified

• Tool execution and state management
• Template selection and import
• User confirmation dialogs
• Error handling and recovery
• File system operations
• Message visibility controls
• Chat continuation fails with tool calls
• Context preservation issues in continued conversations

🔄 Pending

None

❌ Known Issues

Tool Calling is Confusing to the llm with the huge context , need an optimised way to fix this in future
to mitigate that currently tool calling is to only select the template then suppressed via code

Migration

• No breaking changes

Documentation Needed

• Tool system usage
• Template management
• API updates

Preview

bolt.ai.tool.calling.demo.mp4

[coleam00]

Looking great @thecodacus, nice work!!

[thecodacus]

ollama integration is broken for me somehow. not able to test with local llm

[thecodacus]

this feature needs a set of git repo as code templates.
for now I have a dummy list

@chrismahoney , any idea where can we get some good templates for this list to complete ?

[chrismahoney]

I’ve been thinking about it, almost that we should have a single starter repo registry. Once we get this merged, let’s discuss that sort of repository; we can probably assemble it from a bunch of “awesome” repos that curate these starters.

[chrismahoney]

Is this good to review for a merge you think, or still WIP? No rush

[chrismahoney]

ollama integration is broken for me somehow. not able to test with local llm

This has been identified, and @wonderwhy-er has a good provider handling PR that’s also in flight. Should resolve those sorts of issues.

[thecodacus]

I’ve been thinking about it, almost that we should have a single starter repo registry. Once we get this merged, let’s discuss that sort of repository; we can probably assemble it from a bunch of “awesome” repos that curate these starters.

since the template list is hardcoded at the moment shall we just merge it with as it is or remove the dummy templatelists ?

[thecodacus]

I am ready for merge, if we are deciding to add templates later

[thecodacus]

I am ready for merge, if we are deciding to add templates later

@coleam00 any thoughts on this ?

[thecodacus]

Also need a prompt optimisation method to use tool calling for smaller models.
currently all the code content is pushed to the context, for bigger project its too much and also very costly

If we add something like code buffer, where to llm can use tool to load/unload files from buffer, that way only parts of the code that llm wants to know will be in the context, and we can have a limits on how many files you can load into buffer at a time.

this will also setup the base for agentic approch

[wonderwhy-er]

Ouh boy, this is too big...

I wrote elsewhere that I am in favour of doing this in smaller steps.

We not only need templates.
We need ability to star with zip/folder/github-link

That would need to come before templates.

And then templates can come on top of that as its just a way to select a template to start a project from when user does not provide a link.

Also, not all models support tool calling. What is magical about Bolt is that it did not rely on that.
Instead it asks to output specific syntax that it then looks for and parses in answers.
I think that is better way to do it than tool calling.

I am bit afraid that larger non customisable changes would make it hard to work on more foundational issues like file upload in middle of the session.

Also, about not dumping things into context.
I am not user of Aider due to its "manual" context management where you select what to send.

I think such modes should be optional. Most users will not manage context well and will drop using this tool if it forces them to. So its opinionated thing that should be configurable.

[wonderwhy-er]

Did you test it with models that do not support tool use?

Here asked ChatGPT what it thinks:

Tool Usage in AI Models

The tool usage feature is model-dependent, as not all AI models support tools. Here’s a quick rundown of which models are more likely to support tool usage and which do not:

Models Likely to Support Tool Usage:

• OpenAI Models (GPT-4-turbo): These versions often support tools, such as function calling, plugins, and more specific tool invocations.
• Anthropic’s Claude Models (Claude 2 and newer): Claude models are designed with strong capabilities for following tool invocations if configured in the hosting environment.
• Google’s PaLM 2 and Gemini: PaLM and Gemini models are built to handle function calling, making them compatible with tool-based workflows.
• xAI’s Grok (if configured with tools): This model supports tools, but specific setups might vary.

Models Less Likely to Support Tool Usage:

• Older OpenAI Models (GPT-3 and GPT-3.5 without turbo): These do not inherently support direct tool use, plugins, or function calls.
• Meta’s LLaMA models: LLaMA 2 and similar models do not natively support tools unless there’s a specific wrapper or plugin layer added in the hosting infrastructure.
• Mistral Models: These models, such as Mixtral and Mistral-7B, focus on general text generation and lack built-in support for external tool calling.
• Smaller or Open-source Models (e.g., LLaMA derivatives, StableLM): These models are often used without built-in tool integration and typically require extensive modification to enable tool use.

In general, tool usage tends to be a feature of larger, more advanced models specifically designed or configured for interactive workflows. If you’re implementing tool calls, it’s best to check the model documentation or configuration options in the hosting environment to confirm compatibility.

[thecodacus]

@wonderwhy-er , yes I know thats the reason I have added the tool toggle switch, and ts turned off by default..
but even with tool switch on.. I have seen it ignoring tool completely when model does not support tool use except openrouter model as open router tris to find the model with tool support and throws error if none found

[wonderwhy-er]

I wonder if we can detect if current model supports tool use or not.
And then use tool_use if it does.
And if not add to prompt requests for
"to call a tool output syntax like argument1,argument2...

And we would parse response and detect if there was tool use in there.
That is why makes Bolt in general to be model agnostic, weather it has tool support or not.

In reality tool use under the bonnet does the same, outputs structured data in json or xml formats.
And it can be mixed with plain text.

[thecodacus]

initially I though of just prompting a fresh model to select a templete from provided list. but then routed my idea to this approach as the codebase is very much tightly integrated with vercel's ai sdk for its core functions and it fits the tool option perfectly
but for this to work I believe we need multi agent workflow, and use one router model to route the request from code writer and other agents like import template etc

that way we can have plain text prompt with small system instructions with very strict output structures.

so yes I also think whats how the future path should be

[thecodacus]

Also, about not dumping things into context. I am not user of Aider due to its "manual" context management where you select what to send.

I think such modes should be optional. Most users will not manage context well and will drop using this tool if it forces them to. So its opinionated thing that should be configurable.

I am actually not thinking about user managing context but AI managing its context. its like AI does not need all the file content all the time
but a set of files, so I was thinking of having a buffer context that will be dynamic and AI can load and unload files from that withdow.
it will still have the full file structure in the context to always aware of whats available and what that might be

[thecodacus]

Ouh boy, this is too big...

I wrote elsewhere that I am in favour of doing this in smaller steps.

We not only need templates. We need ability to star with zip/folder/github-link

code template is I believe is a very essential, as most of the time people would use bolt to build something from ground up.
not having some template codebase actually makes it difficult and substandard results.

but I totally agree with you on tool calling. plain text parsing is the most desirable approach

[wonderwhy-er]

I agree with most of what you said.
With some clarifications:

we need multi agent workflow

In my experience having multiple smaller, focused prompts works better than one big one
I would not really call that multi agent, rather prompt routing, there is prompt that select what prompt should be loaded
Its kinda like tool use but on prompt level

AI managing its context

Yeah, I am thinking about it too, though I was thinking to chunk, embed and and somehow filter whole chat.
Aka send only things that seem relevant.
But its challenging thing, I would add that as experimental thing.
It will makes things cheaper and better for smaller models but it will degrade performance for models that can handle large context as is
But it is area where I would play around with things... Its very tricky, one can spend a lot of time there...

code template is I believe is a very essential

I agree, I just would start with "import github repo" or "import folder" as first step
And add things on top

I actually want to make two way folder sync work as one of next things I want to do.
It will work in chat kinda like AI assistant responses but coming from local changes

[thecodacus]

I am closing this as I am working on a better solution

[coleam00]

@thecodacus Sorry I missed your ping on this earlier! Sounds good - I really appreciate you diving into an even better solution!

[thecodacus]

Yeah its there #302 , putting it here for who ever stumbles onto this pr

[coleam00]

Perfect, thanks @thecodacus!