feat: Add compact prompt mode for local LLMs #7551
Conversation
- Add compactPromptMode boolean setting to ProviderSettings type - Implement compact prompt generation in SYSTEM_PROMPT function - Add UI toggle in LM Studio and Ollama provider settings - Create reusable CompactPromptControl component - Add translation strings for the new feature - Include comprehensive tests for compact prompt functionality This feature addresses issue #7550 by providing a minimal prompt option that reduces context size and improves response times for local LLMs that have slower token generation speeds.
dosubot
bot
added
size:L
![roomote[bot]](https://avatars.githubusercontent.com/in/1546624?s=60&v=4){kind=small}
| "compactPrompt": { | ||
| "title": "Compact Prompt Mode", | ||
| "description": "Reduces the system prompt size for faster response times with local LLMs. This removes non-essential sections while keeping core functionality.", | ||
| "providerNote": "Recommended for {{provider}} to prevent timeouts with slower local models" |
There was a problem hiding this comment.
I notice there are duplicate translation keys for compact prompt mode - lines 371-376 for LM Studio and lines 384-387 as general keys. Since the CompactPromptControl component already handles provider-specific messages dynamically, could we consolidate these to avoid duplication?
For example, we could keep just the general keys (384-387) and remove the LM Studio specific ones (371-376), as the component already prefixes with the provider name.
| settings?: SystemPromptSettings, | ||
| todoList?: TodoItem[], | ||
| modelId?: string, | ||
| compactPromptMode?: boolean, |
There was a problem hiding this comment.
Consider adding JSDoc comments here to explain when and why to use compact prompt mode:
| compactPromptMode?: boolean, | |
| /** | |
| * @param compactPromptMode - When true, generates a minimal prompt with only essential sections | |
| * to reduce token count for local LLMs with slower processing speeds. | |
| * Excludes MCP, browser tools, capabilities, modes, and verbose instructions. | |
| */ | |
| export const SYSTEM_PROMPT = async ( |
| interface CompactPromptControlProps { | ||
| compactPromptMode?: boolean | ||
| onChange: (value: boolean) => void | ||
| providerName?: string |
There was a problem hiding this comment.
For better type safety, consider using a union type for providerName:
| providerName?: string | |
| interface CompactPromptControlProps { | |
| compactPromptMode?: boolean | |
| onChange: (value: boolean) => void | |
| providerName?: 'LM Studio' | 'Ollama' | |
| } |
|
|
||
| // Check if we should use compact prompt mode for local LLM providers | ||
| const isLocalLLMProvider = | ||
| this.apiConfiguration.apiProvider === "lmstudio" || this.apiConfiguration.apiProvider === "ollama" |
There was a problem hiding this comment.
The compact prompt check happens on every getSystemPrompt() call. Since the provider type doesn't change during a task, could this be cached or memoized for better performance?
You could store the result of this check as a class property when the task is initialized.
| return ( | ||
| <div className="flex flex-col gap-2"> | ||
| <div className="flex items-center justify-between"> | ||
| <label htmlFor="compact-prompt-mode" className="font-medium"> |
There was a problem hiding this comment.
Consider adding a tooltip or help icon to explain the trade-offs of enabling compact mode. Users should understand what features are excluded (MCP, browser tools, modes, etc.) when they enable this option.
You could add a small info icon next to the label that shows this information on hover.
hannesrudolph
added
the
Issue/PR - Triage
Summary
This PR adds a "compact prompt mode" option for local LLM providers (LM Studio and Ollama) to address timeout issues caused by large prompts that take too long to process at slower token generation speeds (7-10 tok/sec).
Changes
compactPromptModeboolean setting to the ProviderSettings typeSYSTEM_PROMPTfunction that creates a minimal prompt with only essential sectionsCompactPromptControlcomponent for the settings UIHow it works
When enabled, the compact prompt mode:
Testing
Screenshots
The new toggle appears in the LM Studio and Ollama settings:
Important
Adds compact prompt mode for local LLMs to reduce prompt size and prevent timeouts, with UI toggle and tests.
compactPromptModeboolean toProviderSettingsinprovider-settings.ts.SYSTEM_PROMPTinsystem.ts, reducing prompts to essential sections.LMStudio.tsxandOllama.tsx.CompactPromptControlcomponent for settings UI.system-prompt.spec.ts.settings.json.This description was created by
for 576864b. You can customize this summary. It will automatically update as commits are pushed.