Skip to content

Add prompt caching to OpenAI-compatible custom model info #1562

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
mrubens wants to merge 1 commit into from
Closed

Add prompt caching to OpenAI-compatible custom model info #1562

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions .changeset/thin-fans-deliver.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
---
"roo-cline": patch
---

Add prompt caching to OpenAI-compatible custom model info
114 changes: 112 additions & 2 deletions webview-ui/src/components/settings/ApiOptions.tsx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -819,7 +819,7 @@ const ApiOptions = ({
style={{ fontSize: "12px" }}
/>
</div>
<div className="text-sm text-vscode-descriptionForeground">
<div className="text-sm text-vscode-descriptionForeground pt-1">
Is this model capable of processing and understanding images?
</div>
</div>
Expand All @@ -842,11 +842,34 @@ const ApiOptions = ({
style={{ fontSize: "12px" }}
/>
</div>
<div className="text-sm text-vscode-descriptionForeground [pt">
<div className="text-sm text-vscode-descriptionForeground pt-1">
Is this model capable of interacting with a browser? (e.g. Claude 3.7 Sonnet).
</div>
</div>

<div>
<div className="flex items-center gap-1">
<Checkbox
checked={apiConfiguration?.openAiCustomModelInfo?.supportsPromptCache ?? false}
onChange={handleInputChange("openAiCustomModelInfo", (checked) => {
return {
...(apiConfiguration?.openAiCustomModelInfo || openAiModelInfoSaneDefaults),
supportsPromptCache: checked,
}
})}>
<span className="font-medium">Prompt Caching</span>
</Checkbox>
<i
className="codicon codicon-info text-vscode-descriptionForeground"
title="Enable if the model supports prompt caching. This can improve performance and reduce costs."
style={{ fontSize: "12px" }}
/>
</div>
<div className="text-sm text-vscode-descriptionForeground pt-1">
Is this model capable of caching prompts?
</div>
</div>

<div>
<VSCodeTextField
value={
Expand Down Expand Up @@ -933,6 +956,93 @@ const ApiOptions = ({
</VSCodeTextField>
</div>

{apiConfiguration?.openAiCustomModelInfo?.supportsPromptCache && (
<>
<div>
<VSCodeTextField
value={
apiConfiguration?.openAiCustomModelInfo?.cacheReadsPrice?.toString() ?? "0"
}
type="text"
style={{
borderColor: (() => {
Copy link
Contributor

@ellipsis-dev ellipsis-dev bot Mar 11, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Consider extracting the repeated inline borderColor styling logic into a helper function to improve readability and maintainability.

Copy link
Collaborator

@cte cte Mar 11, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@ellipsis-dev Show us the equivalent of this logic using cn instead of an inline style object.

Copy link
Contributor

@ellipsis-dev ellipsis-dev bot Mar 11, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@cte, I have addressed your comments in pull request #1563

You can configure Ellipsis to address comments with a direct commit or a side PR, see docs.

Copy link
Collaborator

@cte cte Mar 11, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@mrubens - For reference, it (sort of) produced:

className={cn("w-full", {
  "border-vscode-input-border": !apiConfiguration?.openAiCustomModelInfo?.cacheWritesPrice && apiConfiguration?.openAiCustomModelInfo?.cacheWritesPrice !== 0,
  "border-vscode-charts-green": apiConfiguration?.openAiCustomModelInfo?.cacheWritesPrice >= 0,
  "border-vscode-errorForeground": apiConfiguration?.openAiCustomModelInfo?.cacheWritesPrice < 0,
})}

This is spot on, but pretty verbose. Might want to extract apiConfiguration?.openAiCustomModelInfo?.cacheWritesPrice into a local var.

Copy link
Contributor

@ellipsis-dev ellipsis-dev bot Mar 11, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There is repeated logic for computing the borderColor and for parsing numeric values in the cache price fields. Please consider extracting these into a helper function to reduce duplication and improve maintainability.

const value = apiConfiguration?.openAiCustomModelInfo?.cacheReadsPrice

if (!value && value !== 0) {
return "var(--vscode-input-border)"
}

return value >= 0
? "var(--vscode-charts-green)"
: "var(--vscode-errorForeground)"
})(),
}}
onChange={handleInputChange("openAiCustomModelInfo", (e) => {
Copy link
Contributor

@ellipsis-dev ellipsis-dev bot Mar 11, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Notice that the cache price text fields are using the onChange event, while other similar text inputs in the form use onInput. For consistency and predictable behavior across the form, please review whether onInput should be used instead.

Suggested change
onChange={handleInputChange("openAiCustomModelInfo", (e) => {
onInput={handleInputChange("openAiCustomModelInfo", (e) => {

Copy link
Collaborator Author

@mrubens mrubens Mar 11, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pretty sure there was some reason we don't - based on them truncating decimal points too early or something.

const value = (e.target as HTMLInputElement).value
const parsed = parseFloat(value)

return {
...(apiConfiguration?.openAiCustomModelInfo ??
openAiModelInfoSaneDefaults),
cacheReadsPrice: isNaN(parsed) ? 0 : parsed,
}
})}
placeholder="e.g. 0.0001"
className="w-full">
<div className="flex items-center gap-1">
<span className="font-medium">Cache Reads Price</span>
<i
className="codicon codicon-info text-vscode-descriptionForeground"
title="Cost per million tokens for reading from the cache. This is the price charged when a cached response is retrieved."
style={{ fontSize: "12px" }}
/>
</div>
</VSCodeTextField>
</div>
<div>
<VSCodeTextField
value={
apiConfiguration?.openAiCustomModelInfo?.cacheWritesPrice?.toString() ?? "0"
}
type="text"
style={{
borderColor: (() => {
const value = apiConfiguration?.openAiCustomModelInfo?.cacheWritesPrice

if (!value && value !== 0) {
return "var(--vscode-input-border)"
}

return value >= 0
? "var(--vscode-charts-green)"
: "var(--vscode-errorForeground)"
Copy link
Collaborator

@cte cte Mar 11, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this handling the case when cacheWritesPrice is negative? Do we need to handle this case?

Copy link
Collaborator Author

@mrubens mrubens Mar 11, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah... I basically just copied the inputs for the uncached inputs and outputs. Not sure it's necessary in either.

})(),
}}
onChange={handleInputChange("openAiCustomModelInfo", (e) => {
const value = (e.target as HTMLInputElement).value
const parsed = parseFloat(value)

return {
...(apiConfiguration?.openAiCustomModelInfo ??
openAiModelInfoSaneDefaults),
cacheWritesPrice: isNaN(parsed) ? 0 : parsed,
}
})}
placeholder="e.g. 0.00005"
className="w-full">
<div className="flex items-center gap-1">
<span className="font-medium">Cache Writes Price</span>
<i
className="codicon codicon-info text-vscode-descriptionForeground"
title="Cost per million tokens for writing to the cache. This is the price charged when a prompt is cached for the first time."
style={{ fontSize: "12px" }}
/>
</div>
</VSCodeTextField>
</div>
</>
)}

<Button
variant="secondary"
onClick={() =>
Expand Down