Skip to content

feat: prepare VS Code LM API for image support (#8123) #8132

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 1 commit into from
Closed

feat: prepare VS Code LM API for image support (#8123) #8132

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 8 additions & 3 deletions pnpm-lock.yaml
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

5 changes: 4 additions & 1 deletion src/api/providers/vscode-lm.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -510,7 +510,10 @@ export class VsCodeLmHandler extends BaseProvider implements SingleCompletionHan
typeof this.client.maxInputTokens === "number"
? Math.max(0, this.client.maxInputTokens)
: openAiModelInfoSaneDefaults.contextWindow,
supportsImages: false, // VSCode Language Model API currently doesn't support image inputs
// TODO: Enable image support when VS Code LM API provides capability detection
// The newer VS Code API (1.95+) includes LanguageModelChatInformation.capabilities.imageInput
// but it's not directly accessible from the LanguageModelChat interface yet
supportsImages: false,
Copy link
Contributor Author

@roomote roomote bot Sep 18, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The TODO comment is helpful, but could be more specific. Consider expanding it to:

This would help future maintainers understand exactly what we're waiting for.

supportsPromptCache: true,
inputPrice: 0,
outputPrice: 0,
Expand Down
4 changes: 3 additions & 1 deletion src/api/transform/__tests__/vscode-lm-format.spec.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -170,7 +170,9 @@ describe("convertToVsCodeLmMessages", () => {

expect(result).toHaveLength(1)
const imagePlaceholder = result[0].content[1] as MockLanguageModelTextPart
expect(imagePlaceholder.value).toContain("[Image (base64): image/png not supported by VSCode LM API]")
expect(imagePlaceholder.value).toContain(
"[Image placeholder: base64 (image/png) - VS Code LM API image support pending]",
)
Copy link
Contributor Author

@roomote roomote bot Sep 18, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The test update correctly matches the new placeholder format. Consider adding edge case tests for:

  • Images with missing property
  • Images with or media_type
  • Malformed image data

This would ensure our placeholder handling is robust even with unexpected input.

})
})

Expand Down
11 changes: 9 additions & 2 deletions src/api/transform/vscode-lm-format.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -85,8 +85,12 @@ export function convertToVsCodeLmMessages(
// Convert non-tool messages to TextParts after tool messages
...nonToolMessages.map((part) => {
if (part.type === "image") {
// TODO: When VS Code LM API adds LanguageModelImagePart support, convert images properly
// For now, we provide a placeholder text description
const imageType = part.source?.type || "unknown"
const mediaType = part.source?.media_type || "unknown"
return new vscode.LanguageModelTextPart(
`[Image (${part.source?.type || "Unknown source-type"}): ${part.source?.media_type || "unknown media-type"} not supported by VSCode LM API]`,
`[Image placeholder: ${imageType} (${mediaType}) - VS Code LM API image support pending]`,
Copy link
Contributor Author

@roomote roomote bot Sep 18, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good improvement on the placeholder message! However, I notice the placeholder messages differ between user context (line 93) and assistant context (line 138).

For consistency, should we standardize them? Both could use the same format:

)
}
return new vscode.LanguageModelTextPart(part.text)
Expand Down Expand Up @@ -129,7 +133,10 @@ export function convertToVsCodeLmMessages(
// Convert non-tool messages to TextParts after tool messages
...nonToolMessages.map((part) => {
if (part.type === "image") {
return new vscode.LanguageModelTextPart("[Image generation not supported by VSCode LM API]")
// Assistant-generated images are not yet supported
return new vscode.LanguageModelTextPart(
"[Image generation not yet supported by VS Code LM API]",
Copy link
Contributor Author

@roomote roomote bot Sep 18, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is the different message intentional here? Assistant-generated images use "not yet supported" while user images use "support pending". If we want to distinguish between the two cases, that's fine, but it might be worth a comment explaining why.

)
}
return new vscode.LanguageModelTextPart(part.text)
}),
Expand Down
4 changes: 2 additions & 2 deletions src/package.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@
"theme": "dark"
},
"engines": {
"vscode": "^1.84.0",
"vscode": "^1.95.0",
Copy link
Contributor Author

@roomote roomote bot Sep 18, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Breaking Change Alert: This bumps the minimum VS Code version from 1.84.0 to 1.95.0, which is a significant jump (11 versions). Users on older VS Code versions won't be able to use this extension anymore.

Could we consider:

  1. Adding a migration notice in the README?
  2. Mentioning this prominently in the release notes?
  3. Perhaps maintaining a legacy branch for users who can't upgrade VS Code yet?

"node": "20.19.2"
},
"author": {
Expand Down Expand Up @@ -531,7 +531,7 @@
"@types/string-similarity": "^4.0.2",
"@types/tmp": "^0.2.6",
"@types/turndown": "^5.0.5",
"@types/vscode": "^1.84.0",
"@types/vscode": "^1.104.0",
"@vscode/test-electron": "^2.5.2",
"@vscode/vsce": "3.3.2",
"esbuild": "^0.25.0",
Expand Down
Loading