feat: prepare VS Code LM API for image support (#8123) #8132
Conversation
- Updated VS Code engine requirement to 1.95.0 for image API support - Added TODO comment in vscode-lm.ts for future image capability detection - Improved image placeholder messages in convertToVsCodeLmMessages - Updated tests to match new placeholder format This prepares the codebase for when VS Code fully exposes the image support capabilities through the LanguageModelChat interface.
dosubot
bot
added
the
size:S
hannesrudolph
added
the
Issue/PR - Triage
![roomote[bot]](https://avatars.githubusercontent.com/in/1546624?s=60&v=4){kind=small}
| }, | ||
| "engines": { | ||
| "vscode": "^1.84.0", | ||
| "vscode": "^1.95.0", |
There was a problem hiding this comment.
Could we consider:
- Adding a migration notice in the README?
- Mentioning this prominently in the release notes?
- Perhaps maintaining a legacy branch for users who can't upgrade VS Code yet?
| // TODO: Enable image support when VS Code LM API provides capability detection | ||
| // The newer VS Code API (1.95+) includes LanguageModelChatInformation.capabilities.imageInput | ||
| // but it's not directly accessible from the LanguageModelChat interface yet | ||
| supportsImages: false, |
There was a problem hiding this comment.
The TODO comment is helpful, but could be more specific. Consider expanding it to:
This would help future maintainers understand exactly what we're waiting for.
| const mediaType = part.source?.media_type || "unknown" | ||
| return new vscode.LanguageModelTextPart( | ||
| `[Image (${part.source?.type || "Unknown source-type"}): ${part.source?.media_type || "unknown media-type"} not supported by VSCode LM API]`, | ||
| `[Image placeholder: ${imageType} (${mediaType}) - VS Code LM API image support pending]`, |
There was a problem hiding this comment.
Good improvement on the placeholder message! However, I notice the placeholder messages differ between user context (line 93) and assistant context (line 138).
For consistency, should we standardize them? Both could use the same format:
| expect(imagePlaceholder.value).toContain("[Image (base64): image/png not supported by VSCode LM API]") | ||
| expect(imagePlaceholder.value).toContain( | ||
| "[Image placeholder: base64 (image/png) - VS Code LM API image support pending]", | ||
| ) |
There was a problem hiding this comment.
The test update correctly matches the new placeholder format. Consider adding edge case tests for:
- Images with missing property
- Images with or media_type
- Malformed image data
This would ensure our placeholder handling is robust even with unexpected input.
| return new vscode.LanguageModelTextPart("[Image generation not supported by VSCode LM API]") | ||
| // Assistant-generated images are not yet supported | ||
| return new vscode.LanguageModelTextPart( | ||
| "[Image generation not yet supported by VS Code LM API]", |
There was a problem hiding this comment.
Is the different message intentional here? Assistant-generated images use "not yet supported" while user images use "support pending". If we want to distinguish between the two cases, that's fine, but it might be worth a comment explaining why.
Description
This PR attempts to address Issue #8123 by preparing the codebase for image support in the VS Code Language Model API. While VS Code has added image support capabilities in recent versions, the current API doesn't fully expose these capabilities through the LanguageModelChat interface yet.
What This PR Does
Current Limitations
capabilities.imageInputproperty directlyNext Steps
Once VS Code fully exposes image capabilities through the LanguageModelChat interface, we can:
capabilities.imageInputto setsupportsImagesdynamicallyTesting
Breaking Changes
Feedback and guidance are welcome!
Closes #8123
Important
Prepares VS Code Language Model API for future image support by updating dependencies, adding placeholders, and modifying tests.
1.95.0+inpackage.json.@types/vscodeto1.104.0inpackage.json.vscode-lm.tsfor future image capability detection.convertToVsCodeLmMessages()invscode-lm-format.ts.vscode-lm-format.spec.tsto match new placeholder format.capabilities.imageInputis not exposed byLanguageModelChatinterface.1.84.0to1.95.0.This description was created by
for 5eb87d8. You can customize this summary. It will automatically update as commits are pushed.