Skip to content

feat(amazonq): add image context support #5846

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 24 commits into from
Jul 9, 2025
Merged

feat(amazonq): add image context support #5846

Show file tree
Hide file tree
Changes from 8 commits
Commits
Show all changes
24 commits
Select commit Hold shift + click to select a range
92b30dc
feat(q): enable image context
zuoyaofu Jun 20, 2025
79f86f6
feat(q): add image context support
zuoyaofu Jun 21, 2025
9d0fd61
add change log + fix some styles
zuoyaofu Jun 21, 2025
4621012
pass the dropped files from Kotlin to webView
yzhangok Jun 24, 2025
eb596bd
use params to select different file descriptor
zuoyaofu Jun 25, 2025
2594dbe
Merge branch 'main' into image
zuoyaofu Jun 25, 2025
42016ee
Merge branch 'main' into image
zuoyaofu Jun 25, 2025
3df8ef9
add feature flag
yzhangok Jun 25, 2025
410c68e
Merge branch 'main' into image
zuoyaofu Jun 26, 2025
ac84d65
fixed comments
zuoyaofu Jun 26, 2025
714a4a1
Update plugins/amazonq/shared/jetbrains-community/src/software/aws/to…
yzhangok Jul 8, 2025
dfce2b1
Merge branch 'main' into image
yzhangok Jul 8, 2025
0c18a65
fix comments
yzhangok Jul 8, 2025
024d33b
Merge branch 'main' into image
zuoyaofu Jul 8, 2025
388b428
use old function in order to compile
zuoyaofu Jul 8, 2025
6218560
fix ci
zuoyaofu Jul 8, 2025
c303aac
fix more styles
zuoyaofu Jul 8, 2025
b95ed96
use updated path
zuoyaofu Jul 8, 2025
351b6a1
Merge branch 'main' into image
zuoyaofu Jul 8, 2025
566b82c
more CI fixes
zuoyaofu Jul 8, 2025
8e9fa3a
change to use jacksonObjectMapper
yzhangok Jul 8, 2025
1834f27
fix ci
zuoyaofu Jul 9, 2025
fe318e4
Merge branch 'main' into image
rli Jul 9, 2025
cfe76d5
Merge branch 'main' into image
rli Jul 9, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions .changes/next-release/feature-ddbdde68-769a-468f-bb42-e5aece51c729.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
{
"type" : "feature",
"description" : "Add image context support"
}
115 changes: 111 additions & 4 deletions ...community/src/software/aws/toolkits/jetbrains/services/amazonq/toolwindow/AmazonQPanel.kt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,7 @@

package software.aws.toolkits.jetbrains.services.amazonq.toolwindow

import com.google.gson.Gson
import com.intellij.idea.AppMode
import com.intellij.openapi.Disposable
import com.intellij.openapi.components.service
Expand All @@ -20,6 +21,7 @@
import kotlinx.coroutines.flow.first
import kotlinx.coroutines.launch
import kotlinx.coroutines.withContext
import software.aws.toolkits.core.utils.getLogger
import software.aws.toolkits.jetbrains.core.coroutines.EDT
import software.aws.toolkits.jetbrains.isDeveloperMode
import software.aws.toolkits.jetbrains.services.amazonq.apps.AmazonQAppInitContext
Expand All @@ -44,8 +46,14 @@
import software.aws.toolkits.jetbrains.services.amazonqFeatureDev.auth.isFeatureDevAvailable
import software.aws.toolkits.jetbrains.services.codemodernizer.utils.isCodeTransformAvailable
import software.aws.toolkits.resources.message
import java.awt.datatransfer.DataFlavor
import java.awt.dnd.DropTarget
import java.awt.dnd.DropTargetDragEvent

Check warning on line 51 in plugins/amazonq/chat/jetbrains-community/src/software/aws/toolkits/jetbrains/services/amazonq/toolwindow/AmazonQPanel.kt

View workflow job for this annotation

GitHub Actions / qodana

Unused import directive 

Unused import directive
import java.awt.dnd.DropTargetDropEvent
import java.awt.dnd.DropTargetEvent

Check warning on line 53 in plugins/amazonq/chat/jetbrains-community/src/software/aws/toolkits/jetbrains/services/amazonq/toolwindow/AmazonQPanel.kt

View workflow job for this annotation

GitHub Actions / qodana

Unused import directive 

Unused import directive
import java.util.concurrent.CompletableFuture
import javax.swing.JButton
import javax.imageio.ImageIO.read

class AmazonQPanel(val project: Project, private val scope: CoroutineScope) : Disposable {
private val browser = CompletableFuture<Browser>()
Expand Down Expand Up @@ -122,12 +130,105 @@

withContext(EDT) {
browser.complete(
Browser(this@AmazonQPanel, webUri, project).also {
wrapper.setContent(it.component())
Browser(this@AmazonQPanel, webUri, project).also { browserInstance ->
wrapper.setContent(browserInstance.component())

// Add DropTarget to the browser component
val dropTarget = object : DropTarget() {
override fun drop(dtde: DropTargetDropEvent) {
try {
dtde.acceptDrop(dtde.dropAction)
val transferable = dtde.transferable
if (transferable.isDataFlavorSupported(DataFlavor.javaFileListFlavor)) {
val fileList = transferable.getTransferData(DataFlavor.javaFileListFlavor) as List<*>

val errorMessages = mutableListOf<String>()
val validImages = mutableListOf<java.io.File>()
val allowedTypes = setOf("jpg", "jpeg", "png", "gif", "webp")
val maxFileSize = 3.75 * 1024 * 1024 // 3.75MB in bytes
val maxDimension = 8000
Comment on lines +150 to +152
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why do we need this here, can you please add a comment to the flare location that this picks up the values from?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

JCEF does not propagate OS-level dragenter, dragOver and drop into DOM. As an alternative, enabling the native drag in JCEF, and let the native handling the drop event, and update the UI through JS bridge.
The same logic of filtering the images are implemented on both language server and JetBrain


for (file in fileList) {
val fileObj = file as? java.io.File ?: continue
val fileName = fileObj.name
val ext = fileName.substringAfterLast('.', "").lowercase()

// File type restriction
if (ext !in allowedTypes) {
errorMessages.add("$fileName: File must be an image in JPEG, PNG, GIF, or WebP format.")
continue
}

// Size restriction
if (fileObj.length() > maxFileSize) {
errorMessages.add("$fileName: Image must be no more than 3.75MB in size.")
continue
}

// Width/Height restriction (only for image types)
try {
val img = read(fileObj)
if (img == null) {
errorMessages.add("$fileName: File could not be read as an image.")
continue
}
if (img.width > maxDimension) {
errorMessages.add("$fileName: Image must be no more than 8,000px in width.")
continue
}
if (img.height > maxDimension) {
errorMessages.add("$fileName: Image must be no more than 8,000px in height.")
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Aren't all these checks also in language-server? Do they need to be duplicated on the client?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Due to the restriction of Jetbrain webveiw, we have to handle the drag&drop natively

continue
}
} catch (e: Exception) {
errorMessages.add("$fileName: File could not be read as an image.")
continue
}

validImages.add(fileObj)
}

// File count restriction
if (validImages.size > 20) {
errorMessages.add("A maximum of 20 images can be added to a single message.")
validImages.subList(20, validImages.size).clear()
}

val json = Gson().toJson(validImages)
browserInstance.jcefBrowser.cefBrowser.executeJavaScript(
"window.handleNativeDrop('$json')",
browserInstance.jcefBrowser.cefBrowser.url,
0
)

val errorJson = Gson().toJson(errorMessages)
browserInstance.jcefBrowser.cefBrowser.executeJavaScript(
"window.handleNativeNotify('$errorJson')",
browserInstance.jcefBrowser.cefBrowser.url,
0
)

dtde.dropComplete(true)
} else {
dtde.dropComplete(false)
}
} catch (e: Exception) {
LOG.error("Failed to handle file drop operation", e.message)

Check warning on line 216 in plugins/amazonq/chat/jetbrains-community/src/software/aws/toolkits/jetbrains/services/amazonq/toolwindow/AmazonQPanel.kt

View workflow job for this annotation

GitHub Actions / qodana

Number of placeholders does not match number of arguments in logging call 

More arguments provided (1) than placeholders specified (0)
dtde.dropComplete(false)
}
}
}

// Set DropTarget on the browser component and its children
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

how are the children involved

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We want the whole chat window to be "drag & drop zone"

browserInstance.component()?.let { component ->
component.dropTarget = dropTarget
// Also try setting on parent if needed
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is to make sure all chat window is drop area

component.parent?.dropTarget = dropTarget
}

initConnections()
connectUi(it)
connectApps(it)
connectUi(browserInstance)
connectApps(browserInstance)

loadingPanel.stopLoading()
}
Expand Down Expand Up @@ -211,6 +312,12 @@
}
}

companion object {
private val LOG = getLogger<AmazonQPanel>()

fun getInstance(project: Project) = project.service<AmazonQPanel>()

Check failure on line 318 in plugins/amazonq/chat/jetbrains-community/src/software/aws/toolkits/jetbrains/services/amazonq/toolwindow/AmazonQPanel.kt

View workflow job for this annotation

GitHub Actions / qodana

Incorrect service retrieving 

The 'software.aws.toolkits.jetbrains.services.amazonq.toolwindow.AmazonQPanel' class is not registered as a service
}

override fun dispose() {
}
}
31 changes: 29 additions & 2 deletions ...tbrains-community/src/software/aws/toolkits/jetbrains/services/amazonq/webview/Browser.kt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -85,6 +85,9 @@ class Browser(parent: Disposable, private val webUri: URI, val project: Project)
// setup empty state. The message request handlers use this for storing state
// that's persistent between page loads.
jcefBrowser.setProperty("state", "")
jcefBrowser.jbCefClient.addDragHandler({ browser, dragData, mask ->
true // Allow drag operations
}, jcefBrowser.cefBrowser)
// load the web app
jcefBrowser.loadHTML(
getWebviewHTML(
Expand Down Expand Up @@ -122,7 +125,7 @@ class Browser(parent: Disposable, private val webUri: URI, val project: Project)
<script type="text/javascript" charset="UTF-8" src="$webUri" defer onload="init()"></script>

<script type="text/javascript">

const init = () => {
const hybridChatConnector = connectorAdapter.initiateAdapter(
${MeetQSettings.getInstance().reinvent2024OnboardingCount < MAX_ONBOARDING_PAGE_COUNT},
Expand All @@ -139,7 +142,7 @@ class Browser(parent: Disposable, private val webUri: URI, val project: Project)
},

"${activeProfile?.profileName.orEmpty()}")
amazonQChat.createChat(
const qChat = amazonQChat.createChat(
{
postMessage: message => {
$postMessageToJavaJsCode
Expand All @@ -155,6 +158,30 @@ class Browser(parent: Disposable, private val webUri: URI, val project: Project)
hybridChatConnector,
${CodeWhispererFeatureConfigService.getInstance().getFeatureConfigJsonString()}
);

window.handleNativeDrop = function(filePath) {
const parsedFilePath = JSON.parse(filePath);
const contextArray = parsedFilePath.map(fileObj => {
const fullPath = fileObj.path;
const fileName = fullPath.split(/[\\/]/).pop();
return {
command: fileName,
label: 'image',
route: [fullPath],
description: fullPath
};
});
qChat.addCustomContextToPrompt(qChat.getSelectedTabId(), contextArray);
};

window.handleNativeNotify = function(errorMessages) {
const messages = JSON.parse(errorMessages);
messages.forEach(msg => {
qChat.notify({
content: msg
})
});
};
}
</script>
""".trimIndent()
Expand Down
16 changes: 16 additions & 0 deletions ...ommunity/src/software/aws/toolkits/jetbrains/services/amazonq/webview/BrowserConnector.kt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -77,6 +77,8 @@ import software.aws.toolkits.jetbrains.services.amazonq.lsp.model.aws.chat.GetSe
import software.aws.toolkits.jetbrains.services.amazonq.lsp.model.aws.chat.LIST_MCP_SERVERS_REQUEST_METHOD
import software.aws.toolkits.jetbrains.services.amazonq.lsp.model.aws.chat.LIST_RULES_REQUEST_METHOD
import software.aws.toolkits.jetbrains.services.amazonq.lsp.model.aws.chat.MCP_SERVER_CLICK_REQUEST_METHOD
import software.aws.toolkits.jetbrains.services.amazonq.lsp.model.aws.chat.OPEN_FILE_DIALOG
import software.aws.toolkits.jetbrains.services.amazonq.lsp.model.aws.chat.OPEN_FILE_DIALOG_REQUEST_METHOD
import software.aws.toolkits.jetbrains.services.amazonq.lsp.model.aws.chat.OPEN_SETTINGS
import software.aws.toolkits.jetbrains.services.amazonq.lsp.model.aws.chat.OPEN_WORKSPACE_SETTINGS_KEY
import software.aws.toolkits.jetbrains.services.amazonq.lsp.model.aws.chat.OpenSettingsNotification
Expand Down Expand Up @@ -489,6 +491,19 @@ class BrowserConnector(
)
}
}

OPEN_FILE_DIALOG -> {
handleChat(AmazonQChatServer.showOpenFileDialog, node)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can you walk me through the message flow? does it actually need to transit through flare before requesting the file picker?

Copy link
Contributor Author

@zuoyaofu zuoyaofu Jun 25, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Flow:

  1. chat-client will send openFileDialog request to flare
  2. Flare will send to extension the params, including canSelectMany (multiple files), filters (what file extension types are allowed)
  3. Extension will then handle file selection logic and send back the list of URIs.
  4. Flare will then send to chat-client the file uri to display in chat.

- does it actually need to transit through flare before requesting the file picker?

  • In @yzhangok's design, yes. Flare is also telling client what type of files are supported in file picker as well(step 2)

.whenComplete { response, _ ->
browser.postChat(
FlareUiMessage(
command = OPEN_FILE_DIALOG_REQUEST_METHOD,
params = response
)
)
}
}

LIST_RULES_REQUEST_METHOD -> {
handleChat(AmazonQChatServer.listRules, node)
.whenComplete { response, _ ->
Expand Down Expand Up @@ -517,6 +532,7 @@ class BrowserConnector(
CHAT_PINNED_CONTEXT_REMOVE -> {
handleChat(AmazonQChatServer.pinnedContextRemove, node)
}

}
}

Expand Down
7 changes: 7 additions & 0 deletions ...s-community/src/software/aws/toolkits/jetbrains/services/amazonq/lsp/AmazonQChatServer.kt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -45,6 +45,7 @@ import software.aws.toolkits.jetbrains.services.amazonq.lsp.model.aws.chat.LIST_
import software.aws.toolkits.jetbrains.services.amazonq.lsp.model.aws.chat.LinkClickParams
import software.aws.toolkits.jetbrains.services.amazonq.lsp.model.aws.chat.ListConversationsParams
import software.aws.toolkits.jetbrains.services.amazonq.lsp.model.aws.chat.MCP_SERVER_CLICK_REQUEST_METHOD
import software.aws.toolkits.jetbrains.services.amazonq.lsp.model.aws.chat.OPEN_FILE_DIALOG_REQUEST_METHOD
import software.aws.toolkits.jetbrains.services.amazonq.lsp.model.aws.chat.PROMPT_INPUT_OPTIONS_CHANGE
import software.aws.toolkits.jetbrains.services.amazonq.lsp.model.aws.chat.PromptInputOptionChangeParams
import software.aws.toolkits.jetbrains.services.amazonq.lsp.model.aws.chat.RULE_CLICK_REQUEST_METHOD
Expand Down Expand Up @@ -248,4 +249,10 @@ object AmazonQChatServer : JsonRpcMethodProvider {
TELEMETRY_EVENT,
Any::class.java
)

val showOpenFileDialog = JsonRpcRequest(
OPEN_FILE_DIALOG_REQUEST_METHOD,
Any::class.java,
LSPAny::class.java
)
}
5 changes: 5 additions & 0 deletions ...mmunity/src/software/aws/toolkits/jetbrains/services/amazonq/lsp/AmazonQLanguageClient.kt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,9 @@ import software.aws.toolkits.jetbrains.services.amazonq.lsp.model.aws.chat.GET_S
import software.aws.toolkits.jetbrains.services.amazonq.lsp.model.aws.chat.GetSerializedChatResult
import software.aws.toolkits.jetbrains.services.amazonq.lsp.model.aws.chat.OPEN_FILE_DIFF
import software.aws.toolkits.jetbrains.services.amazonq.lsp.model.aws.chat.OpenFileDiffParams
import software.aws.toolkits.jetbrains.services.amazonq.lsp.model.aws.chat.SHOW_OPEN_FILE_DIALOG_REQUEST_METHOD
import software.aws.toolkits.jetbrains.services.amazonq.lsp.model.aws.chat.SHOW_SAVE_FILE_DIALOG_REQUEST_METHOD
import software.aws.toolkits.jetbrains.services.amazonq.lsp.model.aws.chat.ShowOpenFileDialogParams
import software.aws.toolkits.jetbrains.services.amazonq.lsp.model.aws.chat.ShowSaveFileDialogParams
import software.aws.toolkits.jetbrains.services.amazonq.lsp.model.aws.chat.ShowSaveFileDialogResult
import software.aws.toolkits.jetbrains.services.amazonq.lsp.model.aws.credentials.ConnectionMetadata
Expand All @@ -45,6 +47,9 @@ interface AmazonQLanguageClient : LanguageClient {
@JsonRequest(SHOW_SAVE_FILE_DIALOG_REQUEST_METHOD)
fun showSaveFileDialog(params: ShowSaveFileDialogParams): CompletableFuture<ShowSaveFileDialogResult>

@JsonRequest(SHOW_OPEN_FILE_DIALOG_REQUEST_METHOD)
fun showOpenFileDialog(params: ShowOpenFileDialogParams): CompletableFuture<LSPAny>

@JsonRequest(GET_SERIALIZED_CHAT_REQUEST_METHOD)
fun getSerializedChat(params: LSPAny): CompletableFuture<GetSerializedChatResult>

Expand Down
Loading
Loading