feat(amazonq): add image context support #5846

zuoyaofu · 2025-06-21T00:34:07Z

Types of changes

Bug fix (non-breaking change which fixes an issue)
New feature (non-breaking change which adds functionality)

Description

Checklist

My code follows the code style of this project
I have added tests to cover my changes
A short description of the change has been added to the CHANGELOG if the change is customer-facing in the IDE.
I have added metrics for my changes (if required)

License

I confirm that my contribution is made under the terms of the Apache 2.0 license.

...tware/aws/toolkits/jetbrains/services/amazonq/lsp/model/aws/chat/showOpenFileDialogParams.kt

rli · 2025-06-24T00:23:59Z

...s-community/src/software/aws/toolkits/jetbrains/services/amazonq/webview/BrowserConnector.kt

                    }
            }
+            OPEN_FILE_DIALOG -> {
+                handleChat(AmazonQChatServer.showOpenFileDialog, node)


can you walk me through the message flow? does it actually need to transit through flare before requesting the file picker?

Flow:

chat-client will send openFileDialog request to flare

Flare will send to extension the params, including canSelectMany (multiple files), filters (what file extension types are allowed)

Extension will then handle file selection logic and send back the list of URIs.

Flare will then send to chat-client the file uri to display in chat.

- does it actually need to transit through flare before requesting the file picker?

In @yzhangok's design, yes. Flare is also telling client what type of files are supported in file picker as well(step 2)

...ns-community/src/software/aws/toolkits/jetbrains/services/amazonq/toolwindow/AmazonQPanel.kt

.../jetbrains-community/src/software/aws/toolkits/jetbrains/services/amazonq/webview/Browser.kt

...ns-community/src/software/aws/toolkits/jetbrains/services/amazonq/toolwindow/AmazonQPanel.kt

rli · 2025-06-24T17:24:25Z

...ns-community/src/software/aws/toolkits/jetbrains/services/amazonq/toolwindow/AmazonQPanel.kt

+                            // Set DropTarget on the browser component and its children
+                            browserInstance.component()?.let { component ->
+                                component.dropTarget = dropTarget
+                                // Also try setting on parent if needed


This is to make sure all chat window is drop area

rli · 2025-06-24T17:24:34Z

...ns-community/src/software/aws/toolkits/jetbrains/services/amazonq/toolwindow/AmazonQPanel.kt

+                                }
+                            }
+
+                            // Set DropTarget on the browser component and its children


how are the children involved

We want the whole chat window to be "drag & drop zone"

...ns-community/src/software/aws/toolkits/jetbrains/services/amazonq/toolwindow/AmazonQPanel.kt

rli · 2025-06-24T17:33:21Z

...munity/src/software/aws/toolkits/jetbrains/services/amazonq/lsp/AmazonQLanguageClientImpl.kt

+                    FileChooserDescriptorFactory.createSingleFileNoJarsDescriptor()
+                }
+
+                val chosenFiles = FileChooser.chooseFiles(descriptor, project, null)


why aren't images from this flow subject to constraints?

Yes, they are now with the new commit.

what about the file size and dimensional limits?

The validation of the images chosen from file picker is done in language-server

...ns-community/src/software/aws/toolkits/jetbrains/services/amazonq/toolwindow/AmazonQPanel.kt

...munity/src/software/aws/toolkits/jetbrains/services/amazonq/lsp/AmazonQLanguageClientImpl.kt

...tware/aws/toolkits/jetbrains/services/amazonq/lsp/model/aws/chat/ShowOpenFileDialogParams.kt

...ity/src/software/aws/toolkits/jetbrains/services/amazonq/lsp/model/ExtendedClientMetadata.kt

...ns-community/src/software/aws/toolkits/jetbrains/services/amazonq/toolwindow/AmazonQPanel.kt

rli · 2025-07-01T18:01:28Z

...munity/src/software/aws/toolkits/jetbrains/services/amazonq/lsp/AmazonQLanguageClientImpl.kt

+                    FileChooserDescriptorFactory.createSingleFileNoJarsDescriptor()
+                }
+
+                val chosenFiles = FileChooser.chooseFiles(descriptor, project, null)


what about the file size and dimensional limits?

...ns-community/src/software/aws/toolkits/jetbrains/services/amazonq/toolwindow/AmazonQPanel.kt

manodnyab · 2025-06-30T18:14:28Z

...ns-community/src/software/aws/toolkits/jetbrains/services/amazonq/toolwindow/AmazonQPanel.kt

+                                                        continue
+                                                    }
+                                                    if (img.height > maxDimension) {
+                                                        errorMessages.add("$fileName: Image must be no more than 8,000px in height.")


Aren't all these checks also in language-server? Do they need to be duplicated on the client?

Due to the restriction of Jetbrain webveiw, we have to handle the drag&drop natively

manodnyab · 2025-07-07T06:37:30Z

...ns-community/src/software/aws/toolkits/jetbrains/services/amazonq/toolwindow/AmazonQPanel.kt

+                                            val allowedTypes = setOf("jpg", "jpeg", "png", "gif", "webp")
+                                            val maxFileSize = 3.75 * 1024 * 1024 // 3.75MB in bytes
+                                            val maxDimension = 8000


Why do we need this here, can you please add a comment to the flare location that this picks up the values from?

JCEF does not propagate OS-level dragenter, dragOver and drop into DOM. As an alternative, enabling the native drag in JCEF, and let the native handling the drop event, and update the UI through JS bridge.
The same logic of filtering the images are implemented on both language server and JetBrain

…olkits/jetbrains/services/amazonq/lsp/model/ExtendedClientMetadata.kt not set default value Co-authored-by: Richard Li <[email protected]>

...ns-community/src/software/aws/toolkits/jetbrains/services/amazonq/toolwindow/AmazonQPanel.kt

zuoyaofu · 2025-07-08T22:17:33Z

what about the file size and dimensional limits?

We are checking that in language servers

samgst-amazon · 2025-07-09T15:17:38Z

/retryBuilds

This reverts commit 83ac4f5.

)" This reverts commit 161bf13.

)" (#5892) This reverts commit 161bf13. Co-authored-by: Bryce Ito <[email protected]>

zuoyaofu added 2 commits June 20, 2025 16:40

feat(q): enable image context

92b30dc

feat(q): add image context support

79f86f6

zuoyaofu changed the title ~~Image~~ feat(q): add image context support Jun 21, 2025

zuoyaofu changed the title ~~feat(q): add image context support~~ feat(amazonq): add image context support Jun 21, 2025

zuoyaofu assigned zuoyaofu and yzhangok Jun 21, 2025

add change log + fix some styles

9d0fd61

rli reviewed Jun 24, 2025

View reviewed changes

pass the dropped files from Kotlin to webView

4621012

github-advanced-security bot found potential problems Jun 24, 2025

View reviewed changes

...ns-community/src/software/aws/toolkits/jetbrains/services/amazonq/toolwindow/AmazonQPanel.kt Fixed Show fixed Hide fixed

bryceitoc9 reviewed Jun 24, 2025

View reviewed changes

.../jetbrains-community/src/software/aws/toolkits/jetbrains/services/amazonq/webview/Browser.kt Outdated Show resolved Hide resolved

rli reviewed Jun 24, 2025

View reviewed changes

zuoyaofu and others added 4 commits June 25, 2025 14:03

use params to select different file descriptor

eb596bd

Merge branch 'main' into image

2594dbe

Merge branch 'main' into image

42016ee

add feature flag

3df8ef9

github-advanced-security bot found potential problems Jun 25, 2025

View reviewed changes

Merge branch 'main' into image

410c68e

zuoyaofu marked this pull request as ready for review June 26, 2025 22:38

zuoyaofu requested review from a team as code owners June 26, 2025 22:38

fixed comments

ac84d65

rli reviewed Jul 1, 2025

View reviewed changes

manodnyab reviewed Jul 7, 2025

View reviewed changes

yzhangok and others added 6 commits July 8, 2025 01:47

Update plugins/amazonq/shared/jetbrains-community/src/software/aws/to…

714a4a1

…olkits/jetbrains/services/amazonq/lsp/model/ExtendedClientMetadata.kt not set default value Co-authored-by: Richard Li <[email protected]>

Merge branch 'main' into image

dfce2b1

fix comments

0c18a65

Merge branch 'main' into image

024d33b

use old function in order to compile

388b428

fix ci

6218560

zuoyaofu added 3 commits July 8, 2025 10:56

fix more styles

c303aac

use updated path

b95ed96

Merge branch 'main' into image

351b6a1

manodnyab reviewed Jul 8, 2025

View reviewed changes

...ns-community/src/software/aws/toolkits/jetbrains/services/amazonq/toolwindow/AmazonQPanel.kt Outdated Show resolved Hide resolved

...ns-community/src/software/aws/toolkits/jetbrains/services/amazonq/toolwindow/AmazonQPanel.kt Outdated Show resolved Hide resolved

more CI fixes

566b82c

yzhangok and others added 2 commits July 8, 2025 15:54

change to use jacksonObjectMapper

8e9fa3a

fix ci

1834f27

manodnyab approved these changes Jul 9, 2025

View reviewed changes

Merge branch 'main' into image

fe318e4

Merge branch 'main' into image

cfe76d5

samgst-amazon merged commit 83ac4f5 into aws:main Jul 9, 2025
11 of 12 checks passed

samgst-amazon added a commit that referenced this pull request Jul 9, 2025

Revert "feat(amazonq): add image context support (#5846)"

4874cb2

This reverts commit 83ac4f5.

aws-toolkit-automation pushed a commit that referenced this pull request Jul 9, 2025

Revert "feat(amazonq): add image context support (#5846)" (#5890)

161bf13

This reverts commit 83ac4f5.

samgst-amazon added a commit that referenced this pull request Jul 9, 2025

Revert "Revert "feat(amazonq): add image context support (#5846)" (#5890

a4e3b4d

)" This reverts commit 161bf13.

bryceitoc9 added a commit that referenced this pull request Jul 10, 2025

Revert "Revert "feat(amazonq): add image context support (#5846)" (#5890

8f02eba

)" (#5892) This reverts commit 161bf13. Co-authored-by: Bryce Ito <[email protected]>

feat(amazonq): add image context support #5846

feat(amazonq): add image context support #5846

Uh oh!

Conversation

zuoyaofu commented Jun 21, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Types of changes

Description

Checklist

License

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

zuoyaofu Jun 25, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

zuoyaofu commented Jul 8, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

samgst-amazon commented Jul 9, 2025

Uh oh!

Uh oh!

Uh oh!

zuoyaofu commented Jun 21, 2025 •

edited

Loading

zuoyaofu Jun 25, 2025 •

edited

Loading

zuoyaofu commented Jul 8, 2025 •

edited

Loading