feat(amazonq): add setup for ui-tests #7597

laura-codess · 2025-07-02T18:50:15Z

Changes

This is the initial commit for UI Tests using the framework vscode-extension-tester. It instantiates the VSCode instance which opens Amazon Q and goes through some basic flows to login. This is done using the VSCode-Extension-Tester Framework. (Note: It is missing full authentication and stops working once Amazon Q needs to open the browser).

To run these tests

Please follow the steps outlined in this document:
Setup Instruction for Amazon Q

For information why each dependency / webpack change is needed, please see here.

Demo Video:

UI.Tests.Screen.Recording.mov

Treat all work as PUBLIC. Private feature/x branches will not be squash-merged at release time.
Your code changes must meet the guidelines in CONTRIBUTING.md.
License: I confirm that my contribution is made under the terms of the Apache 2.0 license.

github-actions · 2025-07-02T18:50:31Z

This pull request implements a feat or fix, so it must include a changelog entry (unless the fix is for an unreleased feature). Review the changelog guidelines.
- Note: beta or "experiment" features that have active users should announce fixes in the changelog.
- If this is not a feature or fix, use an appropriate type from the title guidelines. For example, telemetry-only changes should use the telemetry type.

Hweinstock

Nice start!

package.json

Hweinstock · 2025-07-02T18:53:46Z

package.json

        "@types/node": "^22.7.5",
        "jaro-winkler": "^0.2.8",
+        "buffer": "^6.0.3",
+        "mocha": "^11.7.1",


I think most of these can be dev-dependencies. Also same comment as above.

I think mocha should already be getting pulled in (

aws-toolkit-vscode/packages/core/package.json

Line 515 in 4edc815

"mocha": "^10.1.0",

). buffer might be able to be a devDependency but we can look into this as a follow up.

packages/amazonq/package.json

packages/webpack.web.config.js

test.log

tsconfig.tsbuildinfo

packages/amazonq/test/e2e/amazonq/VET.test.ts

Hweinstock · 2025-07-03T12:51:40Z

looks like the buffer dependency was needed:

ERROR in ../../node_modules/buffer-xor/index.js 3:19-25
  Module not found: Error: Can't resolve 'buffer' in '/codebuild/output/src1044368460/src/github.com/aws/aws-toolkit-vscode/node_modules/buffer-xor'

…as webpack.NormalModuleReplacementPlugin that needed to be there in order for the amazonq to compile

Hweinstock · 2025-07-03T18:02:11Z

package.json

        "vscode-nls": "^5.2.0",
        "vscode-nls-dev": "^4.0.4"
+    },
+    "engines": {


curious as to what these actually do, might be worth calling out in PR description with maybe a link to VET docs explaining?

Hweinstock · 2025-07-03T18:13:17Z

I think after removing that try-catch, adding some doc links, and updating the PR description for how to run + what to expect, this looks good to me as a start. We should create tracking issues for:

investigate the webpack and dependencies changes to understand why they are needed. We can't be making changes here we don't understand when trying to merge code to the mainline.
investigate timeout requirements. I suspect someone has solved this problem so might just require some research of finding someone else's solution rather than somewhat arbitrary timeouts between actions.

These PR descriptions can eventually serve a "work-log" for the project so I think its important to make them solid now while we have context.

laura-codess · 2025-07-03T19:13:07Z

It looks like 2 tests are failing even after the webpack addition, but the code has no trouble compiling. My initial commit passed all the checks so I'm going to try adding back some lines to see what was necessary. I'll add all of that information in the PR and work on writing a doc about why each dependency / webpack change is needed.

…he checks. have to commit to check since the build compiles with no errors

…est case

Hweinstock

Nice!

## Changes This is the initial commit for UI Tests using the framework vscode-extension-tester. It instantiates the VSCode instance which opens Amazon Q and goes through some basic flows to login. This is done using the [VSCode-Extension-Tester Framework](https://github.com/redhat-developer/vscode-extension-tester/wiki/). (Note: It is missing full authentication and stops working once Amazon Q needs to open the browser). **To run this test** 1. First compile the test files `npm run testCompile` 2. Then run the test setup `node_modules/.bin/extest setup-tests` 2. Then run the script (NOTE: this js file should be generated when you compile the test files) `node_modules/.bin/extest run-tests aws-toolkit-vscode/packages/amazonq/dist/test/e2e/amazonq/VET.test.js` For information why each dependency / webpack change is needed, please see [here](https://quip-amazon.com/lCuBAOGibHzm/UI-Tests-PR-Explanation). Demo Video: https://github.com/user-attachments/assets/f1c6e59f-d4e3-4ae0-a164-1da389ec1339 --- - Treat all work as PUBLIC. Private `feature/x` branches will not be squash-merged at release time. - Your code changes must meet the guidelines in [CONTRIBUTING.md](https://github.com/aws/aws-toolkit-vscode/blob/master/CONTRIBUTING.md#guidelines). - License: I confirm that my contribution is made under the terms of the Apache 2.0 license.

This is the initial commit for UI Tests using the framework vscode-extension-tester. It instantiates the VSCode instance which opens Amazon Q and goes through some basic flows to login. This is done using the [VSCode-Extension-Tester Framework](https://github.com/redhat-developer/vscode-extension-tester/wiki/). (Note: It is missing full authentication and stops working once Amazon Q needs to open the browser). **To run this test** 1. First compile the test files `npm run testCompile` 2. Then run the test setup `node_modules/.bin/extest setup-tests` 2. Then run the script (NOTE: this js file should be generated when you compile the test files) `node_modules/.bin/extest run-tests aws-toolkit-vscode/packages/amazonq/dist/test/e2e/amazonq/VET.test.js` For information why each dependency / webpack change is needed, please see [here](https://quip-amazon.com/lCuBAOGibHzm/UI-Tests-PR-Explanation). Demo Video: https://github.com/user-attachments/assets/f1c6e59f-d4e3-4ae0-a164-1da389ec1339 --- - Treat all work as PUBLIC. Private `feature/x` branches will not be squash-merged at release time. - Your code changes must meet the guidelines in [CONTRIBUTING.md](https://github.com/aws/aws-toolkit-vscode/blob/master/CONTRIBUTING.md#guidelines). - License: I confirm that my contribution is made under the terms of the Apache 2.0 license.

initial VET commit with baseline setup + dependencies

98688bf

laura-codess requested a review from a team as a code owner July 2, 2025 18:50

Hweinstock reviewed Jul 2, 2025

View reviewed changes

deleting unnecessary dependencies and refactoring some of VET.test.ts

5959a38

laura-codess added 2 commits July 3, 2025 12:57

added back the buffer dependency because it failed the tests without it

4540f16

it was not any of the dependencies that was causing it to break, it w…

f1afe95

…as webpack.NormalModuleReplacementPlugin that needed to be there in order for the amazonq to compile

Hweinstock reviewed Jul 3, 2025

View reviewed changes

laura-codess added 3 commits July 3, 2025 15:27

added back some more node: protocol imports to see if it passes all t…

4fa90ec

…he checks. have to commit to check since the build compiles with no errors

forgot to save a webpack change. hoping it will fix the one failing t…

e1c33d8

…est case

added back basically all the dependencies because 1 test keeps failing

ea46464

Hweinstock approved these changes Jul 7, 2025

View reviewed changes

Hweinstock merged commit d374531 into aws:feature/ui-e2e-tests Jul 8, 2025
22 checks passed

feat(amazonq): add setup for ui-tests #7597

feat(amazonq): add setup for ui-tests #7597

Uh oh!

Conversation

laura-codess commented Jul 2, 2025 • edited by surajrdy-aws Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Changes

Uh oh!

github-actions bot commented Jul 2, 2025

Uh oh!

Hweinstock left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Hweinstock Jul 2, 2025

Choose a reason for hiding this comment

Uh oh!

Hweinstock Jul 3, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Hweinstock commented Jul 3, 2025

Uh oh!

Hweinstock Jul 3, 2025

Choose a reason for hiding this comment

Uh oh!

Hweinstock commented Jul 3, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

laura-codess commented Jul 3, 2025

Uh oh!

Hweinstock left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

laura-codess commented Jul 2, 2025 •

edited by surajrdy-aws

Loading

Hweinstock commented Jul 3, 2025 •

edited

Loading