docs(amazonq): Documentation for setting up and running new UI E2E Tests w/ vscode-extension-tester #7750

surajrdy-aws · 2025-07-24T16:29:48Z

Problem

The AWS Toolkit repository lacks documentation for the UI testing commands (test:ui:prepare, test:ui:install, test:ui:run, test:ui) and the new e2e_new testing suite. Contributors need clear guidance on how to run UI tests and understand the testing framework structure.

Solution

Added comprehensive UI testing documentation covering:

Command usage and workflow for the complete UI test suite
Directory structure and organization of the e2e_new testing framework
Guidelines for writing new UI tests with example code
Troubleshooting common issues and debug techniques
Prerequisites and setup requirements

Treat all work as PUBLIC. Private feature/x branches will not be squash-merged at release time.
Your code changes must meet the guidelines in CONTRIBUTING.md.
License: I confirm that my contribution is made under the terms of the Apache 2.0 license.

laura-codess · 2025-07-24T16:39:11Z

LGTM

Hweinstock

Great start on the docs! Very clear instructions. Left a few small comments.

Hweinstock · 2025-07-24T17:51:06Z

docs/TESTPLAN.md

    -   Do not use mocks.
 -   E2E Tests: **slow** tests
-    -   Live in `src/testE2E`
+    -   Live in `packages/amazonq/test/e2e_new`


wow these docs are outdated, it doesn't even reference the separate packages.

Also, I think there might be a few tests in the old directory that are still active. For example https://github.com/aws/aws-toolkit-vscode/tree/master/packages/amazonq/test/e2e/lsp, won't be replaced with UI tests AFAIK.

Hweinstock · 2025-07-24T17:52:17Z

docs/TESTPLAN.md

+    -   Live in `packages/amazonq/test/e2e_new`
    -   These tests are heavier than Integration tests.
-    -   Some E2E tests have a more complicated architecture, described in [TEST_E2E](./TEST_E2E.md)
+    -   Some E2E tests have a more complicated architecture, described in [UI_E2E_Test](./UI_E2E_Test.md)


do we want to distinguish between e2e and e2e ui tests? We may want to write e2e tests for things like the language server, but not rely on UI tests.

Hweinstock · 2025-07-24T17:53:05Z

docs/UI_E2E_Test.md

+-   Compiles test files with `npm run testCompile`
+-   Runs tests matching `packages/amazonq/dist/test/e2e_new/amazonq/tests/*.js`
+
+#### Directory Structure


I would honestly remove this because its tedious to update as the code evolves.

Sounds good, thanks for pointing it out

Hweinstock · 2025-07-24T17:53:36Z

docs/UI_E2E_Test.md

+3. Use helpers from `../helpers/`
+4. Follow existing patterns for setup/cleanup
+
+#### Example Test Structure


nice, this is super helpful.

Hweinstock · 2025-07-24T17:53:56Z

docs/UI_E2E_Test.md

@@ -0,0 +1,134 @@
+## UI Testing


is it worth mentioning the manual auth details?

Hweinstock · 2025-07-24T18:55:50Z

docs/UI_E2E_Test.md

+
+#### Authentication
+
+Currently, authentication is not configured to be automatically logged into AmazonQ due to issues with Authentication credentials and the VET setup running in a new VSCode instance through a packaged VSIX. To bypass this for now (as of July 24th, 2025), you must click the approve/open button that redirects you to a browser in order for tests to be run in an authenticated environment at the start of a new.


Can we make this more concise? Maybe just drop due to issues with Authentication credentials and the VET setup running in a new VSCode instance through a packaged VSIX?

abhraina-aws · 2025-07-24T19:08:44Z

docs/UI_E2E_Test.md

+
+1. Create test files in `packages/amazonq/test/e2e_new/amazonq/tests/`
+2. Import utilities from `../utils/`
+3. Use helpers from `../helpers/`


Mention about add helpers to .... also. Tell them how to extend the package in future.

abhraina-aws · 2025-07-24T19:09:29Z

docs/UI_E2E_Test.md

+
+### Individual Commands
+
+#### `test:ui:prepare`


Can we also mention why would we need this?

abhraina-aws · 2025-07-24T19:09:44Z

docs/UI_E2E_Test.md

+npm run test:ui:prepare
+```
+
+#### `test:ui:install`


Same for this too.

abhraina-aws · 2025-07-24T19:10:15Z

docs/UI_E2E_Test.md

+
+Note: All of these commands must be run at the root level aws-toolkit-vscode directory.
+
+### Individual Commands


need some context regarding why are these commands here in the first place.

laileni-aws · 2025-07-29T17:53:29Z

/retryBuilds

…sts w/ vscode-extension-tester (aws#7750) ## Problem The AWS Toolkit repository lacks documentation for the UI testing commands (test:ui:prepare, test:ui:install, test:ui:run, test:ui) and the new e2e_new testing suite. Contributors need clear guidance on how to run UI tests and understand the testing framework structure. ## Solution Added comprehensive UI testing documentation covering: - Command usage and workflow for the complete UI test suite - Directory structure and organization of the e2e_new testing framework - Guidelines for writing new UI tests with example code - Troubleshooting common issues and debug techniques - Prerequisites and setup requirements --- - Treat all work as PUBLIC. Private `feature/x` branches will not be squash-merged at release time. - Your code changes must meet the guidelines in [CONTRIBUTING.md](https://github.com/aws/aws-toolkit-vscode/blob/master/CONTRIBUTING.md#guidelines). - License: I confirm that my contribution is made under the terms of the Apache 2.0 license.

surajrdy-aws added 3 commits July 23, 2025 15:18

updated npm scripts

1184b0c

changes to test plan and test doc

aff707d

changes to test doc

f2296b8

surajrdy-aws requested a review from a team as a code owner July 24, 2025 16:29

surajrdy-aws added 3 commits July 24, 2025 12:30

fix package.json

1f5564d

fix package.json new

ea19bb6

fix testplan.md

3930abf

laura-codess approved these changes Jul 24, 2025

View reviewed changes

Hweinstock reviewed Jul 24, 2025

View reviewed changes

fix testplan.md

aa2eadd

Hweinstock reviewed Jul 24, 2025

View reviewed changes

fix testplan.md

4228163

Hweinstock approved these changes Jul 24, 2025

View reviewed changes

abhraina-aws reviewed Jul 24, 2025

View reviewed changes

docs/UI_E2E_Test.md

### Individual Commands

#### `test:ui:prepare`

Copy link

Contributor

abhraina-aws Jul 24, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we also mention why would we need this?

abhraina-aws reviewed Jul 24, 2025

View reviewed changes

docs/UI_E2E_Test.md

npm run test:ui:prepare

```

#### `test:ui:install`

Copy link

Contributor

abhraina-aws Jul 24, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same for this too.

abhraina-aws reviewed Jul 24, 2025

View reviewed changes

surajrdy-aws added 2 commits July 25, 2025 13:08

fix testplan.md

5e511d5

nit: testplan.md

ab2dbf7

abhraina-aws approved these changes Jul 25, 2025

View reviewed changes

abhraina-aws enabled auto-merge (squash) July 29, 2025 17:00

abhraina-aws disabled auto-merge July 29, 2025 17:00

abhraina-aws merged commit 9c5f854 into aws:feature/ui-e2e-tests Jul 29, 2025
22 checks passed


		#### Authentication

		Currently, authentication is not configured to be automatically logged into AmazonQ due to issues with Authentication credentials and the VET setup running in a new VSCode instance through a packaged VSIX. To bypass this for now (as of July 24th, 2025), you must click the approve/open button that redirects you to a browser in order for tests to be run in an authenticated environment at the start of a new.


		Note: All of these commands must be run at the root level aws-toolkit-vscode directory.

		### Individual Commands

docs(amazonq): Documentation for setting up and running new UI E2E Tests w/ vscode-extension-tester #7750

docs(amazonq): Documentation for setting up and running new UI E2E Tests w/ vscode-extension-tester #7750

Uh oh!

Conversation

surajrdy-aws commented Jul 24, 2025

Problem

Solution

Uh oh!

laura-codess commented Jul 24, 2025

Uh oh!

Hweinstock left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

laileni-aws commented Jul 29, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants