From 1184b0c3732041d7b2536019387385ea799b551a Mon Sep 17 00:00:00 2001 From: Suraj Reddy Date: Wed, 23 Jul 2025 15:18:47 -0400 Subject: [PATCH 01/10] updated npm scripts --- package.json | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/package.json b/package.json index fb9ee2fdea3..d5936eec098 100644 --- a/package.json +++ b/package.json @@ -28,7 +28,7 @@ "test": "npm run test -w packages/ --if-present", "testWeb": "npm run testWeb -w packages/ --if-present", "testE2E": "npm run testE2E -w packages/ --if-present", - "test:ui:prepare": "./node_modules/.bin/extest get-vscode -s ~/.vscode-test-resources -n && extest get-chromedriver -s ~/.vscode-test-resources -n", + "test:ui:setup": "./node_modules/.bin/extest get-vscode -s ~/.vscode-test-resources -n && extest get-chromedriver -s ~/.vscode-test-resources -n", "test:ui:install": "cd packages/amazonq && npm run package 2>&1 | grep -o 'VSIX Version: [^ ]*' | cut -d' ' -f3 | xargs -I{} bash -c 'cd ../../ && ./node_modules/.bin/extest install-vsix -f amazon-q-vscode-{}.vsix -e packages/amazonq/test/e2e_new/amazonq/resources -s ~/.vscode-test-resources'", "test:ui:run": "npm run testCompile && ./node_modules/.bin/extest run-tests -s ~/.vscode-test-resources -e packages/amazonq/test/e2e_new/amazonq/resources packages/amazonq/dist/test/e2e_new/amazonq/tests/*.js", "test:ui": "npm run test:ui:prepare && npm run test:ui:install && npm run test:ui:run", From aff707d7b20916b0b1288b3074a4773bf0c8ee3c Mon Sep 17 00:00:00 2001 From: Suraj Reddy Date: Thu, 24 Jul 2025 12:17:15 -0400 Subject: [PATCH 02/10] changes to test plan and test doc --- docs/TESTPLAN.md | 2 +- docs/UI_E2E_Test.md | 139 ++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 140 insertions(+), 1 deletion(-) create mode 100644 docs/UI_E2E_Test.md diff --git a/docs/TESTPLAN.md b/docs/TESTPLAN.md index 56a299c9a5b..0db379a3ced 100644 --- a/docs/TESTPLAN.md +++ b/docs/TESTPLAN.md @@ -41,7 +41,7 @@ The test suite has the following categories of tests: - E2E Tests: **slow** tests - Live in `src/testE2E` - These tests are heavier than Integration tests. - - Some E2E tests have a more complicated architecture, described in [TEST_E2E](./TEST_E2E.md) + - Some E2E tests have a more complicated architecture, described in [UI_E2E_Test](./UI_E2E_Test.md) - Performance Tests: **slow** tests - Live in `src/testInteg/perf`. - A subset of integration tests focused on catching performance regressions. diff --git a/docs/UI_E2E_Test.md b/docs/UI_E2E_Test.md new file mode 100644 index 00000000000..8bdabe9ce80 --- /dev/null +++ b/docs/UI_E2E_Test.md @@ -0,0 +1,139 @@ +## UI Testing + +UI tests use [vscode-extension-tester](https://github.com/redhat-developer/vscode-extension-tester) to test the Amazon Q extension in a real VS Code environment. + +### Quick Start + +```bash +# Run complete UI test suite +npm run test:ui +``` + +### Individual Commands + +#### `test:ui:prepare` + +Downloads VS Code and ChromeDriver to `~/.vscode-test-resources` + +```bash +npm run test:ui:prepare +``` + +#### `test:ui:install` + +Packages the Amazon Q extension and installs it for testing + +```bash +npm run test:ui:install +``` + +- Runs `npm run package` in amazonq directory +- Extracts version from build output +- Installs VSIX using `extest install-vsix` +- Sets up extension in test environment + +#### `test:ui:run` + +Compiles TypeScript and runs UI tests + +```bash +npm run test:ui:run +``` + +- Compiles test files with `npm run testCompile` +- Runs tests matching `packages/amazonq/dist/test/e2e_new/amazonq/tests/*.js` + +### E2E New Test Suite + +Modern UI testing framework located at `packages/amazonq/test/e2e_new/amazonq/` + +#### Directory Structure + +``` +packages/amazonq/test/e2e_new/amazonq/ +├── tests/ # Test files +│ ├── chat.test.ts +│ ├── pinContext.test.ts +│ ├── quickActions.test.ts +│ └── switchModel.test.ts +├── helpers/ # Test helpers +│ ├── pinContextHelper.ts +│ ├── quickActionsHelper.ts +│ └── switchModelHelper.ts +├── utils/ # Testing utilities +│ ├── authUtils.ts +│ ├── cleanupUtils.ts +│ ├── generalUtils.ts +│ ├── setup.ts +│ └── testContext.ts +└── resources/ # Extension config + └── extensions.json +``` + +#### Test Categories + +- **Chat** - Amazon Q chat functionality +- **Pin Context** - Context pinning features +- **Quick Actions** - Quick action commands +- **Switch Model** - Model switching functionality + +### Writing New Tests + +1. Create test files in `packages/amazonq/test/e2e_new/amazonq/tests/` +2. Import utilities from `../utils/` +3. Use helpers from `../helpers/` +4. Follow existing patterns for setup/cleanup + +#### Example Test Structure + +```typescript +import { describe, it, before, after } from 'mocha' +import { setupTestContext, cleanupTestContext } from '../utils/setup' + +describe('Feature Tests', () => { + before(async () => { + await setupTestContext() + }) + + after(async () => { + await cleanupTestContext() + }) + + it('should test functionality', async () => { + // Test implementation + }) +}) +``` + +### Prerequisites + +- Node.js and npm +- VS Code extension development environment +- Chrome/Chromium browser + +### Troubleshooting + +#### Common Issues + +- **VS Code download fails**: Check internet connection, retry `test:ui:prepare` +- **Extension install fails**: Ensure packaging succeeds +- **Tests won't start**: Verify ChromeDriver/Chrome compatibility +- **Permission errors**: Check `~/.vscode-test-resources` permissions + +#### Debug and Reset + +```bash +# Debug mode +DEBUG=true npm run test:ui:run + +# Reset test environment +rm -rf ~/.vscode-test-resources +npm run test:ui:prepare +``` + +### Test Development Tips + +- Tests should be independent and run in any order +- Use existing utilities for common operations +- Clean up resources in `after` hooks +- Follow naming conventions from existing tests From f2296b82b422be5067bf138c0a1bd81d28c26b25 Mon Sep 17 00:00:00 2001 From: Suraj Reddy Date: Thu, 24 Jul 2025 12:28:47 -0400 Subject: [PATCH 03/10] changes to test doc --- docs/UI_E2E_Test.md | 11 +++-------- 1 file changed, 3 insertions(+), 8 deletions(-) diff --git a/docs/UI_E2E_Test.md b/docs/UI_E2E_Test.md index 8bdabe9ce80..e7165fcd68c 100644 --- a/docs/UI_E2E_Test.md +++ b/docs/UI_E2E_Test.md @@ -9,6 +9,8 @@ UI tests use [vscode-extension-tester](https://github.com/redhat-developer/vscod npm run test:ui ``` +Note: All of these commands must be run at the root level aws-toolkit-vscode directory. + ### Individual Commands #### `test:ui:prepare` @@ -43,10 +45,6 @@ npm run test:ui:run - Compiles test files with `npm run testCompile` - Runs tests matching `packages/amazonq/dist/test/e2e_new/amazonq/tests/*.js` -### E2E New Test Suite - -Modern UI testing framework located at `packages/amazonq/test/e2e_new/amazonq/` - #### Directory Structure ``` @@ -120,12 +118,9 @@ describe('Feature Tests', () => { - **Tests won't start**: Verify ChromeDriver/Chrome compatibility - **Permission errors**: Check `~/.vscode-test-resources` permissions -#### Debug and Reset +#### Reset ```bash -# Debug mode -DEBUG=true npm run test:ui:run - # Reset test environment rm -rf ~/.vscode-test-resources npm run test:ui:prepare From 1f5564de832b01db7008821754319a2676382d19 Mon Sep 17 00:00:00 2001 From: Suraj Reddy Date: Thu, 24 Jul 2025 12:30:23 -0400 Subject: [PATCH 04/10] fix package.json --- package.json | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/package.json b/package.json index d5936eec098..fa609457e26 100644 --- a/package.json +++ b/package.json @@ -28,7 +28,7 @@ "test": "npm run test -w packages/ --if-present", "testWeb": "npm run testWeb -w packages/ --if-present", "testE2E": "npm run testE2E -w packages/ --if-present", - "test:ui:setup": "./node_modules/.bin/extest get-vscode -s ~/.vscode-test-resources -n && extest get-chromedriver -s ~/.vscode-test-resources -n", + "test:ui:package": "./node_modules/.bin/extest get-vscode -s ~/.vscode-test-resources -n && extest get-chromedriver -s ~/.vscode-test-resources -n", "test:ui:install": "cd packages/amazonq && npm run package 2>&1 | grep -o 'VSIX Version: [^ ]*' | cut -d' ' -f3 | xargs -I{} bash -c 'cd ../../ && ./node_modules/.bin/extest install-vsix -f amazon-q-vscode-{}.vsix -e packages/amazonq/test/e2e_new/amazonq/resources -s ~/.vscode-test-resources'", "test:ui:run": "npm run testCompile && ./node_modules/.bin/extest run-tests -s ~/.vscode-test-resources -e packages/amazonq/test/e2e_new/amazonq/resources packages/amazonq/dist/test/e2e_new/amazonq/tests/*.js", "test:ui": "npm run test:ui:prepare && npm run test:ui:install && npm run test:ui:run", From ea19bb61a5661d1884f74c8beb33527948c67980 Mon Sep 17 00:00:00 2001 From: Suraj Reddy Date: Thu, 24 Jul 2025 12:30:47 -0400 Subject: [PATCH 05/10] fix package.json new --- package.json | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/package.json b/package.json index fa609457e26..fb9ee2fdea3 100644 --- a/package.json +++ b/package.json @@ -28,7 +28,7 @@ "test": "npm run test -w packages/ --if-present", "testWeb": "npm run testWeb -w packages/ --if-present", "testE2E": "npm run testE2E -w packages/ --if-present", - "test:ui:package": "./node_modules/.bin/extest get-vscode -s ~/.vscode-test-resources -n && extest get-chromedriver -s ~/.vscode-test-resources -n", + "test:ui:prepare": "./node_modules/.bin/extest get-vscode -s ~/.vscode-test-resources -n && extest get-chromedriver -s ~/.vscode-test-resources -n", "test:ui:install": "cd packages/amazonq && npm run package 2>&1 | grep -o 'VSIX Version: [^ ]*' | cut -d' ' -f3 | xargs -I{} bash -c 'cd ../../ && ./node_modules/.bin/extest install-vsix -f amazon-q-vscode-{}.vsix -e packages/amazonq/test/e2e_new/amazonq/resources -s ~/.vscode-test-resources'", "test:ui:run": "npm run testCompile && ./node_modules/.bin/extest run-tests -s ~/.vscode-test-resources -e packages/amazonq/test/e2e_new/amazonq/resources packages/amazonq/dist/test/e2e_new/amazonq/tests/*.js", "test:ui": "npm run test:ui:prepare && npm run test:ui:install && npm run test:ui:run", From 3930abf5492608fb3fe9c4f98efc592a773c447d Mon Sep 17 00:00:00 2001 From: Suraj Reddy Date: Thu, 24 Jul 2025 12:31:50 -0400 Subject: [PATCH 06/10] fix testplan.md --- docs/TESTPLAN.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/TESTPLAN.md b/docs/TESTPLAN.md index 0db379a3ced..f10a44efb42 100644 --- a/docs/TESTPLAN.md +++ b/docs/TESTPLAN.md @@ -39,7 +39,7 @@ The test suite has the following categories of tests: - Trigger VSCode commands and UI elements to test codepaths as from an actual user session, instead of invoking functions directly. - Do not use mocks. - E2E Tests: **slow** tests - - Live in `src/testE2E` + - Live in `packages/amazonq/test/e2e_new` - These tests are heavier than Integration tests. - Some E2E tests have a more complicated architecture, described in [UI_E2E_Test](./UI_E2E_Test.md) - Performance Tests: **slow** tests From aa2eadd7f855ecbeb6af164d3026023765539886 Mon Sep 17 00:00:00 2001 From: Suraj Reddy Date: Thu, 24 Jul 2025 14:45:50 -0400 Subject: [PATCH 07/10] fix testplan.md --- docs/TESTPLAN.md | 8 ++++++-- docs/UI_E2E_Test.md | 25 +++++-------------------- 2 files changed, 11 insertions(+), 22 deletions(-) diff --git a/docs/TESTPLAN.md b/docs/TESTPLAN.md index f10a44efb42..573d41b7144 100644 --- a/docs/TESTPLAN.md +++ b/docs/TESTPLAN.md @@ -39,9 +39,13 @@ The test suite has the following categories of tests: - Trigger VSCode commands and UI elements to test codepaths as from an actual user session, instead of invoking functions directly. - Do not use mocks. - E2E Tests: **slow** tests - - Live in `packages/amazonq/test/e2e_new` + - Live in `src/testE2E` - These tests are heavier than Integration tests. - - Some E2E tests have a more complicated architecture, described in [UI_E2E_Test](./UI_E2E_Test.md) + - Some E2E tests have a more complicated architecture, described in [TEST_E2E](./TEST_E2E.md) +- UI E2E Tests: **slow** tests + - Live in `packages/amazonq/test/e2e_new` + - These tests target the UI testing aspect in an E2E sense. + - Some E2E tests should be set up using pre-defined helpers and architecture, described in [UI_E2E_Test](./TUI_E2E_Test.md) - Performance Tests: **slow** tests - Live in `src/testInteg/perf`. - A subset of integration tests focused on catching performance regressions. diff --git a/docs/UI_E2E_Test.md b/docs/UI_E2E_Test.md index e7165fcd68c..423843f3207 100644 --- a/docs/UI_E2E_Test.md +++ b/docs/UI_E2E_Test.md @@ -45,27 +45,12 @@ npm run test:ui:run - Compiles test files with `npm run testCompile` - Runs tests matching `packages/amazonq/dist/test/e2e_new/amazonq/tests/*.js` -#### Directory Structure +#### Authentication -``` -packages/amazonq/test/e2e_new/amazonq/ -├── tests/ # Test files -│ ├── chat.test.ts -│ ├── pinContext.test.ts -│ ├── quickActions.test.ts -│ └── switchModel.test.ts -├── helpers/ # Test helpers -│ ├── pinContextHelper.ts -│ ├── quickActionsHelper.ts -│ └── switchModelHelper.ts -├── utils/ # Testing utilities -│ ├── authUtils.ts -│ ├── cleanupUtils.ts -│ ├── generalUtils.ts -│ ├── setup.ts -│ └── testContext.ts -└── resources/ # Extension config - └── extensions.json +Currently, authentication is not configured to be automatically logged into AmazonQ due to issues with Authentication credentials and the VET setup running in a new VSCode instance through a packaged VSIX. To bypass this for now (as of July 24th, 2025), you must click the approve/open button that redirects you to a browser in order for tests to be run in an authenticated environment at the start of a new. + +```bash +npm run test:ui:run ``` #### Test Categories From 4228163998b19070e8f92ecb5cbddd7d39ff1b47 Mon Sep 17 00:00:00 2001 From: Suraj Reddy Date: Thu, 24 Jul 2025 14:57:19 -0400 Subject: [PATCH 08/10] fix testplan.md --- docs/UI_E2E_Test.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/UI_E2E_Test.md b/docs/UI_E2E_Test.md index 423843f3207..44d3d0deb95 100644 --- a/docs/UI_E2E_Test.md +++ b/docs/UI_E2E_Test.md @@ -47,7 +47,7 @@ npm run test:ui:run #### Authentication -Currently, authentication is not configured to be automatically logged into AmazonQ due to issues with Authentication credentials and the VET setup running in a new VSCode instance through a packaged VSIX. To bypass this for now (as of July 24th, 2025), you must click the approve/open button that redirects you to a browser in order for tests to be run in an authenticated environment at the start of a new. +Currently, authentication is not configured to be automatically logged into AmazonQ. To bypass this for now (as of July 24th, 2025), you must click the approve/open button that redirects you to a browser in order for tests to be run in an authenticated environment at the start of a new. ```bash npm run test:ui:run From 5e511d59f34e6c73906c904fa32295ffec2da1a2 Mon Sep 17 00:00:00 2001 From: Suraj Reddy Date: Fri, 25 Jul 2025 13:08:36 -0400 Subject: [PATCH 09/10] fix testplan.md --- docs/UI_E2E_Test.md | 12 +++++++++--- 1 file changed, 9 insertions(+), 3 deletions(-) diff --git a/docs/UI_E2E_Test.md b/docs/UI_E2E_Test.md index 44d3d0deb95..7a3cd717587 100644 --- a/docs/UI_E2E_Test.md +++ b/docs/UI_E2E_Test.md @@ -15,7 +15,7 @@ Note: All of these commands must be run at the root level aws-toolkit-vscode dir #### `test:ui:prepare` -Downloads VS Code and ChromeDriver to `~/.vscode-test-resources` +Downloads VS Code and ChromeDriver to `~/.vscode-test-resources` in order to properly hook Selenium/VET to our new VSCode instance we launch. Without this, test cannot be run at all. ```bash npm run test:ui:prepare @@ -23,7 +23,7 @@ npm run test:ui:prepare #### `test:ui:install` -Packages the Amazon Q extension and installs it for testing +Packages the Amazon Q extension VSIX and installs it for testing. This reflects local changes within your immediate aws-toolkit-vscode directory. ```bash npm run test:ui:install @@ -36,7 +36,7 @@ npm run test:ui:install #### `test:ui:run` -Compiles TypeScript and runs UI tests +Compiles TypeScript and runs UI tests within the dist directory. You must run this everytime you make a new change to your tests in order to recompile them. ```bash npm run test:ui:run @@ -67,6 +67,12 @@ npm run test:ui:run 3. Use helpers from `../helpers/` 4. Follow existing patterns for setup/cleanup +#### Writing New Helpers + +1. Plan out abstractions/helpers after confirming current helpers are not viable +2. Create helper file in `../helpers/` +3. Following existing pattens for helper conventions and parameters + #### Example Test Structure ```typescript From ab2dbf7039b3b4d757f3992b01b49a01cfc013ad Mon Sep 17 00:00:00 2001 From: Suraj Reddy Date: Fri, 25 Jul 2025 13:15:13 -0400 Subject: [PATCH 10/10] nit: testplan.md --- docs/UI_E2E_Test.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/UI_E2E_Test.md b/docs/UI_E2E_Test.md index 7a3cd717587..d1ae3af35c4 100644 --- a/docs/UI_E2E_Test.md +++ b/docs/UI_E2E_Test.md @@ -36,7 +36,7 @@ npm run test:ui:install #### `test:ui:run` -Compiles TypeScript and runs UI tests within the dist directory. You must run this everytime you make a new change to your tests in order to recompile them. +Compiles TypeScript and runs UI tests within the dist directory. You must run this everytime you make a new change to your tests in order to recompile the tests. ```bash npm run test:ui:run