Skip to content

Commit 3b510af

Browse files
justinmk3justinmk3
authored
docs: TESTPLAN.md
1 parent 5a8bd62 commit 3b510af

File tree

1 file changed

+0
-0
lines changed

1 file changed

+0
-0
lines changed

‎docs/TESTPLAN.md‎

-1 Bytes

# Test Plan This document describes the testing architecture, goals and current status for the AWS Toolkit for VSCode. ## Test goals In order of priority: 1. Fast feedback from making a code change to seeing a result 2. High confidence in releasing the Toolkit 3. Avoid regressions Ratio of unit to integ tests: 90% unit tests, 10% system/acceptance tests. ## Test categories The test suite has two categories of tests: - Unit Tests: **fast** tests - Live in `src/test/` - The `vscode` API is available. - The Toolkit code is invoked as a library, not as an extension activated in VSCode's typical lifecycle. - Call functions and create objects directly. - May mock state where needed, though this is discouraged in favor of "fake" data/objects/files. - May use the filesystem. - Main property is that [the test is fast](https://pycon-2012-notes.readthedocs.io/en/latest/fast_tests_slow_tests.html). - Global state is shared across tests, thus there is a risk that later tests are polluted by earlier tests. - Integration Tests: **slow** tests - Live in `src/integrationTest/` - Use a full instance of VSCode with an activated instance of the extension. - Global state is shared across tests, thus there is a risk that later tests are polluted by earlier tests. - Trigger VSCode commands and UI elements to test codepaths as from an actual user session, instead of invoking functions directly. - Do not use mocks. ## Test files - `src/test/` : unit tests - `src/test/globalSetup.test.ts` : - defines global setup functions run before and after each test - defines global utility functions such as `getTestLogger()` - `src/integrationTest/` : integration tests - `src/test/testRunner.ts` : used by _both_ the unit tests and integration tests to discover tests, setup the test framework, and run tests. - `src/testFixtures/` : test data (sample projects, SAM templates, ...) - used by both unit and integration tests - `.vscode/launch.json` : defines VSCode launch configs useful for Toolkit developers, e.g. the `Extension Tests` config runs all tests in `src/test/`. ## How we test VSCode documentation describes an [extension testing](https://code.visualstudio.com/api/working-with-extensions/testing-extension) approach. The Toolkit codebase uses that approach, except some modifications/workarounds in `src/test/testRunner.ts`. - We use the [vscode-test](https://github.com/microsoft/vscode-test) package. - [Mocha](https://mochajs.org/) framework is used to write tests. - New code requires new tests. ## Testing Gaps - No handling of case where VSCode crashes. - Test harness hangs forever if VSCode hangs. - No end-to-end testing which make web requests to AWS. - Many failure modes (as opposed to the "happy path") are not tested. - No performance/benchmark regression tests. - No UI tests (to exercise webviews). - https://github.com/redhat-developer/vscode-extension-tester - Missing acceptance tests: - Connect to AWS - Fixed credentials and fixed credentials with assume roles - Testing AWS SDK client functionality is cumbersome, verbose, and low-yield. - Test code uses multiple “mocking” frameworks, which is confusing, error-prone, hard to onboard, and hard to use. - Coverage not counted for integ tests (because of unresolved tooling issue).

Test Plan

This document describes the testing architecture, goals and current status for the AWS Toolkit for VSCode.

Test goals

In order of priority:

  1. Fast feedback from making a code change to seeing a result
  2. High confidence in releasing the Toolkit
  3. Avoid regressions

Ratio of unit to integ tests: 90% unit tests, 10% system/acceptance tests.

Test categories

The test suite has two categories of tests:

  • Unit Tests: fast tests
    • Live in src/test/
    • The vscode API is available.
    • The Toolkit code is invoked as a library, not as an extension activated in VSCode's typical lifecycle.
    • Call functions and create objects directly.
    • May mock state where needed, though this is discouraged in favor of "fake" data/objects/files.
    • May use the filesystem.
    • Main property is that the test is fast.
    • Global state is shared across tests, thus there is a risk that later tests are polluted by earlier tests.
  • Integration Tests: slow tests
    • Live in src/integrationTest/
    • Use a full instance of VSCode with an activated instance of the extension.
    • Global state is shared across tests, thus there is a risk that later tests are polluted by earlier tests.
    • Trigger VSCode commands and UI elements to test codepaths as from an actual user session, instead of invoking functions directly.
    • Do not use mocks.

Test files

  • src/test/ : unit tests
    • src/test/globalSetup.test.ts :
      • defines global setup functions run before and after each test
      • defines global utility functions such as getTestLogger()
  • src/integrationTest/ : integration tests
  • src/test/testRunner.ts : used by both the unit tests and integration tests to discover tests, setup the test framework, and run tests.
  • src/testFixtures/ : test data (sample projects, SAM templates, ...)
    • used by both unit and integration tests
  • .vscode/launch.json : defines VSCode launch configs useful for Toolkit developers, e.g. the Extension Tests config runs all tests in src/test/.

How we test

VSCode documentation describes an extension testing approach. The Toolkit codebase uses that approach, except some modifications/workarounds in src/test/testRunner.ts.

  • We use the vscode-test package.
    • Mocha framework is used to write tests.
  • New code requires new tests.

Testing Gaps

  • No handling of case where VSCode crashes.
  • Test harness hangs forever if VSCode hangs.
  • No end-to-end testing which make web requests to AWS.
  • Many failure modes (as opposed to the "happy path") are not tested.
  • No performance/benchmark regression tests.
  • No UI tests (to exercise webviews).
  • Missing acceptance tests:
    • Connect to AWS
    • Fixed credentials and fixed credentials with assume roles
  • Testing AWS SDK client functionality is cumbersome, verbose, and low-yield.
  • Test code uses multiple “mocking” frameworks, which is confusing, error-prone, hard to onboard, and hard to use.
  • Coverage not counted for integ tests (because of unresolved tooling issue).

0 commit comments

Comments
 (0)