e2e

End-to-end tests

This package contains end-to-end tests for Perses written using Playwright.

Directory structure

• src
  • config - Playwright configurations live here.
    • base.playwright.config.ts - Base configuration with common settings.
    • ci.playwright.config.ts - Configuration used when running in continuous integration.
    • local.playwright.config.ts - Configuration used when running in local development.
  • fixtures - Playwright test fixtures live here. These are useful for managing common setup and teardown patterns across many tests. See pages for managing common page interactions and selectors.
  • pages - Page object models live here. These are classes that wrap selectors, page interactions, and other common patterns associated with a page. This helps reduce code duplication and improve test maintenance. In addition to pages, it also includes classes for large, complex page elements (e.g. panel editor) that benefit from their own wrappers.
  • tests - Playwright tests live here and are named following the pattern testName.spec.ts.

Running tests

Locally without screenshots

Tests are run during local development using the configuration in local.playwright.config.ts. The tests depend on the local development servers (backend and UI) to test against. By default, locally run tests will not take screenshots for visual testing.

• Start the backend server from the project root and disable authentication: ./scripts/api_backend_dev.sh --e2e.
• Change to the ui directory.
• Start the UI server: npm start
  • Important to ensure libraries are built since e2e imports from other Perses packages
• Run the end-to-end tests from the command line: npm run e2e
• (Optional) Run the end-to-end tests in debug mode to walk through a test step by step to debug issues: npm run e2e:debug.
• (Optional) Install Playwright VS Code extension. This extension has a lot of helpful tools for running tests, debugging, and creating selectors. Select local.playwright.config.ts as the profile to use when running locally.

Locally with UI

Example for running a single test with UI:

• ./scripts/api_backend_dev.sh --e2e
• cd ui; npm start
• cd ui/e2e; npx playwright test --config=src/config/local.playwright.config.ts --ui src/tests/statChartPanel.spec.ts

In CI

Tests are automatically run in CI using the workflow configured in e2e.yml with the configuration in ci.playwright.config.ts. In this case, Playwright automatically starts up and waits for the development servers.

You can test the CI configuration locally by running npm run e2e:ci.

Writing tests

Check out Playwright's documentation for general guidance on writing tests.

Test data

• The testing project in dev/data/project.json and associated dashboards in dev/data/dashboard.json should be used for end-to-end tests.
  • Give dashboards names that match the tests they are associated with for ease of debugging and maintenance.
  • Set modifiesDashboard: true in the test fixture configuration for tests that mutate dashboards to ensure these tests can be run in parallel. When this option is enabled, the fixture will automatically generate a duplicate dashboard for the test and clean it up when the test is finished running.
• The project does not currently have a data source that can be used to test consistent rendering in plugins (e.g. a line chart with time series data). You can work around this by mocking network requests. See mockQueryRangeRequests in DashboardPage for an example.

Guidelines

• Tests live in ui/e2e/src/tests and follow the testName.spec.ts naming scheme.
• Tests should be able to run in parallel. Do not write tests that depend on specific order.
• Tests should not be flaky! Flaky tests are frustrating, waste time, and lead to decreased trust in the entire test suite. Ask for help if you are having trouble writing a non-flaky test for specific functionality.
• Use Page Object Models to reduce code duplication and improve test maintenance.
• Use the recommended locators (Playwright's term for element selectors), when possible. These patterns are very similar to React Testing Library, which is used for the project's Jest tests.
• Use unique names for panel groups and panels for ease of writing tests.
• Recommend using usingtoBeCloseTo when asserting on inexact pixel values to allow a margin of error associated with padding/margins/etc. to avoid flaky tests. Note that the precision (second argument) is calculated as Math.pow(10, -precision) / 2 (e.g. precision 1 will allow for a difference of 0.05).

Troubleshooting

Tests failed in CI (GitHub Actions)

• Go to the failing action in GitHub.
• Follow the Playwright instructions for viewing test logs and viewing the html report.

Tests complaining about console errors

Tests that use the dashboardTest fixture check for console errors and fail tests when they are found because they are often a sign of a subtle bug we have not accounted for. When debugging these issues, it can be helpful to run the e2e tests with a headed browser (using headed mode or using the vcode extension with "show browser" checked) with the debugger console open. This will often provide you with a more detailed error message and a stack trace.