Testing improvements

[jf-x-dev]

Current approach to automated testing

1. A dev adds a feature and write automated RSpec Capybara tests for it.
2. The feature is then released onto qa and test where it is manually tested.
3. Automated Playwright tests are created if it would be useful to have them. These tests generate reports with steps and screenshots (currently only used for debugging), which are currently hosted on GitHub pages.

Pros

• Devs get fast feedback from the RSpec Capybara tests which run as part of the CI build

Cons

• Automated tests are written twice, once by the dev and again by the automated tester
• The reports from tests could speed up manual testing, but they are not available to the manual testers when they start testing tickets as the rspec tests do not generate reports and the automated tests usually are not started on until manual testing has already begun.
• There are two separate test suites for the same application. It would be nicer to have just one suite.

Alternative flows (single repository)

Some alternate solutions have been considered which may speed up the assurance process:

Migrate all functional tests to the testing repository

Devs would write Playwright tests in the testing repository instead of rspec tests in the Mavis repository.

Pros

• One suite with all functional tests
• Reports could be generated for the tests written by devs, which would allow manual testers to have reports when they start testing.

Cons

• Dev work is split between multiple repositories - a lot of extra overhead
• These tests would be very slow compared to the current Capybara suite.
  • The Rspec Capybara tests use the Rack driver which is pure Ruby, headless, and very fast. There are 4266 tests which run in 15 minutes (though these are not all functional tests). This also seems to be without any parallelization. Playwright is not as fast. It runs 127 functional tests in 12 minutes on qa, while running up to 4 tests in parallel. However, these are longer e2e tests, and this would take less time on an environment with less patients.
    • There are around 300 Capybara feature tests, and these still finish faster with no parallelization than the Playwright suite
  • A combined test suite of around 200-300 Playwright tests could start taking over 30 minutes to run, which would be pretty long for a CI/CD pipeline. Perhaps only 'crucial' tests would be run as part of CI/CD?
• Devs need to familiarize themselves with Playwright (relatively minor)
• No guarantee that the reports being available to manual testers would speed up any part of the process
  • They may still want to verify everything that appears in each report
  • The Capybara tests cover basic cases, which can often be verified very quickly manually. Issues that are discovered by assurance tend to be edge cases etc. that would not have been covered by these tests.
• The migration itself would require translating many tests from rspec (Capybara) to Playwright, configuring CI to retrieve the tests from the testing repo to run them against Mavis, etc. Would be lengthy - feels like it should only be done if it's a real improvement.
• Not a common approach. Many projects have separate suites for unit tests and E2E tests.

Migrate all the Playwright E2E tests to the main Mavis repository

Migrate all the Playwright E2E tests to the main Mavis repository, kept separate from the unit tests.

Pros

• All code is together - could easily run the E2E tests automatically after deployments
  • Also easy for devs to run the E2E tests and update them themselves
• We can still use Playwright to run tests and Allure for reporting
  • There is a maintained Ruby Playwright client and Capybara Playwright driver. These are unofficial however.
  • Reports could be available to manual testers when they start testing (if the E2E tests cover the right functionality).
• This is a common approach. For example, discourse has an E2E test suite in spec/system which runs using the unofficial Capybara Playwright driver. GitLab has an E2E test suite in qa/ which runs using Selenium.

Cons

• The combined repository size would be large
• Automated testers need to be comfortable with Ruby
• If Mavis was migrated to Python in the future (?), this would need to be undone (see below for more "What if?"s)

Middle ground

• Try generating reports from the Rspec Capybara tests which could be used for assurance?
  • I tried switching out the Capybara driver from Rack to Cuprite/Selenium, which have headed drivers and in theory could take screenshots and generate reports. It doesn't work out of the box however, only making it to the login page and failing to get past the mock CIS2 authentication.
  • There's a nice tool for taking screenshots of what Rack sees during rspec tests. It works, but you only see the raw HTML of the page with none of the styling or JS. Neat but not practical

Gherkin with different adapters

Migrate all functional tests to use Gherkin, and provide both Capybara and Playwright implementations for each test.

Pros

• Would enable us to have a single test suite that could be executed using Rack (for development / speed) and Playwright (for assurance, reports etc.)

Cons

• Would require two migrations and extra maintenance effort compared to one Ruby/Playwright suite.

Considerations ahead of a Python migration

Suppose Mavis is migrated to Python in the future. Let's also suppose further that we are using something like Django:

• We would still have/want two different suites:
  • One (most likely Pytest, but possibly unittest) using Django's headless test client which is much faster than Playwright.
  • Another using Playwright, for E2E/assurance purposes
• In this case we would not be able to "swap out" the unit test driver to use a headed driver - so we could not generate reports from the unit tests
• Then there are the three options which were discussed above:
  • All tests could live in the main Mavis repo
  • All tests could live in the testing repo (with increased development overhead)
  • ...or the E2E tests could live in their own repo, which is where we are now...

Other testing improvements

• Add automated email/SMS testing
  • Looks like we'd need to have an account that can both
    • Receive emails/SMS from Notify using a team key
    • Can be checked using Python (so not requiring NHS 2FA?)
  • For SMS this seems doable (at a cost), unsure about email
• Allow mock CIS2 authentication for automated tests
  • Documentation
• Use real, anonymized data for testing instead of generated data
  • Anonymized patient data could be setup in a new table in the qa database and a new endpoint could be added that retrieves this information.
  • Would increase test coverage and allow other test data quirks to (hopefully) be spotted in tests before they appear in production
• More focus on non-functional testing
  • More on performance testing?
    • We currently use JMeter but this makes collaboration/review very hard. It may make sense to consider other tools e.g. K6)
  • Security testing
    • We could set up an action to regularly run an OWASP Zap against Mavis (probably on QA?)
    • I tried this and it seems to highlight a few areas of interest. I think it'd be worth it.
  • Accessibility testing
    • Playwright works very nicely with axe, this could be fairly simple