diff --git a/quality-checks.md b/quality-checks.md index 7de17875..20a0fd8c 100644 --- a/quality-checks.md +++ b/quality-checks.md @@ -42,7 +42,7 @@ We recommend tracking progress on an Engineering Quality dashboard, for example: | Quality check | Classification | Applicability | What it means | Why we care | Tolerances for green | Endorsed tools / configuration | Further details | | :----------------------------------------------------- | :-------------- | :------------ | :------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | :------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | :------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | :------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | :------------------------------------------------------------------------------------------------------------ | -| Accessibility test | Other | Universal | The practice of making applications usable by as many people as possible. | It is a regulatory requirement that our applications are accessible by as many people as possible. Catching accessibility failures up front is essential to maximise the accessibility | | | | +| Accessibility test | Other | Universal | The practice of making applications usable by as many people as possible. | It is a regulatory requirement that our applications are accessible by as many people as possible. Catching accessibility failures up front is essential to maximise the accessibility | | BrowserStack Automate

- Windows 11: Chrome, Firefox, Edge

- Windows 10: Chrome, Firefox, Edge

- OS X (2020+): Safari

- iOS (2020+): Safari, Chrome

- Android (2020+): Chrome | | | API / contract test | Functionality | Contextual | Check whether the API interface adheres to the agreed contract | Any API interface is an integration point with another component or a software systems. An extra care has to be taken to ensure compatibility and stability of that integration are maintained so that we don't break applications that depend on our APIs | Builds fail if any tests fail | Postman | [Automate Your API Tests with Postman](https://www.postman.com/use-cases/api-testing-automation/) | | Capacity test | Resilience | Contextual | Identify the application's breaking point in terms of an increasingly heavy load. Degradation may manifest itself as
* throughput bottlenecks
* increasing response times
* error rates rising | Without this test, we don't know how much load the application can handle before the application breaks or degrades | | | | | Chaos test | Resilience | Contextual | Cause failures in a system to test the resiliency of that system and its environment, and our ability to respond to failures | Give the team confidence that failures in a given environment will not lead to unplanned downtime or a negative user experience

Ensures that the team has visibility (e.g. dashboards and alerts) to be able to identify issues

Surface performance bottlenecks, weaknesses in system design and tipping points that aren't visible through other types of testing.


Help the team to understand their mean time to recovery (MTTR) and to build muscle memory & confidence for recovery activities | Regular (at least every couple of months) game days, and:

Builds fail if any test fails - note, these tests are slow, and are likely to be part of an infrequently-triggered (e.g. overnight) build

The tests cover whether the system self-heals, auto-scales, and alerts as expected | [aws-fis](tools/aws-fis/) | |