|
| 1 | +# Testing in `wgpu` and `naga` |
| 2 | + |
| 3 | +There exist a large variety of tests within the `wgpu` repository |
| 4 | +to make sure we can easily test all the aspects of our libraries. |
| 5 | +This document serves as a guide to each class of test, and what |
| 6 | +they are used for. |
| 7 | + |
| 8 | +## Requirements |
| 9 | + |
| 10 | +The tests require that the [Vulkan SDK](https://vulkan.lunarg.com/sdk/home) |
| 11 | +is installed on the system and the `bin` folder of the SDK is in your `PATH`. |
| 12 | +Without this some tests may fail to run, or report false negatives. |
| 13 | + |
| 14 | +Additionally you require you run the tests with `cargo-nextest`. |
| 15 | +This is what our xtask calls. You can install it with `cargo install cargo-nextest`. |
| 16 | + |
| 17 | +## Run All Tests |
| 18 | + |
| 19 | +To run all tests, run `cargo xtask test` from the root of the repository. |
| 20 | + |
| 21 | +## Test Breakdown |
| 22 | + |
| 23 | +This is a table of contents, in the form of the repository's directory structure. |
| 24 | + |
| 25 | +- benches |
| 26 | + - [benches](#benchmark-tests) |
| 27 | +- examples |
| 28 | + - [features](#example-tests) |
| 29 | +- naga |
| 30 | + - tests |
| 31 | + - [example_wgsl](#naga-example-tests) |
| 32 | + - [snapshot](#naga-snapshot-tests) |
| 33 | + - [spirv-capabilities](#naga-spirv-capabilities-test) |
| 34 | + - [validation](#naga-validation) |
| 35 | + - [wgsl_errors](#naga-wgsl-error-tests) |
| 36 | +- player |
| 37 | + - [tests](#player-tests) |
| 38 | +- tests |
| 39 | + - [compile-tests](#wgpu-compile-tests) |
| 40 | + - [gpu-tests](#wgpu-gpu-tests) |
| 41 | + - [validation-tests](#wgpu-validation-tests) |
| 42 | + |
| 43 | +And where applicable [unit-tests](#unit-tests) |
| 44 | +are scatteredthroughout the codebase. |
| 45 | + |
| 46 | +## Benchmark Tests |
| 47 | + |
| 48 | +- Located in: `benches/benches` |
| 49 | +- Run with `cargo nextest run --bench wgpu-benchmark` |
| 50 | +- `wgpu` benchmarks for performance testing. |
| 51 | + |
| 52 | +These are benchmarks that test the performance of `wgpu` in various |
| 53 | +scenarios. When run as part of the test suite, they run a single |
| 54 | +iteration of each benchmark to ensure they continue to function. |
| 55 | + |
| 56 | +These tests only run on your system's default GPU. |
| 57 | + |
| 58 | +The benchmarks should be very careful to avoid doing any significant |
| 59 | +work (including connecting to a GPU) outside of the various `benchmark` |
| 60 | +`criterion` functions. If this is done, the benchmarks will take a long |
| 61 | +time to list available tests, slowing down the test suite. |
| 62 | + |
| 63 | +To run the benchmarks for benchmarking purposes, use `cargo bench`. |
| 64 | + |
| 65 | +## Example Tests |
| 66 | + |
| 67 | +- Located in: `examples/features` |
| 68 | +- Run with `cargo xtask test --bin wgpu-examples` |
| 69 | +- Uses a custom `#[gpu_test]` harness. |
| 70 | +- `wgpu` integration tests, with access to `wgpu_test` helpers. |
| 71 | + |
| 72 | +These tests validate that the examples are functioning correctly |
| 73 | +and do not have any regressions. They use the same harness as the |
| 74 | +[gpu tests](#wgpu-gpu-tests), see that section for more information |
| 75 | +on the harness. |
| 76 | + |
| 77 | +These tests use `nv-flip`'s image comparison through the wgpu |
| 78 | +example framework to validate that the images outputted by the |
| 79 | +examples are within tolerance of the expected output. |
| 80 | + |
| 81 | +Examples written in `examples/standalone` do not have tests, as |
| 82 | +they should be easy to copy into a standalone project. |
| 83 | + |
| 84 | +## `naga` Example Tests |
| 85 | + |
| 86 | +- Located in: `naga/tests/example_wgsl` |
| 87 | +- Run with `cargo nextest run --test naga-test example_wgsl` |
| 88 | + |
| 89 | +This simple test ensures that all wgsl files in the `examples` |
| 90 | +directory can be parsed by `naga`'s `wgsl` parser and validate correctly. |
| 91 | + |
| 92 | +## `naga` Snapshot Tests |
| 93 | + |
| 94 | +- Located in: `naga/tests/snapshot`, `naga/tests/in`, and `naga/tests/out` |
| 95 | +- Run with `cargo nextest run --test naga-test snapshots` |
| 96 | +- Data driven snapshot tests for `naga`'s input/output. |
| 97 | + |
| 98 | +These tests are snapshot tests for `naga`s parsers and code generators. |
| 99 | +There are inputs in `wgsl`, `spirv`, and `glsl`. There are outputs for |
| 100 | +`hlsl`, `spirv`, `wgsl`, `msl`, `glsl`, and naga's internal IR. The tests |
| 101 | +can be configured by a sidecar toml file of the same name as the input file. |
| 102 | + |
| 103 | +This is the goto tool for testing all kinds of codegen and parsing features. |
| 104 | + |
| 105 | +To avoid clutter we generally use the following pattern: |
| 106 | + |
| 107 | +- `wgsl` tests generate output to all backends. |
| 108 | +- `spirv`, `glsl` tests generate `wgsl` output |
| 109 | + |
| 110 | +This "butterfly" pattern ensures we don't need to test the |
| 111 | +full matrix of possibilities to get full coverage. |
| 112 | + |
| 113 | +While we do not run the results of the code generators, we do |
| 114 | +test that the generated code is valid. This is done by running |
| 115 | +`cargo xtask validate <backend>` in the `naga` directory and |
| 116 | +will use the respective tool to validate the generated code. |
| 117 | + |
| 118 | +## `naga` SPIR-V Capabilities Tests |
| 119 | + |
| 120 | +- Located in: `naga/tests/spirv_capabilities` |
| 121 | +- Run with `cargo nextest run --test naga-test spirv_capabilities` |
| 122 | +- Uses the standard `#[test]` harness. |
| 123 | + |
| 124 | +These tests convert the given wgsl snippet to spirv and |
| 125 | +then assert that the spirv has enabled the expected capabilities. |
| 126 | + |
| 127 | +## `naga` Validation Tests |
| 128 | + |
| 129 | +- Located in: `naga/tests/validation` |
| 130 | +- Run with `cargo nextest run --test naga-test validation` |
| 131 | + |
| 132 | +These are hand rolled tests against the naga's validator. |
| 133 | +If you don't need to test the validator with a custom module, |
| 134 | +and can use the `wgsl` frontend, you should put the test in |
| 135 | +the [wgsl errors](#naga-wgsl-error-tests) tests. |
| 136 | + |
| 137 | +## `naga` WGSL Error Tests |
| 138 | + |
| 139 | +- Located in: `naga/tests/wgsl_errors` |
| 140 | +- Run with `cargo nextest run --test naga-test wgsl_errors` |
| 141 | + |
| 142 | +These are tests for the error messages that the `wgsl` frontend |
| 143 | +produces. Additionally you can check that a given validation error |
| 144 | +is produced by the validator from a given `wgsl` snippet. |
| 145 | + |
| 146 | +## `player` Tests |
| 147 | + |
| 148 | +- Located in: `player/tests` |
| 149 | +- Run with `cargo nextest run --test player-test` |
| 150 | +- Data driven tests using the `player`'s replay system. |
| 151 | +- `wgpu` integration tests. |
| 152 | + |
| 153 | +These are soft-deprecated tests which are another way to write |
| 154 | +API tests. These use captures of the api calls and replay them |
| 155 | +to assert on the behavior. They are very difficult to write, and |
| 156 | +the trace capturing system is currently broken, so these |
| 157 | +tests exist, but you should not write new ones. |
| 158 | + |
| 159 | +These tests only run on your system's default GPU. |
| 160 | + |
| 161 | +## `wgpu` Compile Tests |
| 162 | + |
| 163 | +- Located in: `tests/compile-tests` |
| 164 | +- Run with `cargo nextest run --test wgpu-compile-test` |
| 165 | +- `trybuild` tests of all rust files in `tests/compile-tests/fail` directory. |
| 166 | + |
| 167 | +These use the `trybuild` crate to test a few scenarios where |
| 168 | +the `wgpu` crate is expected to fail to compile. This mainly |
| 169 | +revolves around ensuring lifetimes are properly handled when |
| 170 | +dropping passes, etc. |
| 171 | + |
| 172 | +## `wgpu` GPU Tests |
| 173 | + |
| 174 | +- Located in: `tests/gpu-tests` |
| 175 | +- Run with `cargo xtask test --test wgpu-gpu-test` |
| 176 | +- Uses a custom `#[gpu_test]` harness. |
| 177 | +- `wgpu` integration tests, with access to `wgpu_test` helpers. |
| 178 | + |
| 179 | +These tests use a custom harness to run each test on all GPUs |
| 180 | +available on the system. They are general integration tests |
| 181 | +that write code against the normal `wgpu` API and assert on the behavior. |
| 182 | + |
| 183 | +These tests are useful to check the runtime behavior of a program, |
| 184 | +validate that there are no validation errors coming from the |
| 185 | +`vulkan`/`dx12`/`metal` validation layers, and ensure behavior |
| 186 | +is the same across GPUs. If the test does not need to run on a |
| 187 | +real GPU, it should be in the [validation tests](#wgpu-validation-tests) instead. |
| 188 | + |
| 189 | +There is a special parameter system that deals with if a GPU |
| 190 | +can support the given test, and dealing with expectation |
| 191 | +management for tests that are expected to fail due to driver or wgpu bugs. |
| 192 | + |
| 193 | +Normal `#[test]`s will not be found in this test crate, as we use a custom harness. |
| 194 | + |
| 195 | +See also the [example tests](#example-tests) for additional GPU tests. |
| 196 | + |
| 197 | +## `wgpu` Validation Tests |
| 198 | + |
| 199 | +- Located in: `tests/validation-tests` |
| 200 | +- Run with `cargo nextest run --test wgpu-validation-test` |
| 201 | +- Use the standard `#[test]` harness. |
| 202 | +- `wgpu` integration tests, with access to `wgpu_test` helpers. |
| 203 | + |
| 204 | +These tests are focused on testing the validation inside of `wgpu-core`. |
| 205 | +They are written against the `wgpu` API, but are targeting a special `noop` |
| 206 | +backend which does not connect to a real GPU. |
| 207 | + |
| 208 | +This is significantly faster and simpler than running on real hardware, |
| 209 | +and allows any validation logic to be checked, even if real hardware |
| 210 | +does not support those features. |
| 211 | + |
| 212 | +## Unit Tests |
| 213 | + |
| 214 | +- Located throughout the codebase. |
| 215 | +- Run with `cargo nextest test -p <package>` |
| 216 | +- Standard `#[test]`s. |
| 217 | + |
| 218 | +Throughout the codebase we have standard `#[test]`s that test individual |
| 219 | +functions or small parts of the codebase. These don't run on the gpu. |
| 220 | + |
0 commit comments