|
1 | | -# Integration test |
| 1 | +# Testing |
2 | 2 |
|
3 | | -Run integration tests with `make test-integration`. |
| 3 | +Spin is tested through several classes of tests that aim to build confidence in the correctness of Spin from multiple angles: |
4 | 4 |
|
5 | | -# E2E tests for spin |
| 5 | +* Unit tests |
| 6 | +* Runtime tests |
| 7 | +* Integration tests |
6 | 8 |
|
7 | | -The goal of these tests is to ensure that spin continues to work with existing apps/examples/templates and there are no regressions introduced as we add more functionality or refactor existing code. |
| 9 | +## Unit tests |
8 | 10 |
|
9 | | -## How to run e2e tests |
| 11 | +Spin is composed of a many different crates that each test their individual functionality using normal Rust unit tests. You can run these tests like you would for any Rust based project using Cargo: |
10 | 12 |
|
11 | | -```sh |
12 | | -## go to root dir of the project, e2e-tests.Dockerfile is located there |
13 | | -make test-spin-up |
14 | | -``` |
15 | | - |
16 | | -The e2e-tests looks for `spin` binary in following folders (in that order): |
17 | | - |
18 | | -- target/debug |
19 | | -- target/release |
20 | | -- $HOME/.cargo/bin |
21 | | -- $PATH |
22 | | - |
23 | | -## How to use `spin` binary with your local changes |
24 | | - |
25 | | -By default, tests use the canary build of `spin` downloaded at the docker image creation time. If you want to test it with your changes, you can use the environment variable E2E_BUILD_SPIN=true |
26 | | - |
27 | | -```sh |
28 | | -E2E_BUILD_SPIN=true make test-spin-up |
| 13 | +```bash |
| 14 | +cargo test -p $CRATE_NAME |
29 | 15 | ``` |
30 | 16 |
|
31 | | -## Important files and their function |
| 17 | +## Runtime tests |
32 | 18 |
|
33 | | -* `crates/e2e-testing` - All the test framework/utilities that are required for `e2e-tests` |
34 | | -* `tests/testcases/mod.rs` - All the testcase definitions should be added here. |
35 | | -* `tests/spinup_tests.rs` - All tests that we want to run with `Spin Up` should be added here |
36 | | -* `tests/testcases/<dirs>` - The testcases which require corresponding `spin app` pre-created should be added here |
| 19 | +Runtime tests are meant to test Spin compliant runtimes to ensure that they conform to expected runtime behavior. |
37 | 20 |
|
38 | | -## Key concepts and types |
| 21 | +The runtime tests are handled through the `runtime-tests` support crate. See the README there for more information. |
39 | 22 |
|
40 | | -### [trait Controller](../crates/e2e-testing/src/controller.rs#L12) |
| 23 | +You can run runtime tests like so: |
41 | 24 |
|
42 | | -This defines a trait which can be implemented by different deployment models (e.g. [`spin up`](../crates/e2e-testing/src/spin_controller.rs#L15) or `Fermyon Cloud`). Using this we can reuse the same testcases, which can be executed against these different deployment models (e.g. they may choose to have different way to start/stop the spin apps.) |
43 | | - |
44 | | -### [TestCase](../crates/e2e-testing/src/testcase.rs#L22) |
45 | | - |
46 | | -This helps us configure the different scenarios/steps which are required for running a specific test app. For example, [TestCase.trigger_type](../crates/e2e-testing/src/testcase.rs#L42) indicates the `trigger` a particular test app uses and [TestCase.plugins](../crates/e2e-testing/src/testcase.rs#L53) indicates which prerequisite [`plugins`](https://developer.fermyon.com/spin/plugin-authoring) are required to run this test app. |
47 | | - |
48 | | -Additionally, [TestCase.assertions](../crates/e2e-testing/src/testcase.rs#L68) is a dynamic function to run testcase-specific assertions. During execution, the assertions function is called with the input parameters of `AppMetadata` as well as handles to the `stdlog/stderr` logs streams. The idea is that inside this function you would trigger your app (`http` or `redis` etc) and then verify if the trigger was successful by verifying either `http response` or `stdout/stderr`. |
49 | | - |
50 | | -A basic assertion function for `http-trigger` |
51 | | - |
52 | | -```rust |
53 | | -async fn checks( |
54 | | - metadata: AppMetadata, |
55 | | - _: Option<BufReader<ChildStdout>>, |
56 | | - _: Option<BufReader<ChildStderr>>, |
57 | | - ) -> Result<()> { |
58 | | - assert_http_response(metadata.base.as_str(), 200, &[], Some("Hello Fermyon!\n")).await |
59 | | -} |
| 25 | +```bash |
| 26 | +cargo test runtime_tests -F e2e-tests |
60 | 27 | ``` |
61 | 28 |
|
62 | | -and for `redis-trigger` |
63 | | - |
64 | | -```rust |
65 | | -async fn checks( |
66 | | - _: AppMetadata, |
67 | | - _: Option<BufReader<ChildStdout>>, |
68 | | - stderr_stream: Option<BufReader<ChildStderr>>, |
69 | | -) -> Result<()> { |
70 | | - //TODO: wait for spin up to be ready dynamically |
71 | | - sleep(Duration::from_secs(10)).await; |
72 | | - |
73 | | - utils::run(vec!["redis-cli", "-u", "redis://redis:6379", "PUBLISH", "redis-go-works-channel", "msg-from-go-channel",], None, None)?; |
74 | | - |
75 | | - let stderr = utils::get_output_from_stderr(stderr_stream, Duration::from_secs(5)).await?; |
76 | | - let expected_logs = vec!["Payload::::", "msg-from-go-channel"]; |
77 | | - |
78 | | - assert!(expected_logs.iter().all(|item| stderr.contains(&item.to_string()))); |
79 | | - |
80 | | - Ok(()) |
81 | | -} |
82 | | - |
83 | | -``` |
84 | | - |
85 | | -### [AppInstance](../crates/e2e-testing/src/controller.rs#L34) |
86 | | - |
87 | | -This object holds the information about the app running as part of the testcase, e.g. it has details of routes for verifying `http trigger`-based apps and has handles to `stdout/stderr` log streams to assert the log messages printed by `redis trigger` templates. |
88 | | - |
89 | | -It also holds a handle to the OS process which was started during the testcase execution. The testcase stops the process after the execution completes using the `controller.stop` method. This gives the control of how an app is run/stopped to the implementer of the specific deployment models. |
90 | | - |
91 | | -## Writing new testcase |
92 | | - |
93 | | -### using pre-existing code scenario |
94 | | - |
95 | | -Let us say we want to add a testcase `foo-env-test` for a specific scenario for which you have already created a `spin app`. Following steps are required to make this happen |
96 | | - |
97 | | -1. You can add the existing app code `tests/testcases/foo-env-test`. |
98 | | -2. Add a new function `pub async fn foo_env_works(controller: &dyn Controller)` in `tests/testcases/mod.rs` as follows: |
99 | | - |
100 | | -```rust |
| 29 | +# Integration tests |
101 | 30 |
|
102 | | -pub async fn foo_env_works(controller: &dyn Controller) { |
103 | | - async fn checks( |
104 | | - metadata: AppMetadata, |
105 | | - _: Option<BufReader<ChildStdout>>, |
106 | | - _: Option<BufReader<ChildStderr>>,) -> Result<()> { |
107 | | - assert_http_response( |
108 | | - get_url(metadata.base.as_str(), "/echo").as_str(), |
109 | | - 200, |
110 | | - &[], |
111 | | - Some("foo-env"), |
112 | | - )?; |
| 31 | +Integration tests are meant to test anything that cannot be tested through some other testing mechanism usually because the scenario under test is complicated and involves the interaction between many different subsystems. Historically, integration tests have been a landing pad for experimentation around testing that have eventually been turned into their own class of tests. |
113 | 32 |
|
114 | | - Ok(()) |
115 | | - } |
| 33 | +Currently, integration tests are split between two different modules that will soon be combined into one: `integration_tests` and `spinup_tests`. |
116 | 34 |
|
117 | | - let tc = TestCaseBuilder::default() |
118 | | - .name("foo-env-test".to_string()) |
119 | | - //the appname should be same as dir where this app exists |
120 | | - .appname(Some("foo-env-test".to_string())) |
121 | | - .template(None) |
122 | | - .assertions( |
123 | | - |metadata: AppMetadata, |
124 | | - stdout_stream: Option<BufReader<ChildStdout>>, |
125 | | - stderr_stream: Option<BufReader<ChildStderr>>| { |
126 | | - Box::pin(checks(metadata, stdout_stream, stderr_stream)) |
127 | | - }, |
128 | | - ) |
129 | | - .build() |
130 | | - .unwrap(); |
131 | | - |
132 | | - tc.run(controller).await.unwrap() |
133 | | -} |
134 | | - |
135 | | -``` |
136 | | - |
137 | | -3. Add the testcase to `tests/spinup_tests.rs` as follows: |
138 | | - |
139 | | - |
140 | | -```rust |
141 | | -#[tokio::test] |
142 | | -async fn foo_env_works() { |
143 | | - testcases::foo_env_works(CONTROLLER).await |
144 | | -} |
145 | | -``` |
146 | | - |
147 | | -4. Run the tests locally to verify |
148 | | - |
149 | | -``` |
150 | | -## go to root dir of the project, e2e-tests.Dockerfile is located there |
151 | | -docker build -t spin-e2e-tests -f e2e-tests.Dockerfile . |
152 | | -docker compose -f e2e-tests-docker-compose.yml run e2e-tests |
153 | | -``` |
154 | | - |
155 | | -### using a template |
156 | | ---------------------- |
157 | | - |
158 | | -Let us say we want to add a testcase for a new template `foo-bar`. Following steps are required to make this happen |
159 | | - |
160 | | -1. Write a function `pub async fn foo_bar_works(controller: &dyn Controller)` as follows: |
161 | | - |
162 | | -```rust |
163 | | -pub async fn foo_bar_works(controller: &dyn Controller) { |
164 | | - async fn checks(metadata: AppMetadata, |
165 | | - _: Option<BufReader<ChildStdout>>, |
166 | | - _: Option<BufReader<ChildStderr>>,) -> Result<()> { |
167 | | - return assert_http_response( |
168 | | - metadata.base.as_str(), |
169 | | - 200, |
170 | | - &[], |
171 | | - Some("Hello foo-bar!\n"), |
172 | | - ); |
173 | | - } |
174 | | - |
175 | | - let tc = TestCaseBuilder::default() |
176 | | - .name("foo-bar template".to_string()) |
177 | | - // for template based tests, appname is generated on the fly |
178 | | - .appname(None) |
179 | | - // this should be the name of the template used to |
180 | | - // create new app using `spin new <template-name> <app-name> |
181 | | - .template("foo-bar".to_string()) |
182 | | - .assertions( |
183 | | - |metadata: AppMetadata, |
184 | | - stdout_stream: Option<BufReader<ChildStdout>>, |
185 | | - stderr_stream: Option<BufReader<ChildStderr>>| { |
186 | | - Box::pin(checks(metadata, stdout_stream, stderr_stream)) |
187 | | - }, |
188 | | - ) |
189 | | - .build() |
190 | | - .unwrap(); |
191 | | - |
192 | | - tc.run(controller).await.unwrap(); |
193 | | -} |
194 | | - |
195 | | -``` |
196 | | - |
197 | | - |
198 | | -2. Add the testcase to `tests/spinup_tests.rs` as follows: |
199 | | - |
200 | | -```rust |
201 | | -#[tokio::test] |
202 | | -async fn foo_bar_works() { |
203 | | - testcases::foo_bar_works(CONTROLLER).await |
204 | | -} |
205 | | -``` |
206 | | - |
207 | | -3. Run the tests locally to verify |
208 | | - |
209 | | -```sh |
210 | | -## go to root dir of the project, e2e-tests.Dockerfile is located there |
211 | | -docker build -t spin-e2e-tests -f e2e-tests.Dockerfile . |
212 | | -docker compose -f e2e-tests-docker-compose.yml run e2e-tests |
| 35 | +You can run integration tests like so: |
| 36 | +```bash |
| 37 | +make test-integration |
| 38 | +make test-spin-up |
213 | 39 | ``` |
0 commit comments