You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: ONBOARDING.md
+45-23Lines changed: 45 additions & 23 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -1,10 +1,26 @@
1
-
##So you want to write Complement tests
1
+
# So you want to write Complement tests
2
2
3
-
If you've not run Complement tests yet, please do so. This document will outline how Complement works and how you can add efficient tests and best practices for Complement itself.
3
+
Complement is a black box integration testing framework for Matrix homeservers.
4
4
5
-
### Architecture
5
+
This document will outline how Complement works and how you can add efficient tests and best practices for Complement itself. If you haven't run Complement tests yet, please see the [README](README.md) and start there!
6
+
7
+
## Terminology
8
+
9
+
*`Blueprint`: a human-readable outline for what should be done prior to a test (such as creating users, rooms, etc).
10
+
*`Deployment`: controls the lifetime of a Docker container (built from a `Blueprint`). It has functions on it for creating deployment-scoped structs such as Client-Server API clients for interacting with specific homeservers in the deployment.
11
+
12
+
## Architecture
13
+
14
+
Each Complement test runs one or more Docker containers for the homeserver(s) involved in the test. These containers are snapshots of the target homeserver at a particular state. The state of each container is determined by the `Blueprint` used. Client-Server and Server-Server API calls can then be made with assertions against the results.
15
+
16
+
In order to actually write a test, Complement needs to:
17
+
18
+
* Create `Deployments`, this is done by calling `deployment := Deploy(...)` (and can subsequently be killed via `deployment.Destroy(...)`).
19
+
* (Potentially) make API calls against the `Deployments`.
20
+
* Make assertions, this can be done via the standard Go testing mechanisms (e.g. `t.Fatalf`), but Complement also provides some helpers in the `must` and `match` packages.
21
+
22
+
For testing outbound federation, Complement implements a bare-bones Federation server for homeservers to talk to. Each test must explicitly declare the functionality of the homeserver. This is done using [functional options](https://dave.cheney.net/2014/10/17/functional-options-for-friendly-apis) and looks something like:
6
23
7
-
Complement runs a Docker container every time you call `deployment := Deploy(...)`, which gets killed on `deployment.Destroy(...)`. These containers are snapshots of the target homeserver at a particular state. The state is determined by the `Blueprint`, which is a human-readable outline for what should be done prior to the test (such as creating users, rooms, etc). Coming from Sytest, a `Blueprint` is similar to a `fixture`. A `deployment` has functions on it for creating deployment-scoped structs such as CS-API clients for interacting with specific Homeservers in the deployment. Assertions are done via the `must` and `match` packages. For testing outbound federation, Complement implements a bare-bones Federation server for homeservers to talk to. Unlike Sytest, you have to explicitly opt-in to attaching core functionality to the server so the reader can clearly see what is and is not being handled automatically. This is done using [functional options](https://dave.cheney.net/2014/10/17/functional-options-for-friendly-apis) and looks something like:
8
24
```go
9
25
// A federation server which handles serving up its own keys when requested,
10
26
// automatically accepts make_join and send_join requests and deals with
@@ -35,11 +51,11 @@ The high numbered ports are randomly chosen, and are for illustrative purposes o
35
51
```
36
52
The mapping of `hs1` to `localhost:port` combinations can be done automatically using a `docker.RoundTripper`.
37
53
38
-
###How do I...
54
+
## How do I...
39
55
40
-
Get a CS API client:
56
+
Get a Client-Server API client:
41
57
```go
42
-
// the user and hs name are from the blueprint
58
+
// the user and homeserver name are from the blueprint
43
59
// automatically maps localhost:12345 to the right container
####How should I name the test files / test functions?
86
+
### How should I name the test files / test functions?
71
87
72
88
Test files have to have `_test.go` else Go won't run the tests in that file. Other than that, there are no restrictions or naming convention.
73
89
If you are converting a sytest be sure to add a comment _anywhere_ in the source code which has the form:
@@ -87,11 +103,11 @@ Adding `// sytest: ...` means `sytest_coverage.go` will know the test is convert
87
103
when run! Use `go run sytest_coverage.go -v` to see the exact string to use, as they may be different to the one produced
88
104
by an actual sytest run due to parameterised tests.
89
105
90
-
####Should I always make a new blueprint for a test?
106
+
### Should I always make a new blueprint for a test?
91
107
92
108
Probably not. Blueprints are costly, and they should only be made if there is a strong case for plenty of reuse among tests. In the same way that we don't always add fixtures to sytest, we should be sparing with adding blueprints.
93
109
94
-
####How should I assert JSON objects?
110
+
### How should I assert JSON objects?
95
111
96
112
Use one of the matchers in the `match` package (which uses `gjson`) rather than `json.Unmarshal(...)` into a struct. There's a few reasons for this:
97
113
- Removes the temptation to use `gomatrixserverlib` structs.
@@ -100,14 +116,14 @@ Use one of the matchers in the `match` package (which uses `gjson`) rather than
100
116
101
117
If you want to extract data from objects, just use `gjson` directly.
102
118
103
-
####How should I assert HTTP requests/responses?
119
+
### How should I assert HTTP requests/responses?
104
120
105
121
Use the corresponding matcher in the `match` package. This allows you to be as specific or as lax as you like on your checks, and allows you to add JSON matchers on
106
122
the HTTP body.
107
123
108
-
####I want to run a bunch of tests in parallel, how do I do this?
124
+
### I want to run a bunch of tests in parallel, how do I do this?
109
125
110
-
If you're familiar with Go testing then you already know how. Add `t.Parallel()` to all tests which you want to run in parallel. For a good example of this, see `registration_test.go` which does:
126
+
This is done using the standard Go testing mechanisms. Add `t.Parallel()` to all tests which you want to run in parallel. For a good example of this, see `registration_test.go` which does:
111
127
```go
112
128
// This will block until the 2 subtests have completed
####I think Complement is doing something weird, can I get more logs?
154
+
### I think Complement is doing something weird, can I get more logs?
139
155
140
156
You can pass `COMPLEMENT_DEBUG=1` to add lots of debug logging. You can also do this via `os.Setenv("COMPLEMENT_DEBUG", "1")` before you make a deployment. This will add trace logging to the clients which logs full HTTP request/responses, amongst other debug info.
141
157
142
-
####How do I set up a bunch of stuff before the tests, e.g before each?
158
+
### How do I set up a bunch of stuff before the tests, e.g before each?
143
159
144
160
There is no syntactically pleasing way to do this. Create a separate function which returns a function. See https://stackoverflow.com/questions/42310088/setup-and-teardown-for-each-test-using-std-testing-package?rq=1
145
161
146
-
####How do I log messages in tests?
162
+
### How do I log messages in tests?
147
163
148
-
Standard Go testing here, use `t.Logf(...)` which will be logged only if the test fails or if `-v` is set. Note that you will not need to log HTTP requests performed using one of the built in deployment clients as they are already wrapped in loggers. For full HTTP logs, use `COMPLEMENT_DEBUG=1`.
164
+
This is done using standard Go testing mechanisms, use `t.Logf(...)` which will be logged only if the test fails or if `-v` is set. Note that you will not need to log HTTP requests performed using one of the built in deployment clients as they are already wrapped in loggers. For full HTTP logs, use `COMPLEMENT_DEBUG=1`.
149
165
150
-
####How do I skip a test?
166
+
### How do I skip a test?
151
167
152
168
Use one of `t.Skipf(...)` or `t.SkipNow()`.
153
169
154
-
####Why do we use `t.Errorf` sometimes and `t.Fatalf` other times?
170
+
### Why do we use `t.Errorf` sometimes and `t.Fatalf` other times?
155
171
156
172
Error will fail the test but continue execution, where Fatal will fail the test and quit. Use Fatal when continuing to run the test will result in programming errors (e.g nil exceptions).
157
173
158
-
####Why do I get the error "Error response from daemon: Conflict. The container name "/complement_rooms_state_alice.hs1_1" is already in use by container "c2d1d90c6cff7b7de2678b56c702bd1ff76ca72b930e8f2ca32eef3f2514ff3b". You have to remove (or rename) that container to be able to reuse that name."?
174
+
### Why do I get the error "Error response from daemon: Conflict. The container name "/complement_rooms_state_alice.hs1_1" is already in use by container "c2d1d90c6cff7b7de2678b56c702bd1ff76ca72b930e8f2ca32eef3f2514ff3b". You have to remove (or rename) that container to be able to reuse that name."?
159
175
160
176
The Docker daemon has a lag time between removing containers and them actually being removed. This means you cannot remove a container called 'foo' and immediately recreate it as 'foo'. To get around this, you need to use a different name. This probably means the namespace you have given the deployment is used by another test. Try changing it to something else e.g `Deploy(t, "rooms_state_2", b.BlueprintAlice.Name)`
161
177
162
-
####How do I run tests inside my IDE?
178
+
### How do I run tests inside my IDE?
163
179
164
180
For VSCode, add to `settings.json`:
165
181
```
@@ -171,3 +187,9 @@ For VSCode, add to `settings.json`:
171
187
For Goland:
172
188
* Under "Run"->"Edit Configurations..."->"Templates"->"Go Test", add `COMPLEMENT_BASE_IMAGE=complement-dendrite:latest`
173
189
* Then you can right-click on any test file or test case and "Run <testname>".
190
+
191
+
### What do I need to know if I'm coming from sytest?
192
+
193
+
Sytest has a concept of a `fixture` to configure the homeserver or test in a particular way, these are replaced with a `Blueprint` in Complement.
194
+
195
+
Unlike Sytest, each test must opt-in to attaching core functionality to the server so the reader can clearly see what is and is not being handled automatically.
Complement is a black box integration testing framework for Matrix homeservers.
6
6
7
-
8
-
#### Getting started
9
-
10
-
To get started developing, see https://github.com/matrix-org/complement/blob/master/ONBOARDING.md
11
-
12
-
If you're looking to run Complement against a local dev instance of Synapse, see [`matrix-org/synapse` -> `scripts-dev/complement.sh`](https://github.com/matrix-org/synapse/blob/develop/scripts-dev/complement.sh).
13
-
14
-
If you want to develop Complement tests while working on a local dev instance of Synapse, use the [`scripts-dev/complement.sh`](https://github.com/matrix-org/synapse/blob/develop/scripts-dev/complement.sh) script and set the `COMPLEMENT_DIR` environment variable to the filepath of your local Complement checkout. A regex that matches against test names can also be supplied as an argument to the script, i.e:
$ COMPLEMENT_BASE_IMAGE=complement-dendrite:latest go test -v ./tests
46
40
```
47
41
48
-
A full list of config options can be found [in the config file](./internal/config/config.go). All normal Go test config
49
-
options will work, so to just run 1 named test and include a timeout for the test run:
50
-
```
51
-
$ COMPLEMENT_BASE_IMAGE=complement-dendrite:latest go test -timeout 30s -run '^(TestOutboundFederationSend)$' -v ./tests
42
+
### Running against Synapse
43
+
44
+
If you're looking to run Complement against a local dev instance of Synapse, see [`matrix-org/synapse` -> `scripts-dev/complement.sh`](https://github.com/matrix-org/synapse/blob/develop/scripts-dev/complement.sh).
45
+
46
+
If you want to develop Complement tests while working on a local dev instance of Synapse, use the [`scripts-dev/complement.sh`](https://github.com/matrix-org/synapse/blob/develop/scripts-dev/complement.sh) script and set the `COMPLEMENT_DIR` environment variable to the filepath of your local Complement checkout. A regex that matches against test names can also be supplied as an argument to the script, i.e:
0 commit comments