1
1
# Overview #
2
2
3
3
` xdist ` works by spawning one or more ** workers** , which are controlled
4
- by the ** master ** . Each ** worker** is responsible for performing
5
- a full test collection and afterwards running tests as dictated by the ** master ** .
4
+ by the ** controller ** . Each ** worker** is responsible for performing
5
+ a full test collection and afterwards running tests as dictated by the ** controller ** .
6
6
7
7
The execution flow is:
8
8
9
- 1 . ** master ** spawns one or more ** workers** at the beginning of
10
- the test session. The communication between ** master ** and ** worker** nodes makes use of
11
- [ execnet] ( http ://codespeak.net/execnet/) and its [ gateways] ( http ://codespeak.net/execnet/basics.html#gateways-bootstrapping-python-interpreters) .
9
+ 1 . ** controller ** spawns one or more ** workers** at the beginning of
10
+ the test session. The communication between ** controller ** and ** worker** nodes makes use of
11
+ [ execnet] ( https ://codespeak.net/execnet/) and its [ gateways] ( https ://codespeak.net/execnet/basics.html#gateways-bootstrapping-python-interpreters) .
12
12
The actual interpreters executing the code for the ** workers** might
13
13
be remote or local.
14
14
15
15
1 . Each ** worker** itself is a mini pytest runner. ** workers** at this
16
16
point perform a full test collection, sending back the collected
17
- test-ids back to the ** master ** which does not
17
+ test-ids back to the ** controller ** which does not
18
18
perform any collection itself.
19
19
20
- 1 . The ** master ** receives the result of the collection from all nodes.
21
- At this point the ** master ** performs some sanity check to ensure that
20
+ 1 . The ** controller ** receives the result of the collection from all nodes.
21
+ At this point the ** controller ** performs some sanity check to ensure that
22
22
all ** workers** collected the same tests (including order), bailing out otherwise.
23
23
If all is well, it converts the list of test-ids into a list of simple
24
24
indexes, where each index corresponds to the position of that test in the
25
25
original collection list. This works because all nodes have the same
26
- collection list, and saves bandwidth because the ** master ** can now tell
26
+ collection list, and saves bandwidth because the ** controller ** can now tell
27
27
one of the workers to just * execute test index 3* index of passing the
28
28
full test id.
29
29
30
- 1 . If ** dist-mode** is ** each** : the ** master ** just sends the full list
30
+ 1 . If ** dist-mode** is ** each** : the ** controller ** just sends the full list
31
31
of test indexes to each node at this moment.
32
32
33
- 1 . If ** dist-mode** is ** load** : the ** master ** takes around 25% of the
33
+ 1 . If ** dist-mode** is ** load** : the ** controller ** takes around 25% of the
34
34
tests and sends them one by one to each ** worker** in a round robin
35
35
fashion. The rest of the tests will be distributed later as ** workers**
36
36
finish tests (see below).
@@ -40,36 +40,36 @@ The execution flow is:
40
40
1 . ** workers** re-implement ` pytest_runtestloop ` : pytest's default implementation
41
41
basically loops over all collected items in the ` session ` object and executes
42
42
the ` pytest_runtest_protocol ` for each test item, but in xdist ** workers** sit idly
43
- waiting for ** master ** to send tests for execution. As tests are
43
+ waiting for ** controller ** to send tests for execution. As tests are
44
44
received by ** workers** , ` pytest_runtest_protocol ` is executed for each test.
45
45
Here it worth noting an implementation detail: ** workers** always must keep at
46
46
least one test item on their queue due to how the ` pytest_runtest_protocol(item, nextitem) `
47
47
hook is defined: in order to pass the ` nextitem ` to the hook, the worker must wait for more
48
- instructions from master before executing that remaining test. If it receives more tests,
48
+ instructions from controller before executing that remaining test. If it receives more tests,
49
49
then it can safely call ` pytest_runtest_protocol ` because it knows what the ` nextitem ` parameter will be.
50
50
If it receives a "shutdown" signal, then it can execute the hook passing ` nextitem ` as ` None ` .
51
51
52
52
1 . As tests are started and completed at the ** workers** , the results are sent
53
- back to the ** master ** , which then just forwards the results to
53
+ back to the ** controller ** , which then just forwards the results to
54
54
the appropriate pytest hooks: ` pytest_runtest_logstart ` and
55
55
` pytest_runtest_logreport ` . This way other plugins (for example ` junitxml ` )
56
- can work normally. The ** master ** (when in dist-mode ** load** )
56
+ can work normally. The ** controller ** (when in dist-mode ** load** )
57
57
decides to send more tests to a node when a test completes, using
58
58
some heuristics such as test durations and how many tests each ** worker**
59
59
still has to run.
60
60
61
- 1 . When the ** master ** has no more pending tests it will
61
+ 1 . When the ** controller ** has no more pending tests it will
62
62
send a "shutdown" signal to all ** workers** , which will then run their
63
63
remaining tests to completion and shut down. At this point the
64
- ** master ** will sit waiting for ** workers** to shut down, still
64
+ ** controller ** will sit waiting for ** workers** to shut down, still
65
65
processing events such as ` pytest_runtest_logreport ` .
66
66
67
67
## FAQ ##
68
68
69
69
> Why does each worker do its own collection, as opposed to having
70
- the master collect once and distribute from that collection to the workers?
70
+ the controller collect once and distribute from that collection to the workers?
71
71
72
- If collection was performed by master then it would have to
72
+ If collection was performed by controller then it would have to
73
73
serialize collected items to send them through the wire, as workers live in another process.
74
74
The problem is that test items are not easily (impossible?) to serialize, as they contain references to
75
75
the test functions, fixture managers, config objects, etc. Even if one manages to serialize it,
0 commit comments