Skip to content

Commit 18cd9b9

Browse files
authored
Merge pull request #725 from bluetech/docs-master
README,OVERVIEW: replace master -> controller
2 parents 10d8198 + 475d7c4 commit 18cd9b9

File tree

2 files changed

+29
-36
lines changed

2 files changed

+29
-36
lines changed

OVERVIEW.md

Lines changed: 19 additions & 19 deletions
Original file line numberDiff line numberDiff line change
@@ -1,36 +1,36 @@
11
# Overview #
22

33
`xdist` works by spawning one or more **workers**, which are controlled
4-
by the **master**. Each **worker** is responsible for performing
5-
a full test collection and afterwards running tests as dictated by the **master**.
4+
by the **controller**. Each **worker** is responsible for performing
5+
a full test collection and afterwards running tests as dictated by the **controller**.
66

77
The execution flow is:
88

9-
1. **master** spawns one or more **workers** at the beginning of
10-
the test session. The communication between **master** and **worker** nodes makes use of
11-
[execnet](http://codespeak.net/execnet/) and its [gateways](http://codespeak.net/execnet/basics.html#gateways-bootstrapping-python-interpreters).
9+
1. **controller** spawns one or more **workers** at the beginning of
10+
the test session. The communication between **controller** and **worker** nodes makes use of
11+
[execnet](https://codespeak.net/execnet/) and its [gateways](https://codespeak.net/execnet/basics.html#gateways-bootstrapping-python-interpreters).
1212
The actual interpreters executing the code for the **workers** might
1313
be remote or local.
1414

1515
1. Each **worker** itself is a mini pytest runner. **workers** at this
1616
point perform a full test collection, sending back the collected
17-
test-ids back to the **master** which does not
17+
test-ids back to the **controller** which does not
1818
perform any collection itself.
1919

20-
1. The **master** receives the result of the collection from all nodes.
21-
At this point the **master** performs some sanity check to ensure that
20+
1. The **controller** receives the result of the collection from all nodes.
21+
At this point the **controller** performs some sanity check to ensure that
2222
all **workers** collected the same tests (including order), bailing out otherwise.
2323
If all is well, it converts the list of test-ids into a list of simple
2424
indexes, where each index corresponds to the position of that test in the
2525
original collection list. This works because all nodes have the same
26-
collection list, and saves bandwidth because the **master** can now tell
26+
collection list, and saves bandwidth because the **controller** can now tell
2727
one of the workers to just *execute test index 3* index of passing the
2828
full test id.
2929

30-
1. If **dist-mode** is **each**: the **master** just sends the full list
30+
1. If **dist-mode** is **each**: the **controller** just sends the full list
3131
of test indexes to each node at this moment.
3232

33-
1. If **dist-mode** is **load**: the **master** takes around 25% of the
33+
1. If **dist-mode** is **load**: the **controller** takes around 25% of the
3434
tests and sends them one by one to each **worker** in a round robin
3535
fashion. The rest of the tests will be distributed later as **workers**
3636
finish tests (see below).
@@ -40,36 +40,36 @@ The execution flow is:
4040
1. **workers** re-implement `pytest_runtestloop`: pytest's default implementation
4141
basically loops over all collected items in the `session` object and executes
4242
the `pytest_runtest_protocol` for each test item, but in xdist **workers** sit idly
43-
waiting for **master** to send tests for execution. As tests are
43+
waiting for **controller** to send tests for execution. As tests are
4444
received by **workers**, `pytest_runtest_protocol` is executed for each test.
4545
Here it worth noting an implementation detail: **workers** always must keep at
4646
least one test item on their queue due to how the `pytest_runtest_protocol(item, nextitem)`
4747
hook is defined: in order to pass the `nextitem` to the hook, the worker must wait for more
48-
instructions from master before executing that remaining test. If it receives more tests,
48+
instructions from controller before executing that remaining test. If it receives more tests,
4949
then it can safely call `pytest_runtest_protocol` because it knows what the `nextitem` parameter will be.
5050
If it receives a "shutdown" signal, then it can execute the hook passing `nextitem` as `None`.
5151

5252
1. As tests are started and completed at the **workers**, the results are sent
53-
back to the **master**, which then just forwards the results to
53+
back to the **controller**, which then just forwards the results to
5454
the appropriate pytest hooks: `pytest_runtest_logstart` and
5555
`pytest_runtest_logreport`. This way other plugins (for example `junitxml`)
56-
can work normally. The **master** (when in dist-mode **load**)
56+
can work normally. The **controller** (when in dist-mode **load**)
5757
decides to send more tests to a node when a test completes, using
5858
some heuristics such as test durations and how many tests each **worker**
5959
still has to run.
6060

61-
1. When the **master** has no more pending tests it will
61+
1. When the **controller** has no more pending tests it will
6262
send a "shutdown" signal to all **workers**, which will then run their
6363
remaining tests to completion and shut down. At this point the
64-
**master** will sit waiting for **workers** to shut down, still
64+
**controller** will sit waiting for **workers** to shut down, still
6565
processing events such as `pytest_runtest_logreport`.
6666

6767
## FAQ ##
6868

6969
> Why does each worker do its own collection, as opposed to having
70-
the master collect once and distribute from that collection to the workers?
70+
the controller collect once and distribute from that collection to the workers?
7171

72-
If collection was performed by master then it would have to
72+
If collection was performed by controller then it would have to
7373
serialize collected items to send them through the wire, as workers live in another process.
7474
The problem is that test items are not easily (impossible?) to serialize, as they contain references to
7575
the test functions, fixture managers, config objects, etc. Even if one manages to serialize it,

README.rst

Lines changed: 10 additions & 17 deletions
Original file line numberDiff line numberDiff line change
@@ -243,11 +243,11 @@ environment this command will send each tests to all
243243
platforms - and report back failures from all platforms
244244
at once. The specifications strings use the `xspec syntax`_.
245245

246-
.. _`xspec syntax`: http://codespeak.net/execnet/basics.html#xspec
246+
.. _`xspec syntax`: https://codespeak.net/execnet/basics.html#xspec
247247

248248
.. _`socketserver.py`: https://raw.githubusercontent.com/pytest-dev/execnet/master/execnet/script/socketserver.py
249249

250-
.. _`execnet`: http://codespeak.net/execnet
250+
.. _`execnet`: https://codespeak.net/execnet
251251

252252
Identifying the worker process during a test
253253
--------------------------------------------
@@ -287,17 +287,6 @@ Since version 2.0, the following functions are also available in the ``xdist`` m
287287
:param request_or_session: the `pytest` `request` or `session` object
288288
"""
289289
290-
def is_xdist_master(request_or_session) -> bool:
291-
"""Return `True` if this is the xdist controller, `False` otherwise
292-
293-
Note: this method also returns `False` when distribution has not been
294-
activated at all.
295-
296-
deprecated alias for is_xdist_controller
297-
298-
:param request_or_session: the `pytest` `request` or `session` object
299-
"""
300-
301290
def is_xdist_controller(request_or_session) -> bool:
302291
"""Return `True` if this is the xdist controller, `False` otherwise
303292
@@ -307,11 +296,15 @@ Since version 2.0, the following functions are also available in the ``xdist`` m
307296
:param request_or_session: the `pytest` `request` or `session` object
308297
"""
309298
299+
def is_xdist_master(request_or_session) -> bool:
300+
"""Deprecated alias for is_xdist_controller."""
301+
310302
def get_xdist_worker_id(request_or_session) -> str:
311303
"""Return the id of the current worker ('gw0', 'gw1', etc) or 'master'
312304
if running on the controller node.
313305
314-
If not distributing tests (for example passing `-n0` or not passing `-n` at all) also return 'master'.
306+
If not distributing tests (for example passing `-n0` or not passing `-n` at all)
307+
also return 'master'.
315308
316309
:param request_or_session: the `pytest` `request` or `session` object
317310
"""
@@ -371,10 +364,10 @@ Additionally, during a test run, the following environment variable is defined:
371364

372365
* ``PYTEST_XDIST_TESTRUNUID``: the unique id of the test run.
373366

374-
Accessing ``sys.argv`` from the master node in workers
375-
------------------------------------------------------
367+
Accessing ``sys.argv`` from the controller node in workers
368+
----------------------------------------------------------
376369

377-
To access the ``sys.argv`` passed to the command-line of the master node, use
370+
To access the ``sys.argv`` passed to the command-line of the controller node, use
378371
``request.config.workerinput["mainargv"]``.
379372

380373

0 commit comments

Comments
 (0)