@@ -10,23 +10,26 @@ want to execute the tests in a specific order to each other, you can use
10
10
``pytest-order ``. If you want to skip or xfail tests dependent on other
11
11
tests you can use ``pytest-dependency ``. If you want to have both behaviors
12
12
combined, you can use both plugins together with the
13
- option :ref: `order-dependencies `, described above . As mentioned before, only
13
+ option :ref: `order-dependencies `. As mentioned before, only
14
14
static ``dependency `` markers are considered for ordering.
15
15
16
16
Usage with other ordering plugins
17
17
---------------------------------
18
18
There is a number of other pytest plugins that change the order in which tests
19
- are executed, the most widely known probably being
20
- `pytest-randomly `_, which
21
- executes tests in a random order to avoid unknown test dependencies.
22
- ``pytest-order `` should still work with these as long as it is executed
23
- *after * the other plugins (which it should by default, except if you use
24
- the option :ref: `indulgent-ordering `).
25
- The marked tests still shall be ordered correctly, but the order of the
26
- unordered tests will change, depending on the order the tests have been
27
- after these other plugins have reordered them.
28
- For example, if you have installed ``pytest-randomly ``, and run the
29
- following tests:
19
+ are executed. Most of these plugins only reorder tests if given some command
20
+ line options or in the presence of specific markers (as does ``pytest-order ``).
21
+ These plugins will not have any effect on ``pytest-order `` if not actively
22
+ used. A few plugins always do reordering, most notably ``pytest-randomly ``.
23
+
24
+ `pytest-randomly `_
25
+ ~~~~~~~~~~~~~~~~~~
26
+ This plugin executes tests in a random order to avoid unknown test
27
+ dependencies. The plugin is effective if installed and not actively disabled.
28
+ ``pytest-order `` should still work correctly, because it is executed after
29
+ ``pytest-randomly `` (except if you use the option
30
+ :ref: `indulgent-ordering `). The marked tests will be ordered correctly, but
31
+ the order of the unordered tests will change unpredictably. For example, if
32
+ you run the following tests:
30
33
31
34
.. code :: python
32
35
64
67
65
68
The same is true for relative ordering. The tests will be correctly ordered
66
69
before and after the tests as configured, but all other tests will be in an
67
- arbitrary order.
68
-
69
- Note that it does not make much sense to use ordering plugins together that
70
- have a similar goal as ``pytest-order ``, as for example ``pytest-ordering ``.
71
- As mentioned, both plugins can co-exist without problems due to the
72
- different marker names, but using markers of both plugins in the same test
70
+ arbitrary order. If you want the tests to execute in a random order except the
71
+ ones that you know depend on each other, this can be a good strategy to
72
+ prevent test dependencies.
73
+
74
+ If you don't want the installed ``pytest-randomly `` be effective in a test
75
+ run, you can exclude it explicitly on the command line::
76
+
77
+ python -m pytest -p no:randomly
78
+
79
+ `pytest-reverse `_
80
+ ~~~~~~~~~~~~~~~~~
81
+ `pytest-reverse `_ is another plugin by the same author to find test
82
+ dependencies by running tests in the reverse order. Other than
83
+ ``pytest-randomly ``, it is only effective if the tests are called with the
84
+ command line option ``--reverse ``.
85
+ As with ``pytest-randomly ``, you can use this plugin combined with
86
+ ``pytest-ordering `` to make sure that all known test dependencies are
87
+ handled, and possible unknown dependencies are found.
88
+
89
+ `pytest-random-order `_
90
+ ~~~~~~~~~~~~~~~~~~~~~~
91
+ `pytest-random-order `_ is very similar to ``pytest-randomly ``, except that the
92
+ tests are only reordered if the option ``--random-order `` is given. Except
93
+ from that, what was mentioned for ``pytest-randomly `` is also true for
94
+ this plugin.
95
+
96
+ `pytest-ordering `_
97
+ ~~~~~~~~~~~~~~~~~~
98
+ As mentioned, `pytest-ordering `_ can coexist with ``pytest-order `` due to
99
+ the different marker names, but using markers of both plugins in the same test
73
100
run is not recommended. One plugin may partially revert the effects of the
74
- other plugin in unpredictable ways. The same is true for other plugins that
75
- define the test order.
101
+ other plugin in unpredictable ways, because the order in which they are
102
+ executed is not deterministic. The same is true for other plugins that define
103
+ the test order and are run last.
104
+
105
+ `pytest-depends `_
106
+ ~~~~~~~~~~~~~~~~~
107
+ `pytest-depends `_ has a goal somewhat similar to `pytest-dependency `_ with
108
+ additional ordering, but due to a known issue it always reorders the tests.
109
+ If you have installed this plugin, the order of the unordered tests will
110
+ change even without using it actively.
111
+
112
+ `pytest-find-dependencies `_
113
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~
114
+ This is a small plugin by the same author as ``pytest-order `` that tries to
115
+ find specific dependencies between tests by running a subset of them repeatedly
116
+ in reverse order until the dependencies are found. This plugin would run the
117
+ tests as ordered by any ordering plugin in the first run, but reverse the
118
+ test order in the second run, so that already ordered tests are not run in
119
+ the correct order. You have the possibility to exclude ordered tests
120
+ completely by using the ``--markers-to-ignore `` option::
121
+
122
+ python -m pytest --find-dependencies --markers-to-ignore=order
76
123
77
124
Usage with pytest-xdist
78
125
-----------------------
@@ -89,3 +136,8 @@ each group of dependent tests in one file, and call pytest with
89
136
.. _`pytest-xdist` : https://pypi.org/project/pytest-xdist/
90
137
.. _`pytest-randomly` : https://pypi.org/project/pytest-randomly/
91
138
.. _`pytest-dependency` : https://pypi.org/project/pytest-dependency/
139
+ .. _`pytest-reverse` : https://pypi.org/project/pytest-reverse/
140
+ .. _`pytest-depends` : https://pypi.org/project/pytest-depends/
141
+ .. _`pytest-random-order` : https://pypi.org/project/pytest-random-order/
142
+ .. _`pytest-find-dependencies` : https://pypi.org/project/pytest-find-dependencies/
143
+ .. _`pytest-ordering` : https://pypi.org/project/pytest-ordering/
0 commit comments