Skip to content

Commit 964b413

Browse files
jbaskojbasko
authored
Merge pull request #9 from jbasko/dev
Docs
2 parents 7e13a71 + 24ef773 commit 964b413

File tree

5 files changed

+668
-63
lines changed

5 files changed

+668
-63
lines changed

‎README.rst

Lines changed: 57 additions & 63 deletions
Original file line numberDiff line numberDiff line change
@@ -4,47 +4,66 @@ pytest-random-order
44
.. image:: https://travis-ci.org/jbasko/pytest-random-order.svg?branch=master
55
:target: https://travis-ci.org/jbasko/pytest-random-order
66

7-
This is a pytest plugin **to randomise the order** in which tests are run **with some control**
8-
over how much randomness one allows.
7+
pytest-random-order is a plugin for `pytest <http://pytest.org>`_ that randomises the order in which
8+
tests are run to reveal unwanted coupling between tests. The plugin allows user to control the level
9+
of randomness they want to introduce and to disable reordering on subsets of tests.
910

10-
Why?
11-
----
1211

13-
It is a good idea to shuffle the order in which your tests run
14-
because a test running early as part of a larger test suite may be leaving
15-
the system under test in a particularly fortunate state for a subsequent test to pass.
12+
Quick Start
13+
-----------
1614

17-
How?
18-
----
15+
::
16+
17+
$ pip install pytest-random-order
18+
19+
The plugin **is enabled by default**. To randomise the order of tests within modules and shuffle the order of
20+
test modules (which is the default behaviour of the plugin), just run pytest as always:
21+
22+
::
23+
24+
$ pytest -v
25+
26+
To change the level of randomness allowed, run pytest with ``--random-order-bucket=<bucket-type>`` option
27+
where ``<bucket-type>`` can be ``class``, ``module``, ``package``, or ``global``:
28+
29+
::
30+
31+
$ pytest -v --random-order-bucket=package
32+
33+
To disable reordering of tests in a module or class, use pytest marker notation:
1934

20-
**pytest-random-order** groups tests in buckets, shuffles them within buckets and then shuffles the buckets.
35+
::
36+
37+
pytestmark = pytest.mark.random_order(disabled=True)
38+
39+
Design
40+
------
41+
42+
pytest-random-order plugin groups tests in buckets, shuffles them within buckets and then shuffles the buckets.
2143

2244
You can choose from four types of buckets:
2345

2446
class
25-
Tests from one class will never have tests from other classes or modules run in-between them.
47+
Tests will be shuffled within a class and classes will be shuffled,
48+
but tests from one class will never have tests from other classes or modules run in-between them.
2649

2750
module
28-
Tests will be shuffled within a module and modules will be shuffled, but tests from one module
29-
will never be separated by tests from other modules.
30-
**This is the default setting**.
51+
Same as above at module level. **This is the default setting**.
3152

3253
package
33-
Same as above, but for package level. Note that modules (and hence tests inside those modules) that
54+
Same as above at package level. Note that modules (and hence tests inside those modules) that
3455
belong to package ``x.y.z`` do not belong to package ``x.y``, so they will fall in different buckets
3556
when randomising with ``package`` bucket type.
3657

3758
global
3859
All tests fall in the same bucket, full randomness, tests probably take longer to run.
3960

4061
If you have three buckets of tests ``A``, ``B``, and ``C`` with three tests ``1`` and ``2``, and ``3`` in each of them,
41-
then here are just two of many potential orderings that non-global randomisation can produce:
62+
then one of many potential orderings that non-global randomisation can produce could be:
4263

4364
::
4465

45-
C2 C1 C3 A3 A1 A2 B3 B2 B1
46-
47-
A2 A1 A3 C1 C2 C3 B2 B1 B3
66+
c2, c1, c3, a3, a1, a2, b3, b2, b1
4867

4968
As you can see, all C tests are executed "next" to each other and so are tests in buckets A and B.
5069
Tests from any bucket X are guaranteed to not be interspersed with tests from another bucket Y.
@@ -54,59 +73,23 @@ By default, your tests will be randomised at ``module`` level which means that
5473
tests within a single module X will be executed in no particular order, but tests from
5574
other modules will not be mixed in between tests of module X.
5675

57-
The plugin also supports **disabling shuffle on module basis** irrespective of the bucket type
58-
chosen for the test run. See Advanced Options below.
59-
60-
----
61-
62-
Installation
63-
------------
64-
65-
::
66-
67-
$ pip install pytest-random-order
68-
69-
70-
Usage
71-
-----
76+
The randomised reordering can be disabled per module or per class irrespective of the chosen bucket type.
7277

73-
The plugin **is enabled by default**.
74-
To randomise the order of tests within modules, just run pytest as always:
78+
Usage and Tips
79+
--------------
7580

76-
::
77-
78-
$ pytest -v
81+
Bucket Type Choice
82+
++++++++++++++++++
7983

8084
It is best to start with smallest bucket type (``class`` or ``module`` depending on whether you have class-based tests),
8185
and switch to a larger bucket type when you are sure your tests handle that.
8286

8387
If your tests rely on fixtures that are module or session-scoped, more randomised order of tests will mean slower tests.
84-
You probably don't want to randomise at ``global`` or ``package`` level while you are developing and need a quick confirmation
88+
You probably don't want to randomise at ``global`` or ``package`` level while you are coding and need a quick confirmation
8589
that nothing big is broken.
8690

87-
::
88-
89-
$ pytest -v --random-order-bucket=class
90-
91-
$ pytest -v --random-order-bucket=module
92-
93-
$ pytest -v --random-order-bucket=package
94-
95-
$ pytest -v --random-order-bucket=global
96-
97-
If the plugin misbehaves or you just want to assure yourself that it is not the plugin making your tests fail or
98-
pass undeservedly, you can disable it:
99-
100-
::
101-
102-
$ pytest -p no:random-order -v
103-
104-
105-
Advanced Options
106-
----------------
107-
108-
Disable Shuffling In a Module
109-
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
91+
Disable Shuffling in Module or Class
92+
++++++++++++++++++++++++++++++++++++
11093

11194
You can disable shuffling of tests within a single module or class by marking the module or class
11295
with ``random_order`` marker and passing ``disabled=True`` to it:
@@ -133,6 +116,17 @@ with ``random_order`` marker and passing ``disabled=True`` to it:
133116
No matter what will be the bucket type for the test run, ``test_number_one`` will always run
134117
before ``test_number_two``.
135118

119+
120+
Disable the Plugin
121+
++++++++++++++++++
122+
123+
If the plugin misbehaves or you just want to assure yourself that it is not the plugin making your tests fail or
124+
pass undeservedly, you can disable it:
125+
126+
::
127+
128+
$ pytest -p no:random-order -v
129+
136130
License
137131
-------
138132

‎docs/Makefile

Lines changed: 192 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,192 @@
1+
# Makefile for Sphinx documentation
2+
#
3+
4+
# You can set these variables from the command line.
5+
SPHINXOPTS =
6+
SPHINXBUILD = sphinx-build
7+
PAPER =
8+
BUILDDIR = _build
9+
10+
# User-friendly check for sphinx-build
11+
ifeq ($(shell which $(SPHINXBUILD) >/dev/null 2>&1; echo $$?), 1)
12+
$(error The '$(SPHINXBUILD)' command was not found. Make sure you have Sphinx installed, then set the SPHINXBUILD environment variable to point to the full path of the '$(SPHINXBUILD)' executable. Alternatively you can add the directory with the executable to your PATH. If you don't have Sphinx installed, grab it from http://sphinx-doc.org/)
13+
endif
14+
15+
# Internal variables.
16+
PAPEROPT_a4 = -D latex_paper_size=a4
17+
PAPEROPT_letter = -D latex_paper_size=letter
18+
ALLSPHINXOPTS = -d $(BUILDDIR)/doctrees $(PAPEROPT_$(PAPER)) $(SPHINXOPTS) .
19+
# the i18n builder cannot share the environment and doctrees with the others
20+
I18NSPHINXOPTS = $(PAPEROPT_$(PAPER)) $(SPHINXOPTS) .
21+
22+
.PHONY: help clean html dirhtml singlehtml pickle json htmlhelp qthelp devhelp epub latex latexpdf text man changes linkcheck doctest coverage gettext
23+
24+
help:
25+
@echo "Please use \`make <target>' where <target> is one of"
26+
@echo " html to make standalone HTML files"
27+
@echo " dirhtml to make HTML files named index.html in directories"
28+
@echo " singlehtml to make a single large HTML file"
29+
@echo " pickle to make pickle files"
30+
@echo " json to make JSON files"
31+
@echo " htmlhelp to make HTML files and a HTML help project"
32+
@echo " qthelp to make HTML files and a qthelp project"
33+
@echo " applehelp to make an Apple Help Book"
34+
@echo " devhelp to make HTML files and a Devhelp project"
35+
@echo " epub to make an epub"
36+
@echo " latex to make LaTeX files, you can set PAPER=a4 or PAPER=letter"
37+
@echo " latexpdf to make LaTeX files and run them through pdflatex"
38+
@echo " latexpdfja to make LaTeX files and run them through platex/dvipdfmx"
39+
@echo " text to make text files"
40+
@echo " man to make manual pages"
41+
@echo " texinfo to make Texinfo files"
42+
@echo " info to make Texinfo files and run them through makeinfo"
43+
@echo " gettext to make PO message catalogs"
44+
@echo " changes to make an overview of all changed/added/deprecated items"
45+
@echo " xml to make Docutils-native XML files"
46+
@echo " pseudoxml to make pseudoxml-XML files for display purposes"
47+
@echo " linkcheck to check all external links for integrity"
48+
@echo " doctest to run all doctests embedded in the documentation (if enabled)"
49+
@echo " coverage to run coverage check of the documentation (if enabled)"
50+
51+
clean:
52+
rm -rf $(BUILDDIR)/*
53+
54+
html:
55+
$(SPHINXBUILD) -b html $(ALLSPHINXOPTS) $(BUILDDIR)/html
56+
@echo
57+
@echo "Build finished. The HTML pages are in $(BUILDDIR)/html."
58+
59+
dirhtml:
60+
$(SPHINXBUILD) -b dirhtml $(ALLSPHINXOPTS) $(BUILDDIR)/dirhtml
61+
@echo
62+
@echo "Build finished. The HTML pages are in $(BUILDDIR)/dirhtml."
63+
64+
singlehtml:
65+
$(SPHINXBUILD) -b singlehtml $(ALLSPHINXOPTS) $(BUILDDIR)/singlehtml
66+
@echo
67+
@echo "Build finished. The HTML page is in $(BUILDDIR)/singlehtml."
68+
69+
pickle:
70+
$(SPHINXBUILD) -b pickle $(ALLSPHINXOPTS) $(BUILDDIR)/pickle
71+
@echo
72+
@echo "Build finished; now you can process the pickle files."
73+
74+
json:
75+
$(SPHINXBUILD) -b json $(ALLSPHINXOPTS) $(BUILDDIR)/json
76+
@echo
77+
@echo "Build finished; now you can process the JSON files."
78+
79+
htmlhelp:
80+
$(SPHINXBUILD) -b htmlhelp $(ALLSPHINXOPTS) $(BUILDDIR)/htmlhelp
81+
@echo
82+
@echo "Build finished; now you can run HTML Help Workshop with the" \
83+
".hhp project file in $(BUILDDIR)/htmlhelp."
84+
85+
qthelp:
86+
$(SPHINXBUILD) -b qthelp $(ALLSPHINXOPTS) $(BUILDDIR)/qthelp
87+
@echo
88+
@echo "Build finished; now you can run "qcollectiongenerator" with the" \
89+
".qhcp project file in $(BUILDDIR)/qthelp, like this:"
90+
@echo "# qcollectiongenerator $(BUILDDIR)/qthelp/pytest-cookiecutterplugin_name.qhcp"
91+
@echo "To view the help file:"
92+
@echo "# assistant -collectionFile $(BUILDDIR)/qthelp/pytest-cookiecutterplugin_name.qhc"
93+
94+
applehelp:
95+
$(SPHINXBUILD) -b applehelp $(ALLSPHINXOPTS) $(BUILDDIR)/applehelp
96+
@echo
97+
@echo "Build finished. The help book is in $(BUILDDIR)/applehelp."
98+
@echo "N.B. You won't be able to view it unless you put it in" \
99+
"~/Library/Documentation/Help or install it in your application" \
100+
"bundle."
101+
102+
devhelp:
103+
$(SPHINXBUILD) -b devhelp $(ALLSPHINXOPTS) $(BUILDDIR)/devhelp
104+
@echo
105+
@echo "Build finished."
106+
@echo "To view the help file:"
107+
@echo "# mkdir -p $$HOME/.local/share/devhelp/pytest-cookiecutterplugin_name"
108+
@echo "# ln -s $(BUILDDIR)/devhelp $$HOME/.local/share/devhelp/pytest-cookiecutterplugin_name"
109+
@echo "# devhelp"
110+
111+
epub:
112+
$(SPHINXBUILD) -b epub $(ALLSPHINXOPTS) $(BUILDDIR)/epub
113+
@echo
114+
@echo "Build finished. The epub file is in $(BUILDDIR)/epub."
115+
116+
latex:
117+
$(SPHINXBUILD) -b latex $(ALLSPHINXOPTS) $(BUILDDIR)/latex
118+
@echo
119+
@echo "Build finished; the LaTeX files are in $(BUILDDIR)/latex."
120+
@echo "Run \`make' in that directory to run these through (pdf)latex" \
121+
"(use \`make latexpdf' here to do that automatically)."
122+
123+
latexpdf:
124+
$(SPHINXBUILD) -b latex $(ALLSPHINXOPTS) $(BUILDDIR)/latex
125+
@echo "Running LaTeX files through pdflatex..."
126+
$(MAKE) -C $(BUILDDIR)/latex all-pdf
127+
@echo "pdflatex finished; the PDF files are in $(BUILDDIR)/latex."
128+
129+
latexpdfja:
130+
$(SPHINXBUILD) -b latex $(ALLSPHINXOPTS) $(BUILDDIR)/latex
131+
@echo "Running LaTeX files through platex and dvipdfmx..."
132+
$(MAKE) -C $(BUILDDIR)/latex all-pdf-ja
133+
@echo "pdflatex finished; the PDF files are in $(BUILDDIR)/latex."
134+
135+
text:
136+
$(SPHINXBUILD) -b text $(ALLSPHINXOPTS) $(BUILDDIR)/text
137+
@echo
138+
@echo "Build finished. The text files are in $(BUILDDIR)/text."
139+
140+
man:
141+
$(SPHINXBUILD) -b man $(ALLSPHINXOPTS) $(BUILDDIR)/man
142+
@echo
143+
@echo "Build finished. The manual pages are in $(BUILDDIR)/man."
144+
145+
texinfo:
146+
$(SPHINXBUILD) -b texinfo $(ALLSPHINXOPTS) $(BUILDDIR)/texinfo
147+
@echo
148+
@echo "Build finished. The Texinfo files are in $(BUILDDIR)/texinfo."
149+
@echo "Run \`make' in that directory to run these through makeinfo" \
150+
"(use \`make info' here to do that automatically)."
151+
152+
info:
153+
$(SPHINXBUILD) -b texinfo $(ALLSPHINXOPTS) $(BUILDDIR)/texinfo
154+
@echo "Running Texinfo files through makeinfo..."
155+
make -C $(BUILDDIR)/texinfo info
156+
@echo "makeinfo finished; the Info files are in $(BUILDDIR)/texinfo."
157+
158+
gettext:
159+
$(SPHINXBUILD) -b gettext $(I18NSPHINXOPTS) $(BUILDDIR)/locale
160+
@echo
161+
@echo "Build finished. The message catalogs are in $(BUILDDIR)/locale."
162+
163+
changes:
164+
$(SPHINXBUILD) -b changes $(ALLSPHINXOPTS) $(BUILDDIR)/changes
165+
@echo
166+
@echo "The overview file is in $(BUILDDIR)/changes."
167+
168+
linkcheck:
169+
$(SPHINXBUILD) -b linkcheck $(ALLSPHINXOPTS) $(BUILDDIR)/linkcheck
170+
@echo
171+
@echo "Link check complete; look for any errors in the above output " \
172+
"or in $(BUILDDIR)/linkcheck/output.txt."
173+
174+
doctest:
175+
$(SPHINXBUILD) -b doctest $(ALLSPHINXOPTS) $(BUILDDIR)/doctest
176+
@echo "Testing of doctests in the sources finished, look at the " \
177+
"results in $(BUILDDIR)/doctest/output.txt."
178+
179+
coverage:
180+
$(SPHINXBUILD) -b coverage $(ALLSPHINXOPTS) $(BUILDDIR)/coverage
181+
@echo "Testing of coverage in the sources finished, look at the " \
182+
"results in $(BUILDDIR)/coverage/python.txt."
183+
184+
xml:
185+
$(SPHINXBUILD) -b xml $(ALLSPHINXOPTS) $(BUILDDIR)/xml
186+
@echo
187+
@echo "Build finished. The XML files are in $(BUILDDIR)/xml."
188+
189+
pseudoxml:
190+
$(SPHINXBUILD) -b pseudoxml $(ALLSPHINXOPTS) $(BUILDDIR)/pseudoxml
191+
@echo
192+
@echo "Build finished. The pseudo-XML files are in $(BUILDDIR)/pseudoxml."

0 commit comments

Comments
 (0)