More documentation updates

- add notes about sparse ordering and handling of unknown dependencies
- add note about test dependencies as bad practice
- add more tests

Author: mrbean-bremen

README.md

@@ -20,8 +20,7 @@ History
 This is a fork of [pytest-ordering](https://github.com/ftobia/pytest-ordering).
 That project is not maintained anymore, and there are several helpful PRs
 that are now integrated into `pytest-order`. The idea and most of the code
-has been created by Frank Tobia, the author of that plugin. In case 
-`pytest-ordering` is revived, this project will be obsolete. 
+has been created by Frank Tobia, the author of that plugin, and contributors.
 
 Compatibility to `pytest_ordering`
 ---------------------------------

docs/source/index.rst

@@ -2,10 +2,16 @@ Introduction
 ============
 ``pytest-order`` is a pytest plugin which allows you to customize the order
 in which your tests are run. It provides the marker ``order``, that has
-attributes that defines when your tests should run in relation to each other.
+attributes that define when your tests should run in relation to each other.
 These attributes can be absolute (i.e. first, or second-to-last) or relative
 (i.e. run this test before this other test).
 
+.. note::
+  It is generally considered bad practice to write tests that depend on each
+  other. However, in reality this may still be needed due to performance
+  problems or legacy code. Still, before using this plugin, you should check
+  if you can refactor your tests to remove the dependencies between tests.
+
 Relationship with pytest-ordering
 ---------------------------------
 ``pytest-order`` is a fork of
@@ -110,11 +116,11 @@ Usage
 =====
 The above is a trivial example, but ordering is respected across test files.
 
-.. note ::
-    The scope of the ordering is always global, e.g. tests with lower ordinal
-    numbers are always executed before tests with higher numbers, regardless of
-    the module and class they reside in. This may be changed to be
-    configurable in a later version.
+.. note::
+  The scope of the ordering is global per default, e.g. tests with lower
+  ordinal numbers are always executed before tests with higher numbers,
+  regardless of the module and class they reside in. This can be changed
+  by using the :ref:`order-scope` option.
 
 There are currently three possibilities to define the order:
 
@@ -257,11 +263,19 @@ by their name:
 
     =========================== 4 passed in 0.02 seconds ===========================
 
+If a test is referenced using the unqualified test name as shown in the
+example, the test is assumed to be in the current module. For tests in other
+modules the qualified test name (e.g. ``my_module.test_something``) has to
+be used.
+
+If an unknown test is referenced, a warning is issued and the test in
+question is ordered behind all other tests.
+
 .. note::
-   The `pytest-dependency <https://pypi.org/project/pytest-dependency/>`__
-   plugin also manages dependencies between tests (skips tests that depend
-   on skipped or failed tests), but doesn't do any ordering. You can combine
-   both plugins if you need both options.
+  The `pytest-dependency <https://pypi.org/project/pytest-dependency/>`__
+  plugin also manages dependencies between tests (skips tests that depend
+  on skipped or failed tests), but doesn't do any ordering. You can combine
+  both plugins if you need both options.
 
 Configuration
 =============
@@ -286,6 +300,8 @@ pytest-order *before* the sort from ``--failed-first``, allowing the failed
 tests to be sorted to the front (note that in pytest versions from 6.0 on,
 this seems not to be needed anymore, at least in this specific case).
 
+.. _order-scope:
+
 ``--order-scope``
 -----------------
 By default, tests are ordered per session, e.g. across all modules in the
@@ -317,6 +333,50 @@ together, we have to put each group of dependent tests in one file, and call
 pytest with ``--dist=loadfile`` (this is taken from
 `this issue <https://github.com/ftobia/pytest-ordering/issues/36>`__).
 
+Sparse ordinal behavior
+-----------------------
+Sparse ordering (e.g. missing some ordinals in your markers) behaves the
+same as if the the ordinals are consecutive. For example, these tests:
+
+.. code:: python
+
+ import pytest
+
+ @pytest.mark.order(4)
+ def test_two():
+     assert True
+
+ def test_three():
+     assert True
+
+ @pytest.mark.order(2)
+ def test_two():
+     assert True
+
+have the same output as:
+
+.. code:: python
+
+ import pytest
+
+ @pytest.mark.order(1)
+ def test_two():
+     assert True
+
+ def test_three():
+     assert True
+
+ @pytest.mark.order(0)
+ def test_two():
+     assert True
+
+namely, the tests are run in the order ``test_one``, ``test_two``,
+``test_three``, regardless of the gaps between numbers, and the starting
+number being not 0. It would be possible to change the ordering behavior to
+fill the gaps between the numbers with tests without a marker, as long as
+any are available. However, this leads to some not very intuitive behavior,
+so it is currently not implemented. If there will be demand for this kind
+of behavior, it can be added in a later version.
 
 .. toctree::
    :maxdepth: 2

tests/test_misc.py

@@ -1,6 +1,8 @@
 # -*- coding: utf-8 -*-
 import re
 
+import pytest
+
 import pytest_order
 
 
@@ -11,3 +13,11 @@ def test_version_exists():
 def test_version_valid():
     assert re.match(r"[0-9]+\.[0-9]+(\.[0-9]+)?(dev)?$",
                     pytest_order.__version__)
+
+
+def test_markers_registered(capsys):
+    pytest.main(["--markers"])
+    out, err = capsys.readouterr()
+    assert "@pytest.mark.order" in out
+    # only order is supported as marker
+    assert out.count("Provided by pytest-order.") == 1

tests/test_ordering.py

@@ -296,6 +296,25 @@ def test_one():
                                             "test_three", "test_four"]
 
 
+def test_sparse_numbers(item_names_for):
+    test_content = """
+     import pytest
+
+     @pytest.mark.order(4)
+     def test_two():
+         assert True
+
+     def test_three():
+         assert True
+
+     @pytest.mark.order(2)
+     def test_one():
+         assert True
+    """
+    assert item_names_for(test_content) == ["test_one", "test_two",
+                                            "test_three"]
+
+
 def test_quickstart(item_names_for):
     test_content = """
     import pytest
@@ -418,12 +437,103 @@ def test_first():
                                             "test_third"]
 
 
-def test_markers_registered(capsys):
-    pytest.main(["--markers"])
+def test_mixed_markers1(item_names_for):
+    test_content = """
+    import pytest
+
+    @pytest.mark.order(2)
+    def test_1():
+        pass
+
+    @pytest.mark.order(after="test_1")
+    def test_2():
+        pass
+
+    @pytest.mark.order(1)
+    def test_3():
+        pass
+    """
+    assert item_names_for(test_content) == ["test_3", "test_1", "test_2"]
+
+
+def test_mixed_markers2(item_names_for):
+    test_content = """
+    import pytest
+
+    @pytest.mark.order(2)
+    def test_1():
+        pass
+
+    @pytest.mark.order(1)
+    def test_2():
+        pass
+
+    @pytest.mark.order(before="test_2")
+    def test_3():
+        pass
+    """
+    assert item_names_for(test_content) == ["test_3", "test_2", "test_1"]
+
+
+def test_dependency_after_unknown_test(item_names_for, capsys):
+    test_content = """
+    import pytest
+
+    @pytest.mark.order(after="some_module.test_2")
+    def test_1():
+        pass
+
+    def test_2():
+        pass
+    """
+    assert item_names_for(test_content) == ["test_2", "test_1"]
+    out, err = capsys.readouterr()
+    warning = ("can not execute test relative to others: "
+               "some_module.test_2 enqueue them behind the others")
+    assert warning in out
+
+
+def test_dependency_before_unknown_test(item_names_for, capsys):
+    test_content = """
+    import pytest
+
+    @pytest.mark.order(before="test_4")
+    def test_1():
+        pass
+
+    def test_2():
+        pass
+    """
+    assert item_names_for(test_content) == ["test_2", "test_1"]
+    out, err = capsys.readouterr()
+    warning = ("can not execute test relative to others: "
+               "test_dependency_before_unknown_test.test_4 enqueue them "
+               "behind the others")
+    assert warning in out
+
+
+def test_dependency_loop(item_names_for, capsys):
+    test_content = """
+    import pytest
+
+    @pytest.mark.order(after="test_3")
+    def test_1():
+        pass
+
+    @pytest.mark.order(1)
+    def test_2():
+        pass
+
+    @pytest.mark.order(before="test_1")
+    def test_3():
+        pass
+    """
+    assert item_names_for(test_content) == ["test_2", "test_3", "test_1"]
     out, err = capsys.readouterr()
-    assert "@pytest.mark.order" in out
-    # only order is supported as marker
-    assert out.count("Provided by pytest-order.") == 1
+    warning = ("can not execute test relative to others: "
+               "test_dependency_loop.test_1 test_dependency_loop.test_3 "
+               "enqueue them behind the others")
+    assert warning in out
 
 
 def test_unsupported_order(item_names_for):