Add option for sparse ordering behavior

Author: mrbean-bremen

CHANGELOG.md

@@ -2,6 +2,12 @@
 
 ## Unreleased
 
+### Added
+- added configuration option for sparse sorting, e.g. the possibility to
+  fill gaps between ordinals with unordered tests (see also 
+  [this issue](https://github.com/ftobia/pytest-ordering/issues/14) in
+  `pytest-ordering`)
+
 ## [Version 0.8.1](https://pypi.org/project/pytest-order/0.8.1/) (2020-11-02)
 
 ### Added

docs/source/index.rst

@@ -17,11 +17,11 @@ Relationship with pytest-ordering
 ``pytest-order`` is a fork of
 `pytest-ordering <https://github.com/ftobia/pytest-ordering>`__, which is
 not maintained anymore. The idea and most of the code has been created by
-Frank Tobia, the author of that plugin.
+Frank Tobia, the author of that plugin, and contributors to the project.
 
 However, ``pytest-order`` is not compatible with ``pytest-ordering`` due to the
 changed marker name (``order`` instead of ``run``) and the removal of all
-other special markers for consistence (see
+other special markers for consistence (as has been discussed in
 `this issue <https://github.com/ftobia/pytest-ordering/issues/38>`__). This
 also avoids clashes between the plugins if they are both installed.
 
@@ -279,9 +279,9 @@ question is ordered behind all other tests.
 
 Configuration
 =============
-Currently there are two command line option that change the behavior of the
-plugin. As for any option, you can add the options to your ``pytest.ini`` if
-you want to have them always applied.
+There are a few command line options that change the behavior of the
+plugin. As with any pytest option, you can add the options to your
+``pytest.ini`` if you want to have them applied to all tests automatically.
 
 ``--indulgent-ordering``
 ------------------------
@@ -320,40 +320,30 @@ separate test functions, these test functions are handled separately from the
 test classes. If a module has no test classes, the effect is the same as
 if using ``--order-scope=module``.
 
-Miscellaneous
-=============
-
-Usage with pytest-xdist
------------------------
-The ``pytest-xdist`` plugin schedules tests unordered, and the order
-configured by ``pytest-order`` will normally not be preserved. But
-if we use the ``--dist=loadfile`` option, provided by ``xdist``, all tests
-from one file will be run in the same thread. So, to make the two plugins work
-together, we have to put each group of dependent tests in one file, and call
-pytest with ``--dist=loadfile`` (this is taken from
-`this issue <https://github.com/ftobia/pytest-ordering/issues/36>`__).
-
-Sparse ordinal behavior
------------------------
-Sparse ordering (e.g. missing some ordinals in your markers) behaves the
-same as if the the ordinals are consecutive. For example, these tests:
+``--sparse-ordering``
+---------------------
+Ordering tests by ordinals where some numbers are missing by default behaves
+the same as if the the ordinals are consecutive. For example, these tests:
 
 .. code:: python
 
  import pytest
 
- @pytest.mark.order(4)
+ @pytest.mark.order(3)
  def test_two():
      assert True
 
  def test_three():
      assert True
 
- @pytest.mark.order(2)
- def test_two():
+ def test_four():
+     assert True
+
+ @pytest.mark.order(1)
+ def test_one():
      assert True
 
-have the same output as:
+are executed in the same order as:
 
 .. code:: python
 
@@ -366,17 +356,42 @@ have the same output as:
  def test_three():
      assert True
 
+ def test_four():
+     assert True
+
  @pytest.mark.order(0)
- def test_two():
+ def test_one():
      assert True
 
-namely, the tests are run in the order ``test_one``, ``test_two``,
-``test_three``, regardless of the gaps between numbers, and the starting
-number being not 0. It would be possible to change the ordering behavior to
-fill the gaps between the numbers with tests without a marker, as long as
-any are available. However, this leads to some not very intuitive behavior,
-so it is currently not implemented. If there will be demand for this kind
-of behavior, it can be added in a later version.
+namely, they are run in the order ``test_one``, ``test_two``, ``test_three``
+and ``test_four``. The gaps between numbers, and the fact that the starting
+number is not 0, are ignored. This is consistent with the current behavior of
+``pytest-ordering``.
+
+If you use the ``--sparse-ordering`` option, the behavior will change: now
+all missing numbers (starting with 0) are filled with unordered tests, as
+long as unordered tests are left. So the shown example will now order as
+``test_three`` (filled in for the missing number 0), ``test_one``,
+``test_four`` (filled in for the missing number 2), and ``test_two``. This
+will also work for tests with negative order numbers (or the respective names).
+The missing ordinals are filled with unordered tests first from the start,
+then from the end if there are negative numbers, and the rest will be in
+between (e.g. between positive and negative numbers), as it is without this
+option.
+
+
+Miscellaneous
+=============
+
+Usage with pytest-xdist
+-----------------------
+The ``pytest-xdist`` plugin schedules tests unordered, and the order
+configured by ``pytest-order`` will normally not be preserved. But
+if we use the ``--dist=loadfile`` option, provided by ``xdist``, all tests
+from one file will be run in the same thread. So, to make the two plugins work
+together, we have to put each group of dependent tests in one file, and call
+pytest with ``--dist=loadfile`` (this is taken from
+`this issue <https://github.com/ftobia/pytest-ordering/issues/36>`__).
 
 .. toctree::
    :maxdepth: 2

pytest_order/__init__.py

@@ -44,7 +44,7 @@ def pytest_configure(config):
     )
     config.addinivalue_line("markers", config_line)
 
-    if config.getoption("indulgent-ordering"):
+    if config.getoption("indulgent_ordering"):
         # We need to dynamically add this `tryfirst` decorator to the plugin:
         # only when the CLI option is present should the decorator be added.
         # Thus, we manually run the decorator on the class function and
@@ -65,14 +65,18 @@ def pytest_addoption(parser):
     """Set up CLI option for pytest"""
     group = parser.getgroup("ordering")
     group.addoption("--indulgent-ordering", action="store_true",
-                    dest="indulgent-ordering",
+                    dest="indulgent_ordering",
                     help="Request that the sort order provided by "
                          "pytest-order be applied before other sorting, "
                          "allowing the other sorting to have priority")
     group.addoption("--order-scope", action="store",
-                    dest="order-scope",
+                    dest="order_scope",
                     help="Defines the scope used for ordering. Possible values"
                          "are 'session' (default), 'module', and 'class'")
+    group.addoption("--sparse-ordering", action="store_true",
+                    dest="sparse_ordering",
+                    help="If there are gaps between ordinals they are filled "
+                         "with unordered tests.")
 
 
 class OrderingPlugin(object):
@@ -126,14 +130,6 @@ def mark_binning(item, keys, start, end, before, after, unordered):
     return False
 
 
-def insert(items, sort):
-    if isinstance(items, tuple):
-        list_items = items[1]
-    else:
-        list_items = items
-    sort += list_items
-
-
 def insert_before(name, items, sort):
     regex_name = re.escape(name) + r"(:?\.\w+)?$"
     for pos, item in enumerate(sort):
@@ -160,7 +156,7 @@ def insert_after(name, items, sort):
     return False
 
 
-def do_modify_items(items):
+def do_modify_items(items, sparse_ordering):
     before_item = {}
     after_item = {}
     start_item = {}
@@ -174,13 +170,8 @@ def do_modify_items(items):
     start_item = sorted(start_item.items())
     end_item = sorted(end_item.items())
 
-    sorted_list = []
-
-    for entries in start_item:
-        insert(entries, sorted_list)
-    insert(unordered_list, sorted_list)
-    for entries in end_item:
-        insert(entries, sorted_list)
+    sorted_list = sort_numbered_items(start_item, end_item, unordered_list,
+                                      sparse_ordering)
 
     still_left = 0
     length = len(before_item) + len(after_item)
@@ -216,24 +207,51 @@ def do_modify_items(items):
     return sorted_list
 
 
+def sort_numbered_items(start_item, end_item, unordered_list, sparse_ordering):
+    sorted_list = []
+    index = 0
+    for entries in start_item:
+        if sparse_ordering:
+            while entries[0] > index and unordered_list:
+                sorted_list.append(unordered_list.pop(0))
+                index += 1
+        sorted_list += entries[1]
+        index += len(entries[1])
+    mid_index = len(sorted_list)
+    index = -1
+    for entries in reversed(end_item):
+        if sparse_ordering:
+            while entries[0] < index and unordered_list:
+                sorted_list.insert(mid_index, unordered_list.pop())
+                index -= 1
+        for item in reversed(entries[1]):
+            sorted_list.insert(mid_index, item)
+        index += len(entries[1])
+    for unordered_item in reversed(unordered_list):
+        sorted_list.insert(mid_index, unordered_item)
+    return sorted_list
+
+
 def modify_items(session, config, items):
-    scope = config.getoption("order-scope")
+    sparse_ordering = config.getoption("sparse_ordering")
+    scope = config.getoption("order_scope")
     if scope not in ("session", "module", "class"):
         if scope is not None:
             warn("Unknown order scope '{}', ignoring it. "
                  "Valid scopes are 'session', 'module' and 'class'."
                  .format(scope))
         scope = "session"
     if scope == "session":
-        sorted_list = do_modify_items(items)
+        sorted_list = do_modify_items(items, sparse_ordering)
     elif scope == "module":
         module_items = OrderedDict()
         for item in items:
             module_path = item.nodeid[:item.nodeid.index("::")]
             module_items.setdefault(module_path, []).append(item)
         sorted_list = []
         for module_item_list in module_items.values():
-            sorted_list.extend(do_modify_items(module_item_list))
+            sorted_list.extend(do_modify_items(
+                module_item_list, sparse_ordering))
     else:  # class scope
         class_items = OrderedDict()
         for item in items:
@@ -244,6 +262,7 @@ def modify_items(session, config, items):
             class_items.setdefault(class_path, []).append(item)
         sorted_list = []
         for class_item_list in class_items.values():
-            sorted_list.extend(do_modify_items(class_item_list))
+            sorted_list.extend(do_modify_items(
+                class_item_list, sparse_ordering))
 
     items[:] = sorted_list

tests/conftest.py

@@ -6,7 +6,6 @@
 @pytest.fixture
 def item_names_for(testdir):
     def _item_names_for(tests_content):
-        # some strange code to extract sorted items
         items = testdir.getitems(tests_content)
         hook = testdir.config.hook
         hook.pytest_collection_modifyitems(session=items[0].session,