Merge pull request #199 from scipy/doctest-only

ENH: add --doctest-only-doctests={true,false} option

Author: ev-br

README.md

@@ -151,6 +151,22 @@ $ pytest --pyargs <your-package> --doctest-modules --doctest-collect=api
 See [More fine-grained control](#more-fine-grained-control) section
 for details on how to customize the behavior.
 
+** NOTE ** Currently, `pytest --doctest-modules` only collects doctests and skips
+'regular' unit tests. This differs from the vanilla `pytest` behavior, which collects
+both doctests and unit tests.
+The behavior will change in a future version: `scipy-doctest==2.0` **will change the
+default** to align with the vanilla `pytest`.
+
+To retain the current behavior, use the `--doctest-only-doctests` CLI option:
+
+```
+$ pytest --doctest-modules --doctest-only-doctests
+```
+
+is unambiguous and will behave identically in the current version 1.8 and upcoming
+versions of `scipy-doctest`. 
+
+
 ### Basic usage
 
 The use of `pytest` is optional, and you can use the `doctest` layer API.
@@ -291,11 +307,11 @@ NumPy wraps `scipy-doctest` with the `spin` command
 $ spin check-docs
 ```
 
-SciPy wraps `scipy-doctest` with custom `dev.py` commands:
+In SciPy, the name of the `spin` command is `smoke-docs`::
 
 ```
-$ python dev.py smoke-docs    # check docstrings
-$ python dev.py smoke-tutorials   # ReST user guide tutorials
+$ spin smoke-docs    # check docstrings
+$ spin smoke-tutorials   # ReST user guide tutorials
 ```
 
 ## Rough edges and sharp bits

scipy_doctest/plugin.py

@@ -29,6 +29,28 @@ def pytest_addoption(parser):
         dest="collection_strategy"
     )
 
+    # We cannot add `--doctest-only` option because of
+    # https://github.com/pytest-dev/pytest/discussions/13435
+    #
+    # Therefore, we add a new option, --doctest-only-doctests,
+    # which is `true` by default, for now.
+    #
+    # In v2.0, it the default will become `false`, so that
+    #
+    # $ pytest --doctest-modules
+    #
+    # will run both doctests and unit tests, and the way to use the
+    # current behavior (only run doctests, skip unit tests) will be
+    #
+    # $ pytest --doctest-modules --doctest-only-doctests=true
+    #
+    group.addoption("--doctest-only-doctests",
+        action="store",
+        default="true",
+        help="Whether to only collect doctests, or also collect unit tests, too.",
+        choices=("true", "false"),
+        dest="doctest_only_doctests"
+    )
 
 def pytest_configure(config):
     """
@@ -49,7 +71,13 @@ def pytest_ignore_collect(collection_path, config):
     This function is used to exclude the 'tests' directory and test modules when
     the `--doctest-modules` option is used.
     """
-    if config.getoption("--doctest-modules"):
+    # XXX: in v2.0, --doctest-modules will mean "run both doctests and unit tests",
+    # (consistent with vanilla pytest), and the way to retain the current behavior
+    # will be to add --doctest-only-doctests=true to the CLI command
+    if (
+        config.getoption("--doctest-modules") and
+        config.getoption("--doctest-only-doctests") == 'true'
+    ):
         path_str = str(collection_path)
         if "tests" in path_str or "test_" in path_str:
             return True