DocTest unittest discovery adapter

[villanyibalint]

I have looked at previous issues and feature requests but have found no mention of this problem. Please let me know if I missed something or have made a mistake below.

Behaviour

When using both unittest tests and doctests in a project the official python docs recommend adapting the doctests into the unittest discovery process, as you can see here:

import unittest
import doctest
import my_module_with_doctests

def load_tests(loader, tests, ignore):
    tests.addTests(doctest.DocTestSuite(my_module_with_doctests))
    return tests

This allows us to run doctests systematically along with regular unit tests.

Under the hood this adapter creates doctest.DocTestCase instances, a subclass of unittest.TestCase. This is not a publicly documented class, but exploring the source we can see that it overrides the id() method to point to the underlying doctest.DocTest instance's name attribute, which in turn is the qualified name of the object that contains the doctest (could be a module, a class or a function).

In vscode-python's unittest adapter script unittest.TestCase.id() is used to build the test tree. This means that doctests discovered in the way described above are displayed "incorrectly" in the test explorer tab:

• Since the script expects a pattern of *folders, filename, class_name, function_name in the id, doctest trees are only correct if the doctest is on a member function of a class. Module and class level doctests have a tree view where a random component of their module name is labeled .py. For example a doctest on a class with a qualified name of some.path.to.module.SomeClass would have a tree of

  some
  └── path
      └── to.py
          └── module
              └── SomeClass
  

• You (usually) cannot open the original source file by double clicking the test, as:
  • This could only work if the doctest is on a member function of a class (as mentioned above)
  • Doesn't work if the module name of the doctest source object doesn't conform to its location in the repository. This can happen if you use a src layout.

To be clear all other functionality of these tests work, they can be run from the test explorer and they report their results correctly. Only the tree and Go to test don't work.

Solution

I think a minor change of creating a special case for doctest.DocTestCase instances in build_test_tree could solve all of these issues. As mentioned doctest.DocTestCase instances have a _dt_test attribute which holds a reference to the underlying doctest.DocTest. This has both a filename and lineno attribute and all necessary information is present to build a correct tree.

I have not researched how the gutter icon is displayed in the editor, but could such a change also automatically fix those?

I'd be happy to take a crack at a PR if possible.

[eleanorjboyd]

Hi! Thank you for your careful and thorough investigation! Your suggested solution is right on! To give you a bit more context on how things are displayed in the editor we use the vscode testing api which handles all displaying in the correct places. This means that the filename and lineno created in build_test_tree is returned to the resultResolver.ts file which then build the test items here. These are vscode interfaces that are then used in vscode-core to render correctly.

Long story short- yes if you fix the filename and lineno there it will resolve the issue! A PR would be great since I can't promise when I could get around to this.

Thank you and I look forward to working with you on this PR! (just FYI I will be out 12/18 - 1/6 so apologies as I wont respond till the new year)

[villanyibalint]

Thanks for the response and sorry for the late reply I was already out for the holidays. I'll take a look at a PR in the following days!

[villanyibalint]

I started drafting a PR and encountered the following issues, I'd appreciate your feedback:

1. I mentioned that the doctest.DocTestCase._dt_test attribute is private. While such usage should be avoided, this PR depends on it. Is it sufficient to add # noqa: SLF001 # pyright: ignore[reportAttributeAccessIssue]?
2. The source file path for doctests could point outside the current test tree, especially if the python.testing.cwd option is set. This discrepancy arises because doctests are not collected directly from their definitions. It could potentially point to a file outside the current workspace. Since we can only return a single tree from the discovery process, I think we have two options:
   • generate a new tree starting at the workspace folder and append the regular test tree under it, along with the doctests. However, this approach changes the path of the returned tree's root element from the discovery root. Whether this has adverse effects is outside my understanding. Also this approach cannot cope with a file outside the current workspace.
   • append doctests under the regular test tree, whatever its root might be. This results in a tree that doesn’t match the filesystem but keeps regular test discovery unaffected. We could even create a separate node for doctests under the regular tree, like so:

test discovery root
└── doctests
    └── workspaceRoot
        └── ...
└── some
    └── test
        └── files
            └── ...
└── other
    └── test
        └── files
            └── ...

3. doctest.DocTest.filename can be None if the source file is unknown (I'm not sure when this happens or if it happens at all). If doctest.DocTest.filename is None, we can add a node directly under the root with a default path like Unknown.