Custom types using typing.NewType are rendered inconsistently

[aaugustin]

Describe the bug

(It is probably easier to look at the reproduction, the screenshot, and the expected behavior than to read this description.)

In the following circumstances:

• A custom type is created with typing.NewType
• This type is autodocumented with the autodata directive
• A function uses this custom type it its signature
• This function is autodocumented and types are rendered in the description

No link to the custom type is generated, even though Sphinx should be able to do so — and does so in some very similar cases.

Also, there are inconsistencies between the rendering of this particular case and some very similar cases, as shown below.

How to Reproduce

1. Start a Sphinx project with default options of sphinx-quickstart
2. Add to conf.py:

   extensions = ['sphinx.ext.autodoc']
   autodoc_typehints = 'both'

   (I'm actually using autodoc_typehints = "description" but setting it to "both" yields interesting observations.)
3. Create mymodule.py:

   from __future__ import annotations
   
   from typing import NewType
   
   class A:
       """
       This class is documented with ``autoclass``.
   
       It gets linked in the docstring of :func:`foo`.
   
       """
   
   B = NewType("B", A)
   """
   This type is documented with ``autoclass``.
   
   It gets linked in the docstring of :func:`foo`.
   
   However, Sphinx doesn't render its definition (alias of...) and displays it
   as "class mymodule.B(x)" rather than "mymodule.B".
   
   """
   
   C = NewType("C", A)
   """
   This type is documented with ``autodata``.
   
   Sphinx renders its definition (alias of...)
   
   However, it doesn't get linked in the docstring of :func:`foo`.
   
   """
   
   def foo(a: A, b: B, c: C) -> None:
       """
       Some function.
   
       :param a: first argument
       :param b: second argument
       :param c: third argument
       """

4. Create index.rst:

   .. automodule:: mymodule
   
     .. autoclass:: A
   
     .. autoclass:: B
   
     .. autodata:: C
   
     .. autofunction:: foo

5. Run:

   PYTHONPATH=. make html

6. Open _build/html/index.hml

Expected behavior

Since A, B, and C are defined in the same module, I expect Sphinx to render their types consistently.

Currently:

• in the signature, NewType instances B and C render differently from regular class A (module name not shown vs. shown); I'm only seeing this in the minimal reproduction, not in my actual use case, so I don't really care;
• in the description, there's the same issue, and further more the link to C isn't generated; this last problem is what bothers me most (because users just get a type name and don't even know whether it comes from the stdlib or my project, namely https://github.com/aaugustin/websockets)

---

Considering that:

• the documentation of C looks better than B;
• Sphinx is able to generate a link (it does so in the signature)

I believe that:

• autoattr is the way to go for NewType instances
• the lack of a link is a bug

---

After a few hours of investigation, I suspect the link to C isn't generated because Sphinx looks up a xref of type "class", but C is documented as "data".

Indeed, this change causes the link to be generated (but obviously isn't the right fix):

diff --git a/sphinx/domains/python.py b/sphinx/domains/python.py
index e8330e81c..bc0667e2e 100644
--- a/sphinx/domains/python.py
+++ b/sphinx/domains/python.py
@@ -1113,7 +1113,7 @@ class PythonDomain(Domain):
     label = 'Python'
     object_types: Dict[str, ObjType] = {
         'function':     ObjType(_('function'),      'func', 'obj'),
-        'data':         ObjType(_('data'),          'data', 'obj'),
+        'data':         ObjType(_('data'),          'data', 'class', 'obj'),
         'class':        ObjType(_('class'),         'class', 'exc', 'obj'),
         'exception':    ObjType(_('exception'),     'exc', 'class', 'obj'),
         'method':       ObjType(_('method'),        'meth', 'obj'),

I think the right fix could be:

• generating xrefs of type "obj" for parameters types in description, so any Python object matches;
• failing that, generating xrefs for parameters types in description like in signature, since the latter appears to work as expected.

Your project

See "How to reproduce"

Screenshots

OS

Mac

Python version

3.10.0rc1+

Sphinx version

v4.2.0+/8fd4373d3

Sphinx extensions

sphinx.ext.autodoc

Extra tools

—

Additional context

—

[aaugustin]

Adding this to conf.py unblocks me:

# Workaround for https://github.com/sphinx-doc/sphinx/issues/9560
from sphinx.domains.python import PythonDomain
assert PythonDomain.object_types['data'].roles == ('data', 'obj')
PythonDomain.object_types['data'].roles = ('data', 'class', 'obj')

🙈🙈🙈

[aaugustin]

The workaround is also required the other way round, else Sphinx fails to resolve type annotations for attributes.

from sphinx.domains.python import PythonDomain
assert PythonDomain.object_types['class'].roles == ('class', 'exc', 'obj')
PythonDomain.object_types['class'].roles = ('class', 'exc', 'data', 'obj')
assert PythonDomain.object_types['data'].roles == ('data', 'obj')
PythonDomain.object_types['data'].roles = ('data', 'class', 'obj')

[tk0miya]

In python3.9, the newtype instance does not have correct information where it's defined.

Python 3.9.5 (default, May  7 2021, 01:47:52)
[Clang 11.0.3 (clang-1103.0.32.59)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> from typing import NewType
>>> T = NewType("T", int)
>>> T.__module__
'typing'

But, it seems improved in 3.10 now:

Python 3.10.0rc1+ (heads/3.10:779b9ae, Aug 29 2021, 13:48:15) [Clang 11.0.3 (clang-1103.0.32.59)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> from typing import NewType
>>> T = NewType("T", int)
>>> T.__module__
'__main__'

So B should be represented as same as A.

On the other hand, I don't understand why case C is needed. Why do you use autodata for the newtype? I'm hesitate to refer "data" via :class: role.

[aaugustin]

Why do you use autodata for the newtype?

Mostly for the reasons I described in my example:

• If I use the data role, Sphinx render the definition (alias of...) of the type, which is really useful. If I use the class role, Sphinx doesn't show the definition.
• Also, if I use the class role, a "class" keyword is rendered for the type. I don't find this helpful because a type is just an annotation in Python, not an actual class ("something that bundles data and behavior").

[aaugustin]

In other words, I tried to find what worked best for me with the current version of Sphinx — but it looked like types are still pretty new and best practices for documenting them haven't settled down yet.