How do I make sphinx understand the documentation information in static methods?

[riclarsson]

Hi any and all,

We decided to switch from pybind11 to nanobind.

This is a growth pain question.

I have a problem getting my sphinx documentation for static data and static methods working in nanobind. In pybind11, static methods produces nice documentation.

You can see this comparing our stable new and stable documentation. This page gives the method get_options, where nanobind seems to think it returns an instance of some of itself, and pybind11 shows the documentation.

Note that if I go into ipython and simply ? on /pyarts.arts.options.SpeciesTagType.get_options: pyarts.arts.options.SpeciesTagType.get_options?, I get the correct documentation. It just doesn't seem to propagate to sphinx.

I am wondering if there are workarounds here so that sphinx can see the documentation?

[wjakob]

How do you list the classes and their members? Have you tried registering them explicitly, like so?

.. autoclass:: class

   .. automethod:: method

It's possible that Sphinx thinks that the static methods are data, not functions.

[wjakob]

If that doesn't do it, then I fear that this is something that only the Sphinx team can fix.

[riclarsson]

Thank you for the advice. The core problem is definitely how sphinx interprets what nanobind types are what type.

The jinja template we are using for generating the class data, which is just a standard template with nothing special in it, does not work with nanobind.

The main reason is that nanobind produces types that are not seen as methods, but as attributes. The link above, the nanobind types never sees the methods tab. As a consequence, sphinx believes it should be using the :autoattribute: command rather than the :automethod: command...

[wjakob]

Have you opening an issue with the Sphinx team? It should be easy to distinguish callable from non-callable objects and use this to display static methods correctly.

[riclarsson]

No, I did not report it.

I am not sure how to report this in a manner that is clear for them to reproduce. Nor if it is just our implementation of the template that is wrong.

I can identify the problem but my only solution has been to completely ignore the automatic code and to sweep through the generated library to manually call the right command depending on the types.

[ianhbell]

I have the same issue. I noticed that when I added a listener in sphinx in my conf.py, like so:

def autodoc_process_handler(app, what, name, obj, options, lines, huh):
    print(what, name, obj, lines, obj.__doc__)
    
def setup(app):
    app.connect('autodoc-process-signature', autodoc_process_handler)

sphinx gives me output like this:

class teqpflsh._teqpflsh_impl.ChebyshevApproximation1D <class 'teqpflsh._teqpflsh_impl.ChebyshevApproximation1D'> (*args, **kwargs) None
attribute teqpflsh._teqpflsh_impl.ChebyshevApproximation1D.count_x_for_y_many <nanobind.nb_method object at 0x104a23560>  count_x_for_y_many(self, arg0: ndarray[dtype=float64, shape=(*), order='C', device='cpu'], arg1: int, arg2: int, arg3: float, arg4: ndarray[dtype=float64, shape=(*), order='C', device='cpu'], /) -> None
attribute teqpflsh._teqpflsh_impl.ChebyshevApproximation1D.eval <nanobind.nb_method object at 0x104a23120>  eval(self, arg: float, /) -> float
attribute teqpflsh._teqpflsh_impl.ChebyshevApproximation1D.eval_many <nanobind.nb_method object at 0x104a23450>  eval_many(self, arg0: ndarray[dtype=float64, shape=(*), order='C', device='cpu'], arg1: ndarray[dtype=float64, shape=(*), order='C', device='cpu'], /) -> None
property teqpflsh._teqpflsh_impl.ChebyshevApproximation1D.expansions <property object at 0x104a02c00>  (self) -> list[teqpflsh._teqpflsh_impl.ChebyshevExpansion]
attribute teqpflsh._teqpflsh_impl.ChebyshevApproximation1D.get_intervals_containing_y <nanobind.nb_method object at 0x104a239a0>  get_intervals_containing_y(self, arg: float, /) -> list[teqpflsh._teqpflsh_impl.IntervalMatch]
attribute teqpflsh._teqpflsh_impl.ChebyshevApproximation1D.get_x_for_y <nanobind.nb_method object at 0x104a23010>  get_x_for_y(self, *, y: float, bits: int = 64, max_iter: int = 100, boundsftol: float = 1e-13) -> list[tuple[float, int]]
property teqpflsh._teqpflsh_impl.ChebyshevApproximation1D.monotonic_intervals <property object at 0x104a02ca0>  (self) -> list[teqpflsh::superancillary::IntervalMatch]
property teqpflsh._teqpflsh_impl.ChebyshevApproximation1D.x_at_extrema <property object at 0x104a02c50>  (self) -> list[float]
property teqpflsh._teqpflsh_impl.ChebyshevApproximation1D.xmax <property object at 0x104a02bb0>  (self) -> float
property teqpflsh._teqpflsh_impl.ChebyshevApproximation1D.xmin <property object at 0x104a02b10>  (self) -> float

but it is inferring incorrectly that the methods are attributes and thus gives no documentation.

[njroussel]

FWIW we worked around this by monkey patching sphinx:
https://github.com/mitsuba-renderer/mitsuba3/blob/master/docs/docs_api/conf.py#L810-L823
This is extremely brittle, as there seems to have been quite a few changes in that part of sphinx's source code in recent versions.

Another solution is to explicitly list the class members yourself, to bypass the "identification" process entirely - instead of relying on the autodoc to find and identify the members.

It's unclear to me if Sphinx already provides a way to alter the "identification" process of objects. I think that would be the correct fix to this issue.

[scottamain]

Thank you so much @njroussel, this patch works great for me! 🙏

Also pasting your conf.py code here for posterity (works with Sphinx 7.4.7):

def setup(app):
    import inspect
    from sphinx.util import inspect as sphinx_inspect

    # Sphinx inspects all objects in the module and tries to resolve their type
    # (attribute, function, class, module, etc.) by using its own functions in
    # `sphinx.util.inspect`. These functions misidentify certain nanobind
    # objects. We monkey patch those functions here.
    def mpatch_ismethod(object):
        if hasattr(object, '__name__') and type(object).__name__ == "nb_method":
            return True
        return inspect.ismethod(object)

    sphinx_inspect_isclassmethod = sphinx_inspect.isclassmethod
    def mpatch_isclassmethod(object, cls=None, name=None):
        if hasattr(object, '__name__') and type(object).__name__ == "nb_method":
            return False
        return sphinx_inspect_isclassmethod(object, cls,name)

    sphinx_inspect.ismethod = mpatch_ismethod
    sphinx_inspect.isclassmethod = mpatch_isclassmethod

[ianhbell]

My solution is also brittle, I wrote a script to parse, since my object hierarchy is flat enough:

import inspect 

class AutodocCollector:
    entries = []
    def __init__(self, module, dunder=False):
        for key in dir(module):
            thing = getattr(module, key)
            if key.startswith('__') and not dunder:
                continue
            
            if 'nanobind.nb_func' in str(type(thing)):
                self.parse_func(thing)
            elif 'enum.EnumType' in str(type(thing)):
                self.parse_enum(thing)
            elif 'nanobind.nb_type_0' in str(type(thing)):
                self.parse_class(thing)
            else:
                print(key, type(thing), thing.__name__, inspect.getmembers(thing))
            
    def add_entry(self, e):
        self.entries.append(e)
        
    def to_file(self, path, title):
        with open(path, 'w') as fp:
            fp.write(title+'\n')
            fp.write('='*len(title)+'\n\n')
            fp.write('\n\n'.join(self.entries))
                            
    def parse_func(self, thing):
        mother = inspect.getmodule(thing)
        self.add_entry(f'.. autofunction:: {mother.__name__}.{thing.__name__}\n\n    :teqpflsh:`{thing.__name__}`')
        
    def parse_enum(self, thing):
        mother = inspect.getmodule(thing)
        self.add_entry(f'.. autoclass:: {mother.__name__}.{thing.__name__}\n    :undoc-members:\n    :members:\n    :show-inheritance:\n\n    :teqpflsh:`{thing.__name__}`')
        
    def parse_class(self, thing):
        mother = inspect.getmodule(thing)
        members = inspect.getmembers(thing)
        e = f'.. autoclass:: {mother.__name__}.{thing.__name__}\n    :show-inheritance:'
        e += f'\n\n    C++ docs: :teqpflsh:`{thing.__name__}`'
        for name, member in members:
            if name.startswith('__'): continue
            if isinstance(member, property):
                e += '\n\n' + ' '*4 + f'.. autoproperty:: {name}\n'
            else:
                e += '\n\n' + ' '*4 + f'.. automethod:: {name}\n'
        self.add_entry(e)
            
if __name__ == '__main__':
    import teqpflsh
    ac = AutodocCollector(teqpflsh._teqpflsh_impl)
    ac.to_file('api/teqpflsh.rst', title='teqpflsh Package')

[VincentRouvreau]

If I understand well what happens in sphinx code, this is somewhere here:
https://github.com/sphinx-doc/sphinx/blob/aeeaeafc8a1198dc1b982d28bd23495a987aeb5d/sphinx/util/inspect.py#L357-L379

And especially when return type(unwrapped).__name__ != 'instancemethod' is tested.
Here, with Nanobind, the name is nb_method, contrary to PyBind11 where it is instancemethod.

@wjakob What was the reason to name it differently ?
I am ok that Sphinx could also add this test to be conform with Nanobind.

I opened sphinx-doc/sphinx#13823 on sphinx side.

UPDATE: Sorry for the noise, my issue was different and not for static method