Add a `collection.abc.Function` ABC for function-like objects

[scoder]

Feature or enhancement

Proposal:

In the Cython project, we generate a function type that is mostly compatible with Python's function type. There is currently no (simple/reasonably fast) way to detect this kind of object, but there are use cases and Cython aware projects have found their workarounds in one way or another. But it still means that all code that wants to specially support both Python and Cython functions in the same (or similar) way needs to do something special to detect both, rather than just saying "is this a function?".

There should be an abc.Function ABC that other function implementations can register with.
That would make the detection a simple and straight forward isinstance(obj, abc.Function).

inspect.isfunction() does not help since it specifically tests that the object is a Python function object, which is good to have, simple and fills a need. The ABC is for use cases where code needs to test whether an object is a callable with a function-like interface, e.g. for introspection, signature, code-like object, etc., however complete that implementation then is.

This is probably also relevant for other tools like mypyc or numba that provide binary functions to Python in one way or another.

I'd also add this ABC to the PyPI package backports_abc which provides missing ABCs for older Pythons.

Has this already been discussed elsewhere?

No response given

Links to previous discussion of this feature:

Make inspect.isfunction() support Cython functions:
cython/cython#6379

Some code trying to detect Cython functions:

• https://github.com/vmware-archive/pop/blob/f3a67f913ee92cf855889719a23f662dd435f39d/pop/loader.py#L222
• https://github.com/sphinx-doc/sphinx/blob/a6d7ae16739bf92a032a7c4df0297db7cf120ec9/sphinx/util/inspect.py#L342-L347
• https://github.com/quora/qcore/blob/f93ccc0f04858f1c606d3b95f05ff315f4fd47bf/qcore/inspection.py#L151-L163

[terryjreedy]

Have you tried callable(f)?

>>> inspect.isfunction(int)
False
>>> callable(int)
True

callable checks for f.__call__, the requirement to be callable.

>>> class Func():
...     def __call__(self): return 'A func'
... 
>>> callable(Func())
True

[scoder]

Have you tried `callable(f)`

Callable only says that an object can be called. It does not say anything about what else the object can do, basically everything that distinguishes a function from an arbitrary callable.

[terryjreedy]

I do not know which distinction you are using. I don't think of any callable as 'doing' anything other than performing side-effects, raising exceptions, and returning values when called. I don't think of having attributes as doing anything. Anyway, some more thoughts.

'function' has various meanings in Python, 'callable in Python code' being the most general. The most narrow is being a callable defined in Python code, specifically with a def statement or a lambda expression. CPython compiles these to instances of class 'function'. This definition is the one used by inspect.isfunction, which returns False for built-in functions such as those listed in the Built-in Functions chapter of the Library reference. (isclass, by contrast does not care about coding language as that does not affect the expected attributes.)

As to the specific proposal, the abc module only has infrastructure, not specific ABCs. The latter are contained in collections.abc. In spite of the title, it is not limited to collection ABCs. In particular, Callable is there. But it would that the entry should be empty; ABCs are about expected methods, whereas your concern about identifying 'function's is about non-function attributes. For most ABCs, So ABCs do not seem the proper mechanism for this.

Since the inspect.isobject functions are about expected non-function attributes, it might seem that asking that inspect.isfunction return True for the objects cython creates from def statements would be more appropriate. Though that raises the question of whether they are sufficiently compatible to do so.

Signatures are not specific to 'functions'; inspect.signature() is the intended method to fetch them. It knows how to do so for the various CPython function types. IDLE calls it for callable calltips (backed up by docstrings). I presume other IDEs do the same or similar. If not now, It would be nice if inspect.signature worked for functions in cythonized modules.

Code looking to use __code__ objects can just access the attribute or use hasattr. One reason that this is not a sufficient replacement for ABCs is that most involve many methods.

[scoder]

inspect.signature() already works and internally calls _signature_from_function() to analyse ("function-like") Cython functions. I'd consider that sufficiently compatible.

it might seem that asking that inspect.isfunction return True for the objects cython creates from def statements would be more appropriate.

I've long argued that it's good that inspect.isfunction() restricts itself to 'real' Python functions that are executed by the interpreter in actual byte code instead of using duck-typing, and that there should be a separate way to ask "does this look like a function to anyone?".

The reason why I think an ABC would be nice is that it's easy to use for both sides. Easy to register with and easy and quick to type-check. A protocol check (as done by inspect._signature_is_functionlike()) seems quite heavy in comparison:

cpython/Lib/inspect.py

Lines 2053 to 2075 in b0f77c4

	def _signature_is_functionlike(obj):
	"""Private helper to test if `obj` is a duck type of FunctionType.
	A good example of such objects are functions compiled with
	Cython, which have all attributes that a pure Python function
	would have, but have their code statically compiled.
	"""
	if not callable(obj) or isclass(obj):
	# All function-like objects are obviously callables,
	# and not classes.
	return False
	name = getattr(obj, '__name__', None)
	code = getattr(obj, '__code__', None)
	defaults = getattr(obj, '__defaults__', _void) # Important to use _void ...
	kwdefaults = getattr(obj, '__kwdefaults__', _void) # ... and not None here
	annotations = getattr(obj, '__annotations__', None)
	return (isinstance(code, types.CodeType) and
	isinstance(name, str) and
	(defaults is None or isinstance(defaults, tuple)) and
	(kwdefaults is None or isinstance(kwdefaults, dict)) and
	(isinstance(annotations, (dict)) or annotations is None) )

If you look at the examples that I linked to, they pretty much list various different callable types that they want to handle (non-)specially, including functions, whether they use byte code or binary code. To be honest, I don't know why they do it that way. I asked a couple of times in the past but never got a concrete answer. What I know is that it is a need that several Python developers have felt in the past and that they resolved by adding knowledge about Cython to their code base. And I don't think it belongs there. If they don't care how a function is implemented, why should they know about the different implementations?

[JelleZijlstra]

Which attributes would you include in this Function ABC?

For comparison, the Coroutine ABC only includes the public methods (send, throw, close), not any of the introspection attributes like __name__ or internal ones like cr_code.

[AA-Turner]

Would alterine isroutine() to regognise Cython functions be appropriate?

[terryjreedy]

A better solution than adding something to the stdlib only for cython users might be a small 'cy_func' module on PYPI. It could cover as many Python and Cython versions as Cython people desired and contain whatever Cython people want and do whatever and change whenever needed without regard to the CPython schedule. I read the linked discussion and did not see any consensus at to what exactly is needed, and I suspect that some experimentation might be needed.

[NicolasT]

I'd be interested in something like this as well. We have a custom function-like type (implemented in a C extension) which quacks like a function object as much as possible: __module__, __name__, __qualname__ and __doc__ attributes, tp_call and tp_vectorcall_offset provided, Py_TPFLAGS_METHOD_DESCRIPTOR support, __annotations__ and __signature__ attributes to pass along typing information,...

However we're required to (monkey)patch at least Sphinx and mypy's stubgen to convince them objects of this type exposed on some module are, well, functions. I believe Sphinx already has some specific code to detect Cython functions, and similarly, stubgen has specific code to detect and handle Pybind11 functions.

Having some "standardized" mechanism to declare a type to be function-like would be useful.

[scoder]

Would altering isroutine() to recognise Cython functions be appropriate?

That sounds reasonable, too. But as mentioned, Cython is not the only target for this. Other tools like pybind11, mypyc, numba, etc. might also have function-like types that they'd want to see recognised. If we change isroutine(), we'd need to make sure function-like objects are either "correctly" detected by it, or tools get a way to register their types as function-like.

Again, an ABC would make this easy.