Prototyping a generic types.FunctionType or built-in function-like class

[bzoracler]

I'm trying to recreate a drop-in replacement of function (types.FunctionType) (and other similar classes for builtins), that will at least work for both functions and instance methods (reflecting the runtime behaviour). I basically got stuck trying to simultaneously preserve the call signature and the instance descriptor protocol:

import typing as tp

FirstT = tp.TypeVar("FirstT")
CallableT = tp.TypeVar("CallableT", bound=tp.Callable[..., tp.Any])
P = tp.ParamSpec("P")
R = tp.TypeVar("R")
S = tp.TypeVar("S")

class nothing: ...

class MethodType_(tp.Generic[CallableT, FirstT, P, R]):
    def __call__(self, *args: P.args, **kwargs: P.kwargs) -> R: ...
    # Actually `types.MemberDescriptorType`s
    __self__: FirstT
    __func__: CallableT

class FunctionType_(tp.Generic[FirstT, P, R]):
    def __call__(self, __first: FirstT, *args: P.args, **kwargs: P.kwargs) -> R: ...
    @tp.overload
    def __get__(self, __instance: None, __type: type) -> FunctionType_[FirstT, P, R]: ...
    @tp.overload
    def __get__(self: CallableT, __instance: S, __type: type[S]) -> MethodType_[CallableT, S, P, R]: ...

@tp.overload
def function_(callable_: tp.Callable[tp.Concatenate[FirstT, P], R], /) -> FunctionType_[FirstT, P, R]: ...
@tp.overload
def function_(callable_: tp.Callable[P, R], /) -> FunctionType_[nothing, P, R]: ...

@function_
def my_func(a: int, b: str) -> None: pass

class C:
    @function_
    def my_method(self, a: int, b: str) -> None: pass

>>> my_func(8, b="")  # OK
>>> my_func(a=8, b="")  # Bad - `a` is positional-only
>>> C().my_method(a=8, b="")  # OK
>>> C.my_method(C(), a=8, b="")  # OK
>>> C.my_method(self=C(), a=8, b="")  # Bad - `self` is positional-only

I have three related questions:

1. Is it possible to statically type a callable with at least one parameter whose overload is considered first, such that type-checkers don't complain? Currently, both mypy and pyright say that the two function_s have overlapping signatures with incompatible returns.
2. Is there any workaround to model "extracting the first parameter of a ParamSpec" (i.e. opposite to what Concatenate does), without using things like mypy_extensions, so that we can exactly preserve the nature of the first parameter being a positional-or-keyword argument?
3. Is there any way to avoid the awkward FirstT generic in FunctionType_, and the __first parameter in FunctionType_.__call__, and to replicate the instance method signature from __get__ using just a FunctionType_(tp.Generic[P, R])?

---

I know that you can instead implement the behaviour separately in two different decorators, e.g. pure function behaviour in @function_ and instance method behaviour in @method_. I'm trying to see if this can be avoided.

[erictraut]

Does this meet your needs?

from typing import Callable, Generic, ParamSpec, Protocol, TypeVar, overload

SelfT = TypeVar("SelfT", contravariant=True)
P = ParamSpec("P")
R = TypeVar("R", covariant=True)
S = TypeVar("S")

class MethodProto(Protocol[SelfT, P, R]):
    def __call__(_self, self: SelfT, *args: P.args, **kwargs: P.kwargs) -> R:
        ...

class MethodType_(Generic[SelfT, P, R]):
    __self__: SelfT
    __func__: Callable[P, R]

    def __call__(self, *args: P.args, **kwargs: P.kwargs) -> R: ...

    @overload
    def __get__(self, __instance: None, __type: type) -> "MethodProto[SelfT, P, R]": ...
    @overload
    def __get__(self: Callable[P, R], __instance: S, __type: type[S]) -> "Callable[P, R]": ...

class FunctionType_(Generic[P, R]):
    def __call__(self, *args: P.args, **kwargs: P.kwargs) -> R: ...

@overload
def function_(callable_: MethodProto[SelfT, P, R]) -> MethodType_[SelfT, P, R]: ...
@overload
def function_(callable_: Callable[P, R], /) -> FunctionType_[P, R]: ...

[bzoracler]

Thanks for that example! Bar the overload overlapping warning for function_, on pyright that works perfectly.

Unfortunately, mypy is not happy with this. Stripped down:

import typing as tp

SelfT = tp.TypeVar("SelfT", contravariant=True)
P = tp.ParamSpec("P")
R = tp.TypeVar("R", covariant=True)

class MethodProto(tp.Protocol[SelfT, P, R]):
    def __call__(_self, self: SelfT, *args: P.args, **kwargs: P.kwargs) -> R: ...

def function_(callable_: MethodProto[SelfT, P, R], /) -> MethodProto[SelfT, P, R]: ...

class C:
    # Argument 1 to "function_" has incompatible type "Callable[[C, int, str], None]"; expected "MethodProto[<nothing>, <nothing>, <nothing>]"  [arg-type]
    @function_
    def my_method(self, a: int, b: str) -> None:
        pass

Looks like mypy interprets callback protocols differently from pyright. mypy wants def __call__(_self, self: SelfT, *args: P.args, **kwargs: P.kwargs) -> R: ... to match exactly

class C:
    def my_method(self, *args, **kwargs):
        pass

so I guess their permissive *args, **kwargs implementation of callback protocols don't currently account for instance or class methods.

[erictraut]

That might be a bug that the mypy maintainers would be willing to address. You could file a bug report and get their thoughts on the matter.

It looks like mypy does accept the following: __call__: tp.Callable[tp.Concatenate[SelfT, P], R], so that might be a workaround to consider.