PEP 612 clarification: ParamSpec that captures a method signature

[erictraut]

I'm a contributor to pyright, and I'm hoping to get some clarification about a particular use case involving ParamSpec (introduced in PEP 612) and __call__ methods.

Consider the following code, which executes with no runtime exceptions.

from functools import cache

class Foo:
    def not_cached(self) -> int:
        return 0

    @cache
    def cached(self) -> int:
        return 0

foo = Foo()
foo.not_cached()
foo.cached()

The functools.cache decorator is defined in typeshed's functools.pyi as follows:

def cache(__user_function: Callable[_P, _T]) -> _lru_cache_wrapper[_P, _T]: ...

And _lru_cache_wrapper is defined as:

class _lru_cache_wrapper(Generic[_P, _T]):
    def __call__(self, *args: _P.args, **kwargs: _P.kwargs) -> _T: ...
    ...

When the @cache decorator is applied to the method cached, the ParamSpec _P captures the input parameters of cached. In other words, it captures the parameter list (self). The type of parameter self in this case is implied to be Foo or some subclass thereof (or if we adopt the terminology in draft PEP 637, its type is Self or Self@Foo).

The decorated type of cached is now _lru_cache_wrapper[(self: Self@Foo), int]. The __call__ method for this specialized class is effectively now typed as:

   def __call__(self: Self@_lru_cache_wrapper, self: Self@Foo) -> int: ...

For the call expression foo.cached() in the sample above, it appears that there is a missing argument (the one corresponding to the second Self@Foo parameter). Pyright therefore generates an error in this case, but clearly this isn't an error at runtime.

How do we avoid a false positive error in this case?

Have the authors of PEP 612 have considered this situation? If so, what is the solution? I haven't found anything in the PEP that describes the intended behavior.

Perhaps this is just the consequence of the functools.cache stub definition, which is attempting to model the runtime behavior but is in effect "lying" about some of the details? In that case, is there a better way to define functools.cache and similar decorators? Maybe through a use of descriptor?

[Akuli]

I think _lru_cache_wrapper should be a descriptor, not just a callable, because the runtime uses a def statement to create it.

[pradeep90]

@erictraut Mark isn't actively working on Pyre anymore, so I can step in for PEP 612 related questions.

The decorated type of cached is now _lru_cache_wrapper[(self: Self@Foo), int].

This implies that we need to leave out the self in the _lru_cache_wrapper.__call__ method since it is already provided by _P.args and _P.kwargs:

class _lru_cache_wrapper(Generic[_P, _T]):
    def __call__(*args: _P.args, **kwargs: _P.kwargs) -> _T: ...

The following code works in Pyre (you'd have to change pyre_extensions to typing, ParameterSpecification to ParamSpec):

from typing import *
from pyre_extensions import ParameterSpecification
    
_P = ParameterSpecification("_P")
_T = TypeVar("_T")
    
class _lru_cache_wrapper(Generic[_P, _T]):
    def __call__(*args: _P.args, **kwargs: _P.kwargs) -> _T: ...

def cache(__user_function: Callable[_P, _T]) -> _lru_cache_wrapper[_P, _T]: ...

class Foo:
    def not_cached(self) -> int:
        return 0

    @cache
    def cached(self, x: str, y: int) -> int:
        return 0

foo = Foo()
foo.not_cached()

foo.cached("hello", 42)

# Missing argument [20]: Call `_lru_cache_wrapper.__call__` expects argument `x`.
foo.cached()
# Incompatible parameter type [6]: In call `_lru_cache_wrapper.__call__`, for 1st positional only parameter expected `str` but got `int`.
foo.cached(1, 2)

If that sounds reasonable, we can probably update the PEP (along with a couple of clarifications that others had brought up).

[Gobot1234]

Would it be possible to implement this behaviour in pyright @erictraut?

[erictraut]

I've created an issue in the pyright issue tracker to track this work. Note that typeshed won't be able to use this technique unless/until mypy also implements this change. None of the mypy maintainers have chimed in on this thread, so I don't know if they will agree to this change. I'd prefer to get a nod and a commitment from them first.

Edit: Updated link to correct issue.

[Gobot1234]

FWIW that link should be for microsoft/pyright#5267

[erictraut]

I've investigated this further, and I'm uncomfortable with the proposed solution because it feels to me like a pretty significant hack. The problem is that the captured signature contains a self parameter with a type associated with it.

In the example posted above by Pradeep, the type of the self parameter in the undecorated cached method is of type Self@Foo (i.e. a Self type that is associated with the Foo class). Once the decorator is applied, the type of cached becomes _lru_cache_wrapper[[self: Self@Foo, x: str, y: int], int]. That means the specialized type of the _lru_cache_wrapper.__call__ method is (self: Self@Foo, x: str, y: int) -> int. The self parameter is now incorrectly typed. It should be Self@_lru_cache_wrapper. Any attempt to bind this method to an _lru_cache_wrapper object will result in an error.

The only way I see to avoid this is to add a hack that detects this particular condition and magically replaces the type of the self parameter with something other than what was captured by the ParamSpec. This sounds fragile to me. How would we document this? Under what specific specific circumstances would this hack apply?

@pradeep90, you mentioned that this works in pyre. How do you work around this issue?

[DevilXD]

There is a similar issue that it appears involves the same problem, in this PR: python/typeshed#6670

The thing is, removing self from _Wrapped.__call__ doesn't fix it. I'm not sure why, trying to follow the same logic as here tells me that it should, yet the error stays the same (at least according to the latest MyPy). I don't know how to debug MyPy so I'm unable to go any further with it.

It seems like it'd be cool to have some kind of a descriptor type, that'd handle inserting the self argument for methods, sort of like the existing classes use descriptors to generate bound methods.