Protocols in decorated function parameters fail type-check somehow

[RedRoserade]

Hello, I have a question regarding decorators and protocols. I am trying to define a decorator that accepts any function that accepts an argument matching a certain protocol, in order to read that argument's values. However, mypy 1.1.1 complains that the function is not compatible (unfortunately not explaining why).

Below is a snippet with a minimal example.

import uuid
from dataclasses import dataclass
from functools import wraps
from typing import Protocol, Callable
from uuid import UUID


class HasBlockUuid(Protocol):
    # @property
    # def block_uuid(self) -> UUID:
    #     ...

    block_uuid: UUID

    # Fails with either the property or the field, expected
    # https://mypy.readthedocs.io/en/latest/common_issues.html?highlight=reveal_type#covariant-subtyping-of-mutable-protocol-members-is-rejected
    # to be the problem


# Objective: create a decorator for functions that receive
# any object with a "block_uuid", property, in order to do something with it.
# The returned function has exactly the same type signature.
def check_block(func: Callable[[HasBlockUuid], None]) -> Callable[[HasBlockUuid], None]:
    @wraps(func)
    def wrapper(event: HasBlockUuid) -> None:
        print("block uuid is", event.block_uuid)
        return func(event)

    return wrapper


@dataclass
class Example:
    block_uuid: UUID


@check_block
# ^ Argument 1 to "check_block" has incompatible type "Callable[[Example], None]"; expected "Callable[[HasBlockUuid], None]"  [arg-type]
def do_something(example: Example) -> None:
    print("doing something with", example)


do_something(Example(block_uuid=uuid.uuid4()))


def do_something_else(has_block_uuid: HasBlockUuid) -> None:
    print("block uuid is", has_block_uuid.block_uuid)
    

do_something_else(Example(block_uuid=uuid.uuid4()))  # OK

Pylance complains on the same line as mypy with:

Argument of type "(example: Example) -> None" cannot be assigned to parameter "func" of type "(HasBlockUuid) -> None" in function "check_block"
  Type "(example: Example) -> None" cannot be assigned to type "(HasBlockUuid) -> None"
    Parameter 1: type "HasBlockUuid" cannot be assigned to type "Example"
      "HasBlockUuid" is incompatible with "Example"

However, creating a second function that uses the protocol in its arguments, and calling it with a concrete type, has no issue, even though HasBlockUuid is apparently incompatible with Example.

Furthermore, pycharm does not complain about this.

I am wondering if I am missing something with parameter variance, but I haven't found what it is yet. Does anyone know?

[erictraut]

Function parameters are contravariant. While Example is a subtype of HasBlockUuid, the converse isn't true.

This makes sense if you think about the decorator check_block. It says that it accepts a function that expects to receive an argument that conforms to the HasBlockUuid protocol. That means check_block would be free to call this function with any object that conforms to HasBlockUuid. But the do_something function has more stringent requirements. It requires that its first parameter is an Example instance (or subtype thereof). If the check_block were to call it with some other object that happens to conform to the HasBlockUuid protocol (but is not a subtype of Example), it could crash. For example, what if do_something were to reference the expression example.__dataclass_fields__ which is valid on Example but is not guaranteed to exist on HasBlockUuid.

[erictraut]

You can make this work by defining a TypeVar that is bound to HasBlockUuid.

THasBlockUuid = TypeVar("THasBlockUuid", bound=HasBlockUuid)

def check_block(
    func: Callable[[THasBlockUuid], None]
) -> Callable[[THasBlockUuid], None]: ...

[RedRoserade]

Wow, 🤦 , that was easier than I expected, I tried too long to figure out a fancy solution.

Thanks!

Now, I still need to make this work with methods, but that's an entirely different question, and beyond the scope of this question.

[RedRoserade]

I decided to continue investigating this situation, and I made it closer to the real situation. check_block now expects an object having params, which in turn must have block_uuid. For reuse purposes (there are other functions like this), I decided to create a HasParams protocol that's generic on params:

TParams = TypeVar("TParams")


class HasParams(Protocol[TParams]):
    params: TParams

And, after some iterations of applying your suggestion, I reached this:

class HasBlockUuidParams(Protocol[THasBlockUuid]):
    params: THasBlockUuid


THasBlockUuidParams = TypeVar("THasBlockUuidParams", bound=HasBlockUuidParams)


def check_block(
    func: Callable[[THasBlockUuidParams], None]
) -> Callable[[THasBlockUuidParams], None]:
    ...

This does seem to type-check the following usage properly:

@dataclass
class Example:
    @dataclass
    class Params:
        block_uuid: UUID
        something_else: bool

    params: Params


@check_block
def do_something(example: Example) -> None:
    ...

However, mypy complains on the definition of THasBlockUuidParams that HasBlockUuidParams is missing type arguments. Pylance doesn't complain.

Even with the missing type argument, the following fails type-check, as expected:

@dataclass
class NotAnExample:
    @dataclass
    class Params:
        block_uuid: UUID

    params: Params


do_something(
    NotAnExample( # NotAnExample is not Example, OK
        params=NotAnExample.Params(
            block_uuid=uuid.uuid4(),
        ),
    )
)

It also fails (correctly) the following case:

class SomethingElse:
    pass


@check_block
def do_something_else(event: SomethingElse) -> None:
    pass

But where mypy does not complain (even on --strict mode), and I expected it to, is here:

class SomethingElseWithParams:
    class Params:
        not_block_uuid: str

    params: Params


@check_block
def do_something_else_2(event: SomethingElseWithParams) -> None:
    pass

Whereas Pylance will complain:

Argument of type "(event: SomethingElseWithParams) -> None" cannot be assigned to parameter "func" of type "(THasBlockUuidParams@check_block) -> None" in function "check_block"
  Type "(event: SomethingElseWithParams) -> None" cannot be assigned to type "(THasBlockUuidParams@check_block) -> None"
    Parameter 1: type "THasBlockUuidParams@check_block" cannot be assigned to type "SomethingElseWithParams"
      "HasBlockUuidParams[Unknown]*" is incompatible with "SomethingElseWithParams"

Now:

• If I don't use THasBlockUuidParams, and instead use the non-generic form, type checks will fail even on the "correct" example, as you explained, with Argument 1 to "check_block" has incompatible type "Callable[[Example], None]"; expected "Callable[[HasBlockUuidParams[<nothing>]], None]"
• If I add more type arguments to the function declaration, through the TypeVars (which is not allowed):

  def check_block(
    func: Callable[[THasBlockUuidParams[HasBlockUuid]], None]
  ) -> Callable[[THasBlockUuidParams[HasBlockUuid]], None]: ...

  Doesn't seem to do much, and only confuses Pylance further.

Either I'm missing something, or this is something not possible to do at the moment?

[erictraut]

mypy complains on the definition of THasBlockUuidParams that HasBlockUuidParams is missing type arguments

Pyright also complains about this if you enable reportMissingTypeArgument (or enable strict-mode checking). You can avoid this error in both mypy and pyright by supplying an Any for the type argument, as in bound=HasBlockUuidParams[Any].

If I add more type arguments to the function declaration, through the TypeVars...

You can't supply type arguments to a TypeVar because they are not generic types and are not parameterizable. There is a notion in some type systems called "Higher Kinded Types" (HKA's for short) that support "generic type variables", but Python's type system doesn't support this concept.

[RedRoserade]

Pyright also complains about this if you enable reportMissingTypeArgument (or enable strict-mode checking). You can avoid this error in both mypy and pyright by supplying an Any for the type argument, as in bound=HasBlockUuidParams[Any].

Indeed, though I was hoping to avoid using Any here so that everything was as explicit as possible. I did it as an exercise to see how far I could go.

There is a notion in some type systems called "Higher Kinded Types" (HKA's for short) that support "generic type variables", but Python's type system doesn't support this concept.

I see.