Skip to content

[PEP 695] Fix incorrect Variance Computation with Polymorphic Methods. #19466

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 10 commits into
base: master
Choose a base branch
from
Open

[PEP 695] Fix incorrect Variance Computation with Polymorphic Methods. #19466

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
10 commits
Select commit Hold shift + click to select a range
d7b88cc
use plain_self as subtitution type for self
randolf-scholz Jul 16, 2025
a0b037e
Update mypy/subtypes.py
randolf-scholz Jul 16, 2025
ab680d1
Merge branch 'master' into fix_variance_polymorphic_constructor
randolf-scholz Jul 16, 2025
0f33c79
Merge branch 'master' into fix_variance_polymorphic_constructor
randolf-scholz Jul 27, 2025
5d16a08
Merge branch 'master' into fix_variance_polymorphic_constructor
randolf-scholz Jul 28, 2025
25e8efa
Merge branch 'master' into fix_variance_polymorphic_constructor
randolf-scholz Aug 4, 2025
9870139
Merge branch 'master' into fix_variance_polymorphic_constructor
randolf-scholz Sep 16, 2025
0b2aead
added unit test for 18334
randolf-scholz Sep 16, 2025
307382e
Merge branch 'master' into fix_variance_polymorphic_constructor
randolf-scholz Oct 6, 2025
01e8651
Merge branch 'master' into fix_variance_polymorphic_constructor
randolf-scholz Oct 14, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion mypy/subtypes.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2212,7 +2212,8 @@ def infer_variance(info: TypeInfo, i: int) -> bool:
settable = False

# TODO: handle settable properties with setter type different from getter.
typ = find_member(member, self_type, self_type)
plain_self = fill_typevars_with_any(info) # self-type without type variables
typ = find_member(member, self_type, plain_self)
if typ:
# It's okay for a method in a generic class with a contravariant type
# variable to return a generic instance of the class, if it doesn't involve
Expand Down
30 changes: 30 additions & 0 deletions test-data/unit/check-python312.test
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -450,6 +450,36 @@ class Contra2[T]:
d1: Contra2[int] = Contra2[float]()
d2: Contra2[float] = Contra2[int]() # E: Incompatible types in assignment (expression has type "Contra2[int]", variable has type "Contra2[float]")

[case testPEP695InferVariancePolymorphicMethod]
class Cov[T]:
def get(self) -> T: ...
def new[S](self: "Cov[S]", arg: list[S]) -> "Cov[S]": ...

cov_pos: Cov[object] = Cov[int]()
Copy link
Member

@ilevkivskyi ilevkivskyi Oct 15, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is not safe, if I continue this example like this I get an error at runtime that is not detected:

class Sub(Cov[int]):
    def new(self, arg: list[int]) -> Sub:
        print(arg[0].to_bytes())
        return self

cov_pos: Cov[object] = Sub()
cov_pos.new([object()])

On a more general level, how exactly this:

class Cov[T]:
    def get(self) -> T: ...
    def new[S](self: Cov[S], arg: list[S]) -> Cov[S]: ...

is different from this

class Cov[T]:
    def get(self) -> T: ...
    def new(self, arg: list[T]) -> Cov[T]: ...

? Maybe I am missing something but these two are literally identical in terms of semantics. Can you give an example of uses of Cov where these two classes behave (or should behave) differently?

Copy link
Contributor Author

@randolf-scholz randolf-scholz Oct 15, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

? Maybe I am missing something but these two are literally identical in terms of semantics.

I think they are only identical when called as a bound method, but not when called on the class and passing self as a regular argument. Note that my original example in #19439 - which is how I came across this issue -- was in the context of classmethods, and the Cov test case was really just breaking this down to the most elementary MWE.

class Cov[T]:
    def get(self) -> T: ...
    def new[S](self: Cov[S], arg: list[S]) -> Cov[S]: ...
    
x: Cov[int]
Cov[str].new(x, [1,2,3])   # OK

whereas

class Cov[T]:
    def get(self) -> T: ...
    def new(self, arg: list[T]) -> Cov[T]: ...

x: Cov[int]
Cov[str].new(x, [1,2,3])   # not OK, self must be subtype of Cov[str]

Copy link
Contributor Author

@randolf-scholz randolf-scholz Oct 15, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Consequently, I'd argue that your Sub example is not a proper subclass of Cov. pyright agrees that Sub.new is does not override Cov.new in a compatible manner. Code sample in pyright playground

So I'd say this is actually a false negative in mypy. https://mypy-play.net/?mypy=latest&python=3.12&gist=f60a4694098406077af0b8627fc78557

Copy link
Member

@ilevkivskyi ilevkivskyi Oct 15, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Access on class object is not a good argument. For two reasons:

  • First, prohibiting overrides that are incompatible on class object only would prohibit a lot of use cases that people expect to be treated as safe. Most notably, overrides involving properties and various custom descriptors will stop working.
  • Second, my example doesn't involve or rely on class object access in any way. So it is at least weird to say the unsafety is caused by the class object access incompatibility.

Finally, things like C[int].method are not really well specified (for good reasons, google type erasure). Support for this was added to mypy only recently, following a popular demand.

Btw #18334 is not a bug, it is a true positive. Coming back to your original issue: the two revealed types here are different (and both IMO correct):

class Foo[T](Sequence[T]):
    @classmethod
    def new[T2](cls: "type[Foo[T2]]", arg: list[T2]) -> "Foo[T2]": ...

    @classmethod
    def new2[T2](cls, arg: list[T2]) -> "Foo[T2]": ...

tfi: type[Foo[int]]

reveal_type(tfi.new)  # def (arg: builtins.list[builtins.int]) -> tstgrp3.Foo[builtins.int]
reveal_type(tfi.new2)  # def [T2] (arg: builtins.list[T2`2]) -> tstgrp3.Foo[T2`2]

and this is the reason why a class with the first method is considered invariant. I understand why do you want to have the first one: the body should include something like return cls(arg) which would give an error with the second method.

TBH I don't think this is solvable without special-casing alternative constructors somehow. Also it looks like a flaw in PEP 695, there should be a simple way to override inferred variance.

Copy link
Collaborator

@sterliakov sterliakov Oct 15, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@ilevkivskyi your unsafety example is very similar to what I pointed out before, and the problem is roughly "yes, this is obviously unsafe, but the spec says so". What's the official mypy stance on the spec conformance? I prefer to read the spec as an advice, not a mandatory requirement, but IDK if that matches the project attitude.

If mypy considers spec conformance its goal, then this PR is correct, and a typing-sig discussion is needed to fix this spec unsoundness. If not, this PR makes the state of affairs worse, trading false positives for false negatives.

there should be a simple way to override inferred variance

There is, it's called typing.TypeVar. AFAIC this ability was one of the reasons to not even consider deprecating "old-style" generics. I think it's rare enough to not warrant any extra syntax (though I'm a bad person to ask about that, IMO PEP695 should have never been implemented at all), so not a PEP omission.

Copy link
Member

@ilevkivskyi ilevkivskyi Oct 15, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@sterliakov

I prefer to read the spec as an advice, not a mandatory requirement, but IDK if that matches the project attitude.

Yes, this is the official mypy stance: internal consistency is more important than consistency with the spec.

AFAIC this ability was one of the reasons to not even consider deprecating "old-style" generics

OK, I see :-)

Copy link
Contributor Author

@randolf-scholz randolf-scholz Oct 15, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

there should be a simple way to override inferred variance.

Feel free to comment or👍 my suggestion for explicit variance spec for PEP695 type hints in discourse.

cov_neg: Cov[int] = Cov[object]() # E: Incompatible types in assignment (expression has type "Cov[object]", variable has type "Cov[int]")

class Contra[T]:
def set(self, arg: T) -> None: ...
def new[S](self: "Contra[S]", arg: list[S]) -> "Contra[S]": ...

contra_pos: Contra[object] = Contra[int]() # E: Incompatible types in assignment (expression has type "Contra[int]", variable has type "Contra[object]")
contra_neg: Contra[int] = Contra[object]()


[case testPEP695SelfAttribute]
# https://github.com/python/mypy/issues/18334
from typing import Self

class Foo[T]:
instance: Self
def foo(self) -> T: ... # type:ignore[empty-body]

class Bar(Foo[int]): ...

# OK
foo: Foo[object] = Bar()


[case testPEP695InheritInvariant]
class Invariant[T]:
x: T
Expand Down