[PEP 696] Fix swapping TypeVars with defaults.

[randolf-scholz]

• Fixes [PEP 696] Assigning to type variables with defaults gives unexpected results #19444. (added testTypeVarDefaultsSwap)
• Fixes Order dependent override error when subclassing with multiple typevars with defaults #19362 (added testTypeVarDefaultsSwap2)

Changed the logic for recursion guards of TypeVarType: Instead of always substituting repl = repl.accept(self), and situationally updating repl.default = repl.default.accept(self) if the result is a TypeVarType, we now always update repl.default = repl.default.accept(self) a priori and then only choose the expanded repl.accept(self) if the result is a concrete type.

New Tests

• testTypeVarDefaultsSwap (https://mypy-play.net/?mypy=1.17.0&python=3.12&gist=d5a025a31ae3c8b9e2a36f4738aa1991)
• testTypeVarDefaultsSwap2 (https://mypy-play.net/?mypy=1.17.0&python=3.12&gist=d3ed42c82f7144967c97d846c4c041ef)

PS: closed earlier PRs #19447, since it contained debugging changes, and #19448 because it didn't solve #19362.

[github-actions]

According to mypy_primer, this change doesn't affect type check results on a corpus of open source code. ✅

[ilevkivskyi]

Types should be never modified in place. I know you didn't introduce it, but this should be fixed ASAP. (I think there is also a chance some of the problems you try to fix may be caused by this).

[randolf-scholz]

Types should be never modified in place.

So, with copy_modified then? If stuff should never be modified in place, it surprises me that the variables are not annotated as Final in the first place.

[randolf-scholz]

Hm, If I change this then now 2 tests fail: testTypeVarDefaultsClassRecursive1 and testTypeVarDefaultsClassRecursiveMultipleFiles

____________________________________________________________________________________________________________________________________ testTypeVarDefaultsClassRecursive1 _____________________________________________________________________________________________________________________________________
[gw0] linux -- Python 3.13.8 /home/rscholz/github/CONTRIBUTING/mypy/.venv/bin/python3
data: /home/rscholz/github/CONTRIBUTING/mypy/test-data/unit/check-typevar-defaults.test:451:
Failed: Unexpected type checker output (/home/rscholz/github/CONTRIBUTING/mypy/test-data/unit/check-typevar-defaults.test, line 451)
------------------------------------------------------------------------------------------------------------------------------------------- Captured stderr call --------------------------------------------------------------------------------------------------------------------------------------------
Expected:
  main:16: note: Revealed type is "__main__.ClassD1[builtins.str, builtins.str]"
  main:17: note: Revealed type is "__main__.ClassD1[builtins.int, builtins.int]"
  main:18: note: Revealed type is "__main__.ClassD1[builtins.int, builtins.float]"
  main:21: note: Revealed type is "__main__.ClassD1[builtins.str, builtins.str]" (diff)
  main:23: note: Revealed type is "__main__.ClassD1[builtins.int, builtins.int]"
  main:25: note: Revealed type is "__main__.ClassD1[builtins.int, builtins.float]"
  main:35: note: Revealed type is "__main__.ClassD2[builtins.str, builtins.str, builtins.str]"
  main:36: note: Revealed type is "__main__.ClassD2[builtins.int, builtins.int, builtins.int]"
  main:37: note: Revealed type is "__main__.ClassD2[builtins.int, builtins.float, builtins.float]"
  main:38: note: Revealed type is "__main__.ClassD2[builtins.int, builtins.float, builtins.str]"
  main:41: note: Revealed type is "__main__.ClassD2[builtins.str, builtins.str, builtins.str]" (diff)
  main:43: note: Revealed type is "__main__.ClassD2[builtins.int, builtins.int, builtins.int]" (diff)
  main:45: note: Revealed type is "__main__.ClassD2[builtins.int, builtins.float, builtins.float]"
  main:47: note: Revealed type is "__main__.ClassD2[builtins.int, builtins.float, builtins.str]"
  main:56: note: Revealed type is "__main__.ClassD3[builtins.str, builtins.list[builtins.str]]"
  ...
  main:58: note: Revealed type is "__main__.ClassD3[builtins.int, builtins.float]"
  main:63: note: Revealed type is "__main__.ClassD3[builtins.int, builtins.list[builtins.int]]"
  main:65: note: Revealed type is "__main__.ClassD3[builtins.int, builtins.float]"
Actual:
  main:16: note: Revealed type is "__main__.ClassD1[builtins.str, builtins.str]"
  main:17: note: Revealed type is "__main__.ClassD1[builtins.int, builtins.int]"
  main:18: note: Revealed type is "__main__.ClassD1[builtins.int, builtins.float]"
  main:21: note: Revealed type is "__main__.ClassD1[builtins.str, T1`1 = builtins.str]" (diff)
  main:23: note: Revealed type is "__main__.ClassD1[builtins.int, builtins.int]"
  main:25: note: Revealed type is "__main__.ClassD1[builtins.int, builtins.float]"
  main:35: note: Revealed type is "__main__.ClassD2[builtins.str, builtins.str, builtins.str]"
  main:36: note: Revealed type is "__main__.ClassD2[builtins.int, builtins.int, builtins.int]"
  main:37: note: Revealed type is "__main__.ClassD2[builtins.int, builtins.float, builtins.float]"
  main:38: note: Revealed type is "__main__.ClassD2[builtins.int, builtins.float, builtins.str]"
  main:41: note: Revealed type is "__main__.ClassD2[builtins.str, T1`1 = builtins.str, T2`2 = T1`1 = builtins.str]" (diff)
  main:43: note: Revealed type is "__main__.ClassD2[builtins.int, builtins.int, T2`2 = builtins.int]" (diff)
  main:45: note: Revealed type is "__main__.ClassD2[builtins.int, builtins.float, builtins.float]"
  main:47: note: Revealed type is "__main__.ClassD2[builtins.int, builtins.float, builtins.str]"
  main:56: note: Revealed type is "__main__.ClassD3[builtins.str, builtins.list[builtins.str]]"
  ...
  main:58: note: Revealed type is "__main__.ClassD3[builtins.int, builtins.float]"
  main:63: note: Revealed type is "__main__.ClassD3[builtins.int, builtins.list[builtins.int]]"
  main:65: note: Revealed type is "__main__.ClassD3[builtins.int, builtins.float]"

Alignment of first line difference:
  E: ..._.ClassD1[builtins.str, builtins.str]"
  A: ..._.ClassD1[builtins.str, T1`1 = builtins.str]"
                                ^
Update the test output using --update-data (implies -n0; you can additionally use the -k selector to update only specific tests)
______________________________________________________________________________________________________________________________ testTypeVarDefaultsClassRecursiveMultipleFiles _______________________________________________________________________________________________________________________________
[gw0] linux -- Python 3.13.8 /home/rscholz/github/CONTRIBUTING/mypy/.venv/bin/python3
data: /home/rscholz/github/CONTRIBUTING/mypy/test-data/unit/check-typevar-defaults.test:518:
Failed: Unexpected type checker output (/home/rscholz/github/CONTRIBUTING/mypy/test-data/unit/check-typevar-defaults.test, line 518)
------------------------------------------------------------------------------------------------------------------------------------------- Captured stderr call --------------------------------------------------------------------------------------------------------------------------------------------
Expected:
  main:15: note: Revealed type is "__main__.ClassG1[builtins.int, builtins.int]"
  main:16: note: Revealed type is "__main__.ClassG1[builtins.str, builtins.str]"
  main:17: note: Revealed type is "__main__.ClassG1[builtins.str, builtins.float]"
  main:20: note: Revealed type is "__main__.ClassG1[builtins.int, builtins.int]" (diff)
  main:22: note: Revealed type is "__main__.ClassG1[builtins.str, builtins.str]"
  main:24: note: Revealed type is "__main__.ClassG1[builtins.str, builtins.float]"
Actual:
  main:15: note: Revealed type is "__main__.ClassG1[builtins.int, builtins.int]"
  main:16: note: Revealed type is "__main__.ClassG1[builtins.str, builtins.str]"
  main:17: note: Revealed type is "__main__.ClassG1[builtins.str, builtins.float]"
  main:20: note: Revealed type is "__main__.ClassG1[builtins.int, T2`1 = builtins.int]" (diff)
  main:22: note: Revealed type is "__main__.ClassG1[builtins.str, builtins.str]"
  main:24: note: Revealed type is "__main__.ClassG1[builtins.str, builtins.float]"

Alignment of first line difference:
  E: ..._.ClassG1[builtins.int, builtins.int]"
  A: ..._.ClassG1[builtins.int, T2`1 = builtins.int]"
                                ^
Update the test output using --update-data (implies -n0; you can additionally use the -k selector to update only specific tests)
========================================================================================================================================== short test summary info ==========================================================================================================================================
FAILED mypy/test/testcheck.py::TypeCheckSuite::check-typevar-defaults.test::testTypeVarDefaultsClassRecursive1
FAILED mypy/test/testcheck.py::TypeCheckSuite::check-typevar-defaults.test::testTypeVarDefaultsClassRecursiveMultipleFiles

I noticed that TypeVarType.__hash__ is given by hash((self.id, self.upper_bound, tuple(self.values))), so changing default does not change the hash.

[ilevkivskyi]

Yes. For historical reasons some types are constructed "gradually", so we can't make type attributes final. Maybe we will at some point, but that would be a big refactoring. For now, mypy contributors should remember a simple rule: types always new, symbols always original.

[ilevkivskyi]

If I change this then now 2 tests fail

Oh well, it means there is another bug. TBH I am not sure if default should be part of the hash conceptually (and what it even means to have a type variable with same id, but a different default). IMO the support for type variable defaults may need to be re-implemented from scratch, as it is problematic on a conceptual level: regular default is not a property of an argument, it is a property of a function; same for type variables, their defaults are not property of type variable type itself, but of the function/class that uses them.

[randolf-scholz]

So the problem seems to be with getting the correct ID's. In the test case, we have class D[T1=str, T2=T1].

>>> self.variables 
{1: T1`604 = builtins.str, 2: T2`605 = T1`1 = builtins.str}
>>> self.recursive_tvar_guard
{604: T1`604 = builtins.str, 605: None}
>>> repl.default = repl.default.accept(self)
>>> self.variables 
{1: T1`604 = builtins.str, 2: T2`605 = T1`604 = builtins.str}
>>> self.recursive_tvar_guard
{604: T1`604 = builtins.str, 605: None}

Modifying repl.default in-place ensures that in the ExpandTypeVisitor.variables dictionary, the default of the dependent type variable gets updated to the new ID.

[ilevkivskyi]

TBH I don't understand why do we need to handle the defaults in expand_type in the first place. As I explained, the default is not a property of a type variable, it is a property of a class/function, so the caller (e.g. applytype etc) should always provide a complete type var -> replacement mapping. So that expand_type() should not be even allowed to look at the defaults.

Yeah, I just looked at the original PR that added this logic #16878, it looks broken. The full complete mapping should be computed either in:

• semanal/typeanal for "static" application like x: C[int]
• in applytype for "dynamic" application like x = C[int]()

before we even call expand_type. I think the only right way to do it is this way.

[randolf-scholz]

I am not sure how to fix this at the moment, but given that this PR

1. Fixes two very annoying bugs with PEP696 checking (essentially prevents typing bijections with defaults)
2. Doesn't introduce inplace modification -- it was already there before (all we really do is make sure to update the default a-priori, which is crucial e.g. when the second typevar defaults to the first one and we expand the first one.)
3. This issue probably needs to be addressed elsewhere

Maybe this can still be merged as-is?

[ilevkivskyi]

The downside is that this is papering over the problem instead of fixing the real root cause. But anyway, if you feel strongly about this, I can merge this, it will not be much more work to axe this all later (and the tests are useful).