Updated conformance tests and results for latest versions of type che… (#1863)

* Updated conformance tests and results for latest versions of type checkers.

One test case in the `generics_syntax_infer_variance` was commented out temporarily until we decide how to handle variance inference for dataclasses in Python 3.13.

Mypy version was updated from 1.11.1 to 1.11.2. Minor error message change but no other substantive changes to the conformance results.

Pyright version was updated from 1.1.374 to 1.1.384. Many error messages changed slightly. Tests also identified three conformance regressions.

Pytype version was updated from 2024.04.11 to 2024.09.13. Error message format changed significantly, requiring updates to the test logic. No other substantive changes to conformance results.

Changed test infrastructure to make sure old versions of type checkers are uninstalled before installing new to guarantee correct version and increase determinism of timings.

* Added "--version" invocation to mypy after installation to eliminate the artificial time associated with malware scanning on MacOS.

Author: erictraut

conformance/results/mypy/annotations_forward_refs.toml

@@ -6,19 +6,18 @@ Incorrectly generates error for quoted type defined in class scope.
 """
 output = """
 annotations_forward_refs.py:41: error: Invalid type comment or annotation  [valid-type]
-annotations_forward_refs.py:42: error: Bracketed expression "[...]" is not valid as a type  [valid-type]
-annotations_forward_refs.py:43: error: Syntax error in type annotation  [syntax]
-annotations_forward_refs.py:43: note: Suggestion: Use Tuple[T1, ..., Tn] instead of (T1, ..., Tn)
+annotations_forward_refs.py:42: error: Invalid type comment or annotation  [valid-type]
+annotations_forward_refs.py:43: error: Invalid type comment or annotation  [valid-type]
 annotations_forward_refs.py:44: error: Invalid type comment or annotation  [valid-type]
 annotations_forward_refs.py:45: error: Invalid type comment or annotation  [valid-type]
 annotations_forward_refs.py:46: error: Invalid type comment or annotation  [valid-type]
 annotations_forward_refs.py:47: error: Invalid type comment or annotation  [valid-type]
 annotations_forward_refs.py:48: error: Invalid type comment or annotation  [valid-type]
 annotations_forward_refs.py:49: error: Variable "annotations_forward_refs.var1" is not valid as a type  [valid-type]
 annotations_forward_refs.py:49: note: See https://mypy.readthedocs.io/en/stable/common_issues.html#variables-vs-type-aliases
-annotations_forward_refs.py:50: error: Invalid type: try using Literal[True] instead?  [valid-type]
-annotations_forward_refs.py:51: error: Invalid type: try using Literal[1] instead?  [valid-type]
-annotations_forward_refs.py:52: error: Invalid type: try using Literal[-1] instead?  [valid-type]
+annotations_forward_refs.py:50: error: Invalid type comment or annotation  [valid-type]
+annotations_forward_refs.py:51: error: Invalid type comment or annotation  [valid-type]
+annotations_forward_refs.py:52: error: Invalid type comment or annotation  [valid-type]
 annotations_forward_refs.py:53: error: Invalid type comment or annotation  [valid-type]
 annotations_forward_refs.py:54: error: Invalid type comment or annotation  [valid-type]
 annotations_forward_refs.py:55: error: Module "types" is not valid as a type  [valid-type]

conformance/results/mypy/generics_syntax_infer_variance.toml

@@ -1,2 +1,2 @@
-version = "mypy 1.11.1"
-test_duration = 13.2
+version = "mypy 1.11.2"
+test_duration = 1.9

conformance/results/mypy/version.toml

@@ -8,7 +8,7 @@ Does not generate error for unquoted type defined in class scope.
 Does not treat triple-quoted forward reference annotation as implicitly parenthesized.
 """
 output = """
-annotations_forward_refs.py:41:8 Undefined or invalid type [11]: Annotation `eval(.join(map(chr, [105, 110, 116])))` is not defined as a type.
+annotations_forward_refs.py:41:8 Invalid type [31]: Expression `eval("".join(map(chr, [105, 110, 116])))` is not a valid type.
 annotations_forward_refs.py:42:8 Invalid type [31]: Expression `"[int, str]"` is not a valid type.
 annotations_forward_refs.py:43:8 Invalid type [31]: Expression `"(int, str)"` is not a valid type.
 annotations_forward_refs.py:44:8 Undefined or invalid type [11]: Annotation `comprehension(int for generators(generator($target$i in range(1) if )))` is not defined as a type.

conformance/results/pyre/annotations_forward_refs.toml

@@ -3,7 +3,7 @@ notes = """
 Type parameter syntax not yet support.
 """
 output = """
-generics_syntax_infer_variance.py:125:25 Parsing failure [404]: invalid syntax
+generics_syntax_infer_variance.py:132:25 Parsing failure [404]: invalid syntax
 """
 conformance_automated = "Fail"
 errors_diff = """
@@ -12,19 +12,18 @@ Line 17: Expected 1 errors
 Line 29: Expected 1 errors
 Line 37: Expected 1 errors
 Line 46: Expected 1 errors
-Line 55: Expected 1 errors
-Line 68: Expected 1 errors
-Line 79: Expected 1 errors
-Line 95: Expected 1 errors
-Line 96: Expected 1 errors
-Line 110: Expected 1 errors
-Line 111: Expected 1 errors
+Line 75: Expected 1 errors
+Line 86: Expected 1 errors
+Line 102: Expected 1 errors
+Line 103: Expected 1 errors
+Line 117: Expected 1 errors
 Line 118: Expected 1 errors
-Line 119: Expected 1 errors
-Line 120: Expected 1 errors
-Line 121: Expected 1 errors
-Line 129: Expected 1 errors
-Line 137: Expected 1 errors
-Line 148: Expected 1 errors
-Line 125: Unexpected errors ['generics_syntax_infer_variance.py:125:25 Parsing failure [404]: invalid syntax']
+Line 125: Expected 1 errors
+Line 126: Expected 1 errors
+Line 127: Expected 1 errors
+Line 128: Expected 1 errors
+Line 136: Expected 1 errors
+Line 144: Expected 1 errors
+Line 155: Expected 1 errors
+Line 132: Unexpected errors ['generics_syntax_infer_variance.py:132:25 Parsing failure [404]: invalid syntax']
 """

conformance/results/pyre/generics_syntax_infer_variance.toml

@@ -1,2 +1,2 @@
 version = "pyre 0.9.22"
-test_duration = 5.4
+test_duration = 4.5

conformance/results/pyre/version.toml

@@ -9,40 +9,40 @@ aliases_explicit.py:71:24 - error: Expected ParamSpec, ellipsis, or list of type
 aliases_explicit.py:79:21 - error: Invalid expression form for type alias definition (reportInvalidTypeForm)
 aliases_explicit.py:79:21 - error: Call expression not allowed in type expression (reportInvalidTypeForm)
 aliases_explicit.py:80:21 - error: Invalid expression form for type alias definition (reportInvalidTypeForm)
-aliases_explicit.py:80:21 - error: List expression not allowed in type annotation
+aliases_explicit.py:80:21 - error: List expression not allowed in type expression
   Use List[T] to indicate a list type or Union[T1, T2] to indicate a union type (reportInvalidTypeForm)
 aliases_explicit.py:80:21 - error: Expected class but received "list[Unknown]" (reportGeneralTypeIssues)
 aliases_explicit.py:81:21 - error: Invalid expression form for type alias definition (reportInvalidTypeForm)
-aliases_explicit.py:81:21 - error: Tuple expression not allowed in type annotation
+aliases_explicit.py:81:21 - error: Tuple expression not allowed in type expression
   Use tuple[T1, ..., Tn] to indicate a tuple type or Union[T1, T2] to indicate a union type (reportInvalidTypeForm)
 aliases_explicit.py:82:21 - error: Invalid expression form for type alias definition (reportInvalidTypeForm)
-aliases_explicit.py:82:21 - error: List expression not allowed in type annotation
+aliases_explicit.py:82:21 - error: List expression not allowed in type expression
   Use List[T] to indicate a list type or Union[T1, T2] to indicate a union type (reportInvalidTypeForm)
 aliases_explicit.py:82:21 - error: Expected class but received "list[type[int]]" (reportGeneralTypeIssues)
 aliases_explicit.py:83:21 - error: Invalid expression form for type alias definition (reportInvalidTypeForm)
-aliases_explicit.py:83:21 - error: Dictionary expression not allowed in type annotation
+aliases_explicit.py:83:21 - error: Dictionary expression not allowed in type expression
   Use Dict[T1, T2] to indicate a dictionary type (reportInvalidTypeForm)
 aliases_explicit.py:83:21 - error: Expected class but received "dict[str, Unknown]" (reportGeneralTypeIssues)
 aliases_explicit.py:83:28 - error: "b" is not defined (reportUndefinedVariable)
 aliases_explicit.py:84:21 - error: Invalid expression form for type alias definition (reportInvalidTypeForm)
 aliases_explicit.py:84:21 - error: Call expression not allowed in type expression (reportInvalidTypeForm)
 aliases_explicit.py:85:21 - error: Invalid expression form for type alias definition (reportInvalidTypeForm)
-aliases_explicit.py:85:21 - error: List expression not allowed in type annotation
+aliases_explicit.py:85:21 - error: List expression not allowed in type expression
   Use List[T] to indicate a list type or Union[T1, T2] to indicate a union type (reportInvalidTypeForm)
 aliases_explicit.py:85:21 - error: Expected class but received "list[type[int]]" (reportGeneralTypeIssues)
 aliases_explicit.py:85:27 - error: Expected class but received "Literal[0]" (reportGeneralTypeIssues)
 aliases_explicit.py:86:21 - error: Invalid expression form for type alias definition (reportInvalidTypeForm)
-aliases_explicit.py:86:21 - error: Ternary expression not allowed in type annotation (reportInvalidTypeForm)
+aliases_explicit.py:86:21 - error: Ternary expression not allowed in type expression (reportInvalidTypeForm)
 aliases_explicit.py:87:21 - error: Variable not allowed in type expression (reportInvalidTypeForm)
+aliases_explicit.py:88:22 - error: Invalid expression form for type alias definition (reportInvalidTypeForm)
 aliases_explicit.py:88:22 - error: Expected class but received "Literal[True]" (reportGeneralTypeIssues)
 aliases_explicit.py:89:22 - error: Invalid expression form for type alias definition (reportInvalidTypeForm)
 aliases_explicit.py:89:22 - error: Expected class but received "Literal[1]" (reportGeneralTypeIssues)
 aliases_explicit.py:90:22 - error: Invalid expression form for type alias definition (reportInvalidTypeForm)
-aliases_explicit.py:90:22 - error: Binary operator not allowed in type annotation (reportInvalidTypeForm)
-aliases_explicit.py:91:22 - error: Type annotations cannot use format string literals (f-strings) (reportGeneralTypeIssues)
+aliases_explicit.py:90:22 - error: Binary operator not allowed in type expression (reportInvalidTypeForm)
+aliases_explicit.py:91:22 - error: Type expressions cannot use format string literals (f-strings) (reportGeneralTypeIssues)
 aliases_explicit.py:100:5 - error: Type "list[Unknown]" is already specialized (reportInvalidTypeArguments)
-aliases_explicit.py:101:6 - error: Object of type "UnionType" is not callable
-  Attribute "__call__" is unknown (reportCallIssue)
+aliases_explicit.py:101:6 - error: Object of type "UnionType" is not callable (reportCallIssue)
 aliases_explicit.py:102:5 - error: Type "list[Unknown]" is already specialized (reportInvalidTypeArguments)
 aliases_explicit.py:102:5 - error: Type "set[Unknown]" is already specialized (reportInvalidTypeArguments)
 """

conformance/results/pyright/aliases_explicit.toml

@@ -7,8 +7,8 @@ aliases_implicit.py:78:29 - error: Too many type arguments provided for "GoodTyp
 aliases_implicit.py:79:29 - error: Too many type arguments provided for "GoodTypeAlias8[T@GoodTypeAlias8]"; expected 1 but received 2 (reportInvalidTypeForm)
 aliases_implicit.py:80:24 - error: Expected ParamSpec, ellipsis, or list of types (reportInvalidTypeForm)
 aliases_implicit.py:81:9 - error: Could not specialize type "GoodTypeAlias12[TFloat@GoodTypeAlias12]"
-  Type "str" is incompatible with type "float"
-    "str" is incompatible with "float" (reportInvalidTypeForm)
+  Type "str" is not assignable to type "float"
+    "str" is not assignable to "float" (reportInvalidTypeForm)
 aliases_implicit.py:106:9 - error: Variable not allowed in type expression (reportInvalidTypeForm)
 aliases_implicit.py:107:9 - error: Variable not allowed in type expression (reportInvalidTypeForm)
 aliases_implicit.py:108:9 - error: Variable not allowed in type expression (reportInvalidTypeForm)
@@ -23,8 +23,7 @@ aliases_implicit.py:116:10 - error: Variable not allowed in type expression (rep
 aliases_implicit.py:117:10 - error: Variable not allowed in type expression (reportInvalidTypeForm)
 aliases_implicit.py:118:10 - error: Variable not allowed in type expression (reportInvalidTypeForm)
 aliases_implicit.py:119:10 - error: Variable not allowed in type expression (reportInvalidTypeForm)
-aliases_implicit.py:133:6 - error: Object of type "UnionType" is not callable
-  Attribute "__call__" is unknown (reportCallIssue)
+aliases_implicit.py:133:6 - error: Object of type "UnionType" is not callable (reportCallIssue)
 aliases_implicit.py:135:5 - error: Type "list[Unknown]" is already specialized (reportInvalidTypeArguments)
 aliases_implicit.py:135:5 - error: Type "set[Unknown]" is already specialized (reportInvalidTypeArguments)
 """

conformance/results/pyright/aliases_implicit.toml

@@ -1,19 +1,19 @@
 conformant = "Pass"
 output = """
 aliases_newtype.py:11:8 - error: Argument of type "Literal['user']" cannot be assigned to parameter "_x" of type "int" in function "__init__"
-  "Literal['user']" is incompatible with "int" (reportArgumentType)
-aliases_newtype.py:12:14 - error: Expression of type "Literal[42]" is incompatible with declared type "UserId"
-  "Literal[42]" is incompatible with "UserId" (reportAssignmentType)
+  "Literal['user']" is not assignable to "int" (reportArgumentType)
+aliases_newtype.py:12:14 - error: Type "Literal[42]" is not assignable to declared type "UserId"
+  "Literal[42]" is not assignable to "UserId" (reportAssignmentType)
 aliases_newtype.py:20:16 - error: Second argument to "isinstance" must be a class or tuple of classes
   Class created with NewType cannot be used with instance and class checks (reportArgumentType)
 aliases_newtype.py:23:21 - error: Base class "UserId" is marked final and cannot be subclassed (reportGeneralTypeIssues)
 aliases_newtype.py:32:1 - error: NewType must be assigned to a variable with the same name (reportGeneralTypeIssues)
 aliases_newtype.py:38:19 - error: Expected no type arguments for class "GoodNewType1" (reportInvalidTypeArguments)
 aliases_newtype.py:44:38 - error: Expected class as second argument to NewType (reportGeneralTypeIssues)
 aliases_newtype.py:47:43 - error: Type variable "T" has no meaning in this context (reportGeneralTypeIssues)
-aliases_newtype.py:49:38 - error: NewType cannot be used with structural type (a protocol or TypedDict class) (reportGeneralTypeIssues)
+aliases_newtype.py:49:38 - error: NewType cannot be used with structural type (a Protocol or TypedDict class) (reportGeneralTypeIssues)
 aliases_newtype.py:51:38 - error: NewType cannot be used with Literal type (reportGeneralTypeIssues)
-aliases_newtype.py:58:38 - error: NewType cannot be used with structural type (a protocol or TypedDict class) (reportGeneralTypeIssues)
+aliases_newtype.py:58:38 - error: NewType cannot be used with structural type (a Protocol or TypedDict class) (reportGeneralTypeIssues)
 aliases_newtype.py:60:15 - error: NewType requires two positional arguments (reportCallIssue)
 aliases_newtype.py:62:38 - error: The second argument to NewType must be a known class, not Any or Unknown (reportGeneralTypeIssues)
 """