You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: conformance/README.md
+38-37Lines changed: 38 additions & 37 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -10,26 +10,26 @@ Accompanying the typing specification is this conformance test suite which valid
10
10
11
11
This project contains test cases for behaviors defined in the Python typing spec. Tests are structured and grouped in accordance with the specification's chapter headings.
A test file is a ".py" file. The file name should start with one of the above names followed by a description of the test (with words separated by underscores). For example, `generics_paramspec_basic_usage.py` would contain the basic usage tests for `ParamSpec`. Each test file can contain multiple individual unit tests, but these tests should be related to each other. If the number of unit tests in a single test file exceeds ten, it may be desirable to split it into separate test files. This will help maintain a consistent level of granularity across tests.
35
35
@@ -43,44 +43,45 @@ The test suite focuses on static type checking not general Python semantics. Tes
43
43
44
44
Test cases use the following conventions:
45
45
46
-
* Lines that are expected to produce a type checker error should have a comment starting with # E",
46
+
- Lines that are expected to produce a type checker error should have a comment starting with # E",
47
47
either by itself or followed by an explanation after a colon (e.g., "# E: int is not a subtype
48
48
of str"). Such explanatory comments are purely for human understanding, but type checkers are not
49
49
expected to use their exact wording. There are several syntactic variations; see "Test Case Syntax"
50
50
below.
51
-
* Lines that may produce an error (e.g., because the spec allows multiple behaviors) should be
51
+
- Lines that may produce an error (e.g., because the spec allows multiple behaviors) should be
52
52
marked with "# E?" instead of "# E".
53
-
* If a test case tests conformance with a specific passage in the spec, that passage should be
53
+
- If a test case tests conformance with a specific passage in the spec, that passage should be
54
54
quoted in a comment prefixed with "# > ".
55
55
56
56
## Test Case Syntax
57
57
58
58
Test cases support the following special comments for declaring where errors should be raised:
59
59
60
-
*`# E`: an error must be raised on this line
61
-
*`# E?`: an error may be raised on this line
62
-
*`# E[tag]`, where `tag` is an arbitrary string: must appear multiple times in a file with the same tag.
60
+
-`# E`: an error must be raised on this line
61
+
-`# E?`: an error may be raised on this line
62
+
-`# E[tag]`, where `tag` is an arbitrary string: must appear multiple times in a file with the same tag.
63
63
Exactly one line with this tag must raise an error.
64
-
*`# E[tag+]`: like `# E[tag]`, but errors may be raised on multiple lines.
64
+
-`# E[tag+]`: like `# E[tag]`, but errors may be raised on multiple lines.
65
65
66
66
Each comment may be followed by a colon plus an explanation of the error; the explanation is ignored
67
67
by the scoring system.
68
68
69
69
## Running the Conformance Test Tool
70
70
71
71
To run the conformance test suite:
72
-
* Clone the https://github.com/python/typing repo.
73
-
* Create and activate a Python 3.12 virtual environment.
74
-
* Switch to the `conformance` subdirectory and install all dependencies (`pip install -r requirements.txt`).
75
-
* Switch to the `src` subdirectory and run `python main.py`.
72
+
73
+
- Clone the https://github.com/python/typing repo.
74
+
- Create and activate a Python 3.12 virtual environment.
75
+
- Switch to the `conformance` subdirectory and install all dependencies (`pip install -r requirements.txt`).
76
+
- Switch to the `src` subdirectory and run `python main.py`.
76
77
77
78
Note that some type checkers may not run on some platforms. If a type checker fails to install, tests will be skipped for that type checker.
78
79
79
80
## Reporting Conformance Results
80
81
81
82
Different type checkers report errors in different ways (with different wording in error messages and different line numbers or character ranges for errors). This variation makes it difficult to fully automate test validation given that tests will want to check for both false positive and false negative type errors. Some level of manual inspection will therefore be needed to determine whether a type checker is fully conformant with all tests in any given test file. This "scoring" process is required only when the output of a test changes — e.g. when a new version of that type checker is released and the tests are rerun. We assume that the output of a type checker will be the same from one run to the next unless/until a new version is released that fixes or introduces a bug. In this case, the output will need to be manually inspected and the conformance results re-scored for those tests whose output has changed.
82
83
83
-
Conformance results are reported and summarized for each supported type checker. Currently, results are reported for mypy, pyrefly, pyright, and zuban. It is the goal and desire to add additional type checkers over time.
84
+
Conformance results are reported and summarized for each supported type checker. Currently, results are reported for mypy, pyrefly, pyright, zuban and ty. It is the goal and desire to add additional type checkers over time.
84
85
85
86
## Adding a New Test Case
86
87
@@ -98,16 +99,16 @@ If a new version of a type checker is released, re-run the test tool with the ne
98
99
99
100
In addition to manual scoring, we provide an experimental tool that automatically checks type checkers for conformance. This tool relies on the "# E" comments present in the stubs and on parsing type checker output. This logic is run automatically as part of the conformance test tool. It produces the following fields in the `.toml` output files:
100
101
101
-
*`errors_diff`: a string describing all issues found with the type checker's behavior: either expected errors that were not emitted, or extra errors that the conformance test suite does not allow.
102
-
*`conformance_automated`: either "Pass" or "Fail" based on whether there are any discrepancies with the expected behavior.
102
+
-`errors_diff`: a string describing all issues found with the type checker's behavior: either expected errors that were not emitted, or extra errors that the conformance test suite does not allow.
103
+
-`conformance_automated`: either "Pass" or "Fail" based on whether there are any discrepancies with the expected behavior.
103
104
104
105
This tool does not yet work reliably on all test cases. The script `conformance/src/unexpected_fails.py` can be run to find all test cases where the automated tool's conformance judgment differs from the manual judgment entered in the `.toml` files.
105
106
106
107
Some common problems with automated checks:
107
108
108
-
* Sometimes the spec is imprecise or allows multiple options. In this case, use "# E?" to mark an error as optional.
109
-
* Type checkers may produce additional errors for issues unrelated to the topic being tested. In this case, add an extra field `ignore_errors` in the type checker's `.toml` file that contains the text of the irrelevant errors. Any error message that contains a substring in the `ignore_errors` list is ignored. For example, if `ignore_errors = ["Too many arguments"]`, then a mypy error `dataclasses_usage.py:127: error: Too many arguments for "DC7" [call-arg]` will be ignored.
110
-
* Type checkers may differ in the line on which they report an error. In this case, on each of the lines where an error could
109
+
- Sometimes the spec is imprecise or allows multiple options. In this case, use "# E?" to mark an error as optional.
110
+
- Type checkers may produce additional errors for issues unrelated to the topic being tested. In this case, add an extra field `ignore_errors` in the type checker's `.toml` file that contains the text of the irrelevant errors. Any error message that contains a substring in the `ignore_errors` list is ignored. For example, if `ignore_errors = ["Too many arguments"]`, then a mypy error `dataclasses_usage.py:127: error: Too many arguments for "DC7" [call-arg]` will be ignored.
111
+
- Type checkers may differ in the line on which they report an error. In this case, on each of the lines where an error could
111
112
reasonably be shown, write `# E[<tag>]`, where `<tag>` is an arbitrary string that is unique in the file. The test will be marked as passing if the type checker produces an error on exactly one of the lines where this tag appears.
Copy file name to clipboardExpand all lines: conformance/results/pyrefly/protocols_explicit.toml
+2-2Lines changed: 2 additions & 2 deletions
Original file line number
Diff line number
Diff line change
@@ -3,14 +3,14 @@ notes = """
3
3
Does not reject call to super method with no default implementation.
4
4
Does not detect stub methods inherited from protocols as abstract.
5
5
"""
6
-
conformance_automated = "Fail"
6
+
conformance_automated = "Pass"
7
7
errors_diff = """
8
-
Line 89: Expected 1 errors
9
8
"""
10
9
output = """
11
10
ERROR protocols_explicit.py:27:16-28: Method `draw` inherited from class `PColor` has no implementation and cannot be accessed via `super()` [missing-attribute]
12
11
ERROR protocols_explicit.py:56:20-36: `tuple[int, int, str]` is not assignable to attribute `rgb` with type `tuple[int, int, int]` [bad-assignment]
13
12
ERROR protocols_explicit.py:60:10-20: Cannot instantiate `Point` because the following members are abstract: `intensity`, `transparency` [bad-instantiation]
13
+
ERROR protocols_explicit.py:89:15-17: Cannot instantiate `Concrete1` because the following members are abstract: `cm1` [bad-instantiation]
14
14
ERROR protocols_explicit.py:134:15-17: Cannot instantiate `Concrete5` because the following members are abstract: `method1` [bad-instantiation]
15
15
ERROR protocols_explicit.py:164:17-19: Cannot instantiate `Concrete7A` because the following members are abstract: `method1` [bad-instantiation]
0 commit comments