You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: conformance/README.md
+1-2Lines changed: 1 addition & 2 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -75,13 +75,12 @@ To run the conformance test suite:
75
75
* Switch to the `src` subdirectory and run `python main.py`.
76
76
77
77
Note that some type checkers may not run on some platforms. If a type checker fails to install, tests will be skipped for that type checker.
78
-
Currently, the only unsupported type checker is Pyre on Windows.
79
78
80
79
## Reporting Conformance Results
81
80
82
81
Different type checkers report errors in different ways (with different wording in error messages and different line numbers or character ranges for errors). This variation makes it difficult to fully automate test validation given that tests will want to check for both false positive and false negative type errors. Some level of manual inspection will therefore be needed to determine whether a type checker is fully conformant with all tests in any given test file. This "scoring" process is required only when the output of a test changes — e.g. when a new version of that type checker is released and the tests are rerun. We assume that the output of a type checker will be the same from one run to the next unless/until a new version is released that fixes or introduces a bug. In this case, the output will need to be manually inspected and the conformance results re-scored for those tests whose output has changed.
83
82
84
-
Conformance results are reported and summarized for each supported type checker. Currently, results are reported for mypy, pyre, and pyright. It is the goal and desire to add additional type checkers over time.
83
+
Conformance results are reported and summarized for each supported type checker. Currently, results are reported for mypy, pyrefly, pyright, and zuban. It is the goal and desire to add additional type checkers over time.
Copy file name to clipboardExpand all lines: conformance/results/mypy/annotations_forward_refs.toml
-3Lines changed: 0 additions & 3 deletions
Original file line number
Diff line number
Diff line change
@@ -31,11 +31,8 @@ annotations_forward_refs.py:96: error: Expression is of type int?, not "int" [a
31
31
"""
32
32
conformance_automated = "Fail"
33
33
errors_diff = """
34
-
Line 22: Expected 1 errors
35
-
Line 23: Expected 1 errors
36
34
Line 24: Expected 1 errors
37
35
Line 25: Expected 1 errors
38
-
Line 66: Expected 1 errors
39
36
Line 87: Unexpected errors ['annotations_forward_refs.py:87: error: Function "annotations_forward_refs.ClassD.int" is not valid as a type [valid-type]']
40
37
Line 96: Unexpected errors ['annotations_forward_refs.py:96: error: Expression is of type int?, not "int" [assert-type]']
0 commit comments