Skip to content

Commit 935f764

Browse files
committed
fixup! feat: add ty
1 parent 04f4a87 commit 935f764

File tree

1 file changed

+36
-37
lines changed

1 file changed

+36
-37
lines changed

conformance/README.md

Lines changed: 36 additions & 37 deletions
Original file line numberDiff line numberDiff line change
@@ -10,26 +10,26 @@ Accompanying the typing specification is this conformance test suite which valid
1010

1111
This project contains test cases for behaviors defined in the Python typing spec. Tests are structured and grouped in accordance with the specification's chapter headings.
1212

13-
- [concepts](https://typing.python.org/en/latest/spec/concepts.html)
14-
- [annotations](https://typing.python.org/en/latest/spec/annotations.html)
15-
- [specialtypes](https://typing.python.org/en/latest/spec/special-types.html)
16-
- [generics](https://typing.python.org/en/latest/spec/generics.html)
17-
- [qualifiers](https://typing.python.org/en/latest/spec/qualifiers.html)
18-
- [classes](https://typing.python.org/en/latest/spec/class-compat.html)
19-
- [aliases](https://typing.python.org/en/latest/spec/aliases.html)
20-
- [literals](https://typing.python.org/en/latest/spec/literal.html)
21-
- [protocols](https://typing.python.org/en/latest/spec/protocol.html)
22-
- [callables](https://typing.python.org/en/latest/spec/callables.html)
23-
- [constructors](https://typing.python.org/en/latest/spec/constructors.html)
24-
- [overloads](https://typing.python.org/en/latest/spec/overload.html)
25-
- [dataclasses](https://typing.python.org/en/latest/spec/dataclasses.html)
26-
- [typeddicts](https://typing.python.org/en/latest/spec/typeddict.html)
27-
- [tuples](https://typing.python.org/en/latest/spec/tuples.html)
28-
- [namedtuples](https://typing.python.org/en/latest/spec/namedtuples.html)
29-
- [narrowing](https://typing.python.org/en/latest/spec/narrowing.html)
30-
- [directives](https://typing.python.org/en/latest/spec/directives.html)
31-
- [distribution](https://typing.python.org/en/latest/spec/distributing.html)
32-
- [historical](https://typing.python.org/en/latest/spec/historical.html)
13+
* [concepts](https://typing.python.org/en/latest/spec/concepts.html)
14+
* [annotations](https://typing.python.org/en/latest/spec/annotations.html)
15+
* [specialtypes](https://typing.python.org/en/latest/spec/special-types.html)
16+
* [generics](https://typing.python.org/en/latest/spec/generics.html)
17+
* [qualifiers](https://typing.python.org/en/latest/spec/qualifiers.html)
18+
* [classes](https://typing.python.org/en/latest/spec/class-compat.html)
19+
* [aliases](https://typing.python.org/en/latest/spec/aliases.html)
20+
* [literals](https://typing.python.org/en/latest/spec/literal.html)
21+
* [protocols](https://typing.python.org/en/latest/spec/protocol.html)
22+
* [callables](https://typing.python.org/en/latest/spec/callables.html)
23+
* [constructors](https://typing.python.org/en/latest/spec/constructors.html)
24+
* [overloads](https://typing.python.org/en/latest/spec/overload.html)
25+
* [dataclasses](https://typing.python.org/en/latest/spec/dataclasses.html)
26+
* [typeddicts](https://typing.python.org/en/latest/spec/typeddict.html)
27+
* [tuples](https://typing.python.org/en/latest/spec/tuples.html)
28+
* [namedtuples](https://typing.python.org/en/latest/spec/namedtuples.html)
29+
* [narrowing](https://typing.python.org/en/latest/spec/narrowing.html)
30+
* [directives](https://typing.python.org/en/latest/spec/directives.html)
31+
* [distribution](https://typing.python.org/en/latest/spec/distributing.html)
32+
* [historical](https://typing.python.org/en/latest/spec/historical.html)
3333

3434
A test file is a ".py" file. The file name should start with one of the above names followed by a description of the test (with words separated by underscores). For example, `generics_paramspec_basic_usage.py` would contain the basic usage tests for `ParamSpec`. Each test file can contain multiple individual unit tests, but these tests should be related to each other. If the number of unit tests in a single test file exceeds ten, it may be desirable to split it into separate test files. This will help maintain a consistent level of granularity across tests.
3535

@@ -43,37 +43,36 @@ The test suite focuses on static type checking not general Python semantics. Tes
4343

4444
Test cases use the following conventions:
4545

46-
- Lines that are expected to produce a type checker error should have a comment starting with # E",
46+
* Lines that are expected to produce a type checker error should have a comment starting with # E",
4747
either by itself or followed by an explanation after a colon (e.g., "# E: int is not a subtype
4848
of str"). Such explanatory comments are purely for human understanding, but type checkers are not
4949
expected to use their exact wording. There are several syntactic variations; see "Test Case Syntax"
5050
below.
51-
- Lines that may produce an error (e.g., because the spec allows multiple behaviors) should be
51+
* Lines that may produce an error (e.g., because the spec allows multiple behaviors) should be
5252
marked with "# E?" instead of "# E".
53-
- If a test case tests conformance with a specific passage in the spec, that passage should be
53+
* If a test case tests conformance with a specific passage in the spec, that passage should be
5454
quoted in a comment prefixed with "# > ".
5555

5656
## Test Case Syntax
5757

5858
Test cases support the following special comments for declaring where errors should be raised:
5959

60-
- `# E`: an error must be raised on this line
61-
- `# E?`: an error may be raised on this line
62-
- `# E[tag]`, where `tag` is an arbitrary string: must appear multiple times in a file with the same tag.
60+
* `# E`: an error must be raised on this line
61+
* `# E?`: an error may be raised on this line
62+
* `# E[tag]`, where `tag` is an arbitrary string: must appear multiple times in a file with the same tag.
6363
Exactly one line with this tag must raise an error.
64-
- `# E[tag+]`: like `# E[tag]`, but errors may be raised on multiple lines.
64+
* `# E[tag+]`: like `# E[tag]`, but errors may be raised on multiple lines.
6565

6666
Each comment may be followed by a colon plus an explanation of the error; the explanation is ignored
6767
by the scoring system.
6868

6969
## Running the Conformance Test Tool
7070

7171
To run the conformance test suite:
72-
73-
- Clone the https://github.com/python/typing repo.
74-
- Create and activate a Python 3.12 virtual environment.
75-
- Switch to the `conformance` subdirectory and install all dependencies (`pip install -r requirements.txt`).
76-
- Switch to the `src` subdirectory and run `python main.py`.
72+
* Clone the https://github.com/python/typing repo.
73+
* Create and activate a Python 3.12 virtual environment.
74+
* Switch to the `conformance` subdirectory and install all dependencies (`pip install -r requirements.txt`).
75+
* Switch to the `src` subdirectory and run `python main.py`.
7776

7877
Note that some type checkers may not run on some platforms. If a type checker fails to install, tests will be skipped for that type checker.
7978

@@ -99,16 +98,16 @@ If a new version of a type checker is released, re-run the test tool with the ne
9998

10099
In addition to manual scoring, we provide an experimental tool that automatically checks type checkers for conformance. This tool relies on the "# E" comments present in the stubs and on parsing type checker output. This logic is run automatically as part of the conformance test tool. It produces the following fields in the `.toml` output files:
101100

102-
- `errors_diff`: a string describing all issues found with the type checker's behavior: either expected errors that were not emitted, or extra errors that the conformance test suite does not allow.
103-
- `conformance_automated`: either "Pass" or "Fail" based on whether there are any discrepancies with the expected behavior.
101+
* `errors_diff`: a string describing all issues found with the type checker's behavior: either expected errors that were not emitted, or extra errors that the conformance test suite does not allow.
102+
* `conformance_automated`: either "Pass" or "Fail" based on whether there are any discrepancies with the expected behavior.
104103

105104
This tool does not yet work reliably on all test cases. The script `conformance/src/unexpected_fails.py` can be run to find all test cases where the automated tool's conformance judgment differs from the manual judgment entered in the `.toml` files.
106105

107106
Some common problems with automated checks:
108107

109-
- Sometimes the spec is imprecise or allows multiple options. In this case, use "# E?" to mark an error as optional.
110-
- Type checkers may produce additional errors for issues unrelated to the topic being tested. In this case, add an extra field `ignore_errors` in the type checker's `.toml` file that contains the text of the irrelevant errors. Any error message that contains a substring in the `ignore_errors` list is ignored. For example, if `ignore_errors = ["Too many arguments"]`, then a mypy error `dataclasses_usage.py:127: error: Too many arguments for "DC7" [call-arg]` will be ignored.
111-
- Type checkers may differ in the line on which they report an error. In this case, on each of the lines where an error could
108+
* Sometimes the spec is imprecise or allows multiple options. In this case, use "# E?" to mark an error as optional.
109+
* Type checkers may produce additional errors for issues unrelated to the topic being tested. In this case, add an extra field `ignore_errors` in the type checker's `.toml` file that contains the text of the irrelevant errors. Any error message that contains a substring in the `ignore_errors` list is ignored. For example, if `ignore_errors = ["Too many arguments"]`, then a mypy error `dataclasses_usage.py:127: error: Too many arguments for "DC7" [call-arg]` will be ignored.
110+
* Type checkers may differ in the line on which they report an error. In this case, on each of the lines where an error could
112111
reasonably be shown, write `# E[<tag>]`, where `<tag>` is an arbitrary string that is unique in the file. The test will be marked as passing if the type checker produces an error on exactly one of the lines where this tag appears.
113112

114113
## Contributing

0 commit comments

Comments
 (0)