[PEP 747] Recognize TypeForm[T] type and values (#9773)

[davidfstr]

(This PR replaces an earlier draft of the same feature: #18690 )

Feedback from @JukkaL integrated since the last PR, by commit title:

• Apply feedback: Change MAYBE_UNRECOGNIZED_STR_TYPEFORM from unaccompanied note to standalone error
• Apply feedback: Refactor extract save/restore of SemanticAnalyzer state to a new context manager
• Apply feedback: Suppress SyntaxWarnings when parsing strings as types at the most-targeted location
• Apply feedback: Add TypeForm profiling counters to SemanticAnalyzer and the --dump-build-stats option
  • Increase efficiency of quick rejection heuristic from 85.8% -> 99.6% in SemanticAnalyzer.try_parse_as_type_expression()
• Apply feedback: Recognize assignment to union of TypeForm with non-TypeForm
• Apply feedback: Alter primitives.pyi fixture rather than tuple.pyi and dict.pyi

Feedback NOT integrated, with rationale:

• ✖️ Add tests related to recursive types
  • Recursive cases are already well-covered by tests related to TypeType (is_type_form=False).
  • I did find an infinite recursion bug affecting garden-variety Type[...], which I can fix in a separate PR.
• ✖️ Define TypeForm(...) in value contexts as a regular function like Callable[[TypeForm[T]], TypeForm[T]] rather than as a special expression node (TypeFormExpr).
  • The special expression node allows mypy to print out better error messages when a user puts an invalid type expression inside TypeForm(...). See case 4 of testTypeFormExpression in check-typeform.test

There is one NOMERGE commit temporarily in this PR so that mypy_primer gives more insightful CI output:

• NOMERGE: mypy_primer: Enable --enable-incomplete-feature=TypeForm when checking open source code

There is one commit unrelated to the core function of this PR that could be split to a separate PR:

• Allow TypeAlias and PlaceholderNode to be stringified/printed

Closes #9773

---

(Most of the following description is copied from the original PR, except for the text in bold)

Implements the TypeForm PEP 747, as an opt-in feature enabled by the CLI flag --enable-incomplete-feature=TypeForm.

Implementation approach:

• The TypeForm[T] is represented as a type using the existing TypeType class, with an is_type_form=True constructor parameter. Type[C] continues to be represented using TypeType, but with is_type_form=False (the default).

• Recognizing a type expression literal such as int | str requires parsing an Expression as a type expression. Only the SemanticAnalyzer pass has the ability to parse arbitrary type expressions (including stringified annotations), using SemanticAnalyzer.expr_to_analyzed_type(). (I've extended the TypeChecker pass to parse all kinds of type expressions except stringified annotations, using the new TypeCheckerAsSemanticAnalyzer adapter.)

• Therefore during the SemanticAnalyzer pass, at certain syntactic locations (i.e. assignment r-values, callable arguments, returned expressions), the analyzer tries to parse the Expression it is looking at using try_parse_as_type_expression() - a new function - and stores the result (a Type) in {IndexExpr, OpExpr, StrExpr}.as_type - a new attribute.

• During the later TypeChecker pass, when looking at an Expression to determine its type, if the expression is in a type context that expects some kind of TypeForm[...] and the expression was successfully parsed as a type expression by the earlier SemanticAnalyzer pass (or can be parsed as a type expression immediately during the type checker pass), the expression will be given the type TypeForm[expr.as_type] rather than using the regular type inference rules for a value expression.

• Key relationships between TypeForm[T], Type[C], and object types are defined in the visitors powering is_subtype, join_types, and meet_types.

• The TypeForm(T) expression is recognized as a TypeFormExpr and has the return type TypeForm[T].

• The new test suite in check-typeform.test is a good reference to the expected behaviors for operations that interact with TypeForm in some way.

Controversial parts of this PR, in @davidfstr 's opinion:

• Type form literals containing stringified annotations are only recognized in certain syntactic locations (and not ALL possible locations). Namely they are recognized as (1) assignment r-values, (2) callable expression arguments, and (3) as returned expressions, but nowhere else. For example they aren't recognized in expressions like dict_with_typx_keys[int | str]. Attempting to use stringified annotations in other locations will emit a MAYBE_UNRECOGNIZED_STR_TYPEFORM error.

• The existing TypeType class is now used to represent BOTH the Type[T] and TypeForm[T] types, rather than introducing a distinct subclass of Type to represent the TypeForm[T] type. This was done to simplify logic that manipulates both Type[T] and TypeForm[T] values, since they are both manipulated in very similar ways.

• The "normalized" form of TypeForm[X | Y] - as returned by TypeType.make_normalized() - is just TypeForm[X | Y] rather than TypeForm[X] | TypeForm[Y], differing from the normalization behavior of Type[X | Y].

[davidfstr]

I fixed all the "only in CI" check issues so this PR is now actually ready for review.

[davidfstr]

@JukkaL Did you want to look at this latest PR draft yourself, or recommend a review from any of the other mypy maintainers?

[JukkaL]

I can have a look. Sorry for the delay, this is a big PR and I need to block some time to review this properly.

[davidfstr]

Sometime between 2-6 weeks from now I'm expecting to become unavailable for a few months due to increased family commitments. If it's possible to get feedback on this PR before then, it will be able to be merged in a timely fashion and mypy will gain TypeForm support in 2025.

[davidfstr]

I have a limited amount of time now each day to review PR feedback, which I expect to continue having over the next 2 weeks. @JukkaL if you find a block of time to review during that window, I should be able to respond to you.

[JukkaL]

I have blocked time in my calendar to review the PR next week. Again, apologies for the slow response.

[JukkaL]

This is looking good now, thank you for all the updates! I measured performance and it's looking good -- I couldn't measure a difference compared to master. Left a few comments that would be good to fix before merging (plus there are a few merge conflicts).

The following things are optional for now -- it's fine to merge the PR after the comments have been addressed and create a follow-up issue (or issues) about these.

• Callable[[], int] isn't accepted as a TypeForm, unless I put in in a string literal.
• Literal[1] is similar -- it isn't accepted.
• It would be good to have tests for other kinds of types that are missing coverage (fixed-length tuple, type[int] as a type form, TypedDict type, NoReturn).
• What should happen if a type variable type such as T is used as a TypeForm -- is it supposed to work? Right now it seems to be rejected.
• It would be good to have one or two incremental and daemon mode tests.

[davidfstr]

Thanks for taking a look @JukkaL ! I'll get started on making revisions over the next few days, with the goal of having a clean squashable branch ready to merge.

[JukkaL]

Note that I will be on vacation from Oct 22, but I'll review your changes once I'm back.

[davidfstr]

Update: I've pushed fixes to every PR comment.

I'm currently looking at a few of the bonus items mentioned by Jukka, since I'm thinking of fixing a few of them as part of this PR.

In particular the inability to assign Callable[...] to a TypeForm variable concerned me. I wrote a new test (not yet pushed) which also revealed that Optional[...] and Union[...] specifically have assignment problems too.

I'm going to see if I can find time within the next 3 days to fix that last issue as part of this PR. The remaining bullets I'm looking at filing as post-merge GitHub issues to investigate/fix.

[github-actions]

According to mypy_primer, this change doesn't affect type check results on a corpus of open source code. ✅

[davidfstr]

These issues are now fixed, by commit fadc994:

• Callable[[], int] isn't accepted as a TypeForm, unless I put in in a string literal.
• Literal[1] is similar -- it isn't accepted.
• It would be good to have tests for other kinds of types that are missing coverage (fixed-length tuple, type[int] as a type form, TypedDict type, NoReturn).

---

• What should happen if a type variable type such as T is used as a TypeForm -- is it supposed to work? Right now it seems to be rejected.

Indeed using a type variable type like T as a TypeForm is currently rejected, as shown in the following 2 tests:

[case testEveryKindOfTypeExpressionIsAssignableToATypeFormVariable]
...
# Begin rules taken from: https://typing.python.org/en/latest/spec/annotations.html#grammar-token-expression-grammar-type_expression
...
# name (where name must refer to a valid in-scope TypeVar)
# NOTE: Unbound TypeVar isn't currently accepted as a TypeForm. Is that OK?
typx = SomeTypeVar  # E: Incompatible types in assignment (expression has type "TypeVar", variable has type "TypeForm[Any]")

-- mypy already refused to recognize TypeVars in value expressions before
-- the TypeForm feature was introduced.
[case testTypeVarTypeFormsAreOnlyRecognizedInStringAnnotation]
...
E = TypeVar('E')
class Box(Generic[E]):
    def foo(self, e: E) -> None:
        list_of_typx: List[TypeForm] = [E]  # E: "E" is a type variable and only valid in type context
        typx1: TypeForm = E  # E: "E" is a type variable and only valid in type context
        typx2: TypeForm = 'E'

However probably type variables like T should be accepted as a TypeForm, to be consistent with TypeForms accepting all other kinds of type expressions. I'll address this in a followup issue. (Edit 1: Issue created) 👈(♯1)

• It would be good to have one or two incremental and daemon mode tests.

I'm not familar with either incremental or daemon mode tests presently. I'll address this in a followup issue. (Edit 2: Issue created) 👈(♯2)

---

@JukkaL I think this branch is ready to merge.