Skip to content

Add constants for Concatenate and Unpack type names #18553

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 2 commits into from
Jan 30, 2025
Merged

Add constants for Concatenate and Unpack type names #18553

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
a97a406
Add constants for Concatenate and Unpack type names
cdce8p Jan 27, 2025
8330492
Use TYPE_VAR_LIKE_NAMES in stubgen
cdce8p Jan 27, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 3 additions & 2 deletions mypy/semanal.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -263,6 +263,7 @@
TYPE_CHECK_ONLY_NAMES,
TYPE_VAR_LIKE_NAMES,
TYPED_NAMEDTUPLE_NAMES,
UNPACK_TYPE_NAMES,
AnyType,
CallableType,
FunctionLike,
Expand Down Expand Up @@ -2281,7 +2282,7 @@ def analyze_unbound_tvar(self, t: Type) -> tuple[str, TypeVarLikeExpr] | None:
return self.analyze_unbound_tvar_impl(t.type, is_unpacked=True)
if isinstance(t, UnboundType):
sym = self.lookup_qualified(t.name, t)
if sym and sym.fullname in ("typing.Unpack", "typing_extensions.Unpack"):
if sym and sym.fullname in UNPACK_TYPE_NAMES:
inner_t = t.args[0]
if isinstance(inner_t, UnboundType):
return self.analyze_unbound_tvar_impl(inner_t, is_unpacked=True)
Expand Down Expand Up @@ -4166,7 +4167,7 @@ def analyze_type_alias_type_params(
base,
code=codes.TYPE_VAR,
)
if sym and sym.fullname in ("typing.Unpack", "typing_extensions.Unpack"):
if sym and sym.fullname in UNPACK_TYPE_NAMES:
self.note(
"Don't Unpack type variables in type_params", base, code=codes.TYPE_VAR
)
Expand Down
10 changes: 2 additions & 8 deletions mypy/stubgen.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -145,6 +145,7 @@
DATACLASS_TRANSFORM_NAMES,
OVERLOAD_NAMES,
TPDICT_NAMES,
TYPE_VAR_LIKE_NAMES,
TYPED_NAMEDTUPLE_NAMES,
AnyType,
CallableType,
Expand Down Expand Up @@ -1090,14 +1091,7 @@ def is_alias_expression(self, expr: Expression, top_level: bool = True) -> bool:
or module alias.
"""
# Assignment of TypeVar(...) and other typevar-likes are passed through
if isinstance(expr, CallExpr) and self.get_fullname(expr.callee) in (
"typing.TypeVar",
"typing_extensions.TypeVar",
"typing.ParamSpec",
"typing_extensions.ParamSpec",
"typing.TypeVarTuple",
"typing_extensions.TypeVarTuple",
):
if isinstance(expr, CallExpr) and self.get_fullname(expr.callee) in TYPE_VAR_LIKE_NAMES:
Comment on lines -1093 to +1094
Copy link
Collaborator Author

@cdce8p cdce8p Jan 27, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

mypy/mypy/types.py

Lines 87 to 94 in 82f4e88

TYPE_VAR_LIKE_NAMES: Final = (
"typing.TypeVar",
"typing_extensions.TypeVar",
"typing.ParamSpec",
"typing_extensions.ParamSpec",
"typing.TypeVarTuple",
"typing_extensions.TypeVarTuple",
)

return True
elif isinstance(expr, EllipsisExpr):
return not top_level
Expand Down
10 changes: 6 additions & 4 deletions mypy/typeanal.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -62,10 +62,12 @@
from mypy.types import (
ANNOTATED_TYPE_NAMES,
ANY_STRATEGY,
CONCATENATE_TYPE_NAMES,
FINAL_TYPE_NAMES,
LITERAL_TYPE_NAMES,
NEVER_NAMES,
TYPE_ALIAS_NAMES,
UNPACK_TYPE_NAMES,
AnyType,
BoolTypeQuery,
CallableArgument,
Expand Down Expand Up @@ -525,7 +527,7 @@ def visit_unbound_type_nonoptional(self, t: UnboundType, defining_literal: bool)
elif node.fullname in TYPE_ALIAS_NAMES:
return AnyType(TypeOfAny.special_form)
# Concatenate is an operator, no need for a proper type
elif node.fullname in ("typing_extensions.Concatenate", "typing.Concatenate"):
elif node.fullname in CONCATENATE_TYPE_NAMES:
# We check the return type further up the stack for valid use locations
return self.apply_concatenate_operator(t)
else:
Expand Down Expand Up @@ -779,7 +781,7 @@ def try_analyze_special_unbound_type(self, t: UnboundType, fullname: str) -> Typ
):
# In most contexts, TypeGuard[...] acts as an alias for bool (ignoring its args)
return self.named_type("builtins.bool")
elif fullname in ("typing.Unpack", "typing_extensions.Unpack"):
elif fullname in UNPACK_TYPE_NAMES:
if len(t.args) != 1:
self.fail("Unpack[...] requires exactly one type argument", t)
return AnyType(TypeOfAny.from_error)
Expand Down Expand Up @@ -1501,7 +1503,7 @@ def analyze_callable_args_for_concatenate(
return None
if sym.node is None:
return None
if sym.node.fullname not in ("typing_extensions.Concatenate", "typing.Concatenate"):
if sym.node.fullname not in CONCATENATE_TYPE_NAMES:
return None

tvar_def = self.anal_type(callable_args, allow_param_spec=True)
Expand Down Expand Up @@ -1650,7 +1652,7 @@ def analyze_callable_args(
return None
elif (
isinstance(arg, UnboundType)
and self.refers_to_full_names(arg, ("typing_extensions.Unpack", "typing.Unpack"))
and self.refers_to_full_names(arg, UNPACK_TYPE_NAMES)
or isinstance(arg, UnpackType)
):
if seen_unpack:
Expand Down
6 changes: 6 additions & 0 deletions mypy/types.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -130,6 +130,12 @@
# Supported Annotated type names.
ANNOTATED_TYPE_NAMES: Final = ("typing.Annotated", "typing_extensions.Annotated")

# Supported Concatenate type names.
CONCATENATE_TYPE_NAMES: Final = ("typing.Concatenate", "typing_extensions.Concatenate")

# Supported Unpack type names.
UNPACK_TYPE_NAMES: Final = ("typing.Unpack", "typing_extensions.Unpack")
Copy link
Collaborator

@JukkaL JukkaL Jan 28, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It would probably be slightly faster with current mypyc to annotated these as Final[tuple[str, ...]], as otherwise I'd expect there to be more boxing/unboxing.

Copy link
Collaborator Author

@cdce8p cdce8p Jan 28, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There are quite a few of these already with just Final. Would it be alright if I explore possible performance gains in a followup?

Just for pure Python, using set or frozenset would probably be even faster than tuple. Would need to check if that's true for mypyc as well.

mypy/mypy/types.py

Lines 112 to 134 in f49a1cb

# Supported names of Protocol base class.
PROTOCOL_NAMES: Final = ("typing.Protocol", "typing_extensions.Protocol")
# Supported TypeAlias names.
TYPE_ALIAS_NAMES: Final = ("typing.TypeAlias", "typing_extensions.TypeAlias")
# Supported Final type names.
FINAL_TYPE_NAMES: Final = ("typing.Final", "typing_extensions.Final")
# Supported @final decorator names.
FINAL_DECORATOR_NAMES: Final = ("typing.final", "typing_extensions.final")
# Supported @type_check_only names.
TYPE_CHECK_ONLY_NAMES: Final = ("typing.type_check_only", "typing_extensions.type_check_only")
# Supported Literal type names.
LITERAL_TYPE_NAMES: Final = ("typing.Literal", "typing_extensions.Literal")
# Supported Annotated type names.
ANNOTATED_TYPE_NAMES: Final = ("typing.Annotated", "typing_extensions.Annotated")
# Supported @deprecated type names
DEPRECATED_TYPE_NAMES: Final = ("warnings.deprecated", "typing_extensions.deprecated")

Copy link
Collaborator Author

@cdce8p cdce8p Jan 29, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

From my initial testing it looks like your hunch was correct. While is faster to construct the Final tuples, they need to be recreated for each call. In contrast Final[tuple[str, ...]] takes slightly longer to setup but will be reused.

The bigger performance gain would be to optimize for contains. They are nearly exclusively used with ... in .... A good example here is try_analyze_special_unbound_type which would benefit from using set instead.

Mypyc doesn't seem to optimize frozenset yet, does it? AFAICT it still uses PySequence_Contains for it, not PySet_Contains.

mypy/mypy/typeanal.py

Lines 610 to 619 in d4e7a81

def try_analyze_special_unbound_type(self, t: UnboundType, fullname: str) -> Type | None:
"""Bind special type that is recognized through magic name such as 'typing.Any'.
Return the bound type if successful, and return None if the type is a normal type.
"""
if fullname == "builtins.None":
return NoneType()
elif fullname == "typing.Any" or fullname == "builtins.Any":
return AnyType(TypeOfAny.explicit, line=t.line, column=t.column)
elif fullname in FINAL_TYPE_NAMES:

--
Will change it in a followup so we can do a proper performance comparison. Although the improvement will probably be fairly small, I suspect.

Copy link
Collaborator

@JukkaL JukkaL Jan 30, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Mypyc doesn't optimize frozenset much. And in small collections linear search may be faster than a set lookup. The best long-term solution is for mypyc to optimize in operations targeting final fixed-length tuples, so unless there is a measurable performance improvement from using variable-length tuples, it's not worth it.

Copy link
Collaborator Author

@cdce8p cdce8p Jan 31, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Did some performance testing today. AFAICT the difference is below the noice. Probably indeed not worth it at this point.

The main advantage this change provided is that devs don't accidentally miss on of the type names if they reuse it. IIRC this happened for typing_extensions.Unpack a while back.


# Supported @deprecated type names
DEPRECATED_TYPE_NAMES: Final = ("warnings.deprecated", "typing_extensions.deprecated")

Expand Down
Loading