Merge branch 'master' into feature/deprecated/descriptors

Author: tyralla

.pre-commit-config.yaml

@@ -1,17 +1,17 @@
 exclude: '^(mypyc/external/)|(mypy/typeshed/)|misc/typeshed_patches'  # Exclude all vendored code from lints
 repos:
   - repo: https://github.com/pre-commit/pre-commit-hooks
-    rev: v4.5.0  # must match test-requirements.txt
+    rev: v4.5.0
     hooks:
       - id: trailing-whitespace
       - id: end-of-file-fixer
   - repo: https://github.com/psf/black-pre-commit-mirror
-    rev: 24.8.0  # must match test-requirements.txt
+    rev: 24.8.0
     hooks:
       - id: black
         exclude: '^(test-data/)'
   - repo: https://github.com/astral-sh/ruff-pre-commit
-    rev: v0.6.9  # must match test-requirements.txt
+    rev: v0.6.9
     hooks:
       - id: ruff
         args: [--exit-non-zero-on-fix]

CONTRIBUTING.md

@@ -65,16 +65,16 @@ However, if you wish to do so, you can run the full test suite
 like this:
 
 ```bash
-python3 runtests.py
+python runtests.py
 ```
 
 Some useful commands for running specific tests include:
 
 ```bash
 # Use mypy to check mypy's own code
-python3 runtests.py self
+python runtests.py self
 # or equivalently:
-python3 -m mypy --config-file mypy_self_check.ini -p mypy
+python -m mypy --config-file mypy_self_check.ini -p mypy
 
 # Run a single test from the test suite
 pytest -n0 -k 'test_name'
@@ -117,7 +117,7 @@ tox -e dev --override testenv:dev.allowlist_externals+=env -- env  # inspect the
 ```
 
 If you don't already have `tox` installed, you can use a virtual environment as
-described above to install `tox` via `pip` (e.g., ``python3 -m pip install tox``).
+described above to install `tox` via `pip` (e.g., ``python -m pip install tox``).
 
 ## First time contributors
 

MANIFEST.in

@@ -9,6 +9,7 @@ recursive-include mypy/typeshed *.pyi
 
 # mypy and mypyc
 include mypy/py.typed
+include mypyc/py.typed
 recursive-include mypy *.py
 recursive-include mypyc *.py
 

docs/source/command_line.rst

@@ -166,6 +166,10 @@ imports.
 
     For more details, see :ref:`ignore-missing-imports`.
 
+.. option:: --follow-untyped-imports
+
+    This flag makes mypy analyze imports without stubs or a py.typed marker.
+
 .. option:: --follow-imports {normal,silent,skip,error}
 
     This flag adjusts how mypy follows imported modules that were not
@@ -537,12 +541,12 @@ potentially problematic or redundant in some way.
 
         This limitation will be removed in future releases of mypy.
 
-.. option:: --report-deprecated-as-error
+.. option:: --report-deprecated-as-note
 
-    By default, mypy emits notes if your code imports or uses deprecated
-    features.    This flag converts such notes to errors, causing mypy to
-    eventually finish with a non-zero exit code.  Features are considered
-    deprecated when decorated with ``warnings.deprecated``.
+    If error code ``deprecated`` is enabled, mypy emits errors if your code
+    imports or uses deprecated features. This flag converts such errors to
+    notes, causing mypy to eventually finish with a zero exit code. Features
+    are considered deprecated when decorated with ``warnings.deprecated``.
 
 .. _miscellaneous-strictness-flags:
 

docs/source/config_file.rst

@@ -315,6 +315,18 @@ section of the command line docs.
     match the name of the *imported* module, not the module containing the
     import statement.
 
+.. confval:: follow_untyped_imports
+
+    :type: boolean
+    :default: False
+
+    Typechecks imports from modules that do not have stubs or a py.typed marker.
+
+    If this option is used in a per-module section, the module name should
+    match the name of the *imported* module, not the module containing the
+    import statement. Note that scanning all unannotated modules might
+    significantly increase the runtime of your mypy calls.
+
 .. confval:: follow_imports
 
     :type: string
@@ -717,6 +729,14 @@ section of the command line docs.
 
     Note: This option will override disabled error codes from the disable_error_code option.
 
+.. confval:: extra_checks
+
+   :type: boolean
+   :default: False
+
+   This flag enables additional checks that are technically correct but may be impractical in real code.
+   See :option:`mypy --extra-checks` for more info.
+
 .. confval:: implicit_reexport
 
     :type: boolean
@@ -739,23 +759,23 @@ section of the command line docs.
 
 .. confval:: strict_concatenate
 
-    :type: boolean
-    :default: False
+   :type: boolean
+   :default: False
 
-    Make arguments prepended via ``Concatenate`` be truly positional-only.
+   Make arguments prepended via ``Concatenate`` be truly positional-only.
 
 .. confval:: strict_equality
 
-    :type: boolean
-    :default: False
+   :type: boolean
+   :default: False
 
    Prohibit equality checks, identity checks, and container checks between
    non-overlapping types.
 
 .. confval:: strict
 
-    :type: boolean
-    :default: False
+   :type: boolean
+   :default: False
 
    Enable all optional error checking flags.  You can see the list of
    flags enabled by strict mode in the full :option:`mypy --help`

docs/source/error_code_list2.rst

@@ -236,13 +236,13 @@ incorrect control flow or conditional checks that are accidentally always true o
 Check that imported or used feature is deprecated [deprecated]
 --------------------------------------------------------------
 
-By default, mypy generates a note if your code imports a deprecated feature explicitly with a
+If you use :option:`--enable-error-code deprecated <mypy --enable-error-code>`,
+mypy generates an error if your code imports a deprecated feature explicitly with a
 ``from mod import depr`` statement or uses a deprecated feature imported otherwise or defined
 locally.  Features are considered deprecated when decorated with ``warnings.deprecated``, as
-specified in `PEP 702 <https://peps.python.org/pep-0702>`_.  You can silence single notes via
-``# type: ignore[deprecated]`` or turn off this check completely via ``--disable-error-code=deprecated``.
-Use the :option:`--report-deprecated-as-error <mypy --report-deprecated-as-error>` option for
-more strictness, which turns all such notes into errors.
+specified in `PEP 702 <https://peps.python.org/pep-0702>`_.
+Use the :option:`--report-deprecated-as-note <mypy --report-deprecated-as-note>` option to
+turn all such errors into notes.
 
 .. note::
 

docs/source/extending_mypy.rst

@@ -245,4 +245,4 @@ Mypy ships ``mypy.plugins.proper_plugin`` plugin which can be useful
 for plugin authors, since it finds missing ``get_proper_type()`` calls,
 which is a pretty common mistake.
 
-It is recommended to enable it is a part of your plugin's CI.
+It is recommended to enable it as a part of your plugin's CI.

docs/source/generics.rst

@@ -146,7 +146,7 @@ example (Python 3.12 syntax):
    from typing import Mapping, Iterator
 
    # This is a generic subclass of Mapping
-   class MyMapp[KT, VT](Mapping[KT, VT]):
+   class MyMap[KT, VT](Mapping[KT, VT]):
        def __getitem__(self, k: KT) -> VT: ...
        def __iter__(self) -> Iterator[KT]: ...
        def __len__(self) -> int: ...
@@ -641,7 +641,7 @@ infer the most flexible variance for each class type variable. Here
 
 .. code-block:: python
 
-   class Box[T]:  # this type is implilicitly covariant
+   class Box[T]:  # this type is implicitly covariant
        def __init__(self, content: T) -> None:
            self._content = content
 
@@ -663,12 +663,12 @@ the attribute as ``Final``, the class could still be made covariant:
 
    from typing import Final
 
-   class Box[T]:  # this type is implilicitly covariant
+   class Box[T]:  # this type is implicitly covariant
        def __init__(self, content: T) -> None:
            self.content: Final = content
 
        def get_content(self) -> T:
-           return self._content
+           return self.content
 
 When using the legacy syntax, mypy assumes that all user-defined generics
 are invariant by default. To declare a given generic class as covariant or

docs/source/running_mypy.rst

@@ -321,6 +321,12 @@ not catch errors in its use.
     recommend avoiding ``--ignore-missing-imports`` if possible: it's equivalent
     to adding a ``# type: ignore`` to all unresolved imports in your codebase.
 
+4.  To make mypy typecheck imports from modules without stubs or a py.typed
+    marker, you can set the :option:`--follow-untyped-imports <mypy --follow-untyped-imports>`
+    command line flag or set the :confval:`follow_untyped_imports` config file option to True,
+    either in the global section of your mypy config file, or individually on a
+    per-module basis.
+
 
 Library stubs not installed
 ---------------------------

mypy/binder.py

@@ -2,7 +2,7 @@
 
 from collections import defaultdict
 from contextlib import contextmanager
-from typing import DefaultDict, Iterator, List, Optional, Tuple, Union, cast
+from typing import DefaultDict, Iterator, List, NamedTuple, Optional, Tuple, Union
 from typing_extensions import TypeAlias as _TypeAlias
 
 from mypy.erasetype import remove_instance_last_known_values
@@ -30,6 +30,11 @@
 BindableExpression: _TypeAlias = Union[IndexExpr, MemberExpr, NameExpr]
 
 
+class CurrentType(NamedTuple):
+    type: Type
+    from_assignment: bool
+
+
 class Frame:
     """A Frame represents a specific point in the execution of a program.
     It carries information about the current types of expressions at
@@ -44,7 +49,7 @@ class Frame:
 
     def __init__(self, id: int, conditional_frame: bool = False) -> None:
         self.id = id
-        self.types: dict[Key, Type] = {}
+        self.types: dict[Key, CurrentType] = {}
         self.unreachable = False
         self.conditional_frame = conditional_frame
         self.suppress_unreachable_warnings = False
@@ -132,18 +137,18 @@ def push_frame(self, conditional_frame: bool = False) -> Frame:
         self.options_on_return.append([])
         return f
 
-    def _put(self, key: Key, type: Type, index: int = -1) -> None:
-        self.frames[index].types[key] = type
+    def _put(self, key: Key, type: Type, from_assignment: bool, index: int = -1) -> None:
+        self.frames[index].types[key] = CurrentType(type, from_assignment)
 
-    def _get(self, key: Key, index: int = -1) -> Type | None:
+    def _get(self, key: Key, index: int = -1) -> CurrentType | None:
         if index < 0:
             index += len(self.frames)
         for i in range(index, -1, -1):
             if key in self.frames[i].types:
                 return self.frames[i].types[key]
         return None
 
-    def put(self, expr: Expression, typ: Type) -> None:
+    def put(self, expr: Expression, typ: Type, *, from_assignment: bool = True) -> None:
         if not isinstance(expr, (IndexExpr, MemberExpr, NameExpr)):
             return
         if not literal(expr):
@@ -153,7 +158,7 @@ def put(self, expr: Expression, typ: Type) -> None:
         if key not in self.declarations:
             self.declarations[key] = get_declaration(expr)
             self._add_dependencies(key)
-        self._put(key, typ)
+        self._put(key, typ, from_assignment)
 
     def unreachable(self) -> None:
         self.frames[-1].unreachable = True
@@ -164,7 +169,10 @@ def suppress_unreachable_warnings(self) -> None:
     def get(self, expr: Expression) -> Type | None:
         key = literal_hash(expr)
         assert key is not None, "Internal error: binder tried to get non-literal"
-        return self._get(key)
+        found = self._get(key)
+        if found is None:
+            return None
+        return found.type
 
     def is_unreachable(self) -> bool:
         # TODO: Copy the value of unreachable into new frames to avoid
@@ -193,7 +201,7 @@ def update_from_options(self, frames: list[Frame]) -> bool:
         If a key is declared as AnyType, only update it if all the
         options are the same.
         """
-
+        all_reachable = all(not f.unreachable for f in frames)
         frames = [f for f in frames if not f.unreachable]
         changed = False
         keys = {key for f in frames for key in f.types}
@@ -207,17 +215,30 @@ def update_from_options(self, frames: list[Frame]) -> bool:
                 # know anything about key in at least one possible frame.
                 continue
 
-            type = resulting_values[0]
-            assert type is not None
+            if all_reachable and all(
+                x is not None and not x.from_assignment for x in resulting_values
+            ):
+                # Do not synthesize a new type if we encountered a conditional block
+                # (if, while or match-case) without assignments.
+                # See check-isinstance.test::testNoneCheckDoesNotMakeTypeVarOptional
+                # This is a safe assumption: the fact that we checked something with `is`
+                # or `isinstance` does not change the type of the value.
+                continue
+
+            current_type = resulting_values[0]
+            assert current_type is not None
+            type = current_type.type
             declaration_type = get_proper_type(self.declarations.get(key))
             if isinstance(declaration_type, AnyType):
                 # At this point resulting values can't contain None, see continue above
-                if not all(is_same_type(type, cast(Type, t)) for t in resulting_values[1:]):
+                if not all(
+                    t is not None and is_same_type(type, t.type) for t in resulting_values[1:]
+                ):
                     type = AnyType(TypeOfAny.from_another_any, source_any=declaration_type)
             else:
                 for other in resulting_values[1:]:
                     assert other is not None
-                    type = join_simple(self.declarations[key], type, other)
+                    type = join_simple(self.declarations[key], type, other.type)
                     # Try simplifying resulting type for unions involving variadic tuples.
                     # Technically, everything is still valid without this step, but if we do
                     # not do this, this may create long unions after exiting an if check like:
@@ -236,8 +257,8 @@ def update_from_options(self, frames: list[Frame]) -> bool:
                         )
                         if simplified == self.declarations[key]:
                             type = simplified
-            if current_value is None or not is_same_type(type, current_value):
-                self._put(key, type)
+            if current_value is None or not is_same_type(type, current_value[0]):
+                self._put(key, type, from_assignment=True)
                 changed = True
 
         self.frames[-1].unreachable = not frames
@@ -374,7 +395,9 @@ def most_recent_enclosing_type(self, expr: BindableExpression, type: Type) -> Ty
         key = literal_hash(expr)
         assert key is not None
         enclosers = [get_declaration(expr)] + [
-            f.types[key] for f in self.frames if key in f.types and is_subtype(type, f.types[key])
+            f.types[key].type
+            for f in self.frames
+            if key in f.types and is_subtype(type, f.types[key][0])
         ]
         return enclosers[-1]