Merge branch 'master' into patch-10

Author: BobTheBuidler

.pre-commit-config.yaml

@@ -41,7 +41,7 @@ repos:
           # actionlint has a shellcheck integration which extracts shell scripts in `run:` steps from GitHub Actions
           # and checks these with shellcheck. This is arguably its most useful feature,
           # but the integration only works if shellcheck is installed
-          - "github.com/wasilibs/go-shellcheck/cmd/shellcheck@v0.10.0"
+          - "github.com/wasilibs/go-shellcheck/cmd/shellcheck@v0.11.0"
   - repo: https://github.com/woodruffw/zizmor-pre-commit
     rev: v1.5.2
     hooks:

docs/source/common_issues.rst

@@ -731,7 +731,7 @@ This example demonstrates both safe and unsafe overrides:
 
     class NarrowerReturn(A):
         # A more specific return type is fine
-        def test(self, t: Sequence[int]) -> List[str]:  # OK
+        def test(self, t: Sequence[int]) -> list[str]:  # OK
             ...
 
     class GeneralizedReturn(A):
@@ -746,7 +746,7 @@ not necessary:
 .. code-block:: python
 
     class NarrowerArgument(A):
-        def test(self, t: List[int]) -> Sequence[str]:  # type: ignore[override]
+        def test(self, t: list[int]) -> Sequence[str]:  # type: ignore[override]
             ...
 
 .. _unreachable:

docs/source/error_code_list2.rst

@@ -676,3 +676,26 @@ Example:
                 print("red")
             case _:
                 print("other")
+
+.. _code-untyped-decorator:
+
+Error if an untyped decorator makes a typed function effectively untyped [untyped-decorator]
+--------------------------------------------------------------------------------------------
+
+If enabled with :option:`--disallow-untyped-decorators <mypy --disallow-untyped-decorators>`
+mypy generates an error if a typed function is wrapped by an untyped decorator
+(as this would effectively remove the benefits of typing the function).
+
+Example:
+
+.. code-block:: python
+
+        def printing_decorator(func):
+            def wrapper(*args, **kwds):
+                print("Calling", func)
+                return func(*args, **kwds)
+            return wrapper
+        # A decorated function.
+        @printing_decorator  # E: Untyped decorator makes function "add_forty_two" untyped  [untyped-decorator]
+        def add_forty_two(value: int) -> int:
+            return value + 42

docs/source/stubtest.rst

@@ -99,8 +99,75 @@ to mypy build errors". In this case, you will need to mitigate those errors
 before stubtest will run. Despite potential overlap in errors here, stubtest is
 not intended as a substitute for running mypy directly.
 
+Allowlist
+*********
+
 If you wish to ignore some of stubtest's complaints, stubtest supports a
-pretty handy allowlist system.
+pretty handy :option:`--allowlist` system.
+
+Let's say that you have this python module called ``ex``:
+
+.. code-block:: python
+
+   try:
+       import optional_expensive_dep
+   except ImportError:
+       optional_expensive_dep = None
+
+   first = 1
+   if optional_expensive_dep:
+       second = 2
+
+Let's say that you can't install ``optional_expensive_dep`` in CI for some reason,
+but you still want to include ``second: int`` in the stub file:
+
+.. code-block:: python
+
+    first: int
+    second: int
+
+In this case stubtest will correctly complain:
+
+.. code-block:: shell
+
+   error: ex.second is not present at runtime
+   Stub: in file /.../ex.pyi:2
+   builtins.int
+   Runtime:
+   MISSING
+
+   Found 1 error (checked 1 module)
+
+To fix this, you can add an ``allowlist`` entry:
+
+.. code-block:: ini
+
+   # Allowlist entries in `allowlist.txt` file:
+
+   # Does not exist if `optional_expensive_dep` is not installed:
+   ex.second
+
+And now when running stubtest with ``--allowlist=allowlist.txt``,
+no errors will be generated anymore.
+
+Allowlists also support regular expressions,
+which can be useful to ignore many similar errors at once.
+They can also be useful for suppressing stubtest errors that occur sometimes,
+but not on every CI run. For example, if some CI workers have
+``optional_expensive_dep`` installed, stubtest might complain with this message
+on those workers if you had the ``ex.second`` allowlist entry:
+
+.. code-block:: ini
+
+   note: unused allowlist entry ex.second
+   Found 1 error (checked 1 module)
+
+Changing ``ex.second`` to be ``(ex\.second)?`` will make this error optional,
+meaning that stubtest will pass whether or not a CI runner
+has``optional_expensive_dep`` installed.
+
+CLI
+***
 
 The rest of this section documents the command line interface of stubtest.
 
@@ -119,15 +186,15 @@ The rest of this section documents the command line interface of stubtest.
 .. option:: --allowlist FILE
 
     Use file as an allowlist. Can be passed multiple times to combine multiple
-    allowlists. Allowlists can be created with --generate-allowlist. Allowlists
-    support regular expressions.
+    allowlists. Allowlists can be created with :option:`--generate-allowlist`.
+    Allowlists support regular expressions.
 
     The presence of an entry in the allowlist means stubtest will not generate
     any errors for the corresponding definition.
 
 .. option:: --generate-allowlist
 
-    Print an allowlist (to stdout) to be used with --allowlist
+    Print an allowlist (to stdout) to be used with :option:`--allowlist`.
 
     When introducing stubtest to an existing project, this is an easy way to
     silence all existing errors.
@@ -141,17 +208,17 @@ The rest of this section documents the command line interface of stubtest.
 
     Note if an allowlist entry is a regex that matches the empty string,
     stubtest will never consider it unused. For example, to get
-    `--ignore-unused-allowlist` behaviour for a single allowlist entry like
+    ``--ignore-unused-allowlist`` behaviour for a single allowlist entry like
     ``foo.bar`` you could add an allowlist entry ``(foo\.bar)?``.
     This can be useful when an error only occurs on a specific platform.
 
 .. option:: --mypy-config-file FILE
 
-    Use specified mypy config file to determine mypy plugins and mypy path
+    Use specified mypy config *file* to determine mypy plugins and mypy path
 
 .. option:: --custom-typeshed-dir DIR
 
-    Use the custom typeshed in DIR
+    Use the custom typeshed in *DIR*
 
 .. option:: --check-typeshed
 

mypy-requirements.txt

@@ -4,3 +4,4 @@ typing_extensions>=4.6.0
 mypy_extensions>=1.0.0
 pathspec>=0.9.0
 tomli>=1.1.0; python_version<'3.11'
+librt>=0.2.1

mypy/build.py

@@ -143,6 +143,10 @@ def __init__(self, manager: BuildManager, graph: Graph) -> None:
         self.errors: list[str] = []  # Filled in by build if desired
 
 
+def build_error(msg: str) -> NoReturn:
+    raise CompileError([f"mypy: error: {msg}"])
+
+
 def build(
     sources: list[BuildSource],
     options: Options,
@@ -241,6 +245,9 @@ def _build(
     errors = Errors(options, read_source=lambda path: read_py_file(path, cached_read))
     plugin, snapshot = load_plugins(options, errors, stdout, extra_plugins)
 
+    # Validate error codes after plugins are loaded.
+    options.process_error_codes(error_callback=build_error)
+
     # Add catch-all .gitignore to cache dir if we created it
     cache_dir_existed = os.path.isdir(options.cache_dir)
 
@@ -1546,7 +1553,7 @@ def write_cache(
     source_hash: str,
     ignore_all: bool,
     manager: BuildManager,
-) -> tuple[str, tuple[dict[str, Any], str, str] | None]:
+) -> tuple[str, tuple[dict[str, Any], str] | None]:
     """Write cache files for a module.
 
     Note that this mypy's behavior is still correct when any given
@@ -1568,7 +1575,7 @@ def write_cache(
 
     Returns:
       A tuple containing the interface hash and inner tuple with cache meta JSON
-      that should be written and paths to cache files (inner tuple may be None,
+      that should be written and path to cache file (inner tuple may be None,
       if the cache data could not be written).
     """
     metastore = manager.metastore
@@ -1662,12 +1669,10 @@ def write_cache(
         "ignore_all": ignore_all,
         "plugin_data": plugin_data,
     }
-    return interface_hash, (meta, meta_json, data_json)
+    return interface_hash, (meta, meta_json)
 
 
-def write_cache_meta(
-    meta: dict[str, Any], manager: BuildManager, meta_json: str, data_json: str
-) -> CacheMeta:
+def write_cache_meta(meta: dict[str, Any], manager: BuildManager, meta_json: str) -> None:
     # Write meta cache file
     metastore = manager.metastore
     meta_str = json_dumps(meta, manager.options.debug_cache)
@@ -1677,8 +1682,6 @@ def write_cache_meta(
         # The next run will simply find the cache entry out of date.
         manager.log(f"Error writing meta JSON file {meta_json}")
 
-    return cache_meta_from_dict(meta, data_json)
-
 
 """Dependency manager.
 
@@ -1864,9 +1867,6 @@ class State:
     # List of (path, line number) tuples giving context for import
     import_context: list[tuple[str, int]]
 
-    # The State from which this module was imported, if any
-    caller_state: State | None = None
-
     # If caller_state is set, the line number in the caller where the import occurred
     caller_line = 0
 
@@ -1917,7 +1917,6 @@ def __init__(
         self.manager = manager
         State.order_counter += 1
         self.order = State.order_counter
-        self.caller_state = caller_state
         self.caller_line = caller_line
         if caller_state:
             self.import_context = caller_state.import_context.copy()
@@ -2008,11 +2007,6 @@ def __init__(
             self.parse_file(temporary=temporary)
             self.compute_dependencies()
 
-    @property
-    def xmeta(self) -> CacheMeta:
-        assert self.meta, "missing meta on allegedly fresh module"
-        return self.meta
-
     def add_ancestors(self) -> None:
         if self.path is not None:
             _, name = os.path.split(self.path)
@@ -2479,7 +2473,7 @@ def valid_references(self) -> set[str]:
 
         return valid_refs
 
-    def write_cache(self) -> tuple[dict[str, Any], str, str] | None:
+    def write_cache(self) -> tuple[dict[str, Any], str] | None:
         assert self.tree is not None, "Internal error: method must be called on parsed file only"
         # We don't support writing cache files in fine-grained incremental mode.
         if (
@@ -3477,14 +3471,13 @@ def process_stale_scc(graph: Graph, scc: list[str], manager: BuildManager) -> No
     for id in stale:
         meta_tuple = meta_tuples[id]
         if meta_tuple is None:
-            graph[id].meta = None
             continue
-        meta, meta_json, data_json = meta_tuple
+        meta, meta_json = meta_tuple
         meta["dep_hashes"] = {
             dep: graph[dep].interface_hash for dep in graph[id].dependencies if dep in graph
         }
         meta["error_lines"] = errors_by_id.get(id, [])
-        graph[id].meta = write_cache_meta(meta, manager, meta_json, data_json)
+        write_cache_meta(meta, manager, meta_json)
 
 
 def sorted_components(

mypy/cache.py

@@ -1,66 +1,23 @@
 from __future__ import annotations
 
 from collections.abc import Sequence
-from typing import TYPE_CHECKING, Final
-
+from typing import Final
+
+from librt.internal import (
+    Buffer as Buffer,
+    read_bool as read_bool,
+    read_float as read_float,
+    read_int as read_int,
+    read_str as read_str,
+    read_tag as read_tag,
+    write_bool as write_bool,
+    write_float as write_float,
+    write_int as write_int,
+    write_str as write_str,
+    write_tag as write_tag,
+)
 from mypy_extensions import u8
 
-try:
-    from native_internal import (
-        Buffer as Buffer,
-        read_bool as read_bool,
-        read_float as read_float,
-        read_int as read_int,
-        read_str as read_str,
-        read_tag as read_tag,
-        write_bool as write_bool,
-        write_float as write_float,
-        write_int as write_int,
-        write_str as write_str,
-        write_tag as write_tag,
-    )
-except ImportError:
-    # TODO: temporary, remove this after we publish mypy-native on PyPI.
-    if not TYPE_CHECKING:
-
-        class Buffer:
-            def __init__(self, source: bytes = b"") -> None:
-                raise NotImplementedError
-
-            def getvalue(self) -> bytes:
-                raise NotImplementedError
-
-        def read_int(data: Buffer) -> int:
-            raise NotImplementedError
-
-        def write_int(data: Buffer, value: int) -> None:
-            raise NotImplementedError
-
-        def read_tag(data: Buffer) -> u8:
-            raise NotImplementedError
-
-        def write_tag(data: Buffer, value: u8) -> None:
-            raise NotImplementedError
-
-        def read_str(data: Buffer) -> str:
-            raise NotImplementedError
-
-        def write_str(data: Buffer, value: str) -> None:
-            raise NotImplementedError
-
-        def read_bool(data: Buffer) -> bool:
-            raise NotImplementedError
-
-        def write_bool(data: Buffer, value: bool) -> None:
-            raise NotImplementedError
-
-        def read_float(data: Buffer) -> float:
-            raise NotImplementedError
-
-        def write_float(data: Buffer, value: float) -> None:
-            raise NotImplementedError
-
-
 # Always use this type alias to refer to type tags.
 Tag = u8
 

mypy/checker.py

@@ -1801,7 +1801,8 @@ def check___new___signature(self, fdef: FuncDef, typ: CallableType) -> None:
                 "but must return a subtype of",
             )
         elif not isinstance(
-            get_proper_type(bound_type.ret_type), (AnyType, Instance, TupleType, UninhabitedType)
+            get_proper_type(bound_type.ret_type),
+            (AnyType, Instance, TupleType, UninhabitedType, LiteralType),
         ):
             self.fail(
                 message_registry.NON_INSTANCE_NEW_TYPE.format(