From d805bae83f5f3e5b0c70f72e85652d15f88ba4a9 Mon Sep 17 00:00:00 2001 From: Zac Hatfield-Dodds Date: Fri, 29 Aug 2025 15:56:12 -0700 Subject: [PATCH 1/2] PEP 789: Note gh issue was closed --- peps/pep-0789.rst | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/peps/pep-0789.rst b/peps/pep-0789.rst index 726fe979453..7ad706d45cb 100644 --- a/peps/pep-0789.rst +++ b/peps/pep-0789.rst @@ -215,9 +215,9 @@ and so we never got a chance to create and raise an ``ExceptionGroup(..., [RuntimeError()])``. To fix this, we need to turn our async generator into an async context manager, -which yields an async iterable - in this case a generator wrapping the queue; in -future `perhaps the queue itself -`__: +which yields an async iterable - in this case a generator wrapping the queue, +although `I'd usually reach for a different interface +`__: .. code-block:: python From ae951784bcd8f070e46dc68f577d00ad3a04b10d Mon Sep 17 00:00:00 2001 From: Zac Hatfield-Dodds Date: Fri, 29 Aug 2025 15:56:12 -0700 Subject: [PATCH 2/2] PEP 785: Update proposal --- peps/pep-0785.rst | 293 +++++++++++++++++++++++++--------------------- 1 file changed, 157 insertions(+), 136 deletions(-) diff --git a/peps/pep-0785.rst b/peps/pep-0785.rst index 3e4052ab94c..65ed48bd916 100644 --- a/peps/pep-0785.rst +++ b/peps/pep-0785.rst @@ -1,12 +1,12 @@ PEP: 785 -Title: New methods for easier handling of ``ExceptionGroup``\ s +Title: ``ExceptionGroup.leaf_exceptions()`` and ``except*`` chaining Author: Zac Hatfield-Dodds Sponsor: Gregory P. Smith Discussions-To: https://discuss.python.org/t/88244 Status: Draft Type: Standards Track Created: 08-Apr-2025 -Python-Version: 3.14 +Python-Version: 3.15 Post-History: `13-Apr-2025 `__, @@ -15,15 +15,15 @@ Abstract As :pep:`654` :class:`ExceptionGroup` has come into widespread use across the Python community, some common but awkward patterns have emerged. We therefore -propose adding two new methods to exception objects: +propose: -- :meth:`!BaseExceptionGroup.leaf_exceptions`, returning the 'leaf' exceptions as - a list, with each traceback composited from any intermediate groups. +- a new :meth:`!BaseExceptionGroup.leaf_exceptions` method, returning a list + of (exception, full_traceback) pairs consisting of each non-group exception + and a traceback composited from any intermediate groups. -- :meth:`!BaseException.preserve_context`, a context manager which - saves and restores the :attr:`!self.__context__` attribute of ``self``, - so that re-raising the exception within another handler does not overwrite - the existing context. +- a narrow language change: if an exception is raised inside an ``except*`` + block, and that exception is (by identity) part of the group, do not attach + the group as ``.__context__``. We expect this to enable more concise expression of error handling logic in many medium-complexity cases. Without them, exception-group handlers will @@ -39,116 +39,127 @@ often write code to process or respond to individual leaf exceptions, for example when implementing middleware, error logging, or response handlers in a web framework. -`Searching GitHub`__ found four implementations of :meth:`!leaf_exceptions` by -various names in the first sixty hits, of which none handle -tracebacks.\ [#numbers]_ The same search found thirteen cases where -:meth:`!.leaf_exceptions` could be used. We therefore believe that providing +In April 2025, `searching GitHub`__ [#naming]_ found four implementations of +:meth:`!leaf_exceptions` by various names in the first sixty hits, of which +none handle tracebacks.\ [#numbers]_ The same search found thirteen cases +where :meth:`!.leaf_exceptions` could be used. In August, `searching`__ +for ``leaf_exceptions`` identified a total of five distinct implementations, +none of which handled tracebacks, and three cases of copying an early +reference implementation from this PEP. We therefore believe that providing a method on the :class:`BaseException` type with proper traceback preservation will improve error-handling and debugging experiences across the ecosystem. __ https://github.com/search?q=%2Ffor+%5Cw%2B+in+%5Beg%5D%5Cw*%5C.exceptions%3A%2F+language%3APython&type=code -The rise of exception groups has also made re-raising exceptions caught by an -earlier handler much more common: for example, web-server middleware might -unwrap ``HTTPException`` if that is the sole leaf of a group: +__ https://github.com/search?q=%2Fdef+.*leaf_exceptions.*%5C%28%2F+language%3APython&type=code + +When Python raises an exception, it automatically sets ``__context__`` to +track what was being handled at the time. This normally provides valuable +debugging information. However, with ``except*``, this behavior can create +a problem. Consider this example middleware, which unwraps a single-exception +group: .. code-block:: python except* HTTPException as group: - first, *rest = group.leaf_exceptions() # get the whole traceback :-) + (exc, tb), *rest = group.leaf_exceptions() if not rest: - raise first + raise exc.with_traceback(tb) raise -However, this innocent-seeming code has a problem: ``raise first`` will do -``first.__context__ = group`` as a side effect. This discards the original +In this case, the implicit ``exc.__context__ = group`` discards the original context of the error, which may contain crucial information to understand why the exception was raised. In many production apps it also causes tracebacks -to balloon from hundreds of lines, to tens or even `hundreds of thousands of +to balloon from hundreds of lines to as many as `hundreds of thousands of lines`__ - a volume which makes understanding errors far more difficult than it should be. __ https://github.com/python-trio/trio/issues/2001#issuecomment-931928509 - -A new :meth:`!BaseException.preserve_context` method would be a discoverable, -readable, and easy-to-use solution for these cases. +We believe that this behavior is almost always unintentional and undesired +when raising a subgroup or leaf from inside an ``except*`` statement, and +therefore propose that in this narrow case Python should not automatically +attach the active exception as ``__context__``. In cases where tracking the +full group is desirable, ``raise leaf from group`` is easy and explicit. Specification ============= -A new method ``leaf_exceptions()`` will be added to ``BaseExceptionGroup``, with the -following signature: +:meth:`!BaseExceptionGroup.leaf_exceptions` +------------------------------------------- + +A new method ``.leaf_exceptions()`` will be added to ``BaseExceptionGroup``, +with the following signature: .. code-block:: python - def leaf_exceptions(self, *, fix_tracebacks=True) -> list[BaseException]: - """ - Return a flat list of all 'leaf' exceptions in the group. + def leaf_exceptions(self) -> list[tuple[BaseException, TracebackType]]: ... - If fix_tracebacks is True, each leaf will have the traceback replaced - with a composite so that frames attached to intermediate groups are - still visible when debugging. Pass fix_tracebacks=False to disable - this modification, e.g. if you expect to raise the group unchanged. - """ +Each traceback is constructed by concatenating the tracebacks of the +corresponding leaf exception with those of each enclosing group in turn. -A new method ``preserve_context()`` will be added to ``BaseException``, with the -following signature: +While exception groups are typically tree-structured, it is possible to +construct an ``ExceptionGroup`` where two child groups both contain a shared +grandchild exception; and with a custom subclass of ``ExceptionGroup`` you +can even form cycles. The ``.leaf_exceptions()`` method will therefore +track already-seen objects by identity, traversing each subgroup once and +outputting each leaf exception once. -.. code-block:: python - def preserve_context(self) -> contextlib.AbstractContextManager[Self]: - """ - Context manager that preserves the exception's __context__ attribute. +``except*`` changes to ``.__context__`` handling +------------------------------------------------ - When entering the context, the current values of __context__ is saved. - When exiting, the saved value is restored, which allows raising an - exception inside an except block without changing its context chain. - """ - -Usage example: +The language change is fairly simple: if an exception is raised inside an +``except*`` block, and that exception is (by identity) part of the group, +do not attach the group as ``.__context__``. Similar semantics can be +implemented using an unwieldy try/finally: .. code-block:: python - # We're an async web framework, where user code can raise an HTTPException - # to return a particular HTTP error code to the client. However, it may - # (or may not) be raised inside a TaskGroup, so we need to use `except*`; - # and if there are *multiple* such exceptions we'll treat that as a bug. - try: - user_code_here() - except* HTTPException as group: - first, *rest = group.leaf_exceptions() - if rest: - raise # handled by internal-server-error middleware - ... # logging, cache updates, etc. - with first.preserve_context(): - raise first - -Without ``.preserve_context()``, this code would have to either: + except* FooError as group: + try: + ... # current body of except* goes here + finally: + exc = sys.exception() + queue = [group] + seen_ids = set() + while queue: + part = queue.pop(0) + if id(part) in seen_ids: + continue + seen_ids.add(id(part)) + if exc is part: + if exc.__cause__ is part.__cause__: + continue # leave `raise x from y` untouched + exc.__context__ = part.__context__ + break + if isinstance(exc, BaseExceptionGroup): + queue.extend(exc.exceptions) -* arrange for the exception to be raised *after* the ``except*`` block, - making code difficult to follow in nontrivial cases, or -* discard the existing ``__context__`` of the ``first`` exception, replacing - it with an ``ExceptionGroup`` which is simply an implementation detail, or -* use ``try/except`` instead of ``except*``, handling the possibility that the - group doesn't contain an ``HTTPException`` at all,\ [#catch-raw-group]_ or -* implement the semantics of ``.preserve_context()`` inline; while this is not - *literally unheard-of*, it remains very rare. +This example implementation is careful to check for re-raised subgroups in +addition to re-raised leaf exceptions, without following cycles. +Checking by identity means that there is at most one match, and thus iteration +order is unimportant - though the tendency of end-user code to re-raise the +first rather than a later subgroup offers a small performance advantage. Backwards Compatibility ======================= -Adding new methods to built-in classes, especially those as widely used as -``BaseException``, can have substantial impacts. However, GitHub search shows -no collisions for these method names (`zero hits`__\ [#naming]_ and -`three unrelated hits`__ respectively). If user-defined methods with these -names exist in private code they will shadow those proposed in the PEP, -without changing runtime behavior. +Adding a new method to a built-in class, especially one as widely used as +``BaseException``, can have substantial impacts. However, `GitHub search`__ +shows zero user-defined ``.leaf_exceptions(`` methods. Any such methods +in private code will shadow the new method proposed in the PEP, and are +therefore backwards-compatible at runtime. __ https://github.com/search?q=%2F%5C.leaf_exceptions%5C%28%2F+language%3APython&type=code -__ https://github.com/search?q=%2F%5C.preserve_context%5C%28%2F+language%3APython&type=code + +Our proposed change to ``except*`` does not affect a bare ``raise`` statement, +nor ``raise exc from cause`` - only ``raise exc``. In cases where the current +behavior is desired, which we expect is very rare, users can +``raise exc from group`` or even ``exc.__context__ = group; raise exc`` for +a truly exact match. How to Teach This @@ -158,22 +169,23 @@ Working with exception groups is an intermediate-to-advanced topic, unlikely to arise for beginning programmers. We therefore suggest teaching this topic via documentation, and via just-in-time feedback from static analysis tools. In intermediate classes, we recommend teaching ``.leaf_exceptions()`` together -with the ``.split()`` and ``.subgroup()`` methods, and mentioning -``.preserve_context()`` as an advanced option to address specific pain points. +with the ``.split()`` and ``.subgroup()`` methods, emphasizing a preference +for the latter. Both the API reference and the existing `ExceptionGroup tutorial`__ -should be updated to demonstrate and explain the new methods. The tutorial -should include examples of common patterns where ``.leaf_exceptions()`` and -``.preserve_context()`` help simplify error handling logic. Downstream -libraries which often use exception groups could include similar docs. +should be updated to demonstrate and explain the new method. The tutorial +might include examples of common patterns where ``.leaf_exceptions()`` helps +simplify error handling logic. __ https://docs.python.org/3/tutorial/errors.html#raising-and-handling-multiple-unrelated-exceptions -We have also designed lint rules for inclusion in ``flake8-async`` which will -suggest using ``.leaf_exceptions()`` when iterating over ``group.exceptions`` -or re-raising a leaf exception, and suggest using ``.preserve_context()`` when -re-raising a leaf exception inside an ``except*`` block would override any -existing context. +We have also designed lint rules for inclusion in ``flake8-async`` which +suggest using ``.split()``, ``.subgroup()``, or ``.leaf_exceptions()`` when +iterating over ``group.exceptions`` or re-raising a leaf exception. + +We recommend mentioning the ``except*``-``__context__`` behavior in the +reference documentation for BaseException, but not in the error-handling +tutorial. Reference Implementation @@ -191,38 +203,38 @@ A ``leaf_exceptions()`` helper function .. code-block:: python - import copy - import types from types import TracebackType - def leaf_exceptions( - self: BaseExceptionGroup, *, fix_traceback: bool = True - ) -> list[BaseException]: + def leaf_exceptions(self) -> list[tuple[BaseException, TracebackType | None]]: """ - Return a flat list of all 'leaf' exceptions. - - If fix_tracebacks is True, each leaf will have the traceback replaced - with a composite so that frames attached to intermediate groups are - still visible when debugging. Pass fix_tracebacks=False to disable - this modification, e.g. if you expect to raise the group unchanged. + Return a list of (leaf_exception, full_traceback) pairs. + + 'Leaf' exceptions are the non-group exceptions contained inside an + exception group, its subgroups, and so on. The 'full traceback' is + constructed by concatenating the traceback of the leaf exception + with that of each containing group in turn. Tracebacks are returned + separately because :meth:`BaseException.with_traceback` mutates the + exception in-place, which is undesirable if e.g. you may raise the + group unchanged. + + Each distinct leaf exception will be included, or group traversed, + only once - even if it appears in multiple subgroups or a cycle. """ - - def _flatten(group: BaseExceptionGroup, parent_tb: TracebackType | None = None): - group_tb = group.__traceback__ - combined_tb = _combine_tracebacks(parent_tb, group_tb) - result = [] - for exc in group.exceptions: - if isinstance(exc, BaseExceptionGroup): - result.extend(_flatten(exc, combined_tb)) - elif fix_tracebacks: - tb = _combine_tracebacks(combined_tb, exc.__traceback__) - result.append(exc.with_traceback(tb)) - else: - result.append(exc) - return result - - return _flatten(self) + queue: list[tuple[BaseException, TracebackType | None]] = [(self, None)] + seen_ids: set[object] = set() + result: list[tuple[BaseException, TracebackType | None]] = [] + while queue: + exc, parent_tb = queue.pop() + if id(exc) in seen_ids: + continue + seen_ids.add(id(exc)) + tb = _combine_tracebacks(parent_tb, exc.__traceback__) + if isinstance(exc, BaseExceptionGroup): + queue.extend((e, tb) for e in exc.exceptions[::-1]) + else: + result.append((exc, tb)) + return result def _combine_tracebacks( @@ -251,7 +263,7 @@ A ``leaf_exceptions()`` helper function # Add frames from tb1 to the beginning (in reverse order) for frame, lasti, lineno in reversed(frames): - new_tb = types.TracebackType( + new_tb = TracebackType( tb_next=new_tb, tb_frame=frame, tb_lasti=lasti, tb_lineno=lineno ) @@ -261,20 +273,26 @@ A ``leaf_exceptions()`` helper function A ``preserve_context()`` context manager ---------------------------------------- -.. code-block:: python +A change to the semantics of ``except*`` cannot be backported in Python +(although see the Specification section for illustration). We hope this +simpler helper function will meet the same needs on older Python versions. - class preserve_context: - def __init__(self, exc: BaseException): - self.__exc = exc - self.__context = exc.__context__ +.. code-block:: python - def __enter__(self): - return self.__exc + @contextlib.contextmanager + def preserve_context(exc): + ctx = exc.__context__ + try: + yield exc + finally: + # assert sys.exception() is exc # optional sanity-check + exc.__context__ = ctx - def __exit__(self, exc_type, exc_value, traceback): - assert exc_value is self.__exc, f"did not raise the expected exception {self.__exc!r}" - exc_value.__context__ = self.__context - del self.__context # break gc cycle + try: + ... + except* Exception as group: + with preserve_context(group.exceptions[0]) as exc: + raise exc Rejected Ideas @@ -300,7 +318,7 @@ kind of exception inside a group (often incorrectly, motivating ``.leaf_exceptions()``). We briefly `proposed `__ -adding ``.split(...)`` and ``.subgroup(...)`` methods too all exceptions, +adding ``.split(...)`` and ``.subgroup(...)`` methods to all exceptions, before considering ``.leaf_exceptions()`` made us feel this was too clumsy. As a cleaner alternative, we sketched out an ``.as_group()`` method: @@ -314,16 +332,26 @@ As a cleaner alternative, we sketched out an ``.as_group()`` method: However, applying this method to refactor existing code was a negligible improvement over writing the trivial inline version. We also hope that many current uses for such a method will be addressed by ``except*`` as older -Python versions reach end-of-life. +Python versions reach end of life. We recommend documenting a "convert to group" recipe for de-duplicated error handling, instead of adding group-related methods to ``BaseException``. +Add a ``with e.preserve_context(): raise e`` context manager +------------------------------------------------------------ + +Every case we identified in existing code was better addressed by the small +language change in the current version of this PEP. While a language change +is in some sense a bigger deal than adding a method, we argue that there is +a substantial advantage in that handling this without user intervention +prevents many possible mistakes, whether misuse or lack-of-use. + + Add ``e.raise_with_preserved_context()`` instead of a context manager --------------------------------------------------------------------- -We prefer the context-manager form because it allows ``raise ... from ...`` +We preferred the context-manager form because it allows ``raise ... from ...`` if the user wishes to (re)set the ``__cause__``, and is overall somewhat less magical and tempting to use in cases where it would not be appropriate. We could be argued around though, if others prefer this form. @@ -382,13 +410,6 @@ Footnotes indicating that more than a quarter of *all* hits for this fairly general search would benefit from the methods proposed in this PEP. -.. [#catch-raw-group] - This remains very rare, and most cases duplicate logic across - ``except FooError:`` and ``except ExceptionGroup: # containing FooError`` - clauses rather than using something like the ``as_group()`` trick. - We expect that ``except*`` will be widely used in such cases by the time - that the methods proposed by this PEP are widely available. - .. [#naming] The name ``leaf_exceptions()`` was `first proposed`__ in an early precursor to :pep:`654`. If the prototype had matched ``except*``