Merge branch 'main' into gh-139269

Author: ashm-dev

Doc/library/collections.rst

@@ -367,9 +367,11 @@ Several mathematical operations are provided for combining :class:`Counter`
 objects to produce multisets (counters that have counts greater than zero).
 Addition and subtraction combine counters by adding or subtracting the counts
 of corresponding elements.  Intersection and union return the minimum and
-maximum of corresponding counts.  Equality and inclusion compare
-corresponding counts.  Each operation can accept inputs with signed
-counts, but the output will exclude results with counts of zero or less.
+maximum of corresponding counts.  Symmetric difference returns the difference
+between the maximum and minimum of the corresponding counts. Equality and
+inclusion compare corresponding counts.  Each operation can accept inputs
+with signed counts, but the output will exclude results with counts of zero
+or below.
 
 .. doctest::
 
@@ -383,6 +385,8 @@ counts, but the output will exclude results with counts of zero or less.
     Counter({'a': 1, 'b': 1})
     >>> c | d                       # union:  max(c[x], d[x])
     Counter({'a': 3, 'b': 2})
+    >>> c ^ d                       # max(c[x], d[x]) - min(c[x], d[x])
+    Counter({'a': 2, 'b': 1})
     >>> c == d                      # equality:  c[x] == d[x]
     False
     >>> c <= d                      # inclusion:  c[x] <= d[x]
@@ -400,6 +404,9 @@ or subtracting from an empty counter.
 .. versionadded:: 3.3
     Added support for unary plus, unary minus, and in-place multiset operations.
 
+.. versionadded:: next
+    Added support for the symmetric difference multiset operation, ``c ^ d``.
+
 .. note::
 
     Counters were primarily designed to work with positive integers to represent

Doc/library/pyexpat.rst

@@ -72,6 +72,13 @@ The :mod:`xml.parsers.expat` module contains two functions:
    *encoding* [1]_ is given it will override the implicit or explicit encoding of the
    document.
 
+   .. _xmlparser-non-root:
+
+   Parsers created through :func:`!ParserCreate` are called "root" parsers,
+   in the sense that they do not have any parent parser attached. Non-root
+   parsers are created by :meth:`parser.ExternalEntityParserCreate
+   <xmlparser.ExternalEntityParserCreate>`.
+
    Expat can optionally do XML namespace processing for you, enabled by providing a
    value for *namespace_separator*.  The value must be a one-character string; a
    :exc:`ValueError` will be raised if the string has an illegal length (``None``
@@ -231,6 +238,111 @@ XMLParser Objects
    .. versionadded:: 3.13
 
 
+:class:`!xmlparser` objects have the following methods to tune protections
+against some common XML vulnerabilities.
+
+.. method:: xmlparser.SetBillionLaughsAttackProtectionActivationThreshold(threshold, /)
+
+   Sets the number of output bytes needed to activate protection against
+   `billion laughs`_ attacks.
+
+   The number of output bytes includes amplification from entity expansion
+   and reading DTD files.
+
+   Parser objects usually have a protection activation threshold of 8 MiB,
+   but the actual default value depends on the underlying Expat library.
+
+   An :exc:`ExpatError` is raised if this method is called on a
+   |xml-non-root-parser| parser.
+   The corresponding :attr:`~ExpatError.lineno` and :attr:`~ExpatError.offset`
+   should not be used as they may have no special meaning.
+
+   .. note::
+
+      Activation thresholds below 4 MiB are known to break support for DITA 1.3
+      payload and are hence not recommended.
+
+   .. versionadded:: next
+
+.. method:: xmlparser.SetBillionLaughsAttackProtectionMaximumAmplification(max_factor, /)
+
+   Sets the maximum tolerated amplification factor for protection against
+   `billion laughs`_ attacks.
+
+   The amplification factor is calculated as ``(direct + indirect) / direct``
+   while parsing, where ``direct`` is the number of bytes read from
+   the primary document in parsing and ``indirect`` is the number of
+   bytes added by expanding entities and reading of external DTD files.
+
+   The *max_factor* value must be a non-NaN :class:`float` value greater than
+   or equal to 1.0. Peak amplifications of factor 15,000 for the entire payload
+   and of factor 30,000 in the middle of parsing have been observed with small
+   benign files in practice. In particular, the activation threshold should be
+   carefully chosen to avoid false positives.
+
+   Parser objects usually have a maximum amplification factor of 100,
+   but the actual default value depends on the underlying Expat library.
+
+   An :exc:`ExpatError` is raised if this method is called on a
+   |xml-non-root-parser| parser or if *max_factor* is outside the valid range.
+   The corresponding :attr:`~ExpatError.lineno` and :attr:`~ExpatError.offset`
+   should not be used as they may have no special meaning.
+
+   .. note::
+
+      The maximum amplification factor is only considered if the threshold
+      that can be adjusted by :meth:`.SetBillionLaughsAttackProtectionActivationThreshold`
+      is exceeded.
+
+   .. versionadded:: next
+
+.. method:: xmlparser.SetAllocTrackerActivationThreshold(threshold, /)
+
+   Sets the number of allocated bytes of dynamic memory needed to activate
+   protection against disproportionate use of RAM.
+
+   Parser objects usually have an allocation activation threshold of 64 MiB,
+   but the actual default value depends on the underlying Expat library.
+
+   An :exc:`ExpatError` is raised if this method is called on a
+   |xml-non-root-parser| parser.
+   The corresponding :attr:`~ExpatError.lineno` and :attr:`~ExpatError.offset`
+   should not be used as they may have no special meaning.
+
+   .. versionadded:: next
+
+.. method:: xmlparser.SetAllocTrackerMaximumAmplification(max_factor, /)
+
+   Sets the maximum amplification factor between direct input and bytes
+   of dynamic memory allocated.
+
+   The amplification factor is calculated as ``allocated / direct``
+   while parsing, where ``direct`` is the number of bytes read from
+   the primary document in parsing and ``allocated`` is the number
+   of bytes of dynamic memory allocated in the parser hierarchy.
+
+   The *max_factor* value must be a non-NaN :class:`float` value greater than
+   or equal to 1.0. Amplification factors greater than 100.0 can be observed
+   near the start of parsing even with benign files in practice. In particular,
+   the activation threshold should be carefully chosen to avoid false positives.
+
+   Parser objects usually have a maximum amplification factor of 100,
+   but the actual default value depends on the underlying Expat library.
+
+   An :exc:`ExpatError` is raised if this method is called on a
+   |xml-non-root-parser| parser or if *max_factor* is outside the valid range.
+   The corresponding :attr:`~ExpatError.lineno` and :attr:`~ExpatError.offset`
+   should not be used as they may have no special meaning.
+
+   .. note::
+
+      The maximum amplification factor is only considered if the threshold
+      that can be adjusted by :meth:`.SetAllocTrackerActivationThreshold`
+      is exceeded.
+
+   .. versionadded:: next
+
+
 :class:`xmlparser` objects have the following attributes:
 
 
@@ -954,3 +1066,6 @@ The ``errors`` module has the following attributes:
    not. See https://www.w3.org/TR/2006/REC-xml11-20060816/#NT-EncodingDecl
    and https://www.iana.org/assignments/character-sets/character-sets.xhtml.
 
+
+.. _billion laughs: https://en.wikipedia.org/wiki/Billion_laughs_attack
+.. |xml-non-root-parser| replace:: :ref:`non-root <xmlparser-non-root>`

Doc/library/xml.rst

@@ -55,7 +55,7 @@ An attacker can abuse XML features to carry out denial of service attacks,
 access local files, generate network connections to other machines, or
 circumvent firewalls.
 
-Expat versions lower that 2.6.0 may be vulnerable to "billion laughs",
+Expat versions lower than 2.6.0 may be vulnerable to "billion laughs",
 "quadratic blowup" and "large tokens". Python may be vulnerable if it uses such
 older versions of Expat as a system-provided library.
 Check :const:`!pyexpat.EXPAT_VERSION`.

Doc/using/configure.rst

@@ -869,9 +869,9 @@ Libraries options
    .. versionchanged:: 3.13
       Default to using the installed ``mpdecimal`` library.
 
-   .. deprecated-removed:: 3.13 3.15
+   .. deprecated-removed:: 3.13 3.16
       A copy of the ``mpdecimal`` library sources will no longer be distributed
-      with Python 3.15.
+      with Python 3.16.
 
    .. seealso:: :option:`LIBMPDEC_CFLAGS` and :option:`LIBMPDEC_LIBS`.
 

Doc/whatsnew/3.15.rst

@@ -294,6 +294,14 @@ New modules
 Improved modules
 ================
 
+collections
+-----------
+
+* Added :meth:`!collections.Counter.__xor__` and
+  :meth:`!collections.Counter.__ixor__` to compute the symmetric difference
+  between :class:`~collections.Counter` objects.
+  (Contributed by Raymond Hettinger in :gh:`138682`.)
+
 collections.abc
 ---------------
 
@@ -528,6 +536,15 @@ tarfile
   (Contributed by Christoph Walcher in :gh:`57911`.)
 
 
+timeit
+------
+
+* The command-line interface now colorizes error tracebacks
+  by default. This can be controlled with
+  :ref:`environment variables <using-on-controlling-color>`.
+  (Contributed by Yi Hong in :gh:`139374`.)
+
+
 types
 ------
 
@@ -545,6 +562,24 @@ unittest
   (Contributed by Garry Cairns in :gh:`134567`.)
 
 
+xml.parsers.expat
+-----------------
+
+* Add :meth:`~xml.parsers.expat.xmlparser.SetAllocTrackerActivationThreshold`
+  and :meth:`~xml.parsers.expat.xmlparser.SetAllocTrackerMaximumAmplification`
+  to :ref:`xmlparser <xmlparser-objects>` objects to tune protections against
+  disproportional amounts of dynamic memory usage from within an Expat parser.
+  (Contributed by Bénédikt Tran in :gh:`90949`.)
+
+* Add :meth:`~xml.parsers.expat.xmlparser.SetBillionLaughsAttackProtectionActivationThreshold`
+  and :meth:`~xml.parsers.expat.xmlparser.SetBillionLaughsAttackProtectionMaximumAmplification`
+  to :ref:`xmlparser <xmlparser-objects>` objects to tune protections against
+  `billion laughs`_ attacks.
+  (Contributed by Bénédikt Tran in :gh:`90949`.)
+
+  .. _billion laughs: https://en.wikipedia.org/wiki/Billion_laughs_attack
+
+
 zlib
 ----