DRAFT: Comprehension Unpacking Implementation

Doc/library/itertools.rst

@@ -208,6 +208,13 @@ loops that truncate the stream.
           for iterable in iterables:
               yield from iterable
 
+   .. note::
+
+      Since version 3.15, the following is also roughly equivalent::
+
+         def chain(*iterables):
+             return (*iterable for iterable in iterables)
+
 
 .. classmethod:: chain.from_iterable(iterable)
 
@@ -219,6 +226,13 @@ loops that truncate the stream.
           for iterable in iterables:
               yield from iterable
 
+   .. note::
+
+      Since version 3.15, the following is also roughly equivalent::
+
+         def from_iterable(iterables):
+             return (*iterable for iterable in iterables)
+
 
 .. function:: combinations(iterable, r)
 

Doc/reference/expressions.rst

@@ -209,17 +209,18 @@ called "displays", each of them in two flavors:
 Common syntax elements for comprehensions are:
 
 .. productionlist:: python-grammar
-   comprehension: `assignment_expression` `comp_for`
+   comprehension: `flexible_expression` `comp_for`
    comp_for: ["async"] "for" `target_list` "in" `or_test` [`comp_iter`]
    comp_iter: `comp_for` | `comp_if`
    comp_if: "if" `or_test` [`comp_iter`]
 
 The comprehension consists of a single expression followed by at least one
-:keyword:`!for` clause and zero or more :keyword:`!for` or :keyword:`!if` clauses.
-In this case, the elements of the new container are those that would be produced
-by considering each of the :keyword:`!for` or :keyword:`!if` clauses a block,
-nesting from left to right, and evaluating the expression to produce an element
-each time the innermost block is reached.
+:keyword:`!for` clause and zero or more :keyword:`!for` or :keyword:`!if`
+clauses.  In this case, the elements of the new container are those that would
+be produced by considering each of the :keyword:`!for` or :keyword:`!if`
+clauses a block, nesting from left to right, and evaluating the expression to
+produce an element each time the innermost block is reached.  If the expression
+is starred, the result will instead be unpacked to produce 0 or more elements.
 
 However, aside from the iterable expression in the leftmost :keyword:`!for` clause,
 the comprehension is executed in a separate implicitly nested scope. This ensures
@@ -232,6 +233,11 @@ leftmost :keyword:`!for` clause cannot be evaluated in the enclosing scope as
 they may depend on the values obtained from the leftmost iterable. For example:
 ``[x*y for x in range(10) for y in range(x, x+10)]``.
 
+.. note::
+   If any of the expressions in a comprehension contain a walrus operator (``:=``),
+   the resulting variable binding happens in the enclosing scope.  See
+   :pep:`572` for more details and examples.
+
 To ensure the comprehension always results in a container of the appropriate
 type, ``yield`` and ``yield from`` expressions are prohibited in the implicitly
 nested scope.
@@ -264,6 +270,9 @@ See also :pep:`530`.
    asynchronous functions. Outer comprehensions implicitly become
    asynchronous.
 
+.. versionchanged:: 3.15
+   Unpacking with the ``*`` operator is now allowed in the expression.
+
 
 .. _lists:
 
@@ -339,8 +348,8 @@ enclosed in curly braces:
 .. productionlist:: python-grammar
    dict_display: "{" [`dict_item_list` | `dict_comprehension`] "}"
    dict_item_list: `dict_item` ("," `dict_item`)* [","]
+   dict_comprehension: `dict_item` `comp_for`
    dict_item: `expression` ":" `expression` | "**" `or_expr`
-   dict_comprehension: `expression` ":" `expression` `comp_for`
 
 A dictionary display yields a new dictionary object.
 
@@ -362,10 +371,21 @@ earlier dict items and earlier dictionary unpackings.
 .. versionadded:: 3.5
    Unpacking into dictionary displays, originally proposed by :pep:`448`.
 
-A dict comprehension, in contrast to list and set comprehensions, needs two
-expressions separated with a colon followed by the usual "for" and "if" clauses.
-When the comprehension is run, the resulting key and value elements are inserted
-in the new dictionary in the order they are produced.
+A dict comprehension may take one of two forms:
+
+The first form  uses two expressions separated with a colon followed by the
+usual "for" and "if" clauses.  When the comprehension is run, the resulting key
+and value elements are inserted in the new dictionary in the order they are
+produced.
+
+The second form uses a single expression prefixed by the ``**`` dictionary
+unpacking operator followed by the usual "for" and "if" clauses.  When the
+comprehension is run, the expression is evaluated and then unpacked, inserting
+0 or more key/value pairs into the new dictionary.
+
+Both forms of dictionary comprehension retain the property that if the same key
+is specified multiple times, the associated value in the resulting dictionary
+will be the last one specified.
 
 .. index:: pair: immutable; object
            hashable
@@ -382,6 +402,8 @@ prevails.
    the key.  Starting with 3.8, the key is evaluated before the value, as
    proposed by :pep:`572`.
 
+.. versionchanged:: 3.15
+   Unpacking with the ``**`` operator is now allowed in dictionary comprehensions.
 
 .. _genexpr:
 
@@ -396,7 +418,7 @@ Generator expressions
 A generator expression is a compact generator notation in parentheses:
 
 .. productionlist:: python-grammar
-   generator_expression: "(" `expression` `comp_for` ")"
+   generator_expression: "(" `starred_expression` `comp_for` ")"
 
 A generator expression yields a new generator object.  Its syntax is the same as
 for comprehensions, except that it is enclosed in parentheses instead of

Doc/tutorial/classes.rst

@@ -926,6 +926,25 @@ Examples::
    >>> list(data[i] for i in range(len(data)-1, -1, -1))
    ['f', 'l', 'o', 'g']
 
+   >>> x = [[1,2,3], [], [4, 5]]
+   >>> g = (*i for i in x)
+   >>> list(g)
+   [1, 2, 3, 4, 5]
+
+In most cases, generator expressions must be wrapped in parentheses.  As a
+special case, however, when provided as the sole argument to a function (as in
+the examples involving ``sum``, ``set``, ``max``, and ``list`` above), the
+generator expression does not need to be wrapped in an additional set of
+parentheses.  That is to say, the following two pieces of code are semantically
+equivalent::
+
+   >>> f(x for x in y)
+   >>> f((x for x in y))
+
+as are the following::
+
+   >>> f(*x for x in y)
+   >>> f((*x for x in y))
 
 
 .. rubric:: Footnotes

Doc/tutorial/datastructures.rst

@@ -334,6 +334,47 @@ The :func:`zip` function would do a great job for this use case::
 
 See :ref:`tut-unpacking-arguments` for details on the asterisk in this line.
 
+Unpacking in Lists and List Comprehensions
+------------------------------------------
+
+The section on :ref:`tut-unpacking-arguments` describes the use of ``*`` to
+"unpack" the elements of an iterable object, providing each one seperately as
+an argument to a function.  Unpacking can also be used in other contexts, for
+example, when creating lists.  When specifying elements of a list, prefixing an
+expression by a ``*`` will unpack the result of that expression, adding each of
+its elements to the list we're creating::
+
+   >>> x = [1, 2, 3]
+   >>> [0, *x, 4, 5, 6]
+   [0, 1, 2, 3, 4, 5, 6]
+
+This only works if the expression following the ``*`` evaluates to an iterable
+object; trying to unpack a non-iterable object will raise an exception::
+
+   >>> x = 1
+   >>> [0, *x, 2, 3, 4]
+   Traceback (most recent call last):
+     File "<python-input-1>", line 1, in <module>
+       [0, *x, 2, 3, 4]
+   TypeError: Value after * must be an iterable, not int
+
+Unpacking can also be used in list comprehensions, as a way to build a new list
+representing the concatenation of an arbitrary number of iterables::
+
+   >>> x = [[1, 2, 3], [4, 5, 6], [], [7], [8, 9]]
+   >>> [*element for element in x]
+   [1, 2, 3, 4, 5, 6, 7, 8, 9]
+
+Note that the effect is that each element from ``x`` is unpacked.  This works
+for arbitrary iterable objects, not just lists::
+
+   >>> x = [[1, 2, 3], 'cat', {'spam': 'eggs'}]
+   >>> [*element for element in x]
+   [1, 2, 3, 'c', 'a', 't', 'spam']
+
+But if the objects in ``x`` are not iterable, this expression would again raise
+an exception.
+
 .. _tut-del:
 
 The :keyword:`!del` statement
@@ -395,7 +436,10 @@ A tuple consists of a number of values separated by commas, for instance::
    >>> v = ([1, 2, 3], [3, 2, 1])
    >>> v
    ([1, 2, 3], [3, 2, 1])
-
+   >>> # they support unpacking just like lists:
+   >>> x = [1,2,3]
+   >>> 0, *x, 4
+   (0, 1, 2, 3, 4)
 
 As you see, on output tuples are always enclosed in parentheses, so that nested
 tuples are interpreted correctly; they may be input with or without surrounding
@@ -480,12 +524,16 @@ Here is a brief demonstration::
    {'r', 'd', 'b', 'm', 'z', 'l'}
 
 Similarly to :ref:`list comprehensions <tut-listcomps>`, set comprehensions
-are also supported::
+are also supported, including comprehensions with unpacking::
 
    >>> a = {x for x in 'abracadabra' if x not in 'abc'}
    >>> a
    {'r', 'd'}
 
+   >>> fruits = [{'apple', 'avocado', 'apricot'}, {'banana', 'blueberry'}]
+   >>> {*fruit for fruit in fruits}
+   {'blueberry', 'banana', 'avocado', 'apple', 'apricot'}
+
 
 .. _tut-dictionaries:
 
@@ -553,6 +601,18 @@ arbitrary key and value expressions::
    >>> {x: x**2 for x in (2, 4, 6)}
    {2: 4, 4: 16, 6: 36}
 
+And dictionary unpacking (via ``**``) can be used to merge multiple
+dictionaries::
+
+   >>> odds = {i: i**2 for i in (1, 3, 5)}
+   >>> evens = {i: i**2 for i in (2, 4, 6)}
+   >>> {**odds, **evens}
+   {1: 1, 3: 9, 5: 25, 2: 4, 4: 16, 6: 36}
+
+   >>> all_values = [odds, evens, {0: 0}]
+   >>> {**i for i in all_values}
+   {1: 1, 3: 9, 5: 25, 2: 4, 4: 16, 6: 36, 0: 0}
+
 When the keys are simple strings, it is sometimes easier to specify pairs using
 keyword arguments::