From eecc6353c2e607f825f17c64abc4b7d44fd12ad1 Mon Sep 17 00:00:00 2001 From: Daniele Procida Date: Sun, 30 Jul 2023 00:05:32 +0200 Subject: [PATCH 1/7] gh-107432 Rework the style of https://docs.python.org/3/howto/functional.html This is not a how-to guide; the title has been changed. A future commit should move the document out of /howto This patch removes comments about what the author thinks the reader is familiar with, and the first-person voice that sometimes appeared. Some wording that dated the document has been changed. --- Doc/howto/functional.rst | 75 +++++++++++++--------------------------- 1 file changed, 24 insertions(+), 51 deletions(-) diff --git a/Doc/howto/functional.rst b/Doc/howto/functional.rst index b0f9d22d74f0e3..a18d5ab3f0db88 100644 --- a/Doc/howto/functional.rst +++ b/Doc/howto/functional.rst @@ -1,34 +1,24 @@ ******************************** - Functional Programming HOWTO +Functional programming in Python ******************************** :Author: A. M. Kuchling :Release: 0.32 -In this document, we'll take a tour of Python's features suitable for -implementing programs in a functional style. After an introduction to the -concepts of functional programming, we'll look at language features such as -:term:`iterator`\s and :term:`generator`\s and relevant library modules such as -:mod:`itertools` and :mod:`functools`. - -Introduction -============ - -This section explains the basic concept of functional programming; if -you're just interested in learning about Python language features, -skip to the next section on :ref:`functional-howto-iterators`. +The basics of functional programming +==================================== Programming languages support decomposing problems in several different ways: -* Most programming languages are **procedural**: programs are lists of +* Many programming languages are **procedural**: programs are lists of instructions that tell the computer what to do with the program's input. C, Pascal, and even Unix shells are procedural languages. * In **declarative** languages, you write a specification that describes the problem to be solved, and the language implementation figures out how to - perform the computation efficiently. SQL is the declarative language you're - most likely to be familiar with; a SQL query describes the data set you want + perform the computation efficiently. SQL is an example of a declarative + language; a SQL query describes the data set you want to retrieve, and the SQL engine decides whether to scan tables or use indexes, which subclauses should be performed first, etc. @@ -57,7 +47,7 @@ functional, for example. In a functional program, input flows through a set of functions. Each function operates on its input and produces some output. Functional style discourages -functions with side effects that modify internal state or make other changes +functions with *side effects* that modify internal state or make other changes that aren't visible in the function's return value. Functions that have no side effects at all are called **purely functional**. Avoiding side effects means not using data structures that get updated as a program runs; every function's @@ -177,7 +167,7 @@ a few functions specialized for the current task. Iterators ========= -I'll start by looking at a Python language feature that's an important +Let's start by considering a Python language feature that's an important foundation for writing functional-style programs: iterators. An iterator is an object representing a stream of data; this object returns the @@ -259,7 +249,6 @@ consume all of the iterator's output, and if you need to do something different with the same stream, you'll have to create a new iterator. - Data Types That Support Iterators --------------------------------- @@ -326,7 +315,6 @@ elements:: 13 - Generator expressions and list comprehensions ============================================= @@ -440,7 +428,7 @@ Generators are a special class of functions that simplify the task of writing iterators. Regular functions compute a value and return it, but generators return an iterator that returns a stream of values. -You're doubtless familiar with how regular function calls work in Python or C. +You may be familiar with how regular function calls work in Python or C. When you call a function, it gets a private namespace where its local variables are created. When the function reaches a ``return`` statement, the local variables are destroyed and the value is returned to the caller. A later call @@ -529,18 +517,15 @@ Passing values into a generator In Python 2.4 and earlier, generators only produced output. Once a generator's code was invoked to create an iterator, there was no way to pass any new -information into the function when its execution is resumed. You could hack -together this ability by making the generator look at a global variable or by -passing in some mutable object that callers then modify, but these approaches -are messy. +information into the function when its execution is resumed. -In Python 2.5 there's a simple way to pass values into a generator. -:keyword:`yield` became an expression, returning a value that can be assigned to -a variable or otherwise operated on:: +In Python 2.5, :keyword:`yield` became an expression, returning a value that +can be assigned to a variable or otherwise operated on, providing a simple +way to pass values into a generator:: val = (yield i) -I recommend that you **always** put parentheses around a ``yield`` expression +It's recommended that you **always** put parentheses around a ``yield`` expression when you're doing something with the returned value, as in the above example. The parentheses aren't always necessary, but it's easier to always add them instead of having to remember when they're needed. @@ -608,8 +593,8 @@ generators: will also be called by Python's garbage collector when the generator is garbage-collected. - If you need to run cleanup code when a :exc:`GeneratorExit` occurs, I suggest - using a ``try: ... finally:`` suite instead of catching :exc:`GeneratorExit`. + If you need to run cleanup code when a :exc:`GeneratorExit` occurs, it's recommended + to us a ``try: ... finally:`` suite instead of catching :exc:`GeneratorExit`. The cumulative effect of these changes is to turn generators from one-way producers of information into both producers and consumers. @@ -639,7 +624,7 @@ features of generator expressions: >>> [upper(s) for s in ['sentence', 'fragment']] ['SENTENCE', 'FRAGMENT'] -You can of course achieve the same effect with a list comprehension. +(You can achieve the same effect with a list comprehension.) :func:`filter(predicate, iter) ` returns an iterator over all the sequence elements that meet a certain condition, and is similarly duplicated by @@ -1131,20 +1116,21 @@ usual way:: def print_assign(name, value): return name + '=' + str(value) -Which alternative is preferable? That's a style question; my usual course is to -avoid using ``lambda``. +Which alternative is preferable? That's mostly a question of style. -One reason for my preference is that ``lambda`` is quite limited in the +You may wish to avoid using ``lambda``, as there are limits to the functions it can define. The result has to be computable as a single expression, which means you can't have multiway ``if... elif... else`` comparisons or ``try... except`` statements. If you try to do too much in a ``lambda`` statement, you'll end up with an overly complicated expression that's -hard to read. Quick, what's the following code doing? :: +hard to read. + +Consider:: import functools total = functools.reduce(lambda a, b: (0, a[1] + b[1]), items)[1] -You can figure it out, but it takes time to disentangle the expression to figure +It takes some mental effort to disentangle the expression to figure out what's going on. Using a short nested ``def`` statements makes things a little bit better:: @@ -1154,7 +1140,7 @@ little bit better:: total = functools.reduce(combine, items)[1] -But it would be best of all if I had simply used a ``for`` loop:: +But it would be best of all would have been to use a ``for`` loop:: total = 0 for a, b in items: @@ -1166,19 +1152,6 @@ Or the :func:`sum` built-in and a generator expression:: Many uses of :func:`functools.reduce` are clearer when written as ``for`` loops. -Fredrik Lundh once suggested the following set of rules for refactoring uses of -``lambda``: - -1. Write a lambda function. -2. Write a comment explaining what the heck that lambda does. -3. Study the comment for a while, and think of a name that captures the essence - of the comment. -4. Convert the lambda to a def statement, using that name. -5. Remove the comment. - -I really like these rules, but you're free to disagree -about whether this lambda-free style is better. - Revision History and Acknowledgements ===================================== From 9bdb87c6e6e6962b44bf6494d175c7a6057eb766 Mon Sep 17 00:00:00 2001 From: Daniele Procida Date: Sun, 30 Jul 2023 00:14:02 +0200 Subject: [PATCH 2/7] Fix typo --- Doc/howto/functional.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/Doc/howto/functional.rst b/Doc/howto/functional.rst index a18d5ab3f0db88..40d1c38ddb488f 100644 --- a/Doc/howto/functional.rst +++ b/Doc/howto/functional.rst @@ -594,7 +594,7 @@ generators: garbage-collected. If you need to run cleanup code when a :exc:`GeneratorExit` occurs, it's recommended - to us a ``try: ... finally:`` suite instead of catching :exc:`GeneratorExit`. + to use a ``try: ... finally:`` suite instead of catching :exc:`GeneratorExit`. The cumulative effect of these changes is to turn generators from one-way producers of information into both producers and consumers. From 49ca6dc098798d6b6e2f1146b0a6625b910e05fc Mon Sep 17 00:00:00 2001 From: Daniele Procida Date: Sun, 30 Jul 2023 00:15:00 +0200 Subject: [PATCH 3/7] Fix typo --- Doc/howto/functional.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/Doc/howto/functional.rst b/Doc/howto/functional.rst index 40d1c38ddb488f..0637c6c8ed4e61 100644 --- a/Doc/howto/functional.rst +++ b/Doc/howto/functional.rst @@ -1140,7 +1140,7 @@ little bit better:: total = functools.reduce(combine, items)[1] -But it would be best of all would have been to use a ``for`` loop:: +But would be best of all would have been to use a ``for`` loop:: total = 0 for a, b in items: From 8e8ce6b12402bc0e9e61e79254400d0ba608fbef Mon Sep 17 00:00:00 2001 From: Daniele Procida Date: Sun, 30 Jul 2023 21:04:56 +0200 Subject: [PATCH 4/7] Update Doc/howto/functional.rst Co-authored-by: Adam Turner <9087854+AA-Turner@users.noreply.github.com> --- Doc/howto/functional.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/Doc/howto/functional.rst b/Doc/howto/functional.rst index 0637c6c8ed4e61..05f2bcf0f7c4a4 100644 --- a/Doc/howto/functional.rst +++ b/Doc/howto/functional.rst @@ -1140,7 +1140,7 @@ little bit better:: total = functools.reduce(combine, items)[1] -But would be best of all would have been to use a ``for`` loop:: +But best of all would have been to use a ``for`` loop:: total = 0 for a, b in items: From a403c17e6ddd126804c636fced0a530dfe474ef3 Mon Sep 17 00:00:00 2001 From: Daniele Procida Date: Sun, 30 Jul 2023 21:05:07 +0200 Subject: [PATCH 5/7] Update Doc/howto/functional.rst Co-authored-by: Adam Turner <9087854+AA-Turner@users.noreply.github.com> --- Doc/howto/functional.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/Doc/howto/functional.rst b/Doc/howto/functional.rst index 05f2bcf0f7c4a4..b7bb44379b1b59 100644 --- a/Doc/howto/functional.rst +++ b/Doc/howto/functional.rst @@ -1,5 +1,5 @@ ******************************** -Functional programming in Python +Functional Programming in Python ******************************** :Author: A. M. Kuchling From d999392fe56cf2c560f55b19ab613e6a558eeaa3 Mon Sep 17 00:00:00 2001 From: Kumar Aditya Date: Sat, 5 Aug 2023 17:48:15 +0530 Subject: [PATCH 6/7] GH-106684: raise `ResourceWarning` when `asyncio.StreamWriter` is not closed (#107650) --- Lib/asyncio/streams.py | 6 +++++ Lib/test/test_asyncio/test_streams.py | 23 +++++++++++++++++++ ...-08-05-05-10-41.gh-issue-106684.P9zRXb.rst | 1 + 3 files changed, 30 insertions(+) create mode 100644 Misc/NEWS.d/next/Library/2023-08-05-05-10-41.gh-issue-106684.P9zRXb.rst diff --git a/Lib/asyncio/streams.py b/Lib/asyncio/streams.py index bf15f517e50dce..b7ad365709b19e 100644 --- a/Lib/asyncio/streams.py +++ b/Lib/asyncio/streams.py @@ -5,6 +5,7 @@ import collections import socket import sys +import warnings import weakref if hasattr(socket, 'AF_UNIX'): @@ -392,6 +393,11 @@ async def start_tls(self, sslcontext, *, self._transport = new_transport protocol._replace_writer(self) + def __del__(self, warnings=warnings): + if not self._transport.is_closing(): + self.close() + warnings.warn(f"unclosed {self!r}", ResourceWarning) + class StreamReader: diff --git a/Lib/test/test_asyncio/test_streams.py b/Lib/test/test_asyncio/test_streams.py index 7f9dc621808358..9c92e75886c593 100644 --- a/Lib/test/test_asyncio/test_streams.py +++ b/Lib/test/test_asyncio/test_streams.py @@ -1074,6 +1074,29 @@ def test_eof_feed_when_closing_writer(self): self.assertEqual(messages, []) + def test_unclosed_resource_warnings(self): + async def inner(httpd): + rd, wr = await asyncio.open_connection(*httpd.address) + + wr.write(b'GET / HTTP/1.0\r\n\r\n') + data = await rd.readline() + self.assertEqual(data, b'HTTP/1.0 200 OK\r\n') + data = await rd.read() + self.assertTrue(data.endswith(b'\r\n\r\nTest message')) + with self.assertWarns(ResourceWarning): + del wr + gc.collect() + + + messages = [] + self.loop.set_exception_handler(lambda loop, ctx: messages.append(ctx)) + + with test_utils.run_test_server() as httpd: + self.loop.run_until_complete(inner(httpd)) + + self.assertEqual(messages, []) + + if __name__ == '__main__': unittest.main() diff --git a/Misc/NEWS.d/next/Library/2023-08-05-05-10-41.gh-issue-106684.P9zRXb.rst b/Misc/NEWS.d/next/Library/2023-08-05-05-10-41.gh-issue-106684.P9zRXb.rst new file mode 100644 index 00000000000000..02c52d714e9df7 --- /dev/null +++ b/Misc/NEWS.d/next/Library/2023-08-05-05-10-41.gh-issue-106684.P9zRXb.rst @@ -0,0 +1 @@ +Raise :exc:`ResourceWarning` when :class:`asyncio.StreamWriter` is not closed leading to memory leaks. Patch by Kumar Aditya. From 2c87aa0f94bc611d5c9e38076d6ef445b48620fa Mon Sep 17 00:00:00 2001 From: Daniele Procida Date: Sat, 5 Aug 2023 14:52:37 +0200 Subject: [PATCH 7/7] Clarify passing values into a generator in 2.4 --- Doc/howto/functional.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/Doc/howto/functional.rst b/Doc/howto/functional.rst index b7bb44379b1b59..5ed6e2d9a72815 100644 --- a/Doc/howto/functional.rst +++ b/Doc/howto/functional.rst @@ -516,7 +516,7 @@ Passing values into a generator ------------------------------- In Python 2.4 and earlier, generators only produced output. Once a generator's -code was invoked to create an iterator, there was no way to pass any new +code was invoked to create an iterator, there was no elegant way to pass any new information into the function when its execution is resumed. In Python 2.5, :keyword:`yield` became an expression, returning a value that