gh-55454: Add IMAP4 IDLE support to imaplib

This extends imaplib with support for the rfc2177 IMAP IDLE command,
as requested in #55454.  It allows events to be pushed to a client as
they occur, rather than having to continually poll for mailbox changes.

The interface is a new idle() method, which returns an iterable context
manager.  Entering the context starts IDLE mode, during which events
(untagged responses) can be retrieved using the iteration protocol.
Exiting the context sends DONE to the server, ending IDLE mode.

An optional time limit for the IDLE session is supported, for use with
servers that impose an inactivity timeout.

The context manager also offers a burst() method, designed for programs
wishing to process events in batch rather than one at a time.

Notable differences from other implementations:

- It's an extension to imaplib, rather than a replacement.
- It doesn't introduce additional threads.
- It doesn't impose new requirements on the use of imaplib's existing methods.
- It passes the unit tests in CPython's test/test_imaplib.py module
  (and adds new ones).
- It works on Windows, Linux, and other unix-like systems.
- It makes IDLE available on all of imaplib's client variants
  (including IMAP4_stream).
- The interface is pythonic and easy to use.

Caveats:

- Due to a Windows limitation, the special case of IMAP4_stream running
  on Windows lacks a duration/timeout feature. (This is the stdin/stdout
  pipe connection variant; timeouts work fine for socket-based
  connections, even on Windows.) I have documented it where appropriate.

- The file-like imaplib instance attributes are changed from buffered to
  unbuffered mode. This could potentially break any client code that
  uses those objects directly without expecting partial reads/writes.
  However, these attributes are undocumented. As such, I think (and
  PEP 8 confirms) that they are fair game for changes.
  https://peps.python.org/pep-0008/#public-and-internal-interfaces

Usage examples:

#55454 (comment)

Original discussion:

https://discuss.python.org/t/gauging-interest-in-my-imap4-idle-implementation-for-imaplib/59272

Earlier requests and suggestions:

#55454

https://mail.python.org/archives/list/[email protected]/thread/C4TVEYL5IBESQQPPS5GBR7WFBXCLQMZ2/

Author: Forest

Doc/library/imaplib.rst

@@ -10,6 +10,7 @@
 .. changes for IMAP4_SSL by Tino Lange <[email protected]>, March 2002
 .. changes for IMAP4_stream by Piers Lauder <[email protected]>,
    November 2002
+.. changes for IDLE by Forest <[email protected]> August 2024
 
 **Source code:** :source:`Lib/imaplib.py`
 
@@ -187,7 +188,7 @@ However, the *password* argument to the ``LOGIN`` command is always quoted. If
 you want to avoid having an argument string quoted (eg: the *flags* argument to
 ``STORE``) then enclose the string in parentheses (eg: ``r'(\Deleted)'``).
 
-Each command returns a tuple: ``(type, [data, ...])`` where *type* is usually
+Most commands return a tuple: ``(type, [data, ...])`` where *type* is usually
 ``'OK'`` or ``'NO'``, and *data* is either the text from the command response,
 or mandated results from the command. Each *data* is either a ``bytes``, or a
 tuple. If a tuple, then the first part is the header of the response, and the
@@ -307,6 +308,48 @@ An :class:`IMAP4` instance has the following methods:
    of the IMAP4 QUOTA extension defined in rfc2087.
 
 
+.. method:: IMAP4.idle([dur])
+
+   Return an iterable context manager implementing the ``IDLE`` command
+   as defined in :rfc:`2177`.
+
+   The optional *dur* argument specifies a maximum duration (in seconds) to
+   keep idling.  It defaults to ``None``, meaning no time limit.
+   To avoid inactivity timeouts on servers that impose them, callers are
+   advised to keep this <= 29 minutes.  See the note below regarding
+   :class:`IMAP4_stream` on Windows.
+
+   The context manager sends the ``IDLE`` command upon entry, produces
+   responses via iteration, and sends ``DONE`` upon exit.
+   It represents responses as ``(type, datum)`` tuples, rather than the
+   ``(type, [data, ...])`` tuples returned by other methods, because only
+   one response is represented at a time.
+
+   Example::
+
+      with M.idle(dur=29*60) as idler:
+          for response in idler:
+              typ, datum = response
+              print(typ, datum)
+
+   It is also possible to process a burst of responses all at once instead
+   of one at a time.  See `IDLE Context Manager`_ for details.
+
+   Responses produced by the iterator are not added to the internal
+   cache for retrieval by :meth:`IMAP4.response`.
+
+   .. note::
+
+      Windows :class:`IMAP4_stream` connections have no way to accurately
+      respect *dur*, since Windows ``select()`` only works on sockets.
+      However, if the server regularly sends status messages during ``IDLE``,
+      they will wake our selector and keep iteration from blocking for long.
+      Dovecot's ``imap_idle_notify_interval`` is two minutes by default.
+      Assuming that's typical of IMAP servers, subtracting it from the 29
+      minutes needed to avoid server inactivity timeouts would make 27
+      minutes a sensible value for *dur* in this situation.
+
+
 .. method:: IMAP4.list([directory[, pattern]])
 
    List mailbox names in *directory* matching *pattern*.  *directory* defaults to
@@ -612,6 +655,62 @@ The following attributes are defined on instances of :class:`IMAP4`:
    .. versionadded:: 3.5
 
 
+.. _idle context manager:
+
+IDLE Context Manager
+--------------------
+
+The object returned by :meth:`IMAP4.idle` implements the context management
+protocol for the :keyword:`with` statement, and the :term:`iterator` protocol
+for retrieving untagged responses while the context is active.
+It also has the following method:
+
+.. method:: IdleContextManager.burst([interval])
+
+   Yield a burst of responses no more than *interval* seconds apart.
+
+   This generator retrieves the next response along with any
+   immediately available subsequent responses (e.g. a rapid series of
+   ``EXPUNGE`` responses after a bulk delete) so they can be efficiently
+   processed as a batch instead of one at a time.
+
+   The optional *interval* argument specifies a time limit (in seconds)
+   for each response after the first.  It defaults to 0.1 seconds.
+   (The ``IDLE`` context's maximum duration is respected when waiting for the
+   first response.)
+
+   Represents responses as ``(type, datum)`` tuples, just as when
+   iterating directly on the context manager.
+
+   Example::
+
+      with M.idle() as idler:
+
+          # get the next response and any others following by < 0.1 seconds
+          batch = list(idler.burst())
+
+          print(f'processing {len(batch)} responses...')
+          for typ, datum in batch:
+              print(typ, datum)
+
+   Produces no responses and returns immediately if the ``IDLE`` context's
+   maximum duration (the *dur* argument to :meth:`IMAP4.idle`) has elapsed.
+   Callers should plan accordingly if using this method in a loop.
+
+   .. note::
+
+      Windows :class:`IMAP4_stream` connections will ignore the *interval*
+      argument, yielding endless responses and blocking indefinitely for each
+      one, since Windows ``select()`` only works on sockets.  It is therefore
+      advised not to use this method with an :class:`IMAP4_stream` connection
+      on Windows.
+
+.. note::
+
+  The context manager's type name is not part of its public interface,
+  and is subject to change.
+
+
 .. _imap4-example:
 
 IMAP4 Example

Doc/whatsnew/3.14.rst

@@ -117,6 +117,12 @@ Added support for converting any objects that have the
 :meth:`!as_integer_ratio` method to a :class:`~fractions.Fraction`.
 (Contributed by Serhiy Storchaka in :gh:`82017`.)
 
+imaplib
+-------
+
+* Add :meth:`~imaplib.IMAP4.idle`, implementing the ``IDLE`` command
+  as defined in :rfc:`2177`.  (Contributed by Forest in :gh:`55454`.)
+
 json
 ----