IDLE - faster writing of user code output

[serwy]

BPO	45804
Nosy	@terryjreedy, @serwy
Files	idlelib_buffer_output.patch: initial patch proposal

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/terryjreedy'
closed_at = None
created_at = <Date 2021-11-14.02:48:01.662>
labels = ['expert-IDLE', 'type-feature', '3.11']
title = 'IDLE - faster shell writing'
updated_at = <Date 2021-11-21.14:29:09.353>
user = 'https://github.com/serwy'

bugs.python.org fields:

activity = <Date 2021-11-21.14:29:09.353>
actor = 'wyz23x2'
assignee = 'terry.reedy'
closed = False
closed_date = None
closer = None
components = ['IDLE']
creation = <Date 2021-11-14.02:48:01.662>
creator = 'roger.serwy'
dependencies = []
files = ['50438']
hgrepos = []
issue_num = 45804
keywords = ['patch']
message_count = 3.0
messages = ['406306', '406308', '406327']
nosy_count = 2.0
nosy_names = ['terry.reedy', 'roger.serwy']
pr_nums = []
priority = 'normal'
resolution = None
stage = 'patch review'
status = 'open'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue45804'
versions = ['Python 3.11']

[serwy]

The shell provided by IDLE uses synchronous sys.stdout.write() calls between the subprocess and the front-end, leading to very slow writes. The provided patch proposes buffering the stdout/stderr streams in the subprocess and then sending a single update after 50ms. The patch also provides back pressure on the buffer so that it doesn't grow without bound.

When trying the behavior of the patch, disable the squeezer extension, or set its limit to 1000. Then in the shell, run:

    for i in range(500): print(i)

The output will instantly appear in the shell.

[terryjreedy]

Thank you Roger. Good to hear from you.

https://stackoverflow.com/questions/66286367/why-is-my-function-faster-than-pythons-print-function-in-idle, Feb 2021, was about this issue. In my answer I verified the claim and then showed in further experiments that batching prints solved the issue. bpo-43283 added a paragraph to the IDLE doc explaining the problem and suggesting that users could work around it by batching and joining before printing. Buffering the streams to do so is an appealing alternative. I intended to make sure that exceptions are completely and not just partly joined and sent in one write.

Some immediate questions:

Can buffering the output streams have any negative consequences. Does it affect isatty, for instance, or anything checking line buffering?

In case a user is interactively developing a tkinter GUI, IDLE already runs tcl.update() in a 50 ms after loop. Could this be used to trigger writes, by calling flush()?

Did you consider using io.TextIOWrapper instead the current TextIOBase as base class for the output classes?

I am guessing that you do not have a python/cpython clone for making PRs. Tomorrow, I will try to find out how to make apply a .patch file to mine and do so.

[serwy]

All good questions, Terry! I do have a git clone of the cpython repo, but I haven't worked through the new commit/patch process since Mercurial. I'm a bit rusty.

The buffering provided is for calls to write. It does not do any line buffering. Calls to isatty will behave the same. The negative side effect is that the subprocess will proceed as if writes have been committed to the PyShell window, so any pending transfers can be lost if the subprocess terminates unexpectedly.

I used a separate OS thread to handle the transfer of the writes rather than using the Tcl/Tk event loop. The Tcl/Tk loop runs on the main thread, so any long-running processes will prevent a .after callback function from being called.

The base class was not changed. I haven't followed all the changes to the stream class (and its implications) since 3.5, so the code subclasses the newer StdOutputFile class to minimize disruption.