New REPL will exit the interpreter due to errors from substituting builtins.__import__

[devdanzin]

Bug report

Bug description:

It's possible to make the new REPL exit the interpreter by replacing builtins.__import__ with a function that takes the wrong number of arguments or returns an invalid object. The basic REPL still works in these cases, even if imports don't. This happens in Linux and Windows with main.

>>> import builtins
>>> builtins.__import__ = lambda: None
>>> Exception ignored in the internal traceback machinery:
TypeError: <lambda>() takes 0 positional arguments but 5 were given
Traceback (most recent call last):
  File "/home/danzin/projects/cpython/Lib/runpy.py", line 198, in _run_module_as_main
  File "/home/danzin/projects/cpython/Lib/runpy.py", line 88, in _run_code
  File "/home/danzin/projects/cpython/Lib/_pyrepl/__main__.py", line 6, in <module>
  File "/home/danzin/projects/cpython/Lib/_pyrepl/main.py", line 59, in interactive_console
  File "/home/danzin/projects/cpython/Lib/_pyrepl/simple_interact.py", line 147, in run_multiline_interactive_console
  File "/home/danzin/projects/cpython/Lib/_pyrepl/readline.py", line 385, in multiline_input
  File "/home/danzin/projects/cpython/Lib/_pyrepl/reader.py", line 773, in readline
  File "/home/danzin/projects/cpython/Lib/_pyrepl/reader.py", line 722, in handle1
  File "/home/danzin/projects/cpython/Lib/_pyrepl/unix_console.py", line 549, in input_hook
TypeError: <lambda>() takes 0 positional arguments but 5 were given

Python 3.14.0a0 (heads/main:9187484dd97, Jul 29 2024, 09:25:22) [GCC 11.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import builtins
>>> builtins.__import__ = lambda x, y, z, a, b: None
>>> Exception ignored in the internal traceback machinery:
Traceback (most recent call last):
  File "/home/danzin/projects/cpython/Lib/traceback.py", line 139, in _print_exception_bltin
    return print_exception(exc, limit=BUILTIN_EXCEPTION_LIMIT, file=file, colorize=colorize)
  File "/home/danzin/projects/cpython/Lib/traceback.py", line 129, in print_exception
    te = TracebackException(type(value), value, tb, limit=limit, compact=True)
  File "/home/danzin/projects/cpython/Lib/traceback.py", line 1045, in __init__
    self.stack = StackSummary._extract_from_extended_frame_gen(
  File "/home/danzin/projects/cpython/Lib/traceback.py", line 492, in _extract_from_extended_frame_gen
    f.line
  File "/home/danzin/projects/cpython/Lib/traceback.py", line 369, in line
    self._set_lines()
  File "/home/danzin/projects/cpython/Lib/traceback.py", line 350, in _set_lines
    lines.append(linecache.getline(self.filename, lineno).rstrip())
  File "/home/danzin/projects/cpython/Lib/linecache.py", line 25, in getline
    lines = getlines(filename, module_globals)
  File "/home/danzin/projects/cpython/Lib/linecache.py", line 41, in getlines
    return updatecache(filename, module_globals)
  File "/home/danzin/projects/cpython/Lib/linecache.py", line 100, in updatecache
    stat = os.stat(fullname)
AttributeError: 'NoneType' object has no attribute 'stat'
Traceback (most recent call last):
  File "/home/danzin/projects/cpython/Lib/runpy.py", line 198, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/home/danzin/projects/cpython/Lib/runpy.py", line 88, in _run_code
    exec(code, run_globals)
  File "/home/danzin/projects/cpython/Lib/_pyrepl/__main__.py", line 6, in <module>
    __pyrepl_interactive_console()
  File "/home/danzin/projects/cpython/Lib/_pyrepl/main.py", line 59, in interactive_console
    run_multiline_interactive_console(console)
  File "/home/danzin/projects/cpython/Lib/_pyrepl/simple_interact.py", line 147, in run_multiline_interactive_console
    statement = multiline_input(more_lines, ps1, ps2)
  File "/home/danzin/projects/cpython/Lib/_pyrepl/readline.py", line 385, in multiline_input
    return reader.readline()
  File "/home/danzin/projects/cpython/Lib/_pyrepl/reader.py", line 773, in readline
    self.handle1()
  File "/home/danzin/projects/cpython/Lib/_pyrepl/reader.py", line 722, in handle1
    input_hook = self.console.input_hook
  File "/home/danzin/projects/cpython/Lib/_pyrepl/unix_console.py", line 552, in input_hook
    if posix._is_inputhook_installed():
AttributeError: 'NoneType' object has no attribute '_is_inputhook_installed'

Not sure this is something that should be guarded against or is just a "then don't do that" case.

If this is deemed worth fixing, we could change _pyrepl.windows_console.[Windows|Unix]Console.input_hook to guard against TypeError and AttributeError, or move the imports to module level. Also it might make sense to guard updatecache in linecache against the AttributeError. But, again, I'm not sure this should be fixed.

CPython versions tested on:

CPython main branch

Operating systems tested on:

Linux, Windows

[terryjreedy]

I think touching builtins is 'at your own risk' territory. I am a little surprised that the repl is apparently trying to execute code after printing the prompt and before anything is entered. Did you hit return to enter a blank line? Or was an entered line overwritten by the exception? (The latter might be considered a bug.)

In 3.12.4, a subsequent import itertools, for instance, is required to get a simple traceback with the same 'takes 0' exception. It is followed by a new prompt.

When I do the same in IDLE's Shell, which executes entered code in a subprocess, I get this unusual output on stdout:

----------------------------------------
Unhandled exception in user code execution server!'
Thread: SockThread
IDLE Client Address: ('127.0.0.1', 49477)
Request: <socket.socket fd=572, family=2, type=1, proto=0, laddr=('127.0.0.1', 60874), raddr=('127.0.0.1', 49477)>

followed by a traceback on stderr from the socket thread of the subproccess, ending with the 'take 0' exception, followed by another traceback from the subprocess exception handler, again with the 'take 0' exception. I do not consider this a bug in IDLE. Replace a support girder with a defective version and the structure collapses? The user's fault.

[devdanzin]

I think touching builtins is 'at your own risk' territory. I am a little surprised that the repl is apparently trying to execute code after printing the prompt and before anything is entered. Did you hit return to enter a blank line? Or was an entered line overwritten by the exception? (The latter might be considered a bug.)

No, I didn't hit return: the issue comes from e.g. WindowsConsole.input_hook, which is a property:

cpython/Lib/_pyrepl/windows_console.py

Lines 204 to 211 in 46f5a4f

	@property
	def input_hook(self):
	try:
	import nt
	except ImportError:
	return None
	if nt._is_inputhook_installed():
	return nt._inputhook

That is checked at the loop in _pyrepl.reader.Reader.handle1:

cpython/Lib/_pyrepl/reader.py

Lines 726 to 737 in 46f5a4f

	while True:
	input_hook = self.console.input_hook
	if input_hook:
	input_hook()
	# We use the same timeout as in readline.c: 100ms
	while not self.console.wait(100):
	input_hook()
	event = self.console.get_event(block=False)
	else:
	event = self.console.get_event(block)
	if not event: # can only happen if we're not blocking
	return False

Here's a diff that fixes both errors in Windows and avoids exiting the interpreter:

diff --git a/Lib/_pyrepl/windows_console.py b/Lib/_pyrepl/windows_console.py
index ba9af36b8be..0d98e4d5da0 100644
--- a/Lib/_pyrepl/windows_console.py
+++ b/Lib/_pyrepl/windows_console.py
@@ -205,10 +205,11 @@ def refresh(self, screen: list[str], c_xy: tuple[int, int]) -> None:
     def input_hook(self):
         try:
             import nt
-        except ImportError:
+            if nt._is_inputhook_installed():
+                return nt._inputhook
+        except (ImportError, AttributeError, TypeError):
             return None
-        if nt._is_inputhook_installed():
-            return nt._inputhook
+

     def __write_changed_line(
         self, y: int, oldline: str, newline: str, px_coord: int
diff --git a/Lib/traceback.py b/Lib/traceback.py
index 6ee1a50ca68..744d9605238 100644
--- a/Lib/traceback.py
+++ b/Lib/traceback.py
@@ -695,7 +695,7 @@ def output_line(lineno):
         return ''.join(row)

     def _should_show_carets(self, start_offset, end_offset, all_lines, anchors):
-        with suppress(SyntaxError, ImportError):
+        with suppress(SyntaxError, ImportError, TypeError):
             import ast
             tree = ast.parse('\n'.join(all_lines))
             statement = tree.body[0]

[picnixz]

@devdanzin Do you want to open a PR or do you want someone else to look at it? (I'm triaging your issues and thanks a lot for your work).

cc @pablogsal

[devdanzin]

If possible, I'd rather have someone else look at it.

Thanks for triaging and for your fixes!

[pablogsal]

Although I agree with @terryjreedy that touching builtins is 'at your own risk' territory, I think a PR protecting against failure of the input hook makes sense as we are already protecting against some other similar scenarios.

On the other hand I don't particularly like the additions to traceback.py. That's a much heavier change that can obscure other errors. Why is TypeError being ignored there?