Refresh Screen Provided By `curses.wrapper` Causes Seg Fault (macOS, xcode 15 Apple supplied ncurses 6.0 breakage)

[timway]

Crash report

What happened?

MacOS began pushing out updates to XCode Command Line Tools to install 15.0 recently. Upon updating I began having issues with curses. This happens with the Python provided by Apple. I'm not aware of the best way to communicate this issue to Apple, hopefully someone here knows who to ping or is watching.

Save the below as curses-segfault.py:

import curses

def main(stdscr):
    stdscr.refresh()

curses.wrapper(main)

Run the script in zsh in MacOS Terminal via:

/usr/bin/python3 curses-segfault.py

An easy way to see if you got the update is via the terminal by running:

softwareupdate --history

CPython versions tested on:

3.9

Operating systems tested on:

macOS

Output from running 'python -VV' on the command line:

python3 -VV Python 3.9.6 (default, Aug 11 2023, 19:44:49) [Clang 15.0.0 (clang-1500.0.40.1)]

Linked PRs

• gh-109617: fix ncurses incompatibility on macOS with Xcode 15 #111258

[brandtbucher]

I've been encountering this on main too after updating (the test_curses tests in our suite seem to trigger it). One really annoying side-effect is that it breaks your terminal after the crash:

% ./python.exe -m test --multiprocess 1 --use curses --verbose test_curses

0:00:00 load avg: 1.27 [1/1/1] test_curses process crashed (Exit code -11)
                                                                          test_has_extended_color_support (test.test_curses.MiscTests.test_has_extended_color_support) ... ok
    test_ncurses_version (test.test_curses.MiscTests.test_ncurses_version) ... ncurses_version = curses.ncurses_version(major=6, minor=0, patch=20150808)
                                                                                                                                                         ok
                                                                                                                                                           test_update_lines_cols (test.test_curses.MiscTests.test_update_lines_cols) ... ok
                                                                   test_alt (test.test_curses.TestAscii.test_alt) ... ok
                                                                                                                        test_ascii (test.test_curses.TestAscii.test_ascii) ... ok
        test_controlnames (test.test_curses.TestAscii.test_controlnames) ... ok
                                                                               test_ctrl (test.test_curses.TestAscii.test_ctrl) ... ok
                                                                                                                                      test_ctypes (test.test_curses.TestAscii.test_ctypes) ... ok
                        test_unctrl (test.test_curses.TestAscii.test_unctrl) ... ok
                                                                                   TERM=xterm-256color
                                                                                                      test_attributes (test.test_curses.TestCurses.test_attributes) ... ok
 test_background (test.test_curses.TestCurses.test_background) ... ok
                                                                     test_beep (test.test_curses.TestCurses.test_beep) ... ok
                                                                                                                             test_borders_and_lines (test.test_curses.TestCurses.test_borders_and_lines) ... ok
                                      test_chgat (test.test_curses.TestCurses.test_chgat) ... ok
                                                                                                test_clear (test.test_curses.TestCurses.test_clear) ... ok
                                                                                                                                                          test_color_attrs (test.test_curses.TestCurses.test_color_attrs) ... ok
                                                       test_color_content (test.test_curses.TestCurses.test_color_content) ... ok
                                                                                                                                 test_create_windows (test.test_curses.TestCurses.test_create_windows) ...

                                 == Tests result: FAILURE ==

                                                            1 test failed:
                                                                              test_curses

                                                                                         Total duration: 198 ms
                                                                                                               Total tests: run=0
                                                                                                                                 Total test files: run=1/1 failed=1
                                                                                                                                                                   Result: FAILURE
         %

I'm not a curses expert, but I'm not sure that this is something we can fix on our end, so this should probably be closed. It might be worth disabling the crashy tests if we can detect the broken curses version, though?

[brandtbucher]

CC @Yhg1s as a curses expert.

[brandtbucher]

The affected tests appear to be test_create_windows, test_move_cursor, test_output_character, test_refresh, and test_refresh_control.

[ned-deily]

FWIW this problem should not arise when using a python from a macOS python.org installer or MacPorts as they supply their own version of ncurses and do not use the system version.

[cdwrobg]

FWIW this problem should not arise when using a python from a macOS python.org installer or MacPorts as they supply their own version of ncurses and do not use the system version.

I can confirm this. I was only able to reproduce it for @timway when using the system version of python.

[brandtbucher]

Downgrading from Xcode 15.0 (curses.ncurses_version(major=6, minor=0, patch=20150808)) to 14.3 (curses.ncurses_version(major=5, minor=7, patch=20081102)) fixes it for me.

[gpshead]

It sounds like they shipped a potentially broken version of ncurses embedded within xcode 15?

We've been building and linking with ncurses >= 6 in the rest of the world since 2015 without this problem...

[gpshead]

It is really strange behavior on Apple's part to "upgrade" to providing 6.0 from 2015 in their 2023 toolchain when ncurses 6.4 came out last year.

We're at the point where it'd be fine for CPython to simply drop the ability to build and link against 6.0 in 3.13+ given that no supported Linux distro platform will be using it anymore by the time we release 3.13. (RHEL 8 shipped with ncurses 6.1 making that a reasonable minimum) -- I do not expect or encourage anyone to go through and do that to the _curses module code.

Recommendation: do what we do with python.org builds, never build against apple's ncurses.

If a workaround for this isn't obvious to someone with a macos debugger, it is reasonable to have a configure check blocklist the xcode 15 supplied ncurses and refuse to use it. Spending our time dealing with brazenly outdated libraries embedded in someone elses toolchain that we don't ship our own builds with isn't worthwhile (it'd never end).

[timway]

Thank you all for looking into this - as next steps I've found the Feedback Assistant and the general feedback form on Apple's website. I'll work to post to those methods and provide any feedback here.

I could also look at writing a small C example and see if I can confirm if the problem is in their ncurses implementation and fully removed from Python or not.

[vstinner]

If you want to require a specific version or recommend libedit or ncurses on macOS, you can update the doc: https://docs.python.org/dev/using/configure.html

[ronaldoussoren]

Issues like this should also be reported to Apple as this is a crash with their copy of Python.

I've filed FB13196764 about this.