PyByteArray_Check(Long) crashes in Python 3.9 and 3.10

[juanjux]

Crash report

What happened?

TEST(PyByteArray_Check, WithNumber)
{
    PyObject* non_text_obj = PyLong_FromLong(42);
    EXPECT_FALSE(PyByteArray_Check(non_text_obj));
    Py_DecRef(non_text_obj);
}

This crashes (segmentation fault) for me on Python 3.9 and 3.10.15. Doesn't crash on 3.7, 3.8, 3.11 or 3.12.

CPython versions tested on:

3.8, 3.9

Operating systems tested on:

Linux

Output from running 'python -VV' on the command line:

No response

[skirpichev]

Hmm, I can only guess what's TEST and EXPECT_FALSE means. I did test on 3.10.14+(e0264a6) with such method on non-debug build:

static PyObject*
bug(PyObject *o, PyObject *Py_UNUSED(ignored))
{
    PyObject* non_text_obj = PyLong_FromLong(42);
    printf("expect false: %d\n", PyByteArray_Check(non_text_obj));
    Py_DecRef(non_text_obj);
    Py_RETURN_NONE;
}

It doesn't crash.

Is this crashed for you on debug builds?

[ZeroIntensity]

Unfortunately, 3.10 is security-only, so there's nothing we can do about this (especially if it doesn't crash on 3.12+)

[juanjux]

Hi @skirpichev and @ZeroIntensity.

Yep, this doesn't crash on 3.11 or 3.12 (or 3.7 or 3.8 either) so if it's security-only already we can close this one.

More info:

• Looks like the crash is because we didn't call Py_Initialize() / Py_Finalize() in this test. Now the odd part is that the other versions work without it...
• The TEST and EXPECT_FALSE stuff is from googletest, which we use here to test native code.
• I did a micro extension module with this function and it doesn't crash either this way.

PyType_IsSubtype () from /usr/local/lib/libpython3.10.so.1.0

_PyObject_TypeCheck (ob=0x7efe660920b0, type=0x7efe66a1a100 <PyByteArray_Type>)
    at /home/user/.pyenv/versions/3.10.5/include/python3.10/object.h:247

[ZeroIntensity]

• Looks like the crash is because we didn't call Py_Initialize() / Py_Finalize() in this test. Now the odd part is that the other versions work without it...

Ah yeah, that will do it. It really depends on what PyObject_TypeCheck is doing. It might not need the interpreter or thread state on all other versions, but it could possibly need it on 3.10 -- calling anything without Py_Initialize first is not something we officially support.