[BUG]: `__new__` does not initialize STL containers and resulting in undefined behavior

[XuehaiPan]

Required prerequisites

• Make sure you've read the documentation. Your issue may be addressed there.
• Search the issue tracker and Discussions to verify that this hasn't already been reported. +1 or comment there if it has.
• Consider asking first in the Gitter chat room or in a Discussion.

What version (or hash if on master) of pybind11 are you using?

2.10.3

Problem description

I'm writing a C++ class with some STL-like containers as its members. Then I bind it with pybind11 and expose it to Python. However, the bind C++ class cannot behave like a normal Python class as expected.

In [1]: !pip3 install optree 
   ...: from optree import PyTreeSpec

In [2]: PyTreeSpec
Out[2]: <class 'optree.PyTreeSpec'>

In [3]: PyTreeSpec.mro()
Out[3]: [<class 'optree.PyTreeSpec'>, <class 'pybind11_builtins.pybind11_object'>, <class 'object'>]

In [4]: PyTreeSpec()  # an error raised as expected
TypeError: optree.PyTreeSpec: No constructor defined!

In [5]: spec = PyTreeSpec.__new__(PyTreeSpec)  # expect to raise an error

In [6]: repr(spec)  # segfault due to invalid memory access
[1]    31095 segmentation fault  ipython3

All bind types created by py::class_<CppClass> are inherit from pybind11_builtins.pybind11_object. As the comment says:

pybind11/include/pybind11/detail/class.h

Lines 346 to 370 in 3cc7e42

	/// Instance creation function for all pybind11 types. It allocates the internal instance layout
	/// for holding C++ objects and holders. Allocation is done lazily (the first time the instance is
	/// cast to a reference or pointer), and initialization is done by an `__init__` function.
	inline PyObject *make_new_instance(PyTypeObject *type) {
	#if defined(PYPY_VERSION)
	// PyPy gets tp_basicsize wrong (issue 2482) under multiple inheritance when the first
	// inherited object is a plain Python type (i.e. not derived from an extension type). Fix it.
	ssize_t instance_size = static_cast<ssize_t>(sizeof(instance));
	if (type->tp_basicsize < instance_size) {
	type->tp_basicsize = instance_size;
	}
	#endif
	PyObject *self = type->tp_alloc(type, 0);
	auto *inst = reinterpret_cast<instance *>(self);
	// Allocate the value/holder internals:
	inst->allocate_layout();
	return self;
	}
	/// Instance creation function for all pybind11 types. It only allocates space for the
	/// C++ object, but doesn't call the constructor -- an `__init__` function must do that.
	extern "C" inline PyObject *pybind11_object_new(PyTypeObject *type, PyObject *, PyObject *) {
	return make_new_instance(type);
	}

pybind11_object_new only allocates space for the C++ object, but doesn't call the constructor. That means if someone calls BoundCppClass.__new__(BoundCppClass), they will get undefined results since the C++ object is not initialized at all. Also, default values in the C++ class definition are not used even if there is a default constructor.

Reproducible example code

1. Clone https://github.com/pybind/cmake_example:

git clone https://github.com/pybind/cmake_example.git
cd cmake_example

2. Paste the following content to src/main.cpp:

#include <string>
#include <sstream>
#include <vector>
#include <pybind11/pybind11.h>

namespace py = pybind11;

using ssize_t = py::ssize_t;

class MyList {
private:
    std::vector<int> data = {0, 1, 2, 3};

public:
    MyList() = default;
    ssize_t size() const { return data.size(); }
    std::string repr() const {
        std::ostringstream os;
        os << "MyList([";
        for (int i = 0; i < data.size(); ++i) {
            if (i != 0) {
                os << ", ";
            }
            os << data[i];
        }
        os << "], size=" << size() << ")";
        return os.str();
    }
};

PYBIND11_MODULE(cmake_example, m) {
    auto cls = py::class_<MyList>(m, "MyList");
    cls.def(py::init<>());
    cls.def("size", &MyList::size);
    cls.def("__repr__", &MyList::repr);
}

3. Create a new virtual environment and install:

python3 -m venv venv
source venv/bin/activate
pip3 install -U pip setuptools ipython
pip3 install -e .

4. Run the following code in ipython:

In [1]: from cmake_example import MyList

In [2]: l = MyList()

In [3]: l.size()
Out[3]: 4

In [4]: l  # calls repr()
Out[4]: MyList([0, 1, 2, 3], size=4)

In [5]: l = MyList.__new__(MyList)

In [6]: l.size()
Out[6]: -23599346664417

In [7]: l  # calls repr()
[1]    31601 segmentation fault  ipython3

In [1]: from cmake_example import MyList

In [2]: class MyAnotherList(MyList):
   ...:     def __new__(cls):
   ...:         inst = super().__new__(cls)
   ...:         inst.default_size = inst.size()  # default size from the C++ default constructor
   ...:         return inst
   ...:         

In [3]: l = MyAnotherList()

In [4]: l  # calls repr()
Out[4]: MyList([0, 1, 2, 3], size=4)

In [5]: l.size()
Out[5]: 4

In [6]: l.default_size  # expect 4
Out[6]: -23607549586738

In [7]: MyAnotherList().default_size  # undefined
Out[7]: -5769946945

In [8]: MyAnotherList().default_size  # undefined
Out[8]: -5769947144

In [9]: MyAnotherList().default_size  # undefined
Out[9]: -23639255465217

Is this a regression? Put the last known working version here if it is.

Not a regression

[Skylion007]

You can override the behavior for new on the C++ side though (we have some tests to demonstrate this). See work #1693 . Then you can have the C++ class call any new method you want, see

pybind11/tests/test_class.cpp

Line 89 in 442261d

.def_static("__new__",

[XuehaiPan]

You can override the behavior for new on the C++ side though (we have some tests to demonstrate this). See work #1693 . Then you can have the C++ class call any new method you want, see

pybind11/tests/test_class.cpp

Line 89 in 442261d

.def_static("__new__",

@Skylion007 Thanks for the example. This works for me:

PyTreeSpecTypeObject
    .def_static(
        "__new__",
        [](const py::type& cls) {
            throw py::type_error(cls.attr("__module__").cast<std::string>() + "." +
                                 cls.attr("__name__").cast<std::string>() +
                                 " cannot be instantiated");
        },
        py::arg("cls"))
    .def(py::init<>())  // no-op

Now it works as expected (raise a TypeError while calling __new__ and do nothing in __init__):

In [1]: !pip3 install -e .
   ...: from optree import PyTreeSpec

In [2]: PyTreeSpec
Out[2]: <class 'optree.PyTreeSpec'>

In [3]: PyTreeSpec.mro()
Out[3]: [<class 'optree.PyTreeSpec'>, <class 'pybind11_builtins.pybind11_object'>, <class 'object'>]

In [4]: PyTreeSpec()  # an error raised as expected
TypeError: optree.PyTreeSpec cannot be instantiated

In [5]: PyTreeSpec.__new__(PyTreeSpec)  # an error raised as expected
TypeError: optree.PyTreeSpec cannot be instantiated

In [6]: spec = tree_structure({'a': 1, 'b': (2, 3)})

In [7]: spec
Out[7]: PyTreeSpec({'a': *, 'b': (*, *)})

In [8]: spec.__new__(PyTreeSpec)  # an error raised as expected
TypeError: optree.PyTreeSpec cannot be instantiated

In [9]: spec.__init__()  # noop

---

Minor bug found in the test case:

pybind11/tests/test_class.cpp

Lines 87 to 90 in 442261d

	py::class_<NoConstructorNew>(m, "NoConstructorNew")
	.def(py::init([](const NoConstructorNew &self) { return self; })) // Need a NOOP __init__
	.def_static("__new__",
	[](const py::object &) { return NoConstructorNew::new_instance(); });

Should be:

-.def(py::init([](const NoConstructorNew &self) { return self; })) // Need a NOOP __init__
+.def(py::init<>()) // Need a NOOP __init__

otherwise, you will get:

>>> print(NoConstructorNew.__init__.__doc__)
__init__(self: NoConstructorNew, arg0: NoConstructorNew) -> None

[XuehaiPan]

This works for me:

PyTreeSpecTypeObject
    .def_static(
        "__new__",
        [](const py::type& cls) {
            throw py::type_error(cls.attr("__module__").cast<std::string>() + "." +
                                 cls.attr("__name__").cast<std::string>() +
                                 " cannot be instantiated");
        },
        py::arg("cls"))
    .def(py::init<>())  // no-op

Now it works as expected (raise a TypeError while calling __new__ and do nothing in __init__):

In [1]: !pip3 install -e .
   ...: from optree import PyTreeSpec

In [2]: PyTreeSpec
Out[2]: <class 'optree.PyTreeSpec'>

In [3]: PyTreeSpec.mro()
Out[3]: [<class 'optree.PyTreeSpec'>, <class 'pybind11_builtins.pybind11_object'>, <class 'object'>]

In [4]: PyTreeSpec()  # an error raised as expected
TypeError: optree.PyTreeSpec cannot be instantiated

In [5]: PyTreeSpec.__new__(PyTreeSpec)  # an error raised as expected
TypeError: optree.PyTreeSpec cannot be instantiated

In [6]: spec = tree_structure({'a': 1, 'b': (2, 3)})

In [7]: spec
Out[7]: PyTreeSpec({'a': *, 'b': (*, *)})

In [8]: spec.__new__(PyTreeSpec)  # an error raised as expected
TypeError: optree.PyTreeSpec cannot be instantiated

In [9]: spec.__init__()  # noop

Reopened for found another issue. If I raise TypeError in __new__ to forbid instantiation, then the serialization support will be broken (__new__ + __setstate__).

In [1]: from optree import PyTreeSpec, tree_structure

In [2]: spec = tree_structure({'a': 1, 'b': (2, 3)})

In [3]: import pickle as pkl

In [4]: pkl.dumps(spec)
Out[4]: b'\x80\x04\x95w\x00\x00\x00\x00\x00\x00\x00\x8c\x06optree\x94\x8c\nPyTreeSpec\x94\x93\x94)\x81\x94((K\x01K\x00NNNK\x01K\x01t\x94(K\x01K\x00NNNK\x01K\x01t\x94(K\x01K\x00NNNK\x01K\x01t\x94(K\x03K\x02NNNK\x02K\x03t\x94(K\x05K\x02]\x94(\x8c\x01a\x94\x8c\x01b\x94eNNK\x03K\x05t\x94t\x94\x89\x8c\x00\x94\x87\x94b.'

In [5]: pkl.loads(pkl.dumps(spec))
TypeError: optree.PyTreeSpec cannot be instantiated

Ref:

• Python docs: obj.__setstate__(state)

• CPython: static PyObject *instantiate(PyObject *cls, PyObject *args)

• Pybind11 docs: Classes - Pickling support

In the Pybind11 pickling example, a new instance is returned in __setstate__ rather than updating the current object state. This means that even if the class can be instantiated, obj1.__setstate__(obj2.__getstate__()) will have no effect on obj1.

Python pickle module uses:

inst = cls.__new__()
inst.__setstate__(state)

The return value for __setstate__ is ignored:

https://github.com/python/cpython/blob/65fb7c4055f280caaa970939d16dd947e6df8a8d/Modules/_pickle.c#L6630-L6644

[Skylion007]

@XuehaiPan That's an oddly specific usecase. you have full access to the object state and you could pass it in the lambda you are using for the __setstate__ function and return a reference to this, right? Or if that still doesn't work, you could just define __setstate__ yourself, although it would be really brittle due to C++ only data that might be needed:

pybind11/include/pybind11/detail/init.h

Line 375 in 4768a6f

void setstate(value_and_holder &v_h, std::pair<T, O> &&result, bool need_alias) {

[XuehaiPan]

• Python docs: obj.__setstate__(state)

• CPython: static PyObject *instantiate(PyObject *cls, PyObject *args)

• Pybind11 docs: Classes - Pickling support

In the Pybind11 pickling example, a new instance is returned in __setstate__ rather than updating the current object state. This means that even if the class can be instantiated, obj1.__setstate__(obj2.__getstate__()) will have no effect on obj1.

Python pickle module uses:

inst = cls.__new__()
inst.__setstate__(state)

The return value for __setstate__ is ignored:

https://github.com/python/cpython/blob/65fb7c4055f280caaa970939d16dd947e6df8a8d/Modules/_pickle.c#L6630-L6644

Finding

• If the instance is allocated but hasn't been initialized yet, then obj.__setstate__(state) will update the instance's internal state.
• If the instance has been initialized or is created by implicit type casting, then obj.__setstate__(state) will take no effect.

---

I investigate Pybind11's setstate implementation. It does update the instance state by setting the value_and_holder.value_ptr().

pybind11/include/pybind11/detail/init.h

Lines 117 to 142 in 442261d

	template <typename Class>
	void construct(value_and_holder &v_h, Cpp<Class> *ptr, bool need_alias) {
	PYBIND11_WORKAROUND_INCORRECT_MSVC_C4100(need_alias);
	no_nullptr(ptr);
	if (Class::has_alias && need_alias && !is_alias<Class>(ptr)) {
	// We're going to try to construct an alias by moving the cpp type. Whether or not
	// that succeeds, we still need to destroy the original cpp pointer (either the
	// moved away leftover, if the alias construction works, or the value itself if we
	// throw an error), but we can't just call `delete ptr`: it might have a special
	// deleter, or might be shared_from_this. So we construct a holder around it as if
	// it was a normal instance, then steal the holder away into a local variable; thus
	// the holder and destruction happens when we leave the C++ scope, and the holder
	// class gets to handle the destruction however it likes.
	v_h.value_ptr() = ptr;
	v_h.set_instance_registered(true); // To prevent init_instance from registering it
	v_h.type->init_instance(v_h.inst, nullptr); // Set up the holder
	Holder<Class> temp_holder(std::move(v_h.holder<Holder<Class>>())); // Steal the holder
	v_h.type->dealloc(v_h); // Destroys the moved-out holder remains, resets value ptr to null
	v_h.set_instance_registered(false);
	construct_alias_from_cpp<Class>(is_alias_constructible<Class>{}, v_h, std::move(*ptr));
	} else {
	// Otherwise the type isn't inherited, so we don't need an Alias
	v_h.value_ptr() = ptr;
	}
	}

But I have no idea why it has no effect on the Python side obj.__setstate__(state):

PyTreeSpecTypeObject
    .def_static(
        "__new__",
        [](const py::type& cls) {
            return std::make_unique<PyTreeSpec>();
        },
        py::arg("cls"))
    .def(py::init<>())  // no-op
    .def(py::pickle([](const PyTreeSpec& t) { return t.ToPicklable(); },
                    // Return a new instance, `setstate` will update `value_and_holder.value_ptr()`
                    [](const py::object& o) { return PyTreeSpec::FromPicklable(o); })

obj.__setstate__(other.__getstate__())
obj  # unchanged

I found:

• If the instance is allocated but hasn't been initialized yet, then obj.__setstate__(state) will update the instance's internal state.
• If the instance has been initialized or is created by implicit type casting, then obj.__setstate__(state) will take no effect.

I tried manually calling vvalue_and_holder.type->init_instance. And I did not use .def_static("__new__", ...) (implicit cast) but set tp_new instead.

auto PyTreeSpecTypeObject =
    py::class_<PyTreeSpec>(mod, "PyTreeSpec", "Representing the structure of the pytree.");
auto PyTreeSpec_Type = *(reinterpret_cast<PyTypeObject*>(PyTreeSpecTypeObject.ptr()));

auto pybind11_object_new = PyTreeSpec_Type.tp_new;
auto new_func = [&pybind11_object_new](
                    PyTypeObject* type, PyObject* args, PyObject* kwargs) -> PyObject* {
    auto* self = pybind11_object_new(type, args, kwargs);
    auto* inst = reinterpret_cast<py::detail::instance*>(self);
    auto v_h = inst->get_value_and_holder(py::detail::get_type_info(typeid(PyTreeSpec)));
    v_h.type->init_instance(inst, nullptr);
    return self;
};
PyTreeSpec_Type.tp_new = reinterpret_cast<decltype(pybind11_object_new)>(&new_func);

Still:

• v_h.type->init_instance(inst, nullptr) does not initialize STL containers.
• __new__ + __setstate__ works as expected (because the instance is not initialized)
• obj.__setstate__(state) has no effect when obj is already initialized. (add v_h.value_ptr() = new PyTreeSpec() in tp_new)

Overall, I cannot let both "always initialize instance in __new__" and "pickling support" work as expected.

[Skylion007]

@XuehaiPan would this PR solve most of your issue? If so, we can try to revive it and get something similar integrated into pybind11: #4116

[rwgk]

In a #4621 comment @Skylion007 asked:

Would this flag help with the set_state problems in #4549?

Not sure. At first glance "probably not", but I could be wrong. I'd need a significant block of time to really understand.

We (@wangxf123456 and myself) hit on something related - I think - the other day. Ultimately I think it has to do with this:

pybind11/include/pybind11/detail/smart_holder_type_casters.h

Lines 658 to 668 in d930de0

	// have_holder() must be true or this function will fail.
	void throw_if_uninitialized_or_disowned_holder() const {
	if (!holder().is_populated) {
	pybind11_fail("Missing value for wrapped C++ type:"
	" Python instance is uninitialized.");
	}
	if (!holder().has_pointee()) {
	throw value_error("Missing value for wrapped C++ type:"
	" Python instance was disowned.");
	}
	}

That's smart_holder code. Pointing that out here because that is meant to clearly identify and cleanly handle the "not initialized" case. I'd have

1. experiment to see how that plays with the MyList.__new__(MyList) use case here.
2. and what we could do on master to prevent the UB.

Sorry, nothing concrete/actionable right now.