A weak_ptr safe type_cast.

[andre-caldas]

I make an extensive use of std::weak_ptr. Thanks to nanobind's excellent documentation, I could soon figure out why my weak_ptrs where becoming invalid prematurely.

It turns out that when nanobind creates a std::shared_ptr for some object it owns, it creates a brand new control block. This makes the weak_ptr a virtually useless thing. There are lots of information on problems with enable_shared_from_this, but of course, the problem does not reside in enable_shared_from_this. The problem is the weak_ptr.

I do understand very well the design decision behind std::shared_ptr's type_cast. This post is a mixture of a "feature request" and a request for advice. As a side effect, I suppose the feature would make all the unsolved problems with enable_shared_from_this, vanish.

As far as I understood, in order to have a single control block, as expected by weak_ptr, I would have to:

1. Provide a custom __new__ to always allocate the objects using shared_ptr, as described here:
   https://nanobind.readthedocs.io/en/latest/classes.html#custom-new
   This custom __new__ would have to save a copy of the shared_ptr, probably using some keep_alive trick, just like the current shared_ptr type caster does in its from_cpp.

2. Provide a shared_ptr type caster that would return a copy of this shared_ptr in its from_python, pretty much as the current type caster does when the class derives from enable_shared_from_this.

So, as a feature request, I would love to have all this automatically provided by nanobind through some special rv_policy::shared_ptr. I do understand if this is denied, though, because I remember reading something about this sort of things being the driving motive for nanobind's existence, in the first place.

As a request for advice, I would love to hear from any interested party, what is your advice for someone in my situation.

[andre-caldas]

So, if I add a .def(nb::new_(...)...) using a method that returns a shared_ptr, how can I access this shared_ptr? When type casting from_python, for example, can I access this shared_ptr and return a copy of it and avoid using the "reference count trick"?

[wjakob]

I'm not sure if this answers your question, but one thing you can do is simply not to use the STL type casters at all. Instead, you can create bindings for shared_ptr<MyType>, weak_ptr<MyType> (or use NB_MAKE_OPAQUE by only binding some of them). This removes some of the convenience of type casting but might allow you to track ownership with only one control block.
With regards to adding extra return value policies or other changes to nanobind to enable this somewhat specific use case, I would say: it's out of scope for this project, and you'd have to maintain your own fork with such changes.