Exposing FFI structs for use by Python runtime managed by non-Rust application

[tychedelia]

Hi!

I'm writing Rust bindings to a visual programming environment called TouchDesigner, which allows users to load plugin DLLs written in C++ (and thus my project in Rust). As part of this plugin API, it's possible to attach a series of PyMethodDef and PyGetSetDef that will be called by a runtime managed by TouchDesigner in the host application.

You can see a brief example of what I'm talking about here: https://twitter.com/tychedelia/status/1715057981690188027.

This works incredibly, and is a hugeee ergonomic improvement over the way TouchDesigner expects you to build this python enabled functionality, using the raw python C api. However, in order to get this to work, I had to fork pyo3 and expose some functionality contained in the no-warranty-implies impl__ module, namely to make https://github.com/PyO3/pyo3/blob/main/src/impl_/pymethods.rs#L173 public. Additional work is also required to create similar functionality for PyGetSetDef.

My question is: is there a better way to do this or other prior art I can see? Or, alternatively, would the maintainers accept some kind of design to enable this functionality. I understand that depending on the internals of proc macro output is not something any library wants to support, but I would love to be able to use pyo3 without hacking into the internals!

Any advice is appreciated and thanks so much!

[tychedelia]

I think it would require a pretty substantial patch on my fork to be able to support the API I want for users, so I'm to pursue this with hand rolling via pyo3_ffi first.