Add template instances via config file instead of headers

[lczech]

Hey Sergey,

yet another suggestion - hope I'm not spamming you too much here. Just trying to get Binder to work nicely for my large code base.

Is there an elegant way to instantiate class and function templates that are not specifically requested anywhere in my code base? My code, being a library, does not actually use instances of all template classes and functions that I'd like to have bindings for.

My current workaround is to have a header that lists those instances, and include that header in the all_includes files. As far as I am aware, that is the recommended way? That header then however becomes a mixture of including my actual library headers, and some custom additions, which makes it harder to, e.g., autogenerate it, adding maintenance burden.

Would it instead be possible to add a config option for that, such as

+template my_function_template int

which would generate a binding for my_function_template<int>, or something like that?

Ideally, for classes, this would also support the renaming issue that I just opened as well.

Thanks and so long
Lucas

[lyskov]

This is known issue for which currently there is no good solution because LLVM does not provide a good way to instantiate a class "manually" (at least I am not aware of it, - if someone contribute PR that will make it work it will be welcome!).

However there is a few workarounds. I would recommend (and that what I use) the following approach:
-- during Binder run specify extra C++ define via command line (say PY_MYPROJECT)
-- later when compiling resulted code also specify the same define
-- in your C++ code in header files with template create a sections like:

#ifdef PY_MYPROJECT
<explicitly instantiate template function/classes with desired types>
#endif

-- that way you will create specific version of your template classes/functions that [a] located in relevant files and [b] that does not interfere with your main C++ code.

[lczech]

Thanks, I see, yeah, I vaguely remember having seen in some other issue that this is due to the way LLVM needs to parse the code - that seems tricky to change indeed.

Maybe I'm overdoing it with the "purity" of my code base here, but I don't like to add Python-related code into my C++... So, I think I will stick with my approach of adding custom extra headers at the stage of the all_includes file instead. Not ideal, but not too bad either.

Thanks again!
Lucas

[lczech]

Hm, opening this again, as I have some more thoughts and questions:

I just noticed the --bind-class-template-specialization option, but I do not understand what it does. The option help says "attempt to bind class template specilizations", and I'm not sure what attempt means in this context - could it fail somehow?! Looking at where it is used, I am also not understanding when this would be triggered. A test with my class template instances shows that they get bindings, independently of this option. So what exactly does that do?

Furthermore, I just found an alternative solution for my use case of keeping my C++ library separate from the Python-bindings-related code: Instead of just providing binder with one all_includes header, I'm giving it two: One of my library, and one custom header that just contains the template specializations that I want bindings for. Is that a good idea, or is there a reason why this might not be what I need?

Cheers and thanks
Lucas

[lczech]

Oh okay, it seems that my approach of providing Binder with two all_include headers does not work. Both get processed, but the second overwrites the module of the first... So that does not work at all then, at least currently. I tried to figure out where in the Binder options in the main these files get processed, and if they could just be concatenated internally instead. Looks like BinderFrontendAction might be the place. But wanted to hear your opinion on that first: What's the reasoning behind the current setup, where Binder can take multiple input files, but then has them overwrite each other?

Furthermore, using a second header that just instantiates templates seems to break the prefix matching that I am using to get the generated bindings code separated into directories that roughly correspond to my original library directory structure. Not the worst, but not nice either... I'll see what I can do about that.

[lyskov]

Hm, opening this again, as I have some more thoughts and questions:

I just noticed the --bind-class-template-specialization option, but I do not understand what it does. The option help says "attempt to bind class template specilizations", and I'm not sure what attempt means in this context - could it fail somehow?! Looking at where it is used, I am also not understanding when this would be triggered. A test with my class template instances shows that they get bindings, independently of this option. So what exactly does that do?

Furthermore, I just found an alternative solution for my use case of keeping my C++ library separate from the Python-bindings-related code: Instead of just providing binder with one all_includes header, I'm giving it two: One of my library, and one custom header that just contains the template specializations that I want bindings for. Is that a good idea, or is there a reason why this might not be what I need?

See the related PR (commit 156eef0) for a test diff that illustrates the mechanism. In short:

• Attempts to generate bindings for template specializations can fail if LLVM has not fully instantiated those templates.
• Prior to this PR, Binder never attempted to bind template specializations automatically.
• With this PR, Binder can attempt such bindings automatically, but you can disable that behavior.

If you want to bind only explicitly instantiated classes (and avoid automatic specialization binding), set the option to false. One reason to disable automatic binding is to reduce binary size and memory footprint.

Hope this helps,