Expose a C-API function to allow custom importers to create a module using an init function

[itamaro]

Feature or enhancement

Proposal:

See full details and discussion in this Discourse thread.

To summarize:

Custom importers may want to create a module object by directly calling the init function (e.g. because the module is a statically linked extension module, and inittab is not desired (i.e. due to performance or other considerations)).
This is currently not possible in all scenarios (i.e. when importing pybind11 submodules), because custom importers can't access the "package context" (_PyRuntime.imports.pkgcontext), or use the internal CPython function to swap package context (_PyImport_SwapPackageContext).

From the Discourse discussion it seems the preferred solution is to introduce a C-API function, PyImport_ImportByInitFunc(const char *fullname, PyObject* (*initfunc)(void)) (exact signature up for bikeshedding, function "stability" should be discussed further).

Proposed implementation:

PyObject*
PyImport_ImportByInitFunc(const char* fullname, PyObject* (*initfunc)(void))
{
    const char *oldcontext;
    PyObject* mod;
    oldcontext = _PyImport_SwapPackageContext(fullname);
    mod = _PyImport_InitFunc_TrampolineCall((PyModInitFunction)initfunc);
    _PyImport_SwapPackageContext(oldcontext);
    return mod;
}

Has this already been discussed elsewhere?

I have already discussed this feature proposal on Discourse

Links to previous discussion of this feature:

https://discuss.python.org/t/c-api-for-initializing-statically-linked-extension-modules/43396/32

Linked PRs

• gh-116146: Add C-API to create module from spec and initfunc #139196

[gvanrossum]

I don't know if I'm speaking out of turn here, but this API looks fine to me, except that it appears there is no (longer a) function named _PyImport_InitFunc_TrampolineCall -- the code in main just calls initfunc, presumably because our WASI support has improved. It certainly looks better than adding (again) a public API for _PyImport_SwapPackageContext.

Also, there may be constituencies that might have a problem with this API, since it seems the modules thus created don't participate in the two-phase init protocol (PEP 489), and hence won't work smoothly in an environment using multiple sub-interpreters. For this I'll call out @ericsnowcurrently and @encukou.

[encukou]

The multi-phase init protocol needs a ModuleSpec object. In your use case, do you have that handy? (Single-phase modules happen to only need spec's name attribute, so that's passed via PackageContext; the new API uses an object with extra info.)
If PyImport_ImportByInitFunc can take a spec, then the new function should check if mod is a PyModuleDef_Type rather than a module, and if so it should call PyModule_FromDefAndSpec and PyModule_ExecDef.

[NicolasT]

For a prototype I'm working on, related to what (e.g.) pybind11 does, this would come in very handy: being able to reuse most of the existing extension loading support in the interpreter, without being tied to individual shared libraries and PyInit_modname symbols, but passing such function in, would be great. Multi-phase init support would be a big plus as well.

One question though: would it make sense to extend the API to something along the lines of

PyObject* PyImport_ImportByInitFunc(const char* fullname, PyObject* (*initfunc)(void *arg), void *arg)

so a loader calling this function can pass some information to this init function? I guess in C++ one could use a lambda to close over some data, then pass that in as initfunc (not sure, not a C++ expert), but in plain C that's quite a bit more difficult. This would allow for initfunc to be more dynamic and be (re)used to load multiple modules.

[itamaro]

I drafted gh-139196 as the proposed C-API, PyImport_CreateBuiltinFromSpecAndInitfunc, that would enable my use-case.
Before I work on tests and docs and news etc, I'd like to get more feedback about the API itself.
For development purposes, I did test it using this demo repo: https://github.com/itamaro/python-static-importer-demo
I also patched our internal 3.14 using this patch and it seems to work.

@encukou wdyt? Since my use-case indeed has a spec already, and what is semantically wants to do is "create a builtin, just like inittab allows, but without using inittab", I thought it would make most sense to make sure it uses the same code path that inittab would use, i.e., go through create_builtin.

@NicolasT sorry I'm not sure I follow what you're suggesting. We don't change the signature of the initfunc (that would break the Stable ABI), so I don't see how we could add arbitrary args.
While I don't fully understand your scenario, perhaps it could be supported without additional changes? The new API is not providing new Finder/Loader, it only exposes a C-API that can be leveraged by custom Finder/Loader (see the demo repo I linked above for a fully working example), where you could do whatever you want.