[OTHER]: Allow modules to be statically linked

[JeWe37]

Please don't use.

As (linux) python modules are shared objects, they typically link to libc. For compatibility, they thus need to be built on a system which uses a libc which is older than the user system, as (g)libc is only forwards compatible. Typically this is addressed by standardizing on one of the manylinux variants as outlined in PEP 513, PEP 600 etc., however this requires that the project can be practically built on a system that old. This can often be challenging.

To address this, libc could be linked statically into the shared library, allowing the resulting shared object to be loaded on any linux system, even if built on a modern distribution, vastly reducing build complexity.

However, statically linking glibc is widely regarded as extremely poorly supported, and in fact distros such as Debian have not even shipped the required PIC variant of libc for years, that would be required for this. Yet, as statically linking libc means that which libc is used is irrelevant, so instead building a project on a modern musl libc based distro(e.g., alpine), will still allow it to be loaded even on glibc based distros.

Long story short, on any musl based system, the following CMake will generate a Python module that can be loaded regardless of distribution:

set(OLD_CMAKE_FIND_LIBRARY_SUFFIXES ${CMAKE_FIND_LIBRARY_SUFFIXES})
set(CMAKE_FIND_LIBRARY_SUFFIXES ".a") # Force finding a static library
find_library(
    LIBC_LIB
    NAMES c
)
set(CMAKE_FIND_LIBRARY_SUFFIXES ${OLD_CMAKE_FIND_LIBRARY_SUFFIXES})

target_link_libraries(my_mod PRIVATE  -nolibc -static-libgcc -static-libstdc++ ${LIBC_LIB})

This will break if any other shared objects are linked into the module, but that can usually be avoided comparatively easily(and is preferable for redistribution anyway), so long as the complexity of libc is taken care of.

I am wondering if there would be interest in integrating this feature into nanobind. The above code can be used to allow the user to configure this, but in so far as I can tell nobody has ever proposed this approach to dealing with libc compatibility for python modules, meaning figuring this out was complex to say the least, but is currently working for my module.

I suspect this would make nanobind the only binder(for any language) that has this feature, at least I have never seen anyone mention it.

[wjakob]

@JeWe37 That's an interesting approach. The restriction to a musl based system seems somewhat strict. Can such a build also be done on a glibc-based OS?

By the way, this seems like more of a discussion than in issue. Will convert.

[JeWe37]

(Ah, I should have noticed that this project actually uses discussions)

I tried, but the general consensus that static linking with glibc is a bad idea, aligns with my experience. Just using the above code will try to compile, but cause linker errors that I have found to be essentially impossible to resolve. That aside, before you can even try that, the libc PIC variant not being shipped on most distros means that to use this, one would have to build their own glibc, extract the libc_pic.a and link with that. Essentially wholly impractical given that a lot of CI already runs on Alpine anyway.

As for Alpine specifically, one minor note that also seems to be going wrong is that the one seems to also need a different variant of the endcode, specifically crtendS.o instead of crtendT.o(which has a hacky workaround, see e.g. https://github.com/lamdera/compiler/blob/685a6d0524ade25b9ad53d7ef7946f15120edafd/Dockerfile#L39-L42), so even there static linking shared libraries is a little challenging.

[hpkfft]

I agree that static linking with glibc is a bad idea. The libc library matches the kernel, so these are installed together on a system.
Delivering a binary (e.g., executable, shared library for python to load) that is statically linked with libc means the binary has its own copy of the libc functions that it uses, which are not necessarily correct for the Linux kernel on the deployment machine.
I found this worth reading: https://blogs.oracle.com/solaris/post/static-linking-where-did-it-go

The GNU C Library, libc.so.6, has symbol versioning. When a binary is linked, the symbol version is recorded, so at runtime the correct function is called. If the GNU C Library changes a function, the symbol gets a new version. The old function remains in the C library (with its original version identifier) so that at runtime a binary originally linked against an old libc will get the old function at runtime even if the deployment machine has installed a newer C library.

For example, using nm -DC myextension.cpython-311-x86_64-linux-gnu.so I see the following:

    ...
    U malloc@GLIBC_2.2.5
    U free@GLIBC_2.2.5
    U dlopen@GLIBC_2.34
...

So, my extension uses malloc and free as the function was designed in GNU C Library verion 2.2.5. If some incompatible update is made to these in some hypothetical version 2.99, then myextension still works on deployment machines having libc version 2.99 installed. If the Linux kernel makes changes to its system calls, that's fine. I can be sure that the libc on the system has adapted to whatever the Linux kernel requires. That's its job.

Note that dlopen() and related functions are only present in version 2.34. (Prior to that, they were in a separate library, libdl.so.2) If I try to run myextension on a machine having a version of libc older that 2.34, it will fail to load.

If you're willing to give up recording symbol versions at link time, I wonder if you can simply not link with libc at all. The python3 binary is linked with libc. So, maybe your extension will find the symbols it needs from whatever is available in the libc that the python3 executable itself has loaded. It's still dangerous--you don't have versioned symbols in myextension--but maybe that's a risk your willing to take if you only use basic, very stable libc functions. I do not know if this works or if there are other problems; I'm just brainstorming ideas.....

[JeWe37]

Well, the thing is, Linux(presumably unlike Solaris going by that old article, but I can't claim to know anything about Solaris), operates based on "don't break user space", so by all accounts the issue outlined in the article should be irrelevant. The issue with static linking libc is static linking glibc specifically, its design(the fact it loads other shared objects for NSS and similar which the article mentions) is inherently hostile to static linking. Those issues don't exist with musl, it's one of its main selling points.

Static linking also completely eliminated any issues with incompatible libc versions, because the system libc isn't referenced at all and just shipped with your binary. So in that sense, "giving up symbol versions" isn't a real problem.

You definitely shouldn't link to other shared objects though as the article points out, that is bound to to create trouble and likely reintroduce the libc dependency.

[hpkfft]

You have a good point about the kernel (now) being stable regarding user space system calls. I see that musl is intended to comply with "POSIX 2008 / Single Unix Standard Version 4", and the Solaris post I mentioned is from way back in 2004. Also, I see "Linux 2.6.39 or later is believed to be sufficient", and that was released in 2011.

But, well, I was working at Sun Microsystems in 2004, and my young, impressionable brain absorbed the fact that "libc is supposed to isolate an application from the user/kernel boundary, a boundary which can undergo changes from release to release." It doesn't seem so long ago. I still have the free Solaris t-shirts they gave out....

[wjakob]

One point may be relevant to mention: the combination of nanobind and cibuildwheel already supports wheel builds where a musllinux image is used in place of manylinux. The traditional application is to make wheels for Alpine Linux-based containers. But based on what you say, it seems that one should be able to re-tag the musllinux wheels as manylinux to benefit from static linking without running into compatibility issues.

Though I guess the main point is actually that the version of GCC or Clang might be newer, rather than static linking actually being the beneficial aspect.

[JeWe37]

I would agree newer clang and GCC are the primary advantage, but having more modern system packages can also be helpful for dependencies you would rather not compile from source every time. And those often are even harder to find for older distros than the two popular compilers.

Personally I've used this with the stable ABI, and I'm admittedly a bit unclear on details of the naming scheme for Python modules, but the .abi3.so extension used for the stable ABI already makes no reference to the libc version anyway.

[hpkfft]

But, aren't we talking about building a nanobind extension that will be dynamically loaded using dlopen() by a python executable?
The python executable will have dynamically loaded libc, right?
That libc will be the GNU C Library, libc.so.6, except on an actual musllinux system, right?

[hpkfft]

-fuse-cxa-atexit

[JeWe37]

Doesn't seem to solve it. Upon closer inspection, running nm on some normal binary with dynamically linked libc we see that:

0000000000b4c040 t atexit

while with and without -fuse-cxa-atexit, the static shared objects from alpine have

                 U atexit

which lends some credibility to my suspicion that indeed this is an incompatibility that I at least can't seem to find a compiler or linker flag to fix easily.

[hpkfft]

https://stackoverflow.com/questions/42912038/what-is-the-difference-between-cxa-atexit-and-atexit

On a musl system you want to see U __cxa_atexit. It seems like the compiler flag (not linker flag, if I understand correctly) ought to tell the compiler to emit calls to __cxa_atexit(). Sorry, I can't help further.

BTW, this illustrates how bad it is to mix and match. What would happen if a statically linked extension is calling a statically-supplied atexit and python itself (and other extensions) are calling __cxa_atexit?

[hpkfft]

Also, you might need to build libnanobind-static.a using -fuse-cxa-atexit (as well as your extension).
I think nanobind checks for memory leaks at the end....

[JeWe37]

Also, you might need to build libnanobind-static.a using -fuse-cxa-atexit (as well as your extension). I think nanobind checks for memory leaks at the end....

Sadly doesn't help. Perhaps one could figure what is actually referring to this and making it get included, but even if that is resolved, there is no telling if no other issues would show up.

BTW, this illustrates how bad it is to mix and match. What would happen if a statically linked extension is calling a statically-supplied atexit and python itself (and other extensions) are calling __cxa_atexit?

Granted, it is difficult to judge the side-effects of this, even if I don't know that atexit specifically would be a problem. One would realistically need a large set of diverse or large programs to trial this with to really verify.

[hpkfft]

C++ One Definition Rule

https://en.cppreference.com/w/cpp/language/definition

One and only one definition of every non-inline function or variable that is odr-used (see below) is required to appear in the entire program (including any standard and user-defined libraries). The compiler is not required to diagnose this violation, but the behavior of the program that violates it is undefined.

Informally,

A function is odr-used if a function call to it is made or its address is taken.

[JeWe37]

Well I suspect this operates at a different level of abstraction. ODR is basically C++-standard-speak to imply that the linker has to know what it's supposed to actually link to(and well, actually have something to link to). That isn't a problem here. After all, the .so is linked statically, so calls to symbols from libc from the .so are not resolved via the dynamic linker, but already at link time when Python's libc isn't in play at all. As for Python itself, its references to libc should be resolved when the Python process starts by ld, so by the time the module is loaded I don't think this matters anymore.

[hpkfft]

The larger point is that it's not a good idea to rely on the implementation details of the tool chain. Undefined behavior may actually do what you'd like, until it doesn't.

[JeWe37]

There is undefined behavior like that in C++, sure, things like why you shouldn't reinterpret cast incompatible types. But I don't think that's quite the right angle to look at this from. This doesn't affect the actual code after all. The standard says nothing of how the code is supposed to be compiled to an actual binary. In fact a C++ interpreter would be entirely compliant as well. That's what I mean when I say this operates at a different level of abstraction. This is simply out-of-scope of the standard.