Problems with musl's ldso on 1.6

[jpsamaroo]

For some reason, musl's loader refuses to reuse libraries that it's already loaded to service later requests for those same libraries. For example, libbz2.so from Bzip2_jll can be successfully dlopen'd on its own, but when trying to dlopen libfreetype.so from FreeType2_jll, we can see (through strace) musl's loader looking through entries in LD_LIBRARY_PATH instead. Trying to modify JLLWrappers.jl to do a withenv("LD_LIBRARY_PATH"=>join(LIBPATH_list, ':')) do ... around the dlopen does not seem to affect the search path, but I can set LD_LIBRARY_PATH manually outside of Julia as usual (so I hypothesize that the environment is cached when ld is first loaded?).

Of course, this doesn't always happen. For me on Alpine Linux, JLLs that do not depend on Bzip2_jll (and a few other problematic JLLs) seem to work fine. @giordano hypothesizes that it's because of a difference between the SONAME and NEEDED entries between dependent libraries, but I've found that even by using patchelf to change these entries, nothing of benefit happens (when it does work, it's only because the loader instead found a matching system library). However, it's totally possible I'm doing this wrong.

If anyone has suggestions on other things to look at or try, please let me know!

[jpsamaroo]

Also likely related is #36458

[staticfloat]

Can you use readelf -d to show what the SONAME that libbz2.so advertises, and what libfreetype.so expects?

[jpsamaroo]

I can set them to match (and I think they do by default), but it doesn't appear that musl actually looks at SONAME, which explains why fiddling with SONAME doesn't really help. All it looks at is NEEDED, and whatever paths were originally passed to dlopen.

[giordano]

Building the library with the right SONAME (see the patch in Alpine) then should help because we'd open the library in the JLL package with the same name as the NEEDED one.

[jpsamaroo]

I'm doing some printf debugging with ldso and julia, and also asking around on the musl IRC. It seems like we may also be doing something "wrong", in that we do a dlopen of .../libbz2.so, whereas the NEEDED path is libbz2.so.1.0, so ldso considers them not to match.

[jpsamaroo]

Ok, so I've spoken a bit with the musl devs, and here's the gist of it:

• We don't know the exact path of dependency lib Y (libbz2) for lib X (libfreetype) during build time, only at runtime.
• We may have multiple such lib Ys that we could choose to link against lib X, and don't know which to use until runtime.
• We may first dlopen libbz2.so, and then want to reuse that exact entry (intentionally avoiding a filesystem search).
• Key point made to me: We should not be using DT_NEEDED at all; instead, strip DT_NEEDED, dlopen things in the right order, and then all the symbols will be available as we expect.

Some relevant quotes:

then you don't want the needed at all
if lib A depends on symbols from lib B1, B2, or B3 but you don't know which, then you don't put a DT_NEEDED on lib B
you just omit the DT_NEEDED and load the appropriate libB[123] before loading A

dlopen without a / in the argument means specifically to look for the library in the search path. not to use something loaded with an explicit path that just happens to have the same name

Link statically

[jpsamaroo]

The musl devs have refused my proposal to patch ldso to look at SONAME and compare it with NEEDED entries. Therefore, I have prepared a patch which adventurous users may apply to their ldso to workaround this issue.

WARNING: I do not make any claims about how safe this patch is to use in general; I recommend only applying this as your Julia interpreter, and that's it (patchelf --set-interpreter /path/to/ld-musl-*.so.1 /path/to/julia). Please let me know if you find any issues with it, and I'll update the patch!

--- ldso/dynlink.c	2021-01-14 20:26:00.000000000 -0600
+++ ldso/dynlink.c	2021-04-22 14:21:02.230108318 -0500
@@ -93,6 +93,7 @@
 	struct td_index *td_index;
 	struct dso *fini_next;
 	char *shortname;
+	char *soname;
 #if DL_FDPIC
 	unsigned char *base;
 #else
@@ -1047,6 +1048,8 @@
 		for (p=head->next; p; p=p->next) {
 			if (p->shortname && !strcmp(p->shortname, name)) {
 				return p;
+			} else if (p->soname && !strcmp(p->soname, name)) {
+				return p;
 			}
 		}
 		if (strlen(name) > NAME_MAX) return 0;
@@ -1153,6 +1156,12 @@
 		return 0;
 	}
 	memcpy(p, &temp_dso, sizeof temp_dso);
+	for (int i=0; p->dynv[i]; i+=2) {
+		if (p->dynv[i] == DT_SONAME) {
+			p->soname = p->strings + p->dynv[i+1];
+			break;
+		}
+	}
 	p->dev = st.st_dev;
 	p->ino = st.st_ino;
 	p->needed_by = needed_by;
@@ -1885,6 +1894,11 @@
 	reclaim_gaps(&app);
 	reclaim_gaps(&ldso);

+	/* Set initial soname */
+	app.soname = NULL;
+	ldso.soname = NULL;
+	vdso.soname = NULL;
+
 	/* Load preload/needed libraries, add symbols to global namespace. */
 	ldso.deps = (struct dso **)no_deps;
 	if (env_preload) load_preload(env_preload);

License is MIT, same as musl.

[staticfloat]

So if I'm understanding correctly, musl already has a fast-path for previously-loaded libraries, which is the p->shortname business.

The fast-path: https://github.com/ifduyue/musl/blob/aad50fcd791e009961621ddfbe3d4c245fd689a3/ldso/dynlink.c#L1048-L1052

Where p->shortname usually gets initialized: https://github.com/ifduyue/musl/blob/aad50fcd791e009961621ddfbe3d4c245fd689a3/ldso/dynlink.c#L1164

It seems to me that possibly, we don't get p->shortname filled in when we dlopen() because we pass in the full pathname? @jpsamaroo can you try experimenting with passing in imperfect (but still valid) pathnames like /path/to/../to/foo.so or ./foo.so and see if p->shortname is properly filled in? If that is the case, can you see if it then fast-paths and never hits the filesystem when loading things that depend on foo.so?

[staticfloat]

Pinging @richfelker, who may have a better idea on how to solve this.

Julia, as a language, makes use of the behavior on glibc Linux, macOS, FreeBSD and Windows, that if libfoo.so depends on libbar.so, and you have already dlopen("/path/to/libbar.so"), then when you dlopen("/other/path/to/libfoo.so"), if the filenames match (on glibc, if the SONAMEs match, on macOS if the dylib ID's match) then the previously-loaded libbar.so is taken to be the dependency for libfoo.so.

This allows us to distribute libraries independently (e.g. we do a nix-like thing where we install binaries into content-addressed directories, and so the paths to different libraries is not knowable at compile-time or even at julia process start-up) with some wrapper code that performs the correct dlopen() calls, in order, to satisfy the dynamic linker. We manage compatibility ranges and whatnot using the typical Julia package resolver.

The difficulty on musl is that when we dlopen("/other/path/to/libfoo.so"), it searches the filesystem for libbar.so, depsite us having loaded libbar.so previously. This appears to be because the only time the short happy path is taken on musl, is when p->shortname is initialized, and this only happens if you don't load a library by its full path. One possibility is the patch @jpsamaroo posted above, where we patch ldso to track the SONAME of objects that are loaded. Another possibility is the workaround I cooked up a few weeks ago where we have Julia manually insert the SONAME as p->shortname, but of course that is fragile and evil and I'd really like it to never see the light of day. (However, it does have the benefit of working on older musl versions).

I'm curious to hear your thoughts about the best way to achieve feature parity here. As far as I can tell, there is no way for a process to satisfy the dynamic linker when loading libraries if it doesn't know the location of all the libraries beforehand, because in musl the only way for us to influence the search locations are by embedding paths into objects (e.g. rpath/runpath, not usable here because we don't know the relative paths between dependencies at compile-time) or setting environment variables (e.g. LD_LIBRARY_PATH, not usable here because we don't know which libraries we're loading when we start up Julia, and modifying environment variables later doesn't work).

Thank you for your time!

[richfelker]

Can you explain what you're trying to achieve? The current behavior is intentional and serves to prevent modules loaded with an explicit pathname from shadowing libraries in the search path just because they happen to have the same name.

If libfoo depends on a version of libbar at a particular location (absolute or relative to libfoo), the intended way to represent that is via rpath in libfoo, using $ORIGIN base if needed. Is there a reason this does not work for what you're trying to achieve?

[staticfloat]

Can you explain what you're trying to achieve?

We ship libraries to users outside of normal distribution channels; they do not get installed into e.g. /usr/local/lib, they get installed into Julia package "depots", which can live anywhere, such as in a system-wide location (like if you're working in a computer lab, and they have a selection of Julia packages that are preinstalled, living in /opt/julia-1.6/depot) or, more typically, in your home directory (e.g. ~/.julia). Libraries are installed into content-addressed locations, and if a library is already available in one depot, it does not get installed into another. So, for example, libfoo.so may exist at ~/.julia/artifacts/<content_hash>/lib/libfoo.so, and libbar.so may exist at /opt/julia-1.6/depot/artifacts/<content_hash>/lib/libbar.so. Because we serve precompiled binaries to users (so they do not have to compile GTK, Cairo, etc....), at compilation time we cannot know the relative paths beforehand. We also can't use LD_LIBRARY_PATH because we would need to set that up before starting the Julia interpreter, and because we expect to be able to install new packages and use them immediately in a single process session, we can't know all the paths that need to be pushed onto LD_LIBRARY_PATH either.

The current behavior is intentional and serves to prevent modules loaded with an explicit pathname from shadowing libraries in the search path just because they happen to have the same name.

This makes sense, however I believe the idea behind the SONAME mechanism is to provide exactly this behavior; e.g. if a library defines an SONAME for itself, it is essentially claiming that part of the library namespace and declaring that anyone that has a DT_NEEDED entry that matches exactly should instead be served by it. If there were no SONAME entries in our libraries, I would agree that this is intended behavior.

[staticfloat]

@richfelker just checking in with you to see if you have any suggestions on how we can get musl to support this use case. To give a really high-level overview of the possible paths forward:

1. We can cook up a patch to musl that is (or even identical) to @jpsamaroo's patch above and submit it for review again. This would cause the dependency resolution at dlopen() time to pay attention to the SONAME for libraries that are loaded by path, as well as those loaded by basename. This is, of course, my preferred route, but I think we may need some more feedback from you or other developers, since the first time Julian submitted it, it was apparently rejected.

2. We can patch musl locally and then statically embed our own patched loader into the julia executable. This is undesirable to me as I can imagine an old Julia version being run in a newer musl environment and causing problems when loading libraries that may depend on newer behaviors of musl.

3. We can perform dynamic runtime modification of the internal dso datastructures as described in my message above. This is undesirable as it's fragile (you need to know the exact dso structure layout, which changes from musl version to musl version) and is not forward-compatible.

4. The musl devs could construct some alternative mechanism by which a program can, at runtime, direct the search path of libraries. Since the fundamental issue is that we don't know the location of all libraries until well after program start, and all mechanisms (such as RPATH, LD_LIBRARY_PATH, etc...) are immutable after process startup, there's no way for us to load a libfoo.so that depends on libbar.so but doesn't know precisely where on disk it is stored. If we could manually instruct the loader to check a certain set of paths, and then change that set before loading each library, that would work as well, even though it's significantly more work than just loading each library individually by path as works on all the other platforms we support.

Thank you again for you time!