Generated deps script for ExecutableProducts breaks curl on 1.4

[maleadt]

This is probably what @SimonDanisch mentioned in #129 (comment)

julia> versioninfo()
Julia Version 1.4.0-DEV.688
Commit c32d166ffe (2020-01-09 09:30 UTC)

shell> curl -o /dev/null https://google.com
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   220  100   220    0     0   2095      0 --:--:-- --:--:-- --:--:--  2095

julia> using GDAL

shell> curl -o /dev/null https://google.com
curl: /opt/julia/bin/../lib/julia/libcurl.so.4: no version information available (required by curl)
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
curl: (60) Cert verify failed: BADCERT_NOT_TRUSTED
More details here: https://curl.haxx.se/docs/sslcerts.html

The linker warning is telling; this is due to a different libcurl getting loaded:

shell> ldd /usr/bin/curl
        linux-vdso.so.1 (0x00007ffea2979000)
        libcurl.so.4 => /usr/lib/x86_64-linux-gnu/libcurl.so.4 (0x00007f7e1a1ec000)
...

julia> using GDAL

shell> ldd /usr/bin/curl
/usr/bin/curl: /opt/julia/bin/../lib/julia/libcurl.so.4: no version information available (required by /usr/bin/curl)
        linux-vdso.so.1 (0x00007ffe5bdfc000)
        libcurl.so.4 => /opt/julia/bin/../lib/julia/libcurl.so.4 (0x00007fc122fad000)

This is due to modifying LD_LIBRARY_PATH in a BinaryProvider-generated deps script:

shell> cat $(dirname(pathof(GDAL)))/../deps/build.jl
write_deps_file(joinpath(@__DIR__, "deps_gdal.jl"), products, verbose=verbose)

shell> cat $(dirname(pathof(GDAL)))/../deps/deps_gdal.jl
ENV["LD_LIBRARY_PATH"] = join(filter(!isempty, libpaths), ":")

BinaryProvider.jl/src/Products.jl

Lines 480 to 502 in d57da3f

	# If any of the products are `ExecutableProduct`s, we need to add Julia's
	# library directory onto the end of {DYLD,LD}_LIBRARY_PATH
	@static if !Sys.iswindows()
	if any(p isa ExecutableProduct for p in products)
	dllist = Libdl.dllist()
	libjulia = filter(x -> occursin("libjulia", x), dllist)[1]
	julia_libdir = repr(joinpath(dirname(libjulia), "julia"))
	envvar_name = @static if Sys.isapple()
	"DYLD_LIBRARY_PATH"
	else Sys.islinux()
	"LD_LIBRARY_PATH"
	end
	envvar_name = repr(envvar_name)
	println(depsjl_file, """
	libpaths = split(get(ENV, $(envvar_name), ""), ":")
	if !($(julia_libdir) in libpaths)
	push!(libpaths, $(julia_libdir))
	end
	ENV[$(envvar_name)] = join(filter(!isempty, libpaths), ":")
	""")
	end
	end

Spotted in PkgEval runs: https://github.com/JuliaCI/NanosoldierReports/blob/f062a47c6bb3b5dd7c7bd29e7a3cc53cc6feeedd/pkgeval/by_hash/c32d166_vs_2e6715c/logs/GeoArrays/1.4.0-DEV-015cd56127.log#L290-L294 https://github.com/JuliaCI/NanosoldierReports/blob/f062a47c6bb3b5dd7c7bd29e7a3cc53cc6feeedd/pkgeval/by_hash/c32d166_vs_2e6715c/logs/DensityRatioEstimation/1.4.0-DEV-015cd56127.log#L336-L349

cc @KristofferC

[staticfloat]

Yeah, there's not really going to be a way to fix this in BinaryProvider; we have to set the LD_LIBRARY_PATH for some executable products, and as long as the interface that BinaryProvider exports is just a string pointing to the executable (as opposed to something more advanced like a function that you call with a do block, as in JLL packages) we can't stop setting LD_LIBRARY_PATH.

As a workaround, you can manually edit the autogenerated build.jl files to remove the setting of LD_LIBRARY_PATH if your binaries don't require anything within the Julia library directory.

[maleadt]

Yeah, I realized that when looking into the problem a bit more... I'm surprised this hasn't been a problem before, with the Julia library path providing widely-used libraries like libz or (here) libcurl.

[staticfloat]

libz is very stable, so you rarely need to worry about that. (It's quite backwards-compatible).

libcurl is less stable, but it's usually pretty backwards-compatible. Are you running on a super-new distribution? In the past, we've generally been shipping newer versions than the OS provides (Which is often okay) but in this case perhaps the OS version is actually the newer one.

[maleadt]

This happened in NewPkgEval, which is running ubuntu:latest (18.04), so nothing particularly new.

[visr]

Isn't the issue in this case not backwards compatibility, but the fact that the libcurl included with the current GDAL build are not full featured builds?

https://github.com/JuliaGeo/GDALBuilder/blob/95d66bd28541acb8e47e2b5ab6f28c72b3d9ba85/build_tarballs.jl#L21-L26