Nuget package 0.22.3.1 can't be loaded inside of docker container.

[flambert860]

I tried updating the nuget package from 0.22.3 to 0.22.3.1 I can run my application fine in debug, but if I run it as a docker container I get the following error:

System.DllNotFoundException: 'Unable to load shared library 'delta_rs_bridge' or one of its dependencies. In order to help diagnose loading problems, consider using a tool like strace. If you're using glibc, consider setting the LD_DEBUG environment variable:
/home/site/wwwroot/bin/Debug/net9.0/runtimes/linux-x64/native/delta_rs_bridge.so: cannot open shared object file: No such file or directory

ls of the dictory returns these files:
libdelta_kerner_ffi.so
libdelta_rs_bridge.so

[OFurnell]

I'm having a similar issue with the image used by by Azure Web Apps for .NET 9.0. I'm getting this error:

/lib/x86_64-linux-gnu/libm.so.6: version `GLIBC_2.38' not found (required by /home/site/wwwroot/runtimes/linux-x64/native/libdelta_rs_bridge.so)

Running ldd --verbose gives the followng output:

/home/site/wwwroot/runtimes/linux-x64/native/libdelta_rs_bridge.so: /lib/x86_64-linux-gnu/libm.so.6: version `GLIBC_2.38' not found (required by /home/site/wwwroot/runtimes/linux-x64/native/libdelta_rs_bridge.so)
/home/site/wwwroot/runtimes/linux-x64/native/libdelta_rs_bridge.so: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.39' not found (required by /home/site/wwwroot/runtimes/linux-x64/native/libdelta_rs_bridge.so)
        linux-vdso.so.1 (0x00007ffdb0bed000)
        libssl.so.3 => /lib/x86_64-linux-gnu/libssl.so.3 (0x00007aab833bd000)
        libcrypto.so.3 => /lib/x86_64-linux-gnu/libcrypto.so.3 (0x00007aab82f37000)
        libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007aab82f17000)
        libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007aab82e38000)
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007aab82c57000)
        /lib64/ld-linux-x86-64.so.2 (0x00007aab8916f000)

        Version information:
        /home/site/wwwroot/runtimes/linux-x64/native/libdelta_rs_bridge.so:
                ld-linux-x86-64.so.2 (GLIBC_2.3) => /lib64/ld-linux-x86-64.so.2
                libgcc_s.so.1 (GCC_4.2.0) => /lib/x86_64-linux-gnu/libgcc_s.so.1
                libgcc_s.so.1 (GCC_3.3) => /lib/x86_64-linux-gnu/libgcc_s.so.1
                libgcc_s.so.1 (GCC_3.0) => /lib/x86_64-linux-gnu/libgcc_s.so.1
                libcrypto.so.3 (OPENSSL_3.0.0) => /lib/x86_64-linux-gnu/libcrypto.so.3
                libm.so.6 (GLIBC_2.35) => /lib/x86_64-linux-gnu/libm.so.6
                libm.so.6 (GLIBC_2.27) => /lib/x86_64-linux-gnu/libm.so.6
                libm.so.6 (GLIBC_2.38) => not found
                libm.so.6 (GLIBC_2.2.5) => /lib/x86_64-linux-gnu/libm.so.6
                libm.so.6 (GLIBC_2.29) => /lib/x86_64-linux-gnu/libm.so.6
                libc.so.6 (GLIBC_2.7) => /lib/x86_64-linux-gnu/libc.so.6
                libc.so.6 (GLIBC_2.32) => /lib/x86_64-linux-gnu/libc.so.6
                libc.so.6 (GLIBC_2.29) => /lib/x86_64-linux-gnu/libc.so.6
                libc.so.6 (GLIBC_2.14) => /lib/x86_64-linux-gnu/libc.so.6
                libc.so.6 (GLIBC_2.18) => /lib/x86_64-linux-gnu/libc.so.6
                libc.so.6 (GLIBC_2.3.2) => /lib/x86_64-linux-gnu/libc.so.6
                libc.so.6 (GLIBC_2.9) => /lib/x86_64-linux-gnu/libc.so.6
                libc.so.6 (GLIBC_2.25) => /lib/x86_64-linux-gnu/libc.so.6
                libc.so.6 (GLIBC_2.33) => /lib/x86_64-linux-gnu/libc.so.6
                libc.so.6 (GLIBC_2.16) => /lib/x86_64-linux-gnu/libc.so.6
                libc.so.6 (GLIBC_2.39) => not found
                libc.so.6 (GLIBC_2.28) => /lib/x86_64-linux-gnu/libc.so.6
                libc.so.6 (GLIBC_2.15) => /lib/x86_64-linux-gnu/libc.so.6
                libc.so.6 (GLIBC_2.3) => /lib/x86_64-linux-gnu/libc.so.6
                libc.so.6 (GLIBC_2.34) => /lib/x86_64-linux-gnu/libc.so.6
                libc.so.6 (GLIBC_2.17) => /lib/x86_64-linux-gnu/libc.so.6
                libc.so.6 (GLIBC_2.4) => /lib/x86_64-linux-gnu/libc.so.6
                libc.so.6 (GLIBC_2.3.4) => /lib/x86_64-linux-gnu/libc.so.6
                libc.so.6 (GLIBC_2.2.5) => /lib/x86_64-linux-gnu/libc.so.6
                libssl.so.3 (OPENSSL_3.0.0) => /lib/x86_64-linux-gnu/libssl.so.3
        /lib/x86_64-linux-gnu/libssl.so.3:
                libc.so.6 (GLIBC_2.4) => /lib/x86_64-linux-gnu/libc.so.6
                libc.so.6 (GLIBC_2.33) => /lib/x86_64-linux-gnu/libc.so.6
                libc.so.6 (GLIBC_2.14) => /lib/x86_64-linux-gnu/libc.so.6
                libc.so.6 (GLIBC_2.3.4) => /lib/x86_64-linux-gnu/libc.so.6
                libc.so.6 (GLIBC_2.2.5) => /lib/x86_64-linux-gnu/libc.so.6
                libcrypto.so.3 (OPENSSL_3.0.3) => /lib/x86_64-linux-gnu/libcrypto.so.3
                libcrypto.so.3 (OPENSSL_3.0.0) => /lib/x86_64-linux-gnu/libcrypto.so.3
        /lib/x86_64-linux-gnu/libcrypto.so.3:
                libc.so.6 (GLIBC_2.15) => /lib/x86_64-linux-gnu/libc.so.6
                libc.so.6 (GLIBC_2.14) => /lib/x86_64-linux-gnu/libc.so.6
                libc.so.6 (GLIBC_2.4) => /lib/x86_64-linux-gnu/libc.so.6
                libc.so.6 (GLIBC_2.3) => /lib/x86_64-linux-gnu/libc.so.6
                libc.so.6 (GLIBC_2.25) => /lib/x86_64-linux-gnu/libc.so.6
                libc.so.6 (GLIBC_2.33) => /lib/x86_64-linux-gnu/libc.so.6
                libc.so.6 (GLIBC_2.7) => /lib/x86_64-linux-gnu/libc.so.6
                libc.so.6 (GLIBC_2.3.4) => /lib/x86_64-linux-gnu/libc.so.6
                libc.so.6 (GLIBC_2.17) => /lib/x86_64-linux-gnu/libc.so.6
                libc.so.6 (GLIBC_2.16) => /lib/x86_64-linux-gnu/libc.so.6
                libc.so.6 (GLIBC_2.34) => /lib/x86_64-linux-gnu/libc.so.6
                libc.so.6 (GLIBC_2.2.5) => /lib/x86_64-linux-gnu/libc.so.6
        /lib/x86_64-linux-gnu/libgcc_s.so.1:
                libc.so.6 (GLIBC_2.35) => /lib/x86_64-linux-gnu/libc.so.6
                libc.so.6 (GLIBC_2.14) => /lib/x86_64-linux-gnu/libc.so.6
                libc.so.6 (GLIBC_2.34) => /lib/x86_64-linux-gnu/libc.so.6
                libc.so.6 (GLIBC_2.2.5) => /lib/x86_64-linux-gnu/libc.so.6
        /lib/x86_64-linux-gnu/libm.so.6:
                ld-linux-x86-64.so.2 (GLIBC_PRIVATE) => /lib64/ld-linux-x86-64.so.2
                libc.so.6 (GLIBC_ABI_DT_RELR) => /lib/x86_64-linux-gnu/libc.so.6
                libc.so.6 (GLIBC_2.4) => /lib/x86_64-linux-gnu/libc.so.6
                libc.so.6 (GLIBC_2.2.5) => /lib/x86_64-linux-gnu/libc.so.6
                libc.so.6 (GLIBC_PRIVATE) => /lib/x86_64-linux-gnu/libc.so.6
        /lib/x86_64-linux-gnu/libc.so.6:
                ld-linux-x86-64.so.2 (GLIBC_2.35) => /lib64/ld-linux-x86-64.so.2
                ld-linux-x86-64.so.2 (GLIBC_2.2.5) => /lib64/ld-linux-x86-64.so.2
                ld-linux-x86-64.so.2 (GLIBC_2.3) => /lib64/ld-linux-x86-64.so.2
                ld-linux-x86-64.so.2 (GLIBC_PRIVATE) => /lib64/ld-linux-x86-64.so.2

[krsma33]

Default dotnet docker images are debian which comes with 2.36 version of glibc. I made it work with noble images, like mcr.microsoft.com/dotnet/runtime:8.0-noble

[mdrakiburrahman]

Hey @flambert860,

Does using a different base image like @krsma33 suggested work for you?

I've personally run this library on Kuberenetes with Azure Linux container. Perhaps what would be helpful for others is having a dockerfile/docker compose sample in this repo with a working setup.

We haven't really tested in the plethora of Dotnet runtimes (like Azure Web Apps @OFurnell linked). Since we have a dependency on Rust (which takes dependency on the base OS), I'm not surprised funky issues are being reprod. All solvable problems, but we basically need to test all various runtime permutations, which is difficult 😊

[locd2o]

Hi @mdrakiburrahman,

I tried mcr.microsoft.com/dotnet/runtime:8.0-noble, then it worked in azure Kubernetes, however it got memory leak issue whenever I inserted new data into delta table.

What is Azure Linux container image that you have used?

Or, Do you know any .net image that include delta_rs_bridge?

Thanks for your help.

[adamreeve]

Assuming the NuGet packages on NuGet.org are the same ones that are built in GitHub actions, it looks like the problem is that the native libraries for x64 are built with the ubuntu-latest runner image.

This means that the package will likely only run on a system with a GLIBC version >= whatever GitHub has on its ubuntu-latest images. Ideally these native libraries would be built with an older GLIBC version for compatibility with older systems. You could specify an older Ubuntu version for the runner, but even better would be to build inside a container for more control over the host environment and to support older GLIBC than what Ubuntu 22.04 has (the oldest Ubuntu runner version).

It looks like the package action already uses cross for building linux-arm64. It might make sense to also use cross for x64, which will build inside a container. According to their docs, the x86_64-unknown-linux-gnu target builds with GLIBC 2.31 by default, which matches Ubuntu 20.04.

I'd be happy to make a PR to update the package action if this would be acceptable.

[adamreeve]

@mightyshazam, it looks like you're doing some work on the packaging workflow already in your rust-upgrades branch. Do you have any thoughts on this?

[mightyshazam]

The latest builds shouldn't have an openssl/libc dependency, so this problem should go away.

[adamreeve]

Unless you're building with no_std, the library will still need glibc. I don't think this is caused by the OpenSSL dependency.

[adamreeve]

Eg. building your rust-upgrades branch on my dev machine, I get libraries that need glibc >= 2.39:

$ ldd src/DeltaLake/bin/Debug/net8.0/libdelta_rs_bridge.so 
        linux-vdso.so.1 (0x00007f4bf1c78000)
        libbz2.so.1 => /lib64/libbz2.so.1 (0x00007f4bf1c39000)
        libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007f4bf1c0d000)
        libm.so.6 => /lib64/libm.so.6 (0x00007f4bf1b1f000)
        libc.so.6 => /lib64/libc.so.6 (0x00007f4be780e000)
        /lib64/ld-linux-x86-64.so.2 (0x00007f4bf1c7a000)

$ objdump -T src/DeltaLake/bin/Debug/net8.0/libdelta_rs_bridge.so | grep GLIBC | sed 's/.*GLIBC_\([.0-9]*\).*/\1/g' | sort -Vu  | tail -n1
2.39

$ ldd src/DeltaLake/bin/Debug/net8.0/libdelta_kernel_ffi.so 
        linux-vdso.so.1 (0x00007fde8d498000)
        libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007fde8d442000)
        libm.so.6 => /lib64/libm.so.6 (0x00007fde8a112000)
        libc.so.6 => /lib64/libc.so.6 (0x00007fde89f20000)
        /lib64/ld-linux-x86-64.so.2 (0x00007fde8d49a000)

$ objdump -T src/DeltaLake/bin/Debug/net8.0/libdelta_kernel_ffi.so | grep GLIBC | sed 's/.*GLIBC_\([.0-9]*\).*/\1/g' | sort -Vu  | tail -n1
2.39

[mightyshazam]

You're correct. lib c will always be required. The rust upgrades branch doesn't really exist to solve the libc problem. It exists to solve the native tls issue. I got my issues mixed up here. My primary concern was no longer having to build gigantic libraries because we were butting up against the maximum size for the nuget package, and having native tls as a dependency limits the environments to the same version of openssl used at build time.
@adamreeve go for it. I appreciate the contribution.

[maxkuzmin]

Hi @mightyshazam, would it be possible to update https://www.nuget.org/packages/DeltaLake.Net? The latest version still has this issue.

[adamreeve]

I think this can be closed as fixed, I can't reproduce the problem now with the NuGet package for version 0.26.2.