Proposal: pack all native libraries into a package

[Yury-Fridlyand]

Proposal: pack all native libraries into a package. On first access to the library, a proper native library for the current OS/platform.

Procs:

1. Single package to be uploaded/downloaded
2. No need to use classifier/arch/platform to resolve the package

Cons:

1. Bigger package size
2. Platform detection in runtime
3. Unsupported platforms (e.g. Windows) will get errors in runtime instead of compile-time

[dw-genesys]

+1

For what it's worth (I haven't been able to dig deep enough yet) platform detection at runtime is required anyway if you need a single jar which can deploy to multiple architectures. I've run into issues discussed in Slack where I worked around the lack of a 'universal' jar by depending on all of them, but glide gets confused while trying to load the correct netty native libraries.

[netofontes]

Can someone from AWS look at this request too? This is pretty important feature when simplifying build pipelines for applications using GLIDE client. This would massively ease and speed up the adoption Valkey and the GLIDE Client across multiple organizations.

One way to do it is enabling NativeUtils to select all necessary dynamic libraries during runtime based on the platform OS and architecture. This would allow multiple architecture JARs to coexist in the classloader.

Benefits

Simplified build process: No need to create separate builds for different architectures

• True multi-architecture support: Single application layer works across all supported architectures
• Aligned with Java philosophy: Write once, run anywhere
• Reduced maintenance overhead: Single artifact to manage and ship in container images.

Implementation Notes

The runtime selection mechanism in NativeUtils should:

1. Detect the current platform OS and architecture
2. Select the appropriate dynamic library from all available libraries in the classpath
3. Load only the compatible library for the current platform

[manjuangadi]

We also faced same issue and good if this can be addressed by Valkey/GLIDE team ASAP.

[arnaudroger]

We just came across that issue, and as a work around create a maven project that repackage the 4 jar, put the native libs in different directory and replace the NativeUtils class to use https://github.com/scijava/native-lib-loader.

<plugin>
             <groupId>org.apache.maven.plugins</groupId>
             <artifactId>maven-dependency-plugin</artifactId>
             <version>3.3.0</version>
             <executions>
                 <execution>
                     <id>unpack_la</id>
                     <phase>prepare-package</phase>
                     <goals>
                         <goal>unpack</goal>
                     </goals>
                     <configuration>
                         <artifactItems>
                             <artifactItem>
                                 <groupId>io.valkey</groupId>
                                 <artifactId>valkey-glide</artifactId>
                                 <classifier>linux-aarch_64</classifier>
                                 <version>${valkey-glide.version}</version>
                                 <includes>
                                     libglide_rs.so
                                 </includes>
                                 <outputDirectory>${project.build.directory}/classes/natives/linux_arm64</outputDirectory>
                             </artifactItem>
                             <artifactItem>
                                 <groupId>io.valkey</groupId>
                                 <artifactId>valkey-glide</artifactId>
                                 <classifier>linux-x86_64</classifier>
                                 <version>${valkey-glide.version}</version>
                                 <includes>
                                     libglide_rs.so
                                 </includes>
                                 <outputDirectory>${project.build.directory}/classes/natives/linux_64</outputDirectory>
                             </artifactItem>
                             <artifactItem>
                                 <groupId>io.valkey</groupId>
                                 <artifactId>valkey-glide</artifactId>
                                 <classifier>osx-aarch_64</classifier>
                                 <version>${valkey-glide.version}</version>
                                 <includes>
                                     libglide_rs.dylib
                                 </includes>
                                 <outputDirectory>${project.build.directory}/classes/natives/osx_arm64</outputDirectory>
                             </artifactItem>
                             <artifactItem>
                                 <groupId>io.valkey</groupId>
                                 <artifactId>valkey-glide</artifactId>
                                 <classifier>osx-x86_64</classifier>
                                 <version>${valkey-glide.version}</version>
                                 <includes>
                                     libglide_rs.dylib
                                 </includes>
                                 <outputDirectory>${project.build.directory}/classes/natives/osx_64</outputDirectory>
                             </artifactItem>

                             <artifactItem>
                                 <groupId>io.valkey</groupId>
                                 <artifactId>valkey-glide</artifactId>
                                 <classifier>linux-aarch_64</classifier>
                                 <version>${valkey-glide.version}</version>
                                 <excludes>
                                     libglide_rs.so,
                                     glide/ffi/resolvers/NativeUtils.class
                                 </excludes>
                                 <outputDirectory>${project.build.directory}/classes/</outputDirectory>
                             </artifactItem>
                         </artifactItems>
                     </configuration>
                 </execution>
             </executions>
         </plugin>

import org.scijava.nativelib.NativeLoader;

import java.io.*;

public class NativeUtils {


    private static boolean glideLibLoaded = false;

    private NativeUtils() {}

    public static synchronized void  loadGlideLib() {

        if (glideLibLoaded) {
            return;
        }
        try {
            NativeLoader.loadLibrary("glide_rs");
            glideLibLoaded = true; // Mark as loaded after successful load
        } catch (IOException e) {
            throw new RuntimeException(e);
        }
    }

}

[yipin-chen]

We are evaluating this proposal.