Migrating from Lettuce java client

[LarsKrogJensen]

We are currently using Lettuce java client when communicating with AWS Elastic Cache Valkey flavor, but are considering moving to Glide client library, but there are a few points that is unclear to me.

We run our workload in alpine based containers on AWS EKS, but obviously also on developer machines (Mac, Linux) and thus needs Glide to:

• Support musl libc
• bundle multi arch (arm/amd) and multi platform (Mac/Linux) dynamic libraries

Is this possible today, and if not any plans?

[xShinnRyuu]

Hi @LarsKrogJensen, sorry for the delayed reply!

Valkey-GLIDE does support musl libc with the Java client.

We currently do not support the bundling of multi arch or platform dynamic libraries. You would need to use classifier in the maven/gradle file to pinpoint a platform-specific jar, otherwise it will not work.

With that being said, I'm not sure if you are using Gradle or Maven, but both of them have plugins/extensions that can auto-detect current system architecture and OS for the classifier field:

For Gradle

plugins {
    id 'com.google.osdetector' version '1.7.3'
}

dependencies {
    implementation "io.valkey:valkey-glide:2.1.1:${osdetector.classifier}"
}

For Maven

<build>
    <extensions>
        <extension>
            <groupId>kr.motd.maven</groupId>
            <artifactId>os-maven-plugin</artifactId>
            <version>1.7.1</version>
        </extension>
    </extensions>
</build>

<dependencies>
    <dependency>
        <groupId>io.valkey</groupId>
        <artifactId>valkey-glide</artifactId>
        <version>2.1.1</version>
        <classifier>${os.detected.classifier}</classifier>
    </dependency>
</dependencies>

Please let us know if these work with your project. We can discuss other options if they can't meet your need.

[LarsKrogJensen]

Hi @xShinnRyuu

Thanks for response and glad to hear musl libc is supported.

We are using maven and I am familiar with classifier concept, but until multi-arch bundle it might will be tricky as CI pipeline runs amd64 and it would not include arm64 :(

Any plans of this?

[xShinnRyuu]

We currently do not have any plans for this as of yet. However, we always welcome new asks for features and will consider making it a priority.

Coincidentally, there is a new issue that was raised earlier today that is quite similar to your ask/situation.

Could you take a look and see if your needs align and provide a comment, so that we can prioritize accordingly? Compatibility with different operating systems #4936

[LarsKrogJensen]

yes this is fundamentally the same issue

Netty library might be a good inspiration (although it does not support muslc) as it figures out at run time which native epoll library to use. Netty is maybe a bit different as it also has a pure java fallback.

[jamesx-improving]

Hi @LarsKrogJensen , sorry for jumping into the conversation half way, but could you please elaborate the CI situation you mentioned above a bit more? Are you saying that the auto-detecting os-maven-plugin won't work in your CI environment?

My understanding is that, as long as your CI only checks out the source code of your project, and pull dependency itself, the plugin will detect the CPU arch of the CI, and generate the corresponding classifier for valkey-glide correctly.

Are you using a different setup for CI, such that you have to use the same .jar for both development and CI?

[LarsKrogJensen]

Hi @jamesx-improving , auto detect is not the problem - it does its job, problem is that our CI typically runs amd64, but at runtime k8s scheduler might schedule a pod on either arm64 or amd64.

There are always work arounds, like running maven builds on both archs - although not especially convenient.

So for smoother adoption to Glide it would be nice if there was a way to bundle multiple native 'drivers'.

[jamesx-improving]

I see.

We are considering the option to offer a single jar that supports all platforms, on top of the existing single platform jars. The primary drawback would be a much larger size, like over 100MB. Are we OK with such library size for the workflow?

[LarsKrogJensen]

Oh, over 100MB would be a lot, probably too much.

What if you could include multiple platform jars and have some smart dynamic mechanism to figure out which one to use.

[jamesx-improving]

Turns out we already have a tracking issue #4596 for single jar supporting all platforms. You can keep an eye on this issue as well. BTW we will be careful on the size as well.

[LarsKrogJensen]

interesting, thanks

must admit I did not realize to native part is so big...