Skip to content

DEVELOPER.md

Latest commit

 

History

History
499 lines (345 loc) · 18.2 KB

DEVELOPER.md

File metadata and controls

499 lines (345 loc) · 18.2 KB

Developer Guide

This document describes how to set up your development environment to build and test the Valkey GLIDE Java wrapper.

Development Overview

The Valkey GLIDE Java wrapper consists of both Java and Rust code. Rust bindings for the Java Native Interface are implemented using jni-rs, and the Java JAR is built using Gradle. The Java and Rust components communicate using the protobuf protocol.

Build from source

Note: See the Troubleshooting section below for possible solutions to problems.

Prerequisites

Software Dependencies

  • git
  • GCC
  • pkg-config
  • cmake
  • protoc (protobuf compiler) >= v29.1
  • openssl
  • openssl-dev
  • rustup
  • Java 11
  • ziglang and zigbuild (for GNU Linux only)
  • valkey (for testing)

Valkey installation

See the Valkey installation guide to install the Valkey server and CLI.

Note

Windows users: Valkey must be installed in WSL (Windows Subsystem for Linux) as the integration tests run Valkey server instances through WSL. See the Dependencies installation for Windows section below for detailed WSL and Valkey installation instructions.

Dependencies installation for Ubuntu

sudo apt update -y
sudo apt install -y openjdk-11-jdk git gcc pkg-config openssl libssl-dev unzip cmake
# Install rust
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
source "$HOME/.cargo/env"
# Check that the Rust compiler is installed
rustc --version

Continue with Install protobuf compiler and Install ziglang and zigbuild below.

Dependencies installation for CentOS

sudo yum update -y
sudo yum install -y java-11-openjdk-devel git gcc pkgconfig openssl openssl-devel unzip cmake
# Install rust
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh

Continue with Install protobuf compiler and Install ziglang and zigbuild below.

Dependencies installation for MacOS

brew update
brew install openjdk@11 git gcc pkgconfig openssl cmake
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
source "$HOME/.cargo/env"

Continue with Install protobuf compiler below. It is not necessary to Install ziglang and zigbuild for MacOS.

Dependencies installation for Windows

Important

Windows users must install WSL (Windows Subsystem for Linux) to run integration tests, as the test suite uses WSL to run Valkey server instances.

Step 1: Install WSL

See the How to install Linux on Windows with WSL for instructions how to install WSL.

Step 2: Install Valkey in WSL

Open your WSL terminal (Ubuntu in this example) and install Valkey:

# Update package list
sudo apt update

# Install dependencies
sudo apt install -y build-essential tcl

# Download and build Valkey
cd /tmp
wget https://github.com/valkey-io/valkey/archive/refs/tags/8.0.1.tar.gz
tar xzf 8.0.1.tar.gz
cd valkey-8.0.1
make
sudo make install

# Verify installation
valkey-server --version
valkey-cli --version

Step 3: Install Windows Dependencies

Option 1: Using winget (Windows Package Manager)

# Install dependencies using winget
winget install --id Microsoft.OpenJDK.11
winget install --id Git.Git
winget install --id Kitware.CMake
winget install --id OpenSSL.OpenSSL

# Install Rust
# Download and run rustup-init.exe from https://rustup.rs/
# Or use winget:
winget install --id Rustlang.Rustup

# Verify Rust installation
rustc --version

Option 2: Using Chocolatey

# Install Chocolatey if not already installed
# Run PowerShell as Administrator and execute:
Set-ExecutionPolicy Bypass -Scope Process -Force; [System.Net.ServicePointManager]::SecurityProtocol = [System.Net.ServicePointManager]::SecurityProtocol -bor 3072; iex ((New-Object System.Net.WebClient).DownloadString('https://community.chocolatey.org/install.ps1'))

# Install dependencies using Chocolatey
choco install openjdk11 -y
choco install git -y
choco install cmake -y
choco install openssl -y

# Install Rust
choco install rustup.install -y

# Verify Rust installation
rustc --version

Continue with Install protobuf compiler below. It is not necessary to Install ziglang and zigbuild for Windows.

Install protobuf compiler

Only protobuf v29.1 is supported. Other versions are not supported and may cause build issues.

Various platform-specific zips can be found here. Choose the appropriate zip for your system and run the commands below, adjusting for the zip you chose:

For Linux-x86_64:

PB_REL="https://github.com/protocolbuffers/protobuf/releases"
curl -LO $PB_REL/download/v29.1/protoc-29.1-linux-x86_64.zip
unzip protoc-29.1-linux-x86_64.zip -d $HOME/.local
export PATH="$PATH:$HOME/.local/bin"
# Check that the protobuf compiler version 29.1 or higher is installed
protoc --version

For Windows (PowerShell):

# Download protoc for Windows
$PB_REL = "https://github.com/protocolbuffers/protobuf/releases"
Invoke-WebRequest -Uri "$PB_REL/download/v29.1/protoc-29.1-win64.zip" -OutFile "protoc-29.1-win64.zip"

# Extract to a directory (e.g., C:\protoc)
Expand-Archive -Path "protoc-29.1-win64.zip" -DestinationPath "C:\protoc" -Force

# Add to PATH (for current session)
$env:PATH += ";C:\protoc\bin"

# To persist PATH, add it to system environment variables:
[Environment]::SetEnvironmentVariable("Path", $env:PATH + ";C:\protoc\bin", [EnvironmentVariableTarget]::User)

# Check that the protobuf compiler version 29.1 or higher is installed
protoc --version

Note

For Linux/MacOS: You may wish to add the entire export PATH line to your shell configuration file to persist this path addition, either .bashrc or .zshrc depending on which shell you are using. For Windows: The PATH is automatically persisted when using the SetEnvironmentVariable command above.

Install ziglang and zigbuild

pip3 install ziglang
cargo install --locked cargo-zigbuild

Building and installation steps

Before starting this step, make sure you've installed all software dependencies.

  1. Clone the repository:

    git clone https://github.com/valkey-io/valkey-glide.git
cd valkey-glide/java

  2. Build the Java wrapper (Choose a build option from the following and run it from the java folder):

    1. Enter the java directory:
    cd java
    1. Build the project:
    ./gradlew :client:buildAll

Linters

Development on the Java wrapper may involve changes in either the Java or Rust code. Each language has distinct linter tests that must be passed before committing changes.

Firstly, install the Rust linter

# Run from the `java` folder
# Will only need to run once during the installation process
rustup component add clippy rustfmt
cargo clippy --all-features --all-targets -- -D warnings
cargo fmt --manifest-path ./Cargo.toml --all

Language-specific Linters and Static Code Analysis

Java: For Java, we use Spotless and SpotBugs.

  1. Spotless

    # Run from the `java` folder
./gradlew :spotlessCheck # run first to see if there are any linting changes to make
./gradlew :spotlessApply # to apply these changes

  2. SpotBugs

    To run SpotBugs and generate reports:

    # Run from the `java` folder
./gradlew spotbugsMain

    This command will generate HTML and XML reports in the build/reports/spotbugs/ directory.

    To view the SpotBugs findings:

    • Open the HTML report located at build/reports/spotbugs/main/spotbugs.html in a web browser.
    • If you are using IntellJ Idea - open build/reports/spotbugs/main/spotbugs.xml in SpotBugs plugin as it will provide better experience.

    Ensure any new findings are addressed and fixed before committing and pushing your changes.

    Note: The spotbugs task is currently configured to not fail the build on findings.

Troubleshooting

Some troubleshooting issues:

  • If the build fails after following the installation instructions, the gradle daemon may need to be restarted (./gradlew --stop) so that it recognizes changes to environment variables (e.g. $PATH). If that doesn't work, you may need to restart your machine. In particular, this may solve the following problems:
    • Failed to find cargo after rustup.
    • No Protobuf compiler (protoc) found.

Running Examples App

An example app (glide.examples.ExamplesApp) is available under examples project. To run the ExamplesApp against a local build of valkey-glide client, you can publish your JAR to local Maven repository as a dependency.

To publish to local maven run (default version 255.255.255):

# Run from the `examples/java` folder
./gradlew publishToMavenLocal

You can then add the valkey-glide dependency to examples project:

repositories {
    mavenLocal()
}
dependencies {
    // Update to use version defined in the previous step
    implementation group: 'io.valkey', name: 'valkey-glide', version: '255.255.255'
}

Optionally: you can specify a snapshot release:

export GLIDE_RELEASE_VERSION=1.0.1-SNAPSHOT
./gradlew publishToMavenLocal

You can then add the valkey-glide dependency to examples/java/build.gradle with the version and classifier:

repositories {
    mavenLocal()
}
dependencies {
    // Update to use version defined in the previous step
    implementation group: 'io.valkey', name: 'valkey-glide', version: '1.0.1-SNAPSHOT', classifier='osx-aarch_64'
}

Test

To run all tests, use the following command:

./gradlew test

To run the unit tests, use the following command:

./gradlew :client:test

To run end-to-end tests, use the following command:

./gradlew :integTest:test

To run integration tests for pubsub feature:

./gradlew :integTest:pubsubTest

IT suite start the server for testing - standalone and cluster installation using cluster_manager script. By default, it starts servers without TLS; to activate TLS add -Dtls=true to the command line:

./gradlew :integTest:test -Dtls=true

To run a single test, use the following command:

./gradlew :integTest:test --tests '*.functionLoad_and_functionList' --rerun

To run one class, use the following command:

./gradlew :client:test --tests 'BatchTests' --rerun

To run IT tests against an existing cluster and/or standalone endpoint, use:

./gradlew :integTest:test -Dcluster-endpoints=localhost:7000 -Dstandalone-endpoints=localhost:6379

If those endpoints use TLS, add -Dtls=true (applied to both endpoints):

./gradlew :integTest:test -Dcluster-endpoints=localhost:7000 -Dstandalone-endpoints=localhost:6379 -Dtls=true

You can combine this with test filter as well:

./gradlew :integTest:test -Dcluster-endpoints=localhost:7000 -Dstandalone-endpoints=localhost:6379 --tests 'BatchTests' -Dtls=true

To run server modules test (it doesn't start servers):

./gradlew :integTest:modulesTest -Dcluster-endpoints=localhost:7000 -Dtls=true

DNS Tests

To run DNS tests locally:

  1. Add the following entries to your hosts file:

    • Linux/macOS: /etc/hosts
    • Windows: C:\Windows\System32\drivers\etc\hosts
    127.0.0.1 valkey.glide.test.tls.com
127.0.0.1 valkey.glide.test.no_tls.com
::1 valkey.glide.test.tls.com
::1 valkey.glide.test.no_tls.com

  2. Set the environment variable:

    export VALKEY_GLIDE_DNS_TESTS_ENABLED=1

If the environment variable is not set, DNS tests will be skipped.

JaCoCo Code Coverage Results

JaCoCo results are automatically generated just by running the tests (due to the finalizedBy jacocoTestReport task). The generated files are located in client/build/reports/jacoco and integTest/build/reports/jacoco

Generate files

To (re)generate protobuf code, use the following command:

./gradlew protobuf

Contributing new ValKey commands

A Valkey command can either have a standalone or cluster implementation which is dependent on their specifications.

  • A node is an instance of a Valkey server, and a valkey cluster is composed of multiple nodes working in tandem.
  • A cluster command will require a note to indicate a node will follow a specific routing. Refer to https://valkey.io/docs/topics/cluster-spec for more details on how hash slots work for cluster commands.

When you start implementing a new command, check the command_request.proto and request_type.rs files to see whether the command has already been implemented in another language such as Python or Node.js.

Standalone and cluster clients both extend BaseClient.java and implement methods from the interfaces listed in java/client/src/main/java/glide/api/commands. The return types of these methods are in the form of a CompletableFuture, which fulfill the purpose of the asynchronous features of the program.

Tests

When implementing a command, include both a unit test and an integration test.

Implement unit tests in the following files:

Implement integration tests in the following files:

BaseBatch.java will add the command to the Batches API. Refer to this link to view the interface directory. Refer to https://valkey.io/docs/topics/transactions/ and https://valkey.io/topics/pipelining for more details about how Batches work in Valkey.

Javadocs

BaseBatch.java and the methods within the command interfaces will both contain documentation on how the command operates. In the command interface each command's javadoc should contain:

  • Detail on when Valkey started supporting the command (if it wasn't initially implemented in 6.0.0 or before).
  • A link to the Valkey documentation.
  • Information about the function parameters.
  • Any glide-core implementation details, such as how glide-core manages default routing for the command. Reference this link for an example.
  • The command's return type. In the BaseBatch.java file, include "Command Response" before specifying the return type.

Previous PR's

Refer to closed-PRs to see commands that have been previously merged.

JNI naming and signatures

The Java client now uses JNI (Java Native Interface) for direct communication with the Rust core library. Native methods are declared in GlideNativeBridge.java and implemented in the Rust lib.rs file. The JNI layer handles marshalling data between Java and Rust, including command serialization, response parsing, and callback management.

Module Information

  • The module-info.java (glide.api) contains a list of all of the directories the user can access.
  • Ensure to update the exports list if there are more directories the user will need to access.

Recommended extensions for VS Code

Recommended extensions for IntelliJ

Community and Feedback

We encourage you to join our community to support, share feedback, and ask questions. You can approach us for anything on our Valkey Slack: Join Valkey Slack.