Skip to content

DEVELOPER.md

Latest commit

 

History

History
655 lines (464 loc) · 23.4 KB

DEVELOPER.md

File metadata and controls

655 lines (464 loc) · 23.4 KB

Developer Guide

This document describes how to set up your development environment to build and test the Valkey GLIDE Python wrapper.

The Valkey GLIDE Python wrapper consists of both Python and Rust code. Rust bindings for Python are implemented using PyO3, and the Python package is built using maturin. The Python and Rust components communicate using the protobuf protocol.

📁 Python Project Structure

The python/ directory contains three separate components:

🔹 glide-async/

  • Purpose: Async client for Valkey, implemented as a hybrid Python/Rust project.
  • Rust bindings: via PyO3, defined in valkey-glide/python/glide-async/src/lib.rs.
  • Communication Layer: Communicates with Glide's Rust core using a Unix Domain Socket (UDS).
  • Import path: import glide
  • PyPI package name: valkey-glide
  • Build backend: Maturin (Rust-based)

🔹 glide-sync/

  • Purpose: Sync client for Valkey, implemented via CFFI, built with setuptools.
  • Rust bindings: via CFFI, defined in valkey-glide/ffi/src/lib.rs.
  • Communication Layer: Communicates with Glide's Rust core via direct FFI.
  • Import path: import glide_sync
  • PyPI package name: valkey-glide-sync
  • Build backend: setuptools (C-based)

🔹 glide-shared/

  • Purpose: Shared Python logic used by both clients — includes command builders, exceptions, constants, protobuf message handling, and more.
  • Import path: import glide_shared
  • Installation: Installed locally via pip install valkey-glide/python/glide-shared during each client’s build process. Not published separately to PyPI.

🧱 High-Level Folder Structure

python/
├── glide-async/            # Async client (PyO3 + Maturin)
│   ├── Cargo.toml
│   ├── pyproject.toml
│   ├── python/glide/       # Python code for async client
│   └── src/                # Rust code with PyO3 bindings
├── glide-sync/             # Sync client (CFFI + setuptools)
│   ├── pyproject.toml
│   └── glide_sync/         # Python code for sync client
├── glide-shared/           # Shared logic used by both clients
│   ├── pyproject.toml
│   └── glide_shared/       # Shared source code
└── tests/                  # Shared test suite
ffi/
├── src/                    # Rust code provides a C-compatible FFI (used in glide-sync)

📦 Packaging and Installation Notes

  • During development, glide_shared must be installed locally (editable or standard install).
  • Each client is built and released independently, with its own isolated package name and pyproject.toml.
  • Shared logic remains fully decoupled from both clients and reusable between them.

Prerequisites

Before building the package from source, make sure that you have installed the listed dependencies below:

  • python3 virtualenv
  • git
  • GCC
  • pkg-config
  • cmake
  • protoc (protobuf compiler) >= v3.20.0
  • openssl
  • openssl-dev
  • rustup
  • ziglang and zigbuild (for GNU Linux only)
  • valkey (for testing)

Valkey installation

See the Valkey installation guide to install the Valkey server and CLI.

For your convenience, we wrapped the steps in a "copy-paste" code blocks for common operating systems:

Ubuntu / Debian 
sudo apt update -y
sudo apt install -y python3 python3-venv git gcc pkg-config openssl libssl-dev unzip cmake
# Install rust
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
source "$HOME/.cargo/env"
# Check that the Rust compiler is installed
rustc --version

Continue with Install protobuf compiler and Install ziglang and zigbuild below.

CentOS 
sudo yum update -y
sudo yum install -y python3 git gcc pkgconfig openssl openssl-devel unzip cmake
pip3 install virtualenv
# Install rust
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
source "$HOME/.cargo/env"
# Check that the Rust compiler is installed
rustc --version

Continue with Install protobuf compiler and Install ziglang and zigbuild below.

MacOS 
brew update
brew install python3 git gcc pkgconfig openssl virtualenv cmake
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
source "$HOME/.cargo/env"
# Check that the Rust compiler is installed
rustc --version

Continue with Install protobuf compiler below. It is not necessary to Install ziglang and zigbuild for MacOS.

Install protobuf compiler

Various platform-specific zips can be found here. Choose the appropriate zip for your system and run the commands below, adjusting for the zip you chose:

PB_REL="https://github.com/protocolbuffers/protobuf/releases"
curl -LO $PB_REL/download/v3.20.3/protoc-3.20.3-linux-x86_64.zip
unzip protoc-3.20.3-linux-x86_64.zip -d $HOME/.local
export PATH="$PATH:$HOME/.local/bin"
# Check that the protobuf compiler is installed. A minimum version of 3.20.0 is required.
protoc --version

[!NOTE] You may wish to add the entire export PATH line to your shell configuration file to persist this path addition, either .bashrc or .zshrc depending on which shell you are using.

Install `ziglang` and `zigbuild` 
pip3 install ziglang
cargo install --locked cargo-zigbuild

Cloning the Repository

To begin development, clone the repository:

mkdir -p $HOME/src
cd $HOME/src
git clone https://github.com/valkey-io/valkey-glide.git
cd valkey-glide

Build

After installing prerequisites and cloning the repository, you can build the async Python client using the dev.py CLI utility that can be found in the root python/ directory.

Examples

# Build the async client in release mode
python3 dev.py build --client async --mode release

These commands handle environment setup, dependency installation, and consistent build logic.

Notes on Build Modes

  • Use --mode debug (default) when developing or debugging.
  • Use --mode release when measuring performance or preparing production builds.

Run the following to see all available commands:

python3 dev.py --help

Tests

Ensure you have installed valkey-server and valkey-cli on your host (or redis-server and redis-cli). See the Valkey installation guide to install the Valkey server and CLI.

You can run all tests from the root python/ directory using:

python3 dev.py test

To pass additional arguments to pytest, use the --args flag:

# Run a specific test
python3 dev.py test --args -k <test_name>

# Run against existing servers with tls enabled
python3 dev.py test --args \
  --tls \
  --cluster-endpoints=localhost:7000 \
  --standalone-endpoints=localhost:6379

Manually Running Tests

If needed, you can invoke pytest directly from the root python/ directory for custom workflows:

Run all tests manually

source .env/bin/activate
pytest -v

Running with different async backends

Python GLIDE supports both trio and asyncio. Pass the --async-backend flag to pytest with either trio, asyncio or uvloop to run tests on the specified async backend. You can pass multiple async backends to run tests on all of them.

Example:

source .env/bin/activate
pytest -v --async-backend=trio --async-backend=asyncio

DNS Tests

To run async and sync DNS tests locally:

  1. Add the following entries to your hosts file:

    • Linux/macOS: /etc/hosts
    • Windows: C:\Windows\System32\drivers\etc\hosts
    127.0.0.1 valkey.glide.test.tls.com
127.0.0.1 valkey.glide.test.no_tls.com
::1 valkey.glide.test.tls.com
::1 valkey.glide.test.no_tls.com

  2. Set the environment variable:

    export VALKEY_GLIDE_DNS_TESTS_ENABLED=1

If the environment variable is not set, DNS tests will be skipped.

Protobuf

During the initial build, Python protobuf files were created in python/glide-shared/glide_shared/protobuf. If modifications are made to the protobuf definition files (.proto files located in glide-core/src/protofuf), it becomes necessary to regenerate the Python protobuf files.

Protobuf files are automatically regenerated as part of the build process. If you only need to regenerate the protobuf files (e.g., after editing .proto files), you can use the standalone command the root python/ directory:

python3 dev.py protobuf

This generates .py and .pyi interface files for type checking and places them in the python/glide-shared/glide_shared/protobuf folder.

Linters

Development on the Python wrapper may involve changes in either the Python or Rust code. Each language has distinct linter tests that must be passed before committing changes.

Python Linters

This project uses the following Python linters and formatters:

  • isort
  • black
  • flake8
  • mypy

To check formatting and run static analysis for Python code, use the dev.py utility from the root python/ directory:

python3 dev.py lint

By default, this will auto-fix formatting issues using isort and black. If you want to only check formatting without modifying any files, pass the --check flag:

python3 dev.py lint --check

Rust Linters

For Rust code, run manually:

rustup component add clippy rustfmt
cargo clippy --all-features --all-targets -- -D warnings
cargo fmt --manifest-path ./Cargo.toml --all

These are not included in the dev.py utility and should be run separately from the python root folder.

Packaging

This section explains how the valkey-glide (async client) and valkey-glide-sync (sync client) packages are built into Python wheels for local use and PyPI publishing.

Async Client (valkey-glide)

  1. Stage shared code Before packaging, we copy the glide-shared/glide_shared package directory into the glide-async/python/ folder. This ensures the shared code is included in the final wheel.

  2. Make it discoverable In glide-async/pyproject.toml, we add the appropriate include rule under [tool.maturin] to make sure the copied files are bundled with the wheel. For example:

    include = ["python/glide_shared/**/*.py"]

  3. Build the wheel We use maturin build from the glide-async directory to create a Python wheel that includes the compiled Rust extension and all Python code.

  4. Multiplatform packaging for PyPI To publish wheels to PyPI, we use the PyO3/maturin-action GitHub Action, which builds manylinux and macOS wheels for different Python versions (both CPython and PyPy). This action runs in CI and uses prebuilt Docker containers for compatibility.

  5. Local testing You can test building a wheel and installing it locally using:

    python dev.py build --client async --wheel

Sync Client (valkey-glide-sync)

  1. Vendoring dependencies (for both sdist and wheel) During the build process (for both source distributions and wheels), we vendor (copy) all required Rust crates and Python modules into the glide-sync directory. This includes:

    • ffi
    • glide-core
    • logger_core
    • glide_shared (Python module)

    Vendoring ensures that all dependencies are self-contained within the package at build time, which is required for reproducible builds and compatibility with tools like cibuildwheel.

  2. Build steps

    • The necessary dependencies are vendored into the glide-sync directory before the build starts.
    • We run cargo build in the glide-sync/ffi/ folder to compile the Rust FFI library.
    • The resulting shared object file (e.g. libglide_ffi.so) is copied into the glide-sync/glide_sync/ directory.
    • We then build the Python package, either as a wheel or as an sdist.

  3. Multiplatform packaging for PyPI To support building wheels for multiple platforms (Linux, macOS, and different Python versions), we use the cibuildwheel tool. This tool installs all required Python versions and runs the build inside isolated Docker containers (e.g., manylinux2014). Because the sync client depends on external Rust code (glide-core, ffi, logger_core), we run cibuildwheel from the project root and specify python/glide-sync as the build target. This allows the tool to copy the full project context into the container.

  4. Local testing

    You can building a wheel and install it locally using:

    python dev.py build --client sync --wheel

Documentation

NOTE: We are currently in process of switching our documentation tool from sphinx to mkdocs. Currently the files located in python/docs are required for sphinx's CI validation step (docs-test) as they are the configuration files for how sphinx works in documenting Valkey GLIDE. Once we switch to mkdocs, sphinx related files and validation should be removed, and mkdocs's files and validation should be used instead.

By default, mkdocs should still be using Google's Python Docstring Style so the "Documentation Style" section below will still be valid.

We follow the Google Style Python Docstrings format in our documentation. For our documentation tool, we use sphinx.

Note: docs/index.rst has manual modifications to it and should NOT be deleted. Modify this file with caution.

To run this tool, execute the following:

cd $HOME/src/valkey-glide/python
source .env/bin/activate
pip install -r dev_requirements.txt
cd docs
sphinx-apidoc -o . ../python/glide
make clean
make html # or run make help to see a list of available options

In docs/_build you will find the index.html page. Open this file in your browser and you should see all the documented functions.

However, some stylings may not be implemented by this Google format. In such cases, we revert back to the default style that sphinx uses: reStructuredText. An example of this is shown for hyperlinks below.

Documentation Style

Example of a Properly Formatted Docstring in Functions

"""
Reads entries from the given streams.

See https://valkey.io/commands/xread for more details.

Note:
    When in cluster mode, all keys in `keys_and_ids` must map to the same hash slot.

Warning:
    If we wanted to provide a warning message, we would format it like this.

Args:
    keys_and_ids (Mapping[TEncodable, TEncodable]): A mapping of keys and entry
        IDs to read from.
    options (Optional[StreamReadOptions]): Options detailing how to read the stream.

Returns:
    Optional[Mapping[bytes, Mapping[bytes, List[List[bytes]]]]]: A mapping of stream keys, to a mapping of stream IDs,
    to a list of pairings with format `[[field, entry], [field, entry], ...]`.

    None will be returned under the following conditions:

        - All key-ID pairs in `keys_and_ids` have either a non-existing key or a non-existing ID, or there are no
            entries after the given ID.
        - The `BLOCK` option is specified and the timeout is hit.

Examples:
    >>> await client.xadd("mystream", [("field1", "value1")], StreamAddOptions(id="0-1"))
    >>> await client.xadd("mystream", [("field2", "value2"), ("field2", "value3")], StreamAddOptions(id="0-2"))
    >>> await client.xread({"mystream": "0-0"}, StreamReadOptions(block_ms=1000))
        {
            b"mystream": {
                b"0-1": [[b"field1", b"value1"]],
                b"0-2": [[b"field2", b"value2"], [b"field2", b"value3"]],
            }
        }
        # Indicates the stream entries for "my_stream" with IDs greater than "0-0". The operation blocks up to
        # 1000ms if there is no stream data.
"""

Example of Properly Formatted Documentation for a Class

class BitOffsetMultiplier(BitFieldOffset):
    """
    Represents an offset in an array of bits for the `BITFIELD` or `BITFIELD_RO` commands. The bit offset index is
    calculated as the numerical value of the offset multiplied by the encoding value. Must be greater than or equal
    to 0.

    For example, if we have the binary 01101001 with offset multiplier of 1 for an unsigned encoding of size 4, then
    the value is 9 from `0110(1001)`.

    Attributes:
        offset (int): The offset in the array of bits, which will be multiplied by the encoding value to get the
            final bit index offset.
    """

    #: Prefix specifying that the offset uses an encoding multiplier.
    OFFSET_MULTIPLIER_PREFIX = "#"

    def __init__(self, offset: int):
        self._offset = f"{self.OFFSET_MULTIPLIER_PREFIX}{str(offset)}"

    def to_arg(self) -> str:
        return self._offset

Example of Properly Formatted Documentation for Enums

class ConditionalChange(Enum):
    """
    A condition to the `SET`, `ZADD` and `GEOADD` commands.
    """

    ONLY_IF_EXISTS = "XX"
    """ Only update key / elements that already exist. Equivalent to `XX` in the Valkey API. """

    ONLY_IF_DOES_NOT_EXIST = "NX"
    """ Only set key / add elements that does not already exist. Equivalent to `NX` in the Valkey API. """

Indentation and Spaces

Args or Attributes

To provide documentation for arguments or attributes, we can have each argument or attribute next to each other with descriptions that exceed the max line length indented on the next line:

Args:
    some_num (int): If this line ever gets too long, what we could do is break this line
        to the next line here, and indent it so that sphinx can see the line as part of
        the same argument.
    options (Optional[StreamReadOptions]): For other arguments, we start by having the indent
        match up with the indent of the first line of the previous argument, and then follow
        the same rule.
    some_bool (bool): And then one-line descriptions are as usual.


Attributes:
    some_num (int): If this line ever gets too long, what we could do is break this line
        to the next line here, and indent it so that sphinx can see the line as part of
        the same argument.
    options (Optional[StreamReadOptions]): For other arguments, we start by having the indent
        match up with the indent of the first line of the previous argument, and then follow
        the same rule.
    some_bool (bool): And then one-line descriptions are as usual.

Return value(s)

Return values are a little special for sphinx. If we wanted to provide more context or multiple possible return values, the convention we will go for is that we should add a space between every different return value.

We start by adding the return type on the first line, followed by a description of the return value.

Note: for each return value, we should not indent the docs like args to show that it is part of the same return value. For example:

Returns:
    List[int]: Some description here regarding the purpose of the list of ints being
    returned. Notice how this new line is not indented but it is still apart of the same
    description.

    If we ever want to provide more context or another description of another return value
    (ex. None, -1, True/False, etc.) we add a space between this description and the
    previous description.

Lists

We have to add a space between the previous line and a line after the list ends and also indent the list by one indent level:

Args:
    key (TEncodable): The key of the stream.
    start (StreamRangeBound): The starting stream ID bound for the range.

        - Use `IdBound` to specify a stream ID.
        - Use `ExclusiveIdBound` to specify an exclusive bounded stream ID.
        - Use `MinId` to start with the minimum available ID.

    end (StreamRangeBound): The ending stream ID bound for the range.

        - Use `IdBound` to specify a stream ID.
        - Use `ExclusiveIdBound` to specify an exclusive bounded stream ID.
        - Use `MaxId` to end with the maximum available ID.

    count (Optional[int]): An optional argument specifying the maximum count of stream entries to return.

Examples or Code Blocks

For examples, we can use the Example: or Examples: keyword and indent the following code block:

Examples:
    >>> await client.xadd("mystream", [("field1", "value1")], StreamAddOptions(id="0-1"))
    >>> await client.xadd("mystream", [("field2", "value2"), ("field2", "value3")], StreamAddOptions(id="0-2"))
    >>> await client.xread({"mystream": "0-0"}, StreamReadOptions(block_ms=1000))
        {
            b"mystream": {
                b"0-1": [[b"field1", b"value1"]],
                b"0-2": [[b"field2", b"value2"], [b"field2", b"value3"]],
            }
        }
        # Indicates the stream entries for "my_stream" with IDs greater than "0-0". The operation blocks up to
        # 1000ms if there is no stream data.

    ... # More examples here

If we wanted to add a code block in a place other than the Examples block, we have to use a double colon syntax and indent the code block.

Attributes:
    addresses (List[NodeAddress]): DNS Addresses and ports of known nodes in the cluster.
        If the server is in cluster mode the list can be partial, as the client will attempt to map out
        the cluster and find all nodes.
        If the server is in standalone mode, only nodes whose addresses were provided will be used by the
        client.
        For example::

            [
                {address:sample-address-0001.use1.cache.amazonaws.com, port:6379},
                {address: sample-address-0002.use2.cache.amazonaws.com, port:6379}
            ]

    use_tls (bool): True if communication with the cluster should use Transport Level Security.
        Should match the TLS configuration of the server/cluster, otherwise the connection attempt will fail

Constants

We want to use #: to add documentation for constants:

#: "GET" subcommand string for use in the `BITFIELD` or `BITFIELD_RO` commands.
GET_COMMAND_STRING = "GET"

Enums

We provide a general description at the top, and follow each enum value with a description beneath. Refer to the example.

Links and Hyperlinks

If we wanted to show a regular link, we can add it as is. If we wanted to show hyperlinks, follow the reStructuredText (rst) link format:

Format: `text <link>`_

Example: `SORT <https://valkey.io/commands/sort/>`_

Recommended extensions for VS Code

Community Support and Feedback

We encourage you to join our community to support, share feedback, and ask questions. You can approach us for anything on our Valkey Slack: Join Valkey Slack.