Skip to content

Expose deterministic partitioner functions in Python #2116

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
wants to merge 2 commits into
base: master
Choose a base branch
from
Open

Expose deterministic partitioner functions in Python #2116

wants to merge 2 commits into from
+163 −0

Conversation

@fangnx
Copy link
Member

@fangnx fangnx commented Oct 30, 2025

What

Why

Users need to manually calculate partition IDs for validation purposes (e.g., verifying messages are delivered to the correct partition). Currently, they must use ctypes to directly call librdkafka functions, which is platform-dependent and not officially supported.

What

Exposes three deterministic partitioner functions from librdkafka:

  • murmur2(key: bytes, partition_count: int) -> int - Java-compatible Murmur2 hash
  • consistent(key: bytes, partition_count: int) -> int - CRC32-based hash
  • fnv1a(key: bytes, partition_count: int) -> int - FNV-1a hash

Changes

  • C extension: Added partitioner wrappers with shared helper function (DRY)
  • Type hints: Added function signatures in cimpl.pyi
  • Exports: Added to __init__.py for public API
  • Tests: Added 3 focused tests covering correctness, deterministic behavior, and input validation

Usage

from confluent_kafka import murmur2

# Calculate expected partition
partition = murmur2(b"user_12345", 10)  # Returns: 3

Checklist

  • Contains customer facing changes? Including API/behavior changes
  • Did you add sufficient unit test and/or integration test coverage for this PR?
    • If not, please explain why it is not required

References

JIRA: https://confluentinc.atlassian.net/browse/NONJAVACLI-4098
Issue: #1759

Test & Review

Open questions / Follow-ups

@fangnx fangnx requested a review from MSeal as a code owner October 30, 2025 19:00
Copilot AI review requested due to automatic review settings October 30, 2025 19:00
@fangnx fangnx requested review from a team as code owners October 30, 2025 19:00
@confluent-cla-assistant
Copy link

🎉 All Contributor License Agreements have been signed. Ready to merge.
Please push an empty commit if you would like to re-run the checks to verify CLA status for all contributors.

@fangnx fangnx changed the title expose Expose deterministic partitioner functions in Python Oct 30, 2025
Copilot AI reviewed Oct 30, 2025
View reviewed changes
Copy link

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

This PR exposes three deterministic partitioner functions from librdkafka to allow users to manually calculate partition IDs for validation purposes, eliminating the need for platform-dependent ctypes calls.

  • Added C extension wrappers for murmur2, consistent, and fnv1a partitioner functions
  • Exposed functions through Python API with proper type hints and documentation
  • Added comprehensive tests covering correctness, deterministic behavior, and input validation

Reviewed Changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 1 comment.

File Description
tests/test_partitioners.py Added unit tests for the three partitioner functions
src/confluent_kafka/src/confluent_kafka.c Implemented C extension wrappers with shared helper function
src/confluent_kafka/cimpl.pyi Added type hints for the new partitioner functions
src/confluent_kafka/init.py Exported the new functions to the public API

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

src/confluent_kafka/src/confluent_kafka.c Outdated
" Calculate partition using Murmur2 hash (Java-compatible).\n"
"\n"
" Deterministic function using Murmur2 hashing algorithm.\n"
" assigned to.\n"
Copy link

Copilot AI Oct 30, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This line appears to be a documentation fragment that doesn't make sense in context. It should be removed or completed with proper context.

Suggested change
" assigned to.\n"

Copilot uses AI. Check for mistakes.
@sonarqube-confluent
Copy link

Quality Gate failed Quality Gate failed

Failed conditions
6 New issues
0.0% Coverage on New Code (required ≥ 80%)

See analysis details on SonarQube

Catch issues before they fail your Quality Gate with our IDE extension SonarQube for IDE SonarQube for IDE

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

Copilot code review Copilot Copilot left review comments

@MSeal MSeal Awaiting requested review from MSeal MSeal is a code owner

At least 1 approving review is required to merge this pull request.

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@fangnx