Skip to content

Conversation

mesteruh
Copy link

@mesteruh mesteruh commented May 28, 2025

feat: add FetchRequest v10 support (Kafka ≥ 2.6, KIP-110/320)

Problem
aiokafka always sends FetchRequest v4.
When a topic or broker is configured with compression.type=zstd (default since Kafka 2.6), the broker must deliver Zstd-compressed batches.
Because v4 does not advertise Zstd capability, the broker returns error_code 76 (UNSUPPORTED_COMPRESSION_TYPE) and the consumer stalls with
Unexpected error while fetching data: UnknownError.

What this PR adds

Module Change
Protocol • Implement FetchRequest_v10 / FetchResponse_v10
• Set rack_id capability bit → declares Zstd support
• Add per-partition current_leader_epoch & log_start_offset
• Introduce top-level responses array
Fetcher • Auto-select v10 when broker ApiVersion ≥ 2.1.0 (Kafka 2.6+)
• Inject leader_epoch (-1 fallback) per partition
• Parse v10 responses and decode Zstd batches
• Treat error codes 74 / 75 (FENCED / UNKNOWN_LEADER_EPOCH) as retriable → triggers metadata refresh
Tests test_fetch_v10_format covers request/response round-trip and happy-path fetch with Zstd

Result

aiokafka consumers now fetch successfully from Kafka 2.6+ clusters where Zstandard is enabled at broker or topic level-no more silent hangs.
For older brokers (ApiVersion < 2.1.0) the client keeps using v4, so behaviour remains backward-compatible.


Checklist

  • Implementation & lint
  • Unit tests
  • Docs update (will add after review)
  • News fragment fetch_v10.feature

* Implement FetchRequest_v10 / FetchResponse_v10 (KIP-110, KIP-320)
  · rack_id field, ZStd capability flag
  · per-partition current_leader_epoch in request
  · new “responses” field in response schema
* Fetcher
  · builds v10 request when broker API ≥ 2.1.0
  · supplies leader_epoch (-1 fallback) and log_start_offset
  · parses v10 response, handles ZStd records
  · treats FENCED_LEADER_EPOCH (74) & UNKNOWN_LEADER_EPOCH (75)
    as retriable → triggers metadata refresh

* Tests
  · `test_fetch_v10_format` covers request/response ser-des
    and Fetcher happy-path
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant