Skip to content

EmbeddedKafkaBroker consume with seekToEnd does not reliably seek to the end #3660

@coekie

Description

@coekie

In what version(s) of Spring for Apache Kafka are you seeing this issue?

3.3.0

Describe the bug

Calling EmbeddedKafkaBroker#consumeFromAnEmbeddedTopic with seekToEnd set to true may cause the consumer to miss messages that were sent to the topic after calling that method, because the seeking is done asynchronously.
That seems unexpected and leads to flaky, unpredictable test results.

That is because consumeFromEmbeddedTopics calls the kafka Consumer#seekToEnd method which is documented as "This function evaluates lazily, seeking to the final offset in all partitions only when poll(Duration) or position(TopicPartition) are called.".

To Reproduce

Call consumeFromAnEmbeddedTopic with seekToEnd, then send a message to the topic, and then try to consume it. If the consumer sees the message or not depends on timing.

Concrete use case: Calling consumeFromAnEmbeddedTopic at the start of a test, then calling application code that is expected to send the message, and then using KafkaTestUtils.getSingleRecord in the test may randomly fail.
Workarounds: Calling Thread.sleep or Consumer.position after calling consumeFromAnEmbeddedTopic.

Expected behavior

The method should seek to the end before returning.

Sample

I will submit a PR including a failing test case.

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions