Skip to content
New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

#

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

Jump to bottom

EmbeddedKafkaBroker consume with seekToEnd does not reliably seek to the end #3660

Closed
coekie opened this issue Dec 8, 2024 · 0 comments
Closed

EmbeddedKafkaBroker consume with seekToEnd does not reliably seek to the end #3660

coekie opened this issue Dec 8, 2024 · 0 comments
Labels
for: backport-to-3.2.x status: backported type: bug
Milestone
3.3.1

Comments

@coekie
Copy link
Contributor

coekie commented Dec 8, 2024

In what version(s) of Spring for Apache Kafka are you seeing this issue?

3.3.0

Describe the bug

Calling EmbeddedKafkaBroker#consumeFromAnEmbeddedTopic with seekToEnd set to true may cause the consumer to miss messages that were sent to the topic after calling that method, because the seeking is done asynchronously.
That seems unexpected and leads to flaky, unpredictable test results.

That is because consumeFromEmbeddedTopics calls the kafka Consumer#seekToEnd method which is documented as "This function evaluates lazily, seeking to the final offset in all partitions only when poll(Duration) or position(TopicPartition) are called.".

To Reproduce

Call consumeFromAnEmbeddedTopic with seekToEnd, then send a message to the topic, and then try to consume it. If the consumer sees the message or not depends on timing.

Concrete use case: Calling consumeFromAnEmbeddedTopic at the start of a test, then calling application code that is expected to send the message, and then using KafkaTestUtils.getSingleRecord in the test may randomly fail.
Workarounds: Calling Thread.sleep or Consumer.position after calling consumeFromAnEmbeddedTopic.

Expected behavior

The method should seek to the end before returning.

Sample

I will submit a PR including a failing test case.
coekie added a commit to coekie/spring-kafka that referenced this issue Dec 8, 2024
@coekie
spring-projectsGH-3660: Make seekToEnd in EmbeddedKafkaBrokers reliable
aba143d 
Problem:
consumeFromEmbeddedTopics calls Consumer.seekToEnd, which
"evaluates lazily, seeking to the final offset in all partitions only
when poll(Duration) or position(TopicPartition) are called".
This means that the consumer may or may not see future messages,
depending on timing.

This fix calls `position` so that the seek happens before
consumeFromEmbeddedTopics returns. That was it is ensured that any
message sent to the topic after the call to consumeFromEmbeddedTopics
are seen by the consumer.

Issue: spring-projects#3660
coekie added a commit to coekie/spring-kafka that referenced this issue Dec 8, 2024
@coekie
spring-projectsGH-3660: Fix EmbeddedKafkaBrokers.seekToEnd
a7a6086 
Problem:
consumeFromEmbeddedTopics calls Consumer.seekToEnd, which
"evaluates lazily, seeking to the final offset in all partitions only
when poll(Duration) or position(TopicPartition) are called".
This means that the consumer may or may not see future messages,
depending on timing.

This fix calls `position` so that the seek happens before
consumeFromEmbeddedTopics returns. That was it is ensured that any
message sent to the topic after the call to consumeFromEmbeddedTopics
are seen by the consumer.

Issue: spring-projects#3660
coekie added a commit to coekie/spring-kafka that referenced this issue Dec 8, 2024
@coekie
spring-projectsGH-3660: Fix EmbeddedKafkaBrokers.seekToEnd
8e0302b 
Problem:
consumeFromEmbeddedTopics calls Consumer.seekToEnd, which
"evaluates lazily, seeking to the final offset in all partitions only
when poll(Duration) or position(TopicPartition) are called".
This means that the consumer may or may not see future messages,
depending on timing.

This fix calls `position` so that the seek happens before
consumeFromEmbeddedTopics returns. That was it is ensured that any
message sent to the topic after the call to consumeFromEmbeddedTopics
are seen by the consumer.

Issue: spring-projects#3660
@coekie coekie mentioned this issue Dec 8, 2024
Merged
@sobychacko sobychacko added this to the 3.3.1 milestone Dec 10, 2024
coekie added a commit to coekie/spring-kafka that referenced this issue Dec 10, 2024
@coekie
spring-projectsGH-3660: Fix EmbeddedKafkaBroker seekToEnd
e47d829 
Problem:
consumeFromEmbeddedTopics calls Consumer.seekToEnd, which
"evaluates lazily, seeking to the final offset in all partitions only
when poll(Duration) or position(TopicPartition) are called".
This means that the consumer may or may not see future messages,
depending on timing.

This fix calls `position` so that the seek happens before
consumeFromEmbeddedTopics returns. That was it is ensured that any
message sent to the topic after the call to consumeFromEmbeddedTopics
are seen by the consumer.

Issue: spring-projects#3660
sobychacko pushed a commit that referenced this issue Dec 13, 2024
@coekie @sobychacko
GH-3660: Fix EmbeddedKafkaBroker seekToEnd (#3661)
71b571c 
Fixes: #3660

Problem: consumeFromEmbeddedTopics calls Consumer.seekToEnd, which
"evaluates lazily, seeking to the final offset in all partitions only
when poll(Duration) or position(TopicPartition) are called".
This means that the consumer may or may not see future messages,
depending on timing.

This fix calls `position` so that the seek happens before
consumeFromEmbeddedTopics returns. That was it is ensured that any
message sent to the topic after the call to consumeFromEmbeddedTopics
are seen by the consumer.

Issue: #3660
@spring-builds spring-builds mentioned this issue Dec 13, 2024
Closed
# for free to join this conversation on GitHub. Already have an account? # to comment
Assignees
No one assigned
Labels
for: backport-to-3.2.x status: backported type: bug
Projects
None yet
Milestone
3.3.1
Development

No branches or pull requests
3 participants
@coekie @sobychacko @spring-builds