[Bug]: Cannot read from Kafka due to short poll timeout of consumer in KafkaIO #30870
Open
1 of 16 tasks
Comments
|
.take-issue |
|
This also affects the Python SDK. |
xianhualiu
added a commit
to xianhualiu/beam
that referenced
this issue
Apr 9, 2024
…ansion service and Python
xianhualiu
added a commit
to xianhualiu/beam
that referenced
this issue
Apr 10, 2024
apache#30870: support consumer polling timeout in KafkaIO expansion service
xianhualiu
added a commit
to xianhualiu/beam
that referenced
this issue
Apr 10, 2024
damccorm
pushed a commit
that referenced
this issue
Apr 16, 2024
…ce (#30915) * [#30870]: support consumer polling timeout in KafkaIO expansion service * fixed spotless complains * fixed python format complains * Update sdks/python/apache_beam/io/kafka.py Co-authored-by: Jonathan Sabbagh <108473809+jbsabbagh@users.noreply.github.com> * Update sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaIO.java Co-authored-by: Jonathan Sabbagh <108473809+jbsabbagh@users.noreply.github.com> * Update sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaIO.java Co-authored-by: Jonathan Sabbagh <108473809+jbsabbagh@users.noreply.github.com> * fixed formating issue * fixed pylint and pydoc issues * shorten the variable name * fixed format and upgrade test * fixed test * fixed test --------- Co-authored-by: Jonathan Sabbagh <108473809+jbsabbagh@users.noreply.github.com>
Abacn
added a commit
to Abacn/beam
that referenced
this issue
Apr 16, 2024
…xpansion service (apache#30915)" This reverts commit a44c4f1.
# for free
to join this conversation on GitHub.
Already have an account?
# to comment
What happened?
The default Kafka consumer poll timeout is set to 1 second. It works fine when the the response can get messages from the kafka broker server within this 1 second, such as when client accesses broker within the same region. But if the responding time is more than 1 second, the consumer will not retrieve any messages. One customer reported that throughput of processed messages was extremely low in cross-region read since most of the time the responding time takes more than 1 second.
As a solution, the Kafka consumer polling timeout needs to be configurable, so customer can adjust it according to their needs.
Issue Priority
Priority: 2 (default / most bugs should be filed as P2)
Issue Components
The text was updated successfully, but these errors were encountered: