New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

#

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

Jump to bottom

Duplicate message when partition revoking or rebalancing #2316

Open

doubleverify-measurement opened this issue Aug 15, 2022 · 1 comment

doubleverify-measurement commented Aug 15, 2022

Hi,
We see a lot of duplicates message exactly when Re-balancing occurs.

When rebalancing partitions are revoked and redistributed
Trim files occurring
Duplicate message at this moment

do you've any idea how to troubleshoot this behavior ? it seems that instead of trimming files it make more files

We use

confluent last version
Secor deployed on kubernetes
Run on Google cloud write files on GCS bucket

Partition Revoking events

Trimming file events

Number of files uploaded watch out the picks on exact time trim files occurs

Thanks for your help

ambud commented Oct 18, 2022

Are you using transactions in Kafka? if not consumer will be replaying messages.

# for free to join this conversation on GitHub. Already have an account? # to comment