Skip to content
New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

Multi-Region: Conflicting exporter names result in lost data #82

Open
lenaschoenburg opened this issue Jan 24, 2024 · 1 comment · May be fixed by #71
Open

Multi-Region: Conflicting exporter names result in lost data #82

lenaschoenburg opened this issue Jan 24, 2024 · 1 comment · May be fixed by #71
Labels
bug Something isn't working

Comments

@lenaschoenburg
Copy link

The Multi-Region profiles configure each region's Zeebe to use their local ES as exporter named "elasticsearch" and the remote ES as exporter named "elasticsearch2".

As a result, Zeebe will distribute conflicting metadata about the exporters. Zeebe brokers from region 0 share something like es1={pos=15, sequence=10},es2={pos=30, sequence=27} while Zeebe brokers from region 1 flip it and distribute es1={pos=30, sequence=27},es2={pos=15, sequence=10}.

Depending on the ordering in which updates are distributed, some leaders will run with incorrect metadata which can result in missing data in ES (i.e. dataloss) or wrong sequence numbers (i.e. data corruption).

@lenaschoenburg lenaschoenburg added the bug Something isn't working label Jan 24, 2024
@hamza-m-masood
Copy link

Thanks for creating the issue. Here is the fix within this commit

@falko falko linked a pull request Jan 26, 2024 that will close this issue
34 tasks
# for free to join this conversation on GitHub. Already have an account? # to comment
Labels
bug Something isn't working
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants