Skip to content
New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

#

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

Jump to bottom

DB archival can cause slowness in full reload over time #154

Open
justjkk opened this issue Mar 5, 2021 · 2 comments
Open

DB archival can cause slowness in full reload over time #154

justjkk opened this issue Mar 5, 2021 · 2 comments
Labels
enhancement New feature or request performance Monitoring, Metrics, Logs, Benchmarks

Comments

@justjkk
Copy link
Contributor

justjkk commented Mar 5, 2021

Multiple tables like communicator.genericsms archive data older than a few months to S3 in order to save space. Over time this can cause more entries in Kafka that slows down a full reload. Need to figure out a way to periodically refresh the Debezium data so that data is stored as snapshots rather than events.
@justjkk
Copy link
Contributor Author

justjkk commented Mar 5, 2021

This issue can also happen due to some tables that are doing a lot of updates like sessions table last_accessed_at column.

@justjkk justjkk added enhancement New feature or request performance Monitoring, Metrics, Logs, Benchmarks labels Mar 5, 2021
@alok87
Copy link
Contributor

alok87 commented Jul 29, 2021

Log compaction can be used here.
It can help keep only the latest record for key. @justjkk

# for free to join this conversation on GitHub. Already have an account? # to comment
Assignees
No one assigned
Labels
enhancement New feature or request performance Monitoring, Metrics, Logs, Benchmarks
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@justjkk @alok87