Skip to content

Rebatch aggressively while dumping the store to a stream of LogMsg #1894

New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

#

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

Jump to bottom
Closed
Tracked by #1619
, #1898
...
teh-cmc opened this issue Apr 18, 2023 · 1 comment · Fixed by #6570
Closed
Tracked by #1619
, #1898
...

Rebatch aggressively while dumping the store to a stream of LogMsg #1894

teh-cmc opened this issue Apr 18, 2023 · 1 comment · Fixed by #6570
Assignees
@teh-cmc
Labels
📉 performance Optimization, memory use, etc ⛃ re_datastore affects the datastore itself

Comments

@teh-cmc
Copy link
Member

teh-cmc commented Apr 18, 2023 •
edited by emilk
Loading

The save-feature walks the store and converts each bucket back into a DataTable that is then serialized and dumped as a LogMsg.

This is a great opportunity to batch things even further.
@teh-cmc teh-cmc mentioned this issue Apr 18, 2023
Closed
18 tasks
@teh-cmc teh-cmc changed the title Rebatch aggressively while dumping to disk Rebatch aggressively while dumping the store to a stream of LogMsg Apr 18, 2023
@teh-cmc teh-cmc added ⛃ re_datastore affects the datastore itself 📉 performance Optimization, memory use, etc labels Apr 18, 2023
This was referenced Apr 18, 2023
Closed
Closed
@teh-cmc
Copy link
Member Author

teh-cmc commented Nov 7, 2023 •
edited
Loading

More importantly, we need the save process to be RowId-driven so that we can forbid re-use of RowIds once and for all, which will remove the need to count unique RowIds for downstream subscribers of store events, which is a blocker for the StoreView/StoreEvent stuff.

@teh-cmc teh-cmc mentioned this issue Nov 10, 2023
Closed
@teh-cmc teh-cmc mentioned this issue Dec 11, 2023
Closed
@teh-cmc teh-cmc mentioned this issue Dec 22, 2023
Closed
@teh-cmc teh-cmc self-assigned this May 16, 2024
@teh-cmc teh-cmc mentioned this issue Jun 18, 2024
Merged
5 tasks
@teh-cmc teh-cmc closed this as completed in ed20cba Jul 4, 2024
# for free to join this conversation on GitHub. Already have an account? # to comment
Assignees

@teh-cmc teh-cmc

Labels
📉 performance Optimization, memory use, etc ⛃ re_datastore affects the datastore itself
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Implement ChunkStore and integrate it everywhere rerun-io/rerun
1 participant
@teh-cmc