Skip to content

re_datastore: index bucket compaction #512

New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

Closed
3 tasks
Tracked by #1898
teh-cmc opened this issue Dec 9, 2022 · 1 comment
Closed
3 tasks
Tracked by #1898

re_datastore: index bucket compaction #512

teh-cmc opened this issue Dec 9, 2022 · 1 comment
Labels
⛃ re_datastore affects the datastore itself

Comments

@teh-cmc
Copy link
Member

teh-cmc commented Dec 9, 2022

(mentioned by @jondo2010 during retro)

Compact sparse index buckets (because they were filled on a per-component basis rather than multiple components at a time).

Basically automatically going from this

┌──────┬───────────┬───────┬───────────┐
│ time ┆ instances ┆ rects ┆ positions │
│ ---  ┆ ---       ┆ ---   ┆ ---       │
│ str  ┆ u64       ┆ u64   ┆ u64       │
╞══════╪═══════════╪═══════╪═══════════╡
│ #42  ┆ null      ┆ 1     ┆ null      │
├╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌┤
│ #42  ┆ 3         ┆ null  ┆ null      │
├╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌┤
│ #42  ┆ null      ┆ null  ┆ 2         │
└──────┴───────────┴───────┴───────────┘

to this

┌──────┬───────────┬───────┬───────────┐
│ time ┆ instances ┆ rects ┆ positions │
│ ---  ┆ ---       ┆ ---   ┆ ---       │
│ str  ┆ u64       ┆ u64   ┆ u64       │
╞══════╪═══════════╪═══════╪═══════════╡
│ #42  ┆ 3         ┆ 1     ┆ 2         │
└──────┴───────────┴───────┴───────────┘

TODO:

  • decide when (on write? on read? both? neither?)
  • implement
  • test
@teh-cmc
Copy link
Member Author

teh-cmc commented Oct 19, 2024

Doesn't make much sense in the new chunk based model.

It does make sense to do in the range cache, but it is in fact already done there.

@teh-cmc teh-cmc closed this as not planned Won't fix, can't repro, duplicate, stale Oct 19, 2024
# for free to join this conversation on GitHub. Already have an account? # to comment
Labels
⛃ re_datastore affects the datastore itself
Projects
None yet
Development

No branches or pull requests

1 participant