Skip to content

re_datastore: properly handle already sorted data during insertion #433

New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

#

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

Jump to bottom
Closed
Tracked by #443
, #1898
...
teh-cmc opened this issue Dec 2, 2022 · 0 comments · Fixed by #3088
Closed
Tracked by #443
, #1898
...

re_datastore: properly handle already sorted data during insertion #433

teh-cmc opened this issue Dec 2, 2022 · 0 comments · Fixed by #3088
Labels
🏹 arrow Apache Arrow enhancement New feature or request ⛃ re_datastore affects the datastore itself
Milestone
0.8.2

Comments

@teh-cmc
Copy link
Member

teh-cmc commented Dec 2, 2022

Do not mark an index as dirty (unsorted) when newly inserted data happens to already be
time-ordered.

  • this is the happy path and in fact most common path: data arrives in time order, and so
    inserting it at the end of the bucket doesn't actually result in the bucket becoming
    unsorted
@teh-cmc teh-cmc added enhancement New feature or request 🏹 arrow Apache Arrow ⛃ re_datastore affects the datastore itself labels Dec 2, 2022
@teh-cmc teh-cmc mentioned this issue Dec 12, 2022
Closed
11 tasks
@teh-cmc teh-cmc assigned teh-cmc and unassigned teh-cmc Feb 28, 2023
@teh-cmc teh-cmc mentioned this issue Apr 18, 2023
Closed
23 tasks
@teh-cmc teh-cmc mentioned this issue Apr 26, 2023
Closed
@emilk emilk mentioned this issue Aug 23, 2023
Closed
@emilk emilk added this to the 0.8.2 milestone Aug 23, 2023
@emilk emilk mentioned this issue Aug 23, 2023
Merged
3 tasks
emilk added a commit that referenced this issue Aug 23, 2023
@emilk
Fix quadratic slowdown when ingesting data with uniform time (#3088)
4bc05e5 
### What
* Closes #3086
* Closes #433

This should also overall just speed up data insertion for the common
case of already-sorted data

### Checklist
* [x] I have read and agree to [Contributor
Guide](https://github.com/rerun-io/rerun/blob/main/CONTRIBUTING.md) and
the [Code of
Conduct](https://github.com/rerun-io/rerun/blob/main/CODE_OF_CONDUCT.md)
* [x] I've included a screenshot or gif (if applicable)
* [x] I have tested [demo.rerun.io](https://demo.rerun.io/pr/3088) (if
applicable)

- [PR Build Summary](https://build.rerun.io/pr/3088)
- [Docs
preview](https://rerun.io/preview/e5adb1aa580de2274b4eca9f6c5de38ae503b521/docs)
<!--DOCS-PREVIEW-->
- [Examples
preview](https://rerun.io/preview/e5adb1aa580de2274b4eca9f6c5de38ae503b521/examples)
<!--EXAMPLES-PREVIEW--><!--EXAMPLES-PREVIEW-->
- [Recent benchmark results](https://ref.rerun.io/dev/bench/)
- [Wasm size tracking](https://ref.rerun.io/dev/sizes/)
jleibs pushed a commit that referenced this issue Aug 31, 2023
@emilk @jleibs
Fix quadratic slowdown when ingesting data with uniform time (#3088)
62b5061 
### What
* Closes #3086
* Closes #433

This should also overall just speed up data insertion for the common
case of already-sorted data

### Checklist
* [x] I have read and agree to [Contributor
Guide](https://github.com/rerun-io/rerun/blob/main/CONTRIBUTING.md) and
the [Code of
Conduct](https://github.com/rerun-io/rerun/blob/main/CODE_OF_CONDUCT.md)
* [x] I've included a screenshot or gif (if applicable)
* [x] I have tested [demo.rerun.io](https://demo.rerun.io/pr/3088) (if
applicable)

- [PR Build Summary](https://build.rerun.io/pr/3088)
- [Docs
preview](https://rerun.io/preview/e5adb1aa580de2274b4eca9f6c5de38ae503b521/docs)
<!--DOCS-PREVIEW-->
- [Examples
preview](https://rerun.io/preview/e5adb1aa580de2274b4eca9f6c5de38ae503b521/examples)
<!--EXAMPLES-PREVIEW--><!--EXAMPLES-PREVIEW-->
- [Recent benchmark results](https://ref.rerun.io/dev/bench/)
- [Wasm size tracking](https://ref.rerun.io/dev/sizes/)
jleibs pushed a commit that referenced this issue Aug 31, 2023
@emilk @jleibs
Fix quadratic slowdown when ingesting data with uniform time (#3088)
c7ea722 
### What
* Closes #3086
* Closes #433

This should also overall just speed up data insertion for the common
case of already-sorted data

### Checklist
* [x] I have read and agree to [Contributor
Guide](https://github.com/rerun-io/rerun/blob/main/CONTRIBUTING.md) and
the [Code of
Conduct](https://github.com/rerun-io/rerun/blob/main/CODE_OF_CONDUCT.md)
* [x] I've included a screenshot or gif (if applicable)
* [x] I have tested [demo.rerun.io](https://demo.rerun.io/pr/3088) (if
applicable)

- [PR Build Summary](https://build.rerun.io/pr/3088)
- [Docs
preview](https://rerun.io/preview/e5adb1aa580de2274b4eca9f6c5de38ae503b521/docs)
<!--DOCS-PREVIEW-->
- [Examples
preview](https://rerun.io/preview/e5adb1aa580de2274b4eca9f6c5de38ae503b521/examples)
<!--EXAMPLES-PREVIEW--><!--EXAMPLES-PREVIEW-->
- [Recent benchmark results](https://ref.rerun.io/dev/bench/)
- [Wasm size tracking](https://ref.rerun.io/dev/sizes/)
jleibs pushed a commit that referenced this issue Aug 31, 2023
@emilk @jleibs
Fix quadratic slowdown when ingesting data with uniform time (#3088)
1f7537d 
### What
* Closes #3086
* Closes #433

This should also overall just speed up data insertion for the common
case of already-sorted data

### Checklist
* [x] I have read and agree to [Contributor
Guide](https://github.com/rerun-io/rerun/blob/main/CONTRIBUTING.md) and
the [Code of
Conduct](https://github.com/rerun-io/rerun/blob/main/CODE_OF_CONDUCT.md)
* [x] I've included a screenshot or gif (if applicable)
* [x] I have tested [demo.rerun.io](https://demo.rerun.io/pr/3088) (if
applicable)

- [PR Build Summary](https://build.rerun.io/pr/3088)
- [Docs
preview](https://rerun.io/preview/e5adb1aa580de2274b4eca9f6c5de38ae503b521/docs)
<!--DOCS-PREVIEW-->
- [Examples
preview](https://rerun.io/preview/e5adb1aa580de2274b4eca9f6c5de38ae503b521/examples)
<!--EXAMPLES-PREVIEW--><!--EXAMPLES-PREVIEW-->
- [Recent benchmark results](https://ref.rerun.io/dev/bench/)
- [Wasm size tracking](https://ref.rerun.io/dev/sizes/)
# for free to join this conversation on GitHub. Already have an account? # to comment
Assignees
No one assigned
Labels
🏹 arrow Apache Arrow enhancement New feature or request ⛃ re_datastore affects the datastore itself
Projects
None yet
Milestone
0.8.2
Development

Successfully merging a pull request may close this issue.

Fix quadratic slowdown when ingesting data with uniform time rerun-io/rerun
2 participants
@emilk @teh-cmc