Pre-calculate distance values #5251
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
6 tasks
danpaz
reviewed
Oct 29, 2018
danpat
force-pushed
the
danpat_cache_distances
branch
3 times, most recently
from
363eac2 to
ef302e8
Compare
ghoshkaj
approved these changes
Oct 30, 2018
danpaz
reviewed
Oct 30, 2018
… out what happened.
…ated distances on shortcuts, avoiding unpacking cost. Adds approx 10% to total data size. Speeds up large table requests by 2 orders of magnitude. Co-authored-by: Kajari Ghosh <ghoshkaj@gmail.com>
…t we can avoid accumulating distances if they're not requested.
danpat
force-pushed
the
danpat_cache_distances
branch
from
4135b0e to
a67c4bf
Compare
This was referenced Oct 30, 2018
# for free
to join this conversation on GitHub.
Already have an account?
# to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
After a long and sordid history, we added
annotations=distancesto thetableplugin back in #4990 and #5019These PRs implemented on-the-fly distance calculations - the table results are obtained, then the paths are unpacked to calculate the distances.
The upside to this is that it costs nothing in terms of resources (no extra preprocessing, no extra RAM) if you don't use this feature.
The downside is that it's a bit slow. With CH, large matrices that take 1-2 seconds to complete returning just durations can take upwards of 60 seconds to complete when distances are included.
@niemeier-PSI submitted #2764 way back which took the other approach - pre-calculate edge distances, and store them as an additional edge property. This approach is nice and fast, but has the downside that it requires quite a bit more RAM.
This PR is a re-implementation of #2764 - and I will open a follow-up ticket to make this data optional.
Tasklist