Merge block databases #2829
Merged
Merge block databases #2829
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
wezrule
added
documentation
wezrule
added
database structure
wezrule
added
the
blocker
|
Had a bootstrap in progress, stopped and upgraded the database with this PR (~21M block count, 30M unchecked) on Ryzen 3600, 16GB RAM and NVME SSD: Though painful the benefits are significant. I agree with vacuuming pre-upgrade, possibly even a rebuild as we've seen that speeds up upgrades considerably and reduces the maximum size reached during the upgrade. @zhyatt Should this have the removal label instead of semantic? |
zhyatt
added
removal
|
@guilhermelawless Yes, just swapped out the labels. |
|
Windows 10 - SSD (NVME), Ryzen 3700X, 64GB RAM (11 minutes) I did some experimenting with a rebuild vacuum before the upgrade and afterwards. After first rebuild |
guilhermelawless
previously approved these changes
Jul 17, 2020
|
Ubuntu 20.04 - NVMe SSD Samsung 970 EVO Plus 512GB, Ryzen 3900X, 64GB RAM (5 mins) |
SergiySW
approved these changes
Jul 24, 2020
|
Ubuntu 20.04 - NVMe Optane, Ryzen 3900X, 64GB RAM (2 mins) |
# for free
to join this conversation on GitHub.
Already have an account?
# to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
send/change/open/receive&stateblock databases have been removed. They are merged into ablocksdatabase, and are serialized as follows:block_type -> block -> sidebandThe
block_typeis a new addition which allows the block and sideband to be correctly deserialized. There isblock_detailsinsidebandwhich could be used, but it would require shrinking the number of bits for theepochand alot more other interface changes which didn't seem worth it, an extra byte for each block is used instead for theblock_type. We already follow this approach for serializing blocks in unchecked for instance so it allowed code re-use there too.Removes
block_count_typeRPC as it's pretty useless now as there's no distinction possible in LMDB without extra IO to store the counts. This is possible with RocksDB as we are storing the count anyway, but I think having consistent RPC interface is more important.The upgrade path is interesting because we do not have a way to merge 2 databases with different value types. For this I used in-memory sorting of smallish databases (legacy open/receive/change) first and then creating temporary databases for that and the send/state blocks which added the new value type (extra
block_type). The smallest databases were then merged before the larger ones to reduce as much iteration.The benefits this provides, will mean RocksDB does not need to worry about stale memtables anymore. Checking if a block exists sometimes took up to 5 database reads, now only 1 is required. It has also reduced complexity in various areas.
The database upgrade was tested on a few systems, logging info shown below:
Windows 10 - SSD, Ryzen 3700X, 64GB RAM (20 minutes)
Windows 10 - SSD (NVME), Ryzen 3700X, 64GB RAM (11 minutes)
Ubuntu - SSD, Ryzen 2600, 16GB RAM (19 mins)
The starting ledger was 36GB, unvacuumed after upgrade it becomes 61GB, vacuumed 22GB. Currently it is set up to automatically vacuum after upgrade, however this might be difficult for some users with storage constraints, perhaps we should make this step optional, or also vacuum the pre-upgraded ledger first?
Also did some benchmarking of LMDB/RocksDB performance & ledger size when using fixed and variable sized keys:
Didn't see any real difference between fixed and variable sized values from LMDB or RocksDB, or a difference in ledger size. All
getswere recorded after a computer restart to prevent any OS caching affected the results.