feat: Restore zstd-compressed database backup #273
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
dlipovetsky
force-pushed
the
restore-compressed
branch
from
7b3ff2d to
f6b85ed
Compare
|
I removed an unused function, and force-pushed. |
The database backup is compressed using zstd. This allows us to load the backup without having to decompress it beforehand.
dlipovetsky
force-pushed
the
restore-compressed
branch
from
f6b85ed to
9466d72
Compare
dlipovetsky
commented
Apr 22, 2024
Comment on lines
+29
to
+40
| err = db.Load(zr, runtime.NumCPU()) | ||
| if errors.Is(err, zstd.ErrMagicMismatch) { | ||
| glog.V(2).Infof("database file is not compressed with zstd, will load without decompressing") | ||
|
|
||
| // We already loaded the database, advancing the file offset. To load the database again, | ||
| // we must reset the offset to the start. | ||
| if _, err := file.Seek(0, io.SeekStart); err != nil { | ||
| return errors.Wrapf(err, "failed to to rewind to start of database restore file: %q", filename) | ||
| } | ||
| err = db.Load(file, runtime.NumCPU()) | ||
| } | ||
| if err != nil { |
There was a problem hiding this comment.
I wanted to check if the file is compressed or not before loading the database, but could not find a way to do that using the zstd package. Attempting to load the database is the best alternative I found.
|
CI failure appears to be unrelated to this change. |
# for free
to join this conversation on GitHub.
Already have an account?
# to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
The database backup is compressed using zstd. This allows us to load the backup without having to decompress it beforehand.
Please note that this change is backward compatible: both compressed and uncompressed backups are supported. I have added an integration test (using an on-disk database) to ensure this.