Skip to content
New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

[no-release-notes] Add Archive Backup and AWS support #8892

Merged
merged 16 commits into from
Feb 27, 2025

Conversation

macneale4
Copy link
Contributor

These changes enable Dolt databases containing archive storage to perform file based backups/restores, and AWS backups/restores, push, pull, and clone operations.

Currently all incremental pushes and fetches produce table files, not archives. The use of backup
allows us to get archive files pushed into S3 for testing.

@coffeegoddd
Copy link
Contributor

@macneale4 DOLT

comparing_percentages
100.000000 to 100.000000
version result total
87b0454 ok 5937457
version total_tests
87b0454 5937457
correctness_percentage
100.0

@macneale4 macneale4 requested a review from reltuk February 26, 2025 17:37
@coffeegoddd
Copy link
Contributor

@macneale4 DOLT

comparing_percentages
100.000000 to 100.000000
version result total
9d74c21 ok 5937457
version total_tests
9d74c21 5937457
correctness_percentage
100.0

Copy link
Contributor

@reltuk reltuk left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Followed up offline asking to get rid of flexibleTableReader and make archiveReader carry a tableReaderAt instead. Some things fall out more nicely as a result.

Copy link
Contributor

@reltuk reltuk left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. One suggestion about copyright header hygiene.

@coffeegoddd
Copy link
Contributor

@coffeegoddd DOLT

comparing_percentages
100.000000 to 100.000000
version result total
99df522 ok 5937457
version total_tests
99df522 5937457
correctness_percentage
100.0

@coffeegoddd
Copy link
Contributor

@macneale4 DOLT

comparing_percentages
100.000000 to 100.000000
version result total
8b4f72e ok 5937457
version total_tests
8b4f72e 5937457
correctness_percentage
100.0

@coffeegoddd
Copy link
Contributor

@macneale4 DOLT

comparing_percentages
100.000000 to 100.000000
version result total
2e9862a ok 5937457
version total_tests
2e9862a 5937457
correctness_percentage
100.0

@macneale4 macneale4 merged commit 529fbe7 into main Feb 27, 2025
21 checks passed
@macneale4 macneale4 deleted the macneale4/archive-aws branch February 27, 2025 00:43
Copy link

@coffeegoddd DOLT

test_name detail row_cnt sorted mysql_time sql_mult cli_mult
batching LOAD DATA 10000 1 0.07 1.29
batching batch sql 10000 1 0.09 1.33
batching by line sql 10000 1 0.1 1.2
blob 1 blob 200000 1 0.92 3.88 4.6
blob 2 blobs 200000 1 0.91 4.44 4.85
blob no blob 200000 1 0.92 2.46 2.84
col type datetime 200000 1 0.86 2.37 2.73
col type varchar 200000 1 0.72 3.39 3.72
config width 2 cols 200000 1 0.78 2.63 2.96
config width 32 cols 200000 1 1.91 2.04 2.74
config width 8 cols 200000 1 0.96 2.48 2.78
pk type float 200000 1 0.84 2.5 2.8
pk type int 200000 1 0.94 2.3 2.54
pk type varchar 200000 1 2.76 0.95 1.04
row count 1.6mm 1600000 1 5.75 2.98 2.99
row count 400k 400000 1 1.59 2.62 2.74
row count 800k 800000 1 2.95 2.87 2.99
secondary index four index 200000 1 3.58 1.43 1.31
secondary index no secondary 200000 1 0.95 2.4 2.78
secondary index one index 200000 1 1.15 2.46 2.52
secondary index two index 200000 1 2.02 1.8 1.86
sorting shuffled 1mm 1000000 0 5.34 2.73 2.55
sorting sorted 1mm 1000000 1 5.24 2.77 2.58

Copy link

@coffeegoddd DOLT

name detail mean_mult
dolt_blame_basic system table 1.21
dolt_blame_commit_filter system table 2.99
dolt_commit_ancestors_commit_filter system table 0.65
dolt_commits_commit_filter system table 1.11
dolt_diff_log_join_from_commit system table 2.83
dolt_diff_log_join_to_commit system table 2.78
dolt_diff_table_from_commit_filter system table 1.2
dolt_diff_table_to_commit_filter system table 1.16
dolt_diffs_commit_filter system table 1.03
dolt_history_commit_filter system table 1.31
dolt_log_commit_filter system table 1.16

Copy link

@coffeegoddd DOLT

name add_cnt delete_cnt update_cnt latency
adds_only 60000 0 0 1.14
adds_updates_deletes 60000 60000 60000 4.57
deletes_only 0 60000 0 2.45
updates_only 0 0 60000 3.06

# for free to join this conversation on GitHub. Already have an account? # to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants