Skip to content
New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

#

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

Jump to bottom

Big value serialization in migration #4100

Open
adiholden opened this issue Nov 10, 2024 · 0 comments · May be fixed by #4119
Open

Big value serialization in migration #4100

adiholden opened this issue Nov 10, 2024 · 0 comments · May be fixed by #4119
Assignees
@chakaz
Milestone
dfly cluster v4

Comments

@adiholden
Copy link
Collaborator

No description provided.

@adiholden adiholden added this to the dfly cluster v4 milestone Nov 10, 2024
chakaz added a commit that referenced this issue Nov 12, 2024
@chakaz
chore: Reduce memory consumption when migrating huge values
3fb8b46 
Before this PR:

We serialized a `RESTORE` command for each entry into a string, and then
push that string to the wire.
This means that, when serializing an entry of size X, we consume 2X
memory during the migration.

This PR:

Instead of serializing into a string, we serialize into the wire
directly. Luckily, only a small modification is needed in the way we
interact with `crc64`, which works well even in chunks.

Fixes #4100
@chakaz chakaz linked a pull request Nov 12, 2024 that will close this issue
Open
# for free to join this conversation on GitHub. Already have an account? # to comment
Assignees

@chakaz chakaz

Labels
None yet
Projects
None yet
Milestone
dfly cluster v4
Development

Successfully merging a pull request may close this issue.

chore: Reduce memory consumption when migrating huge values dragonflydb/dragonfly
2 participants
@chakaz @adiholden