Skip to content
New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

Decompression speed improvements #41

Open
mhluska opened this issue Aug 31, 2023 · 0 comments
Open

Decompression speed improvements #41

mhluska opened this issue Aug 31, 2023 · 0 comments

Comments

@mhluska
Copy link
Contributor

mhluska commented Aug 31, 2023

Currently when loading a compressed solution file, it takes very long because zstd doesn't support multi-threaded decompression. Only compression.

One way around this would be to instead serialize into multiple files instead of one large file. Particularly the Vec[u8] data from storage1 and node_arena which can then be deserialized in parallel into a single vector in memory.

It looks like pzstd uses a similar strategy.

I can submit a PR if the library author is interested. I have something working now which looks promising:

  • loading a 6GB solution compressed with zstd level 3: 10s
  • loading the same solution split into file chunks compressed with zstd level 3, decompressed in parallel: 3.9s
# for free to join this conversation on GitHub. Already have an account? # to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant