Skip to content
New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

Use buffer pools for decompression #2484

Merged
merged 1 commit into from
Jul 25, 2023
Merged

Conversation

ronanh
Copy link
Contributor

@ronanh ronanh commented Jul 4, 2023

While profiling my application, I've noticed that Sarama spends a significant amount of time growing slices when it's decompressing messages:

growSlice

This patch uses an intermediate buffer from a pool for decompression, to avoid these allocations.
When the final size is know, the result buffer can be allocated and copied.

After the patch:

withpool2

@ronanh
Copy link
Contributor Author

ronanh commented Jul 4, 2023

I have signed the CLA!

@dnwe dnwe force-pushed the improve_decompress branch 3 times, most recently from 13ec122 to 832e358 Compare July 17, 2023 11:15
@dnwe dnwe force-pushed the improve_decompress branch from 832e358 to c2e8186 Compare July 23, 2023 10:24
Signed-off-by: Ronan Harmegnies <ronan.harmegnies@gmail.com>
@dnwe dnwe force-pushed the improve_decompress branch from c2e8186 to cf5b296 Compare July 24, 2023 11:09
@hindessm
Copy link
Collaborator

Looks very good to me:

before:
BenchmarkDecompression/lz4_2048_compressable-4          578866    9368 ns/op    8533 B/op   11 allocs/op
BenchmarkDecompression/lz4_2048_lessCompressable-4      830926    6521 ns/op    8506 B/op   11 allocs/op
BenchmarkDecompression/lz4_2048_compressable-16         804277    7391 ns/op    8544 B/op   11 allocs/op
BenchmarkDecompression/lz4_2048_lessCompressable-16    1221556    4904 ns/op    8538 B/op   11 allocs/op

after:
BenchmarkDecompression/lz4_2048_compressable-4          792148    7512 ns/op    2373 B/op    7 allocs/op
BenchmarkDecompression/lz4_2048_lessCompressable-4     1302768    4614 ns/op    2395 B/op    7 allocs/op
BenchmarkDecompression/lz4_2048_compressable-16         920294    5854 ns/op    2364 B/op    7 allocs/op
BenchmarkDecompression/lz4_2048_lessCompressable-16    1544532    3909 ns/op    2420 B/op    7 allocs/op

@dnwe dnwe merged commit 735f33b into IBM:main Jul 25, 2023
@dnwe dnwe added the feat label Jul 25, 2023
ioanzicu pushed a commit to ioanzicu/sarama that referenced this pull request Jul 31, 2023
Signed-off-by: Ronan Harmegnies <ronan.harmegnies@gmail.com>
Signed-off-by: Ioan Zicu <ioan.zicu@nokia.com>
# for free to join this conversation on GitHub. Already have an account? # to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants