Skip to content
New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

Use Pool to store and reuse GZIPWriters #1329

Open
wants to merge 1 commit into
base: release-branch.v6
Choose a base branch
from

Conversation

trams
Copy link

@trams trams commented May 13, 2020

GZIPWriter are very expensive to create and leads
to allocating a lot of memory. They are perfect for reusing
(they have .reset method for this)

I profiled a web service inside my company and I noticed that creation of GZipWriter is responsible for 20% of memory allocation (in bytes).

Inspired by
https://www.akshaydeo.com/blog/2017/12/23/How-did-I-improve-latency-by-700-percent-using-syncPool/
https://dzone.com/articles/memory-pooling-in-go-where-why-and-how

Could you help me to run unit tests?

I noticed there is a shell script ./run-es.sh which worked after I did
sudo sysctl -w vm.max_map_count=262144 (my workstation is linux)

But go test . still fails for a lot of tests with

--- FAIL: TestSearchFilterPath (0.01s)
    setup_test.go:348: elastic: Error 400 (Bad Request): request [/elastic-test] contains unrecognized parameter: [include_type_name] [type=illegal_argument_exception]

I am quite puzzled

I also learned that I probably misuse defer (cause it is in the loop)
Current proposal with additional parameter looks a bit ugly too.
I will appreciate your feedback. This is my first pull request in go project (I am new to go)

GZIPWriter are very expensive to create and leads
to allocating a lot of memory. They are perfect for reusing
(they have .reset method for this)
# for free to join this conversation on GitHub. Already have an account? # to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant