-
Notifications
You must be signed in to change notification settings - Fork 1.4k
New issue
Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? # to your account
Use hardware accelerated CRC32C function if available #1389
Conversation
Oh interesting. Two thoughts:
|
@jeffwidman Yea, will patch the docs. |
@@ -70,3 +70,16 @@ Install the `python-snappy` module | |||
.. code:: bash | |||
|
|||
pip install python-snappy | |||
|
|||
|
|||
Optional crc32c install |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@jeffwidman Could you read this through, good enough?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Looks great
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM, I held off on merging this @tvoinarovskyi in case you want to make any more changes first... when you're ready please press the green button.
@@ -70,3 +70,16 @@ Install the `python-snappy` module | |||
.. code:: bash | |||
|
|||
pip install python-snappy | |||
|
|||
|
|||
Optional crc32c install |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Looks great
@dpkp @jeffwidman Hehe, I'm an idiot for this =) So, here I was pushing my butt to accelerate varint implementation, but did not look properly at the profiler.
Long story short, while accelerating the V2 implementation for
aiokafka
I found that_crc32c.py
is super slow and takes more time for 1 call than 100append
. So as a fast fix I propose to use https://github.com/ICRAR/crc32c if available on the system. I will look into_crc32c.py
script later to make it a bit faster.Benchmark without HW crc32:
With HW crc32:
So now V2 is relatively the same speed as V1 and V0. Now we can move on to TransactionalProducer =)