Skip to content
New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

added 'Kafka tiny client' #551

Open
wants to merge 1 commit into
base: master
Choose a base branch
from
Open

added 'Kafka tiny client' #551

wants to merge 1 commit into from

Conversation

sqlly
Copy link

@sqlly sqlly commented Apr 14, 2020

I've created a most lightweight implementation of Kafka binary protocol for Espruino. Kafka is now almost de-facto standard for stream messaging and Espruino definitely lack such implementation.

@gfwilliams
Copy link
Member

This is great, thanks! Does the crc32 code you have there actually work?

On new Espruino builds (cutting edge, and 2v05 when released) there's actually a E.CRC32 function which would probably help out: http://www.espruino.com/Reference#l_E_CRC32

If you need it I also have an implementation that doesn't need the table...

@sqlly
Copy link
Author

sqlly commented Apr 16, 2020

Probably my Espruino build was not so fresh (actually I've created that code 3 months ago) and at that time I've found that build-in CRC32 value does not equal to what Kafka expects. After a deep dive to that I've found working solution.
As for implementation that does not need a table - I've tried one with dynamic CRC table generation - it was extremly slow. If your version is really fast, I will be glad to remove CRC table, it is very resource expensive.

@gfwilliams
Copy link
Member

This should do it hopefully:

function CRC32(data) {
  var crc = 0xFFFFFFFF;
  data.forEach(function(d) {
    crc^=d;
    crc=(crc>>>1)^(0xEDB88320&-(crc&1));
    crc=(crc>>>1)^(0xEDB88320&-(crc&1));
    crc=(crc>>>1)^(0xEDB88320&-(crc&1));
    crc=(crc>>>1)^(0xEDB88320&-(crc&1));
    crc=(crc>>>1)^(0xEDB88320&-(crc&1));
    crc=(crc>>>1)^(0xEDB88320&-(crc&1));
    crc=(crc>>>1)^(0xEDB88320&-(crc&1));
    crc=(crc>>>1)^(0xEDB88320&-(crc&1));
  });
  return ~crc;
}
print(CRC32([2,4,65,7,5,4,3,5,6]))

# for free to join this conversation on GitHub. Already have an account? # to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants