node-rdkafka 3.2.0 Fails with "Invalid Version for API Key FETCH: 12" Error

[aryamohanan]

Environment Information

OS: Ubuntu 20.04.6 LTS
Node Version: v20
NPM Version: 10.9.0
node-rdkafka version: 3.2.0

Steps to Reproduce

1. Spin up a Kafka broker version 2.1.1 (I used Docker image wurstmeister/kafka:2.12-2.1.1)
2. Consume a topic using the latest node-rdkafka version (3.2.0).
3. On the Kafka broker, the exception is logged:

nodejs-kafka-1 | Caused by: java.lang.IllegalArgumentException: Invalid version for API key FETCH: 12
nodejs-kafka-1 | at org.apache.kafka.common.protocol.ApiKeys.schemaFor(ApiKeys.java:316)
nodejs-kafka-1 | at org.apache.kafka.common.protocol.ApiKeys.requestSchema(ApiKeys.java:286)
nodejs-kafka-1 | at org.apache.kafka.common.protocol.ApiKeys.parseRequest(ApiKeys.java:294)
nodejs-kafka-1 | at org.apache.kafka.common.requests.RequestContext.parseRequest(RequestContext.java:63)
nodejs-kafka-1 | at kafka.network.RequestChannel$Request.(RequestChannel.scala:89)
nodejs-kafka-1 | at kafka.network.Processor.$anonfun$processCompletedReceives$1(SocketServer.scala:858)
nodejs-kafka-1 | at kafka.network.Processor.$anonfun$processCompletedReceives$1$adapted(SocketServer.scala:840)
nodejs-kafka-1 | at scala.collection.Iterator.foreach(Iterator.scala:941)
nodejs-kafka-1 | at scala.collection.Iterator.foreach$(Iterator.scala:941)
nodejs-kafka-1 | at scala.collection.AbstractIterator.foreach(Iterator.scala:1429)
nodejs-kafka-1 | at scala.collection.IterableLike.foreach(IterableLike.scala:74)
nodejs-kafka-1 | at scala.collection.IterableLike.foreach$(IterableLike.scala:73)
nodejs-kafka-1 | at scala.collection.AbstractIterable.foreach(Iterable.scala:56)
nodejs-kafka-1 | at kafka.network.Processor.processCompletedReceives(SocketServer.scala:840)
nodejs-kafka-1 | at kafka.network.Processor.run(SocketServer.scala:731)
nodejs-kafka-1 | at java.lang.Thread.run(Thread.java:748)

After upgrading node-rdkafka from version 3.1.1 to 3.2.0, our tests began failing with an error.
The issue occurs with versions of Apache Kafka earlier than 2.7. I tested Kafka version 2.8.1, and it worked without any issues.

Additional context

Upon further investigation, I discovered that this issue is related to the version of librdkafka bundled with node-rdkafka 3.2.0. Specifically, the issue is fixed in librdkafka version 2.6.1 (as noted in librdkafka issue #4870), but node-rdkafka 3.2.0 currently uses librdkafka version 2.6.0, which does not include this fix.

Please upgrade the librdkafka dependency in node-rdkafka to version 2.6.1 to resolve this compatibility issue.

Thank you for maintaining this library!

[andrewstanovsky]

We faced the same error, trying to build 3.2.0 with rdkafka 2.6.1 fixed the issue, as was pointed out

[andrewstanovsky]

fixed in 3.2.1

[aryamohanan]

Release 3.2.1 has fixed the issue and is now working properly.