Add support for KAFKA-4148 / KIP-79 / search offsets by timestamp in consumer

[anthonyjchriste]

I've looked all around, but I can't seem to find if this library supports Kafka 10.1 or greater yet. I'm specifically trying to access offsetsForTimes as described in https://kafka.apache.org/0102/javadoc/org/apache/kafka/clients/consumer/KafkaConsumer.html#offsetsForTimes(java.util.Map)

This method appears to have been added in Kafka 10.1.

I did not see the method mentioned under the KafkaConsumer for the API docs at http://kafka-python.readthedocs.io/en/master/apidoc/KafkaConsumer.html.

I also looked through the KafkaConsumer class in the master branch, but did not see it there either.

Is it safe to assume that 10.1 is not supported yet or am I just looking in the wrong place?

Thank you for your time and I look forward to your response.

[dpkp]

Looks like you're looking for KIP-79. The new api is available and supported at the protocol level, but no one has added an equivalent to the KafkaConsumer.offsetsForTimes method. I renamed the issue to flag that this is needed.

[anthonyjchriste]

That's exactly it.

Thank you for renaming the topic. I really appreciate the work you (and others) put into this project.

[ZmeiGorynych]

I was looking for the very same thing :)

Looking at https://github.com/dpkp/kafka-python/blob/master/CHANGES.md, under March 14 I see a line saying 'Add send_list_offset_request for searching offset by timestamp (charsyam 1001)' - is that not the same task, and if it is, does that mean it's done? And if so, how do I use it? :)

Thanks a lot for any update you can give.

[tvoinarovskyi]

@ZmeiGorynych No, that PR is about legacy code, no relation to timestamp search. As far as I know timestamp search is yet to be implemented.

[anthonyjchriste]

I'm interested in attempting to implement this feature. Is there any developer documentation and are there any examples in the code that show the best (or recommended) way to implement a public API method by utilizing the protocol? Thank you for your time.

[dpkp]

This python code still follows the java code fairly closely, so a reasonable first step is to just take a look at the java implementation. The java code is KAFKA-4148 / apache/kafka@eaaa433f

[anthonyjchriste]

Mahalo @dpkp! I'll fork and give it a go over the next few weeks.

[Xaelias]

That would be awesome to have :-) Thanks guys!

[anthonyjchriste]

I noticed this was closed with a reference to another issue and the other issue references this issue and is also closed. I'm curious to know if the closed status means KIP-79 is now officially supported in kafka-python or if there are any plans to implement it?

[dpkp]

This issue is still open. The one that closed implemented the lower level protocol apis necessary to support this feature.

[anthonyjchriste]

My apologies. I was confused when I saw the closed tag next to the referenced issue and mistakenly thought it for this issue.

[tvoinarovskyi]

@anthonyjchriste Now there's no mistake in that merged tag above )

[jeffwidman]

So @tvoinarovskyi can this be closed then? Not sure if there are more related APIs in KIP-79 that still need to be implemented...

[tvoinarovskyi]

Nope, its done