producer of 1.4.2 version may occupy large memory

[joe-zhan]

I use 1.4.2 kafka-python to collect my data into kafka, but the memory use of the python process may increase progressively until the ops send a low-memory alarm then I have to restart it.I roll back the version of kafka-python to 1.3.4, then everything is ok.

my env is: centos 7, python 2.7.5

[jeffwidman]

What version of Kafka?

Do you have example code showing how you're producing?

Was the rollback to 1.3.4 the only thing you changed?

[joe-zhan]

here is my python script, just tail -f the files that filename like 'test_2018042700.txt' then send new lines to kafka. My kafka version is 0.10.2 with scala 2.12.
new_monitor_topic_withmark_v0.10.2.py.txt

I didn't do anything else, only rollback the kafka-python version to 1.3.4,then the memory use stays at 19MB.

[joe-zhan]

It seems that every second I new a KafkaProducer object then close it after the new lines sended,and the 1.4.2 version doesn't release the memory after I close the producer.I don't know whether this is an issue or not.Maybe just I use the kafka-python in the wrong way.

[Ronniexie]

I have the same problem
I use 1.4.2 kafka-python.
@namasamitabha Have you solved this problem?

[joe-zhan]

@Ronniexie Not really.I just rollback the kafka-python to pre version which I used before,and these days I am trying to change the collect script from python to filebeat.

[Ronniexie]

#1412
Is it related to this problem?

[jeffwidman]

I haven't tested myself, but from the problem description, it sounds like #1412 is a CPU problem, not a memory problem.

If you're instantiating/tearing down a new KafkaProducer instance every time, that is a huge waste of resources, and would certainly use additional memory.

For most scenarios, you should create a single long-lived instance per Python program and then use that single instance to send all messages to Kafka... you can send to multiple topics with the same instance, it's effectively just a database connection. The only exception is if you need separate instances with different configs that are can only be set on instantiation.

I'm going to close this ticket, as it is most likely user error. Happy to re-open if someone has a code snippet showing problems with a single instance.