Log a warning when bootstrap server is not in list of brokers returned by metadata request

[motherhubbard]

Kafka is configured so that it is bound to all network interfaces (ie the following line is commented out in server.properties):

host.name=localhost

and the advertised.host.name is not set.

I use the python client to connect to the Remote Kafka:
client = KafkaClient("remotehost:port")
producer = SimpleProducer(client, async=asynchronous)
producer.send_messages("topic", "test").

the following error is then seen:
snip...........
line 60, in init
[Fri Apr 11 08:43:19.328326 2014] [:error] [pid 81444:tid 4352339968] self.reinit()
[Fri Apr 11 08:43:19.328331 2014] [:error] [pid 81444:tid 4352339968] File kafka/conn.py", line 153, in reinit
[Fri Apr 11 08:43:19.328334 2014] [:error] [pid 81444:tid 4352339968] self._sock.connect((self.host, self.port))
[Fri Apr 11 08:43:19.328343 2014] [:error] [pid 81444:tid 4352339968] File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/socket.py", line 224, in meth
[Fri Apr 11 08:43:19.328347 2014] [:error] [pid 81444:tid 4352339968] return getattr(self._sock,name)(*args)
[Fri Apr 11 08:43:19.328350 2014] [:error] [pid 81444:tid 4352339968] error: [Errno 61] Connection refused

After debugging the code it can be seen that the client is trying to connect to "localhost".

The "localhost" value has been obtained by decoding the metadata response back from Kafka.

If I set the host.name (or I guess the advertised.host.name) then all is well.

It is my belief that the name used in the response from Kafka should not be used by the client (it may not be reachable by the client) but rather the initial value used when querying Kafka.

What do you think?

[wizzat]

I don't understand what you mean by "the initial value used when querying Kafka". Are you suggesting that we look for localhost as a specific response code and inline replace it with the original connection information?

[motherhubbard]

I meant the value used here:
client = KafkaClient("remotehost:port")

I was not suggesting a find and replace on response value but to always use the value "known" to the client

[wizzat]

I thought you said that the Kafka server was returning "localhost" inside the metadata response? I would imagine the correct response would be to connect to whatever the Kafka server says is the correct location to connect to.

[motherhubbard]

It is - Kafka is returning localhost in the response.

The client then breaks as it uses "localhost" for the send_message request.
My suggestion was to use the remotehost:port as used in the KafkaClient rather than the name from the metadata response (not only in the case of "localhost" being returned as I thought you were suggesting). This will ensure the host / port used by the client is always correct, rather than hoping the client can correctly resolve the host / port returned by Kafka

[wizzat]

But that fundamentally breaks connecting to a multiple node kafka cluster.

On Apr 11, 2014, at 2:36, motherhubbard notifications@github.com wrote:

It is - Kafka is returning localhost in the response.

The client then breaks as it uses "localhost" for the send_message request.
My suggestion was to use the remotehost:port as used in the KafkaClient rather than the name from the metadata response.

—
Reply to this email directly or view it on GitHub.

[motherhubbard]

I didnt have these issues using the node.js client. Are you saying thats fundamentally broken?
If i'd tried the Java / Scala clients and they had worked (i.e. used my initial values for the subsequent send request rather than "localhost") they are fundamentally broken too?

I've not tried these clients so I dont know what the result will be. Should be an easy enough test though.

[motherhubbard]

PS. I dont mean to sound confrontational here and I understand what you are saying. I'm just walking through the implications

[wizzat]

From the Kafka documentation:
https://cwiki.apache.org/confluence/display/KAFKA/FAQ#FAQ-Whycan'tmyconsumers/producersconnecttothebrokers?

When a broker starts up, it registers its ip/port in ZK. You need to make sure the
registered ip is consistent with what's listed in metadata.broker.list in the producer
config. By default, the registered ip is given by InetAddress.getLocalHost.getHostAddress.
Typically, this should return the real ip of the host. However, sometimes (e.g., in EC2), the
returned ip is an internal one and can't be connected to from outside. The solution is to
explicitly set the host ip to be registered in ZK by setting the "hostname" property in
server.properties. In another rare case where the binding host/port is different from the
host/port for client connection, you can set advertised.host.name and advertised.port for
client connection.

[wizzat]

Which node client are you using? It's possible they special cased a size=1 cluster returning "localhost" to connect to the original host.

[motherhubbard]

"kafka-node": "0.1.7"

from:
https://github.com/SOHU-Co/kafka-node/

[motherhubbard]

PS i'm happy with your explanation and this can be closed if you like. I wanted to leave a note so that others wont be caught out by this also ;-)

[wizzat]

I'm not a collaborator on the project, just a very interested party. I'm maintaining the kafka-python fork here at KIXEYE. ;-)

On Apr 11, 2014, at 9:55, motherhubbard notifications@github.com wrote:

PS i'm happy with your explanation and this can be closed if you like. I wanted to leave a note so that others wont be caught out by this also ;-)

—
Reply to this email directly or view it on GitHub.

[kudkudak]

I would add this to the WARNING that is being prompted, as I had to spend quite a lot of time searching for that ;) Host returned by kafka was equal to computer name, which was quite disturbing

[wizzat]

I don't really understand what you're saying. Are you asking for a warning to be emitted? What's the criteria when this warning would be logged?

[kudkudak]

Basically the motivation for this warning is that it seems that simplest kafka version might not work out of the box, which is quite bad for beginners (like me for instance). So you could add warning if host advertised from kafka server we have connected to is different than host that we were connecting to. Or simply you could mention this problem in appropriate wiki page, or README.md.

[wizzat]

It might be reasonable to issue a warning if the host you are connected to is not in the list of brokers received from the metadata request. However, that's still a legitimate thing.

Would a logging.warn be good enough?

On Apr 15, 2014, at 23:52, Stanislaw Jastrzebski notifications@github.com wrote:

Basically the motivation for this warning is that it seems that simplest kafka version might not work out of the box, which is quite bad for beginners (like me for instance). So you could add warning if host advertised from kafka server we have connected to is different than host that we were connecting to. Or simply you could mention this problem in appropriate wiki page, or README.md.

—
Reply to this email directly or view it on GitHub.

[kudkudak]

Yes of course :) Thanks for quick answer

[dpkp]

The root cause here is a misconfigured kafka cluster. Currently we will log a warning if the advertised host name does not resolve via dns. That may not help for 'localhost' but I'm not inclined to do much more since it seems very likely to hit edge cases and annoy a lot of working usages, for example, where the "bootstrap" name is a simple dns round robin but the metadata names are specific hosts.