You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Sometimes users uses kafka client_id to set quotas for input/output IO of kafka consumers/producers. so just some specific client ids can read/write from kafka cluster with a high rate.
Now logstash-integration-kafka plugin appends the thread-id to client_id without any documentation and there isn't any config to disable it to just send the client_id to kafka cluster.
The text was updated successfully, but these errors were encountered:
Thanks Arman, this seems like a legit use-case.
Believe the plugin should not change client_id esp. if it's set to an explicit value by the user.
Now, I am not 100% how to approach this one but #55 seems a bit 'ugly' with an option to change another option.
Maybe we can do some kind of formatting on the client_id thus respecting an explicitly set value while also allowing to use the current behavior (not changing current behavior when client_id is not set - should be the common case) e.g. :
and then sprintf-ing client_id against the thread counter.
It might still needs more tuning as this might seem too similar to LS' sprintf-ing of event fields (using the %{} syntax) and thus might lead to a bit of confusion.
A reminder on how Kafka uses client_id ...
sole purpose of this is to be able to track the source of requests beyond just ip and port by allowing a logical application name to be included in Kafka logs
Sometimes users uses kafka client_id to set quotas for input/output IO of kafka consumers/producers. so just some specific client ids can read/write from kafka cluster with a high rate.
Now logstash-integration-kafka plugin appends the thread-id to client_id without any documentation and there isn't any config to disable it to just send the client_id to kafka cluster.
The text was updated successfully, but these errors were encountered: