Skip to content
New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

Add conf to choose if appending thread number to client_id or not #54

Open
arman1371 opened this issue Aug 28, 2020 · 1 comment · May be fixed by #55
Open

Add conf to choose if appending thread number to client_id or not #54

arman1371 opened this issue Aug 28, 2020 · 1 comment · May be fixed by #55

Comments

@arman1371
Copy link

  • Version: 10.5.1

Sometimes users uses kafka client_id to set quotas for input/output IO of kafka consumers/producers. so just some specific client ids can read/write from kafka cluster with a high rate.
Now logstash-integration-kafka plugin appends the thread-id to client_id without any documentation and there isn't any config to disable it to just send the client_id to kafka cluster.

@kares
Copy link
Contributor

kares commented Oct 21, 2020

Thanks Arman, this seems like a legit use-case.
Believe the plugin should not change client_id esp. if it's set to an explicit value by the user.

Now, I am not 100% how to approach this one but #55 seems a bit 'ugly' with an option to change another option.
Maybe we can do some kind of formatting on the client_id thus respecting an explicitly set value while also allowing to use the current behavior (not changing current behavior when client_id is not set - should be the common case) e.g. :

config :client_id, :validate => :string, :default => "logstash-%i"

and then sprintf-ing client_id against the thread counter.
It might still needs more tuning as this might seem too similar to LS' sprintf-ing of event fields (using the %{} syntax) and thus might lead to a bit of confusion.

A reminder on how Kafka uses client_id ...

sole purpose of this is to be able to track the source of requests beyond just ip and port by allowing a logical application name to be included in Kafka logs

# for free to join this conversation on GitHub. Already have an account? # to comment
Labels
None yet
Projects
None yet
2 participants