feat(outputs.kafka): Option to add metric name as record header

[Mrflatt]

Use Case

Option to add metric name as record header, which allows consumer to check if record should be skipped or processed. Record value is deserialized only when it's needed to be processed. Our usecase have around a million messages per second and not every message is currently processed (every message is deserialized and name field is checked)

Expected behavior

Producer adds header when option is set. Header key is specified by user and value is metric name

Actual behavior

Not implemented, opening pr

Additional info

No response

[powersj]

Hi,

allows consumer to check if record should be skipped or processed

When you say skipped, does that mean dropped? Is there a reason to not do this filtering at the telegraf level with metric filtering?

[Mrflatt]

Hi,

allows consumer to check if record should be skipped or processed

When you say skipped, does that mean dropped? Is there a reason to not do this filtering at the telegraf level with metric filtering?

We currently collect pm data from network devices using gnmi input plugin. All the data ends in shared kafka cluster, where users can use it for alerts, link traffic data for ai/ml stuff, checking fiber rx/rx leves, etc.
All the data is needed but not with every party involved :D

Edit. We could use topic_suffix, but it doesn't cover all the usecases currently

[powersj]

All the data is needed but not with every party involved :D

ok thanks for the further background. Happy to see a PR that exposes the option. This would need to be opt-in of course.