added path_key to include read path into the event under this key

[szhem]

When sending the events to the elasticsearch it''s quite useful to be able to understand what file is the origin of the received event.
The patch adds additional optional 'path_key' tag to the configuration, to hold the filename

<source>
  type tail
  path_key path
  path /var/log/syslog
  tag syslog
</source>

All the changes are backward compatible as the path is inserted into the event only if path_key is specified.

[sonots]

I think, using https://github.com/sonots/fluent-plugin-record-reformer is fine enough. You can add path field to records like

<source>
  type tail
  path /var/log/syslog
  tag syslog
</source>
<match syslog>
  type record_reformer
  path /var/log/syslog
  output_tag reformed.${tag}
</match>

[szhem]

In case of many files using record_reformer is hardly possible, especially, if tail_ex will be merged into the in_tail (#269, #277)

For example, how to add path of the read file into the event in case of configuration like the following one?

<source>
  type tail
  path /var/log/**.log,/var/log/by-date/%Y/messages.%m/%Y%m%d
  tag input_tag
</source>

[sonots]

Ah, understand.

[sonots]

@szhem It seems that, after merging tail_ex, it becomes possible to append file path into tag like

<source>
  type tail
  path /var/log/**.log,/var/log/by-date/%Y/messages.%m/%Y%m%d
  tag input_tag.*
</source>

The * is replaced with the file path tailing. See https://github.com/yosisa/fluent-plugin-tail-ex

Then, you should be able to put the file path into a record field using record-reformer. Is it fine enough for your purpose?

[szhem]

after merging tail_ex, it becomes possible to append file path into tag like tag input_tag.*

Well, and yes, and no.
Here is how the tag is created in tail-ex

def _receive_lines(lines)
  tag = @path.tr('/', '.').gsub(/\.+/, '.').gsub(/^\./, '')
  @parent_receive_lines.call(lines, tag)
end

In case of path ./some/../../complex/../path the tag value will be input_tag.some.complex.path that cannot be translated into the original path, for example, to analyze original log files.

Honestly speaking, the field that can be copy-pasted directly into the shell is pretty useful (splunk, logstash and other log-aggregating software use the full path of the source)

[sonots]

Okay, understand, but we (fluentd committers) still do not know how many people need this functionality.

Could it possible to create another extension plugin for this purpose?
If we find many people are using it, we would merge into in_tail plugin in the future.

[sonots]

@szhem I created extension plugin based on your patch at https://github.com/sonots/fluent-plugin-tail_path. If you like to be a maintainer of this, I can transfer this repository to you. Please let me know.

NOTICE: This plugin requires NewTailInput class of Fluentd which is not released yet. We have to wait until new fluentd gem is released.

[szhem]

@sonots, thanks a lot for the extension plugin. For now, I have already modified tail_ex plugin for that purpose.

[sonots]

Okay, let me close. I will release gem soon to see user's demands.

[sonots]

@szhem Released fluent-plugin-tail_path http://rubygems.org/gems/fluent-plugin-tail_path since fluentd v0.10.45 was released.

[szhem]

Released fluent-plugin-tail_path http://rubygems.org/gems/fluent-plugin-tail_path since fluentd v0.10.45 was released.

Thanks so much!