From dfaf16d3311888d36f5c73e107709dac50e8a5df Mon Sep 17 00:00:00 2001 From: Fabian Ruff Date: Mon, 10 Mar 2025 17:14:27 +0100 Subject: [PATCH 1/2] filter_logs_to_metrics: document field_value usage for counters This documents a feature added with this PR: https://github.com/fluent/fluent-bit/pull/10055 Signed-off-by: Fabian Ruff --- pipeline/filters/log_to_metrics.md | 20 ++++++++++++++++---- 1 file changed, 16 insertions(+), 4 deletions(-) diff --git a/pipeline/filters/log_to_metrics.md b/pipeline/filters/log_to_metrics.md index 82bc245bc..7ca25716a 100644 --- a/pipeline/filters/log_to_metrics.md +++ b/pipeline/filters/log_to_metrics.md @@ -8,7 +8,7 @@ description: Generate metrics from logs ## Log To Metrics -The _Log To Metrics Filter_ plugin allows you to generate log-derived metrics. It currently supports modes to count records, provide a gauge for field values or create a histogram. You can also match or exclude specific records based on regular expression patterns for values or nested values. This filter plugin does not actually act as a record filter and does not change or drop records. All records will pass this filter untouched and generated metrics will be emitted into a seperate metric pipeline. +The _Log To Metrics Filter_ plugin allows you to generate log-derived metrics. It currently supports modes to count records or field values, provide a gauge for field values or create a histogram. You can also match or exclude specific records based on regular expression patterns for values or nested values. This filter plugin does not actually act as a record filter and does not change or drop records. All records will pass this filter untouched and generated metrics will be emitted into a seperate metric pipeline. _Please note that this plugin is an experimental feature and is not recommended for production use. Configuration parameters and plugin functionality are subject to change without notice._ @@ -25,7 +25,7 @@ The plugin supports the following configuration parameters: | bucket | Defines a bucket for `histogram` | Yes, for mode `histogram` | e.g. 0.75 | | add\_label | Add a custom label NAME and set the value to the value of KEY | | | | label\_field | Includes a record field as label dimension in the metric. | | Name of record key. Supports [Record Accessor](../../administration/configuring-fluent-bit/classic-mode/record-accessor.md) notation for nested fields. | -| value\_field | Specify the record field that holds a numerical value | Yes, for modes \[`gauge` and `histogram`] | Name of record key. Supports [Record Accessor](../../administration/configuring-fluent-bit/classic-mode/record-accessor.md) notation for nested fields. | +| value\_field | Specify the record field that holds a numerical value | Yes, for modes \[`gauge` and `histogram`, optional for `counter`] | Name of record key. Supports [Record Accessor](../../administration/configuring-fluent-bit/classic-mode/record-accessor.md) notation for nested fields. | | kubernetes\_mode | If enabled, it will automatically put pod\_id, pod\_name, namespace\_name, docker\_id and container\_name into the metric as labels. This option is intended to be used in combination with the [kubernetes](kubernetes.md) filter plugin, which fills those fields. | | | | Regex | Include records in which the content of KEY matches the regular expression. | | KEY REGEX | | Exclude | Exclude records in which the content of KEY matches the regular expression. | | KEY REGEX | @@ -34,7 +34,7 @@ The plugin supports the following configuration parameters: ### Getting Started -The following example takes records from two dummy inputs and counts all messages passing through the `log_to_metrics` filter. It then generates metric records which are provided to the `prometheus_exporter`: +The following example takes records from two dummy inputs and generates two different counter metrics: It counts all messages passing through the `log_to_metrics` filter and it counts the total time spend according to the duration field of each message. It then generates metric records which are provided to the `prometheus_exporter`: #### Configuration - Counter @@ -61,6 +61,15 @@ The following example takes records from two dummy inputs and counts all message metric_name count_all_dummy_messages metric_description This metric counts dummy messages +[FILTER] + name log_to_metrics + match dummy.log* + tag test_metric + metric_mode counter + metric_name total_duration + metric_description This metric counts the total time spend + value_field duration + [OUTPUT] name prometheus_exporter match * @@ -76,7 +85,10 @@ You can then use e.g. curl command to retrieve the generated metric: # HELP log_metric_counter_count_all_dummy_messages This metric counts dummy messages # TYPE log_metric_counter_count_all_dummy_messages counter -log_metric_counter_count_all_dummy_messages 49 +log_metric_counter_count_all_dummy_messages 24 +# HELP log_metric_counter_total_duration This metric counts the total time spend +# TYPE log_metric_counter_total_duration counter +log_metric_counter_total_duration 960 ``` #### Configuration - Gauge From 13f50c00aff53f44aab45a5898744e20038559a2 Mon Sep 17 00:00:00 2001 From: Fabian Ruff Date: Wed, 12 Mar 2025 20:55:15 +0100 Subject: [PATCH 2/2] Apply suggestions from code review Co-authored-by: Alexa Kreizinger Signed-off-by: Fabian Ruff --- pipeline/filters/log_to_metrics.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/pipeline/filters/log_to_metrics.md b/pipeline/filters/log_to_metrics.md index 7ca25716a..54bd697bf 100644 --- a/pipeline/filters/log_to_metrics.md +++ b/pipeline/filters/log_to_metrics.md @@ -8,7 +8,7 @@ description: Generate metrics from logs ## Log To Metrics -The _Log To Metrics Filter_ plugin allows you to generate log-derived metrics. It currently supports modes to count records or field values, provide a gauge for field values or create a histogram. You can also match or exclude specific records based on regular expression patterns for values or nested values. This filter plugin does not actually act as a record filter and does not change or drop records. All records will pass this filter untouched and generated metrics will be emitted into a seperate metric pipeline. +The _Log To Metrics Filter_ plugin allows you to generate log-derived metrics. It currently supports modes to create counters based on the number of records or the value of a field within those records, create a gauge for field values, or create a histogram. You can also match or exclude specific records based on regular expression patterns for values or nested values. This filter plugin does not actually act as a record filter and does not change or drop records. All records will pass this filter untouched and generated metrics will be emitted into a separate metric pipeline. _Please note that this plugin is an experimental feature and is not recommended for production use. Configuration parameters and plugin functionality are subject to change without notice._ @@ -25,7 +25,7 @@ The plugin supports the following configuration parameters: | bucket | Defines a bucket for `histogram` | Yes, for mode `histogram` | e.g. 0.75 | | add\_label | Add a custom label NAME and set the value to the value of KEY | | | | label\_field | Includes a record field as label dimension in the metric. | | Name of record key. Supports [Record Accessor](../../administration/configuring-fluent-bit/classic-mode/record-accessor.md) notation for nested fields. | -| value\_field | Specify the record field that holds a numerical value | Yes, for modes \[`gauge` and `histogram`, optional for `counter`] | Name of record key. Supports [Record Accessor](../../administration/configuring-fluent-bit/classic-mode/record-accessor.md) notation for nested fields. | +| value\_field | Specify the record field that holds a numerical value | Yes, for modes `gauge` and `histogram`, optional for `counter` | Name of record key. Supports [Record Accessor](../../administration/configuring-fluent-bit/classic-mode/record-accessor.md) notation for nested fields. | | kubernetes\_mode | If enabled, it will automatically put pod\_id, pod\_name, namespace\_name, docker\_id and container\_name into the metric as labels. This option is intended to be used in combination with the [kubernetes](kubernetes.md) filter plugin, which fills those fields. | | | | Regex | Include records in which the content of KEY matches the regular expression. | | KEY REGEX | | Exclude | Exclude records in which the content of KEY matches the regular expression. | | KEY REGEX | @@ -34,7 +34,7 @@ The plugin supports the following configuration parameters: ### Getting Started -The following example takes records from two dummy inputs and generates two different counter metrics: It counts all messages passing through the `log_to_metrics` filter and it counts the total time spend according to the duration field of each message. It then generates metric records which are provided to the `prometheus_exporter`: +The following example takes records from two dummy inputs and generates two different counter metrics: the number of messages that pass through the `log_to_metrics` filter, and the combined value of the `duration` fields included in each of those messages. It then generates metric records which are provided to the `prometheus_exporter`: #### Configuration - Counter @@ -67,7 +67,7 @@ The following example takes records from two dummy inputs and generates two diff tag test_metric metric_mode counter metric_name total_duration - metric_description This metric counts the total time spend + metric_description This metric counts the total time spent across all messages value_field duration [OUTPUT]