You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
There are a few key concepts that are really important to understand how
4
-
Fluent Bit operates.
2
+
description: Learn these key concepts to understand how Fluent Bit operates.
5
3
---
6
4
7
-
# Key Concepts
5
+
# Key concepts
8
6
9
-
Before diving into [Fluent Bit](https://fluentbit.io) it’s good to get acquainted with some of the key concepts of the service. This document provides a gentle introduction to those concepts and common [Fluent Bit](https://fluentbit.io) terminology. We’ve provided a list below of all the terms we’ll cover, but we recommend reading this document from start to finish to gain a more general understanding of our log and stream processor.
7
+
Before diving into [Fluent Bit](https://fluentbit.io) you might want to get acquainted
8
+
with some of the key concepts of the service. This document provides an
9
+
introduction to those concepts and common [Fluent Bit](https://fluentbit.io)
10
+
terminology. Reading this document will help you gain a more general understanding of the
11
+
following topics:
10
12
11
-
* Event or Record
12
-
* Filtering
13
-
* Tag
14
-
* Timestamp
15
-
* Match
16
-
* Structured Message
13
+
- Event or Record
14
+
- Filtering
15
+
- Tag
16
+
- Timestamp
17
+
- Match
18
+
- Structured Message
17
19
18
20
## Event or Record
19
21
20
-
Every incoming piece of data that belongs to a log or a metric that is retrieved by Fluent Bit is considered an Event or a Record.
22
+
Every incoming piece of data that belongs to a log or a metric that's retrieved by
23
+
Fluent Bit is considered an _Event_ or a _Record_.
21
24
22
-
As an example consider the following content of a Syslog file:
25
+
As an example, consider the following content of a Syslog file:
23
26
24
27
```text
25
28
Jan 18 12:52:16 flb systemd[2222]: Starting GNOME Terminal Server
@@ -28,30 +31,31 @@ Jan 18 12:52:16 flb systemd[2222]: Started GNOME Terminal Server.
It contains four lines and all of them represents **four** independent Events.
34
+
It contains four lines that represent four independent Events.
32
35
33
-
Internally an Event is comprised of:
36
+
An Event is comprised of:
34
37
35
-
* timestamp
36
-
* key/value metadata (since v2.1.0)
37
-
* payload
38
+
- timestamp
39
+
- key/value metadata (v2.1.0 and greater)
40
+
- payload
38
41
39
42
### Event format
40
43
41
-
The Fluent Bit wire protocol represents an Event as a 2-element array
44
+
The Fluent Bit wire protocol represents an Event as a two-element array
42
45
with a nested array as the first element:
43
46
44
-
```javascript
47
+
```javascript copy
45
48
[[TIMESTAMP, METADATA], MESSAGE]
46
49
```
47
50
48
51
where
49
52
50
-
* TIMESTAMP is a timestamp in seconds as an integer or floating point value (not a string);
51
-
* METADATA is a possibly-empty object containing event metadata; and
52
-
* MESSAGE is an object containing the event body.
53
+
-_`TIMESTAMP`_ is a timestamp in seconds as an integer or floating point value
54
+
(not a string).
55
+
-_`METADATA`_ is an object containing event metadata, and might be empty.
56
+
-_`MESSAGE`_ is an object containing the event body.
53
57
54
-
Fluent Bit versions prior to v2.1.0 instead used:
58
+
Fluent Bit versions prior to v2.1.0 used:
55
59
56
60
```javascript
57
61
[TIMESTAMP, MESSAGE]
@@ -62,74 +66,79 @@ streams.
62
66
63
67
## Filtering
64
68
65
-
In some cases it is required to perform modifications on the Events content, the process to alter, enrich or drop Events is called Filtering.
69
+
You might need to perform modifications on an Event's content. The process to alter,
70
+
append to, or drop Events is called [_filtering_](data-pipeline/filter.md).
66
71
67
-
There are many use cases when Filtering is required like:
72
+
Use filtering to:
68
73
69
-
* Append specific information to the Event like an IP address or metadata.
70
-
* Select a specific piece of the Event content.
71
-
* Drop Events that matches certain pattern.
74
+
- Append specific information to the Event like an IP address or metadata.
75
+
- Select a specific piece of the Event content.
76
+
- Drop Events that match a certain pattern.
72
77
73
78
## Tag
74
79
75
-
Every Event that gets into Fluent Bit gets assigned a Tag. This tag is an internal string that is used in a later stage by the Router to decide which Filter or Output phase it must go through.
80
+
Every Event ingested by Fluent Bit is assigned a Tag. This tag is an internal string
81
+
used in a later stage by the Router to decide which Filter or
82
+
[Output](data-pipeline/output.md) phase it must go through.
76
83
77
-
Most of the tags are assigned manually in the configuration. If a tag is not specified, Fluent Bit will assign the name of the Input plugin instance from where that Event was generated from.
84
+
Most tags are assigned manually in the configuration. If a tag isn't specified,
85
+
Fluent Bit assigns the name of the [Input](data-pipeline/input.md) plugin
86
+
instance where that Event was generated from.
78
87
79
88
{% hint style="info" %}
80
-
The only input plugin that **does NOT** assign tags is [Forward](../pipeline/inputs/forward.md) input. This plugin speaks the Fluentd wire protocol called Forward where every Event already comes with a Tag associated. Fluent Bit will always use the incoming Tag set by the client.
89
+
The [Forward](../pipeline/inputs/forward.md) input plugin doesn't assign tags. This
90
+
plugin speaks the Fluentd wire protocol called Forward where every Event already
91
+
comes with a Tag associated. Fluent Bit will always use the incoming Tag set by the
92
+
client.
81
93
{% endhint %}
82
94
83
-
A Tagged record must always have a Matching rule. To learn more about Tags and Matches check the [Routing](data-pipeline/router.md) section.
95
+
A tagged record must always have a Matching rule. To learn more about Tags and
96
+
Matches, see [Routing](data-pipeline/router.md).
84
97
85
98
## Timestamp
86
99
87
-
The Timestamp represents the _time_ when an Event was created. Every Event contains a Timestamp associated. The Timestamp is a numeric fractional integer in the format:
100
+
The timestamp represents the time an Event was created. Every Event contains an
101
+
associated timestamps. All events have timestamps, and they're set by the input plugin or
102
+
discovered through a data parsing process.
103
+
104
+
The timestamp is a numeric fractional integer in the format:
88
105
89
106
```javascript
90
107
SECONDS.NANOSECONDS
91
108
```
92
109
93
-
### Seconds
94
-
95
-
It is the number of seconds that have elapsed since the _Unix epoch._
96
-
97
-
### Nanoseconds
110
+
where:
98
111
99
-
Fractional second or one thousand-millionth of a second.
100
-
101
-
{% hint style="info" %}
102
-
A timestamp always exists, either set by the Input plugin or discovered through a data parsing process.
103
-
{% endhint %}
112
+
-`_SECONDS_` is the number of seconds that have elapsed since the Unix epoch.
113
+
-`_NANOSECONDS_` is a fractional second or one thousand-millionth of a second.
104
114
105
115
## Match
106
116
107
-
Fluent Bit allows to deliver your collected and processed Events to one or multiple destinations, this is done through a routing phase. A Match represent a simple rule to select Events where it Tags matches a defined rule.
117
+
Fluent Bit lets you route your collected and processed Events to one or multiple
118
+
destinations. A _Match_ represents a rule to select Events
119
+
where a Tag matches a defined rule.
108
120
109
-
To learn more about Tags and Matches check the [Routing](data-pipeline/router.md) section.
121
+
To learn more about Tags and Matches, see [Routing](data-pipeline/router.md).
110
122
111
-
## Structured Messages
123
+
## Structured messages
112
124
113
-
Source events can have or not have a structure. A structure defines a set of _keys_ and _values_ inside the Event message. As an example consider the following two messages:
125
+
Source events can have a structure. A structure defines a set of `keys` and `values`
126
+
inside the Event message to implement faster operations on data modifications.
127
+
Fluent Bit treats every Event message as a structured message.
114
128
115
-
### No structured message
129
+
Consider the following two messages:
116
130
117
-
```javascript
118
-
"Project Fluent Bit created on 1398289291"
119
-
```
120
-
121
-
### Structured Message
131
+
- No structured message
122
132
123
-
```javascript
124
-
{"project":"Fluent Bit", "created":1398289291}
125
-
```
133
+
```javascript
134
+
"Project Fluent Bitcreated on 1398289291"
135
+
```
126
136
127
-
At a low level both are just an array of bytes, but the Structured message defines _keys_ and _values_, having a structure helps to implement faster operations on data modifications.
137
+
- With a structured message
128
138
129
-
{% hint style="info" %}
130
-
Fluent Bit **always** handles every Event message as a structured message.
131
-
For performance reasons, we use a binary serialization data format called [MessagePack](https://msgpack.org/).
132
-
133
-
Consider [MessagePack](https://msgpack.org/) as a binary version of JSON on steroids.
134
-
{% endhint %}
139
+
```javascript
140
+
{"project":"Fluent Bit", "created":1398289291}
141
+
```
135
142
143
+
For performance reasons, Fluent Bit uses a binary serialization data format called
0 commit comments