Skip to content

development: developer-guide: general cleanup #1666

New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

#

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

Open
wants to merge 2 commits into
base: master
Choose a base branch
from
Open

development: developer-guide: general cleanup #1666

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
c7645c3
development: developer-guide: general cleanup
alexakreizinger May 16, 2025
042f085
typo
alexakreizinger May 16, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
94 changes: 45 additions & 49 deletions development/developer-guide.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,42 +1,43 @@
# Developer guide for beginners on contributing to Fluent Bit

Assuming you have some basic knowledge of C, this guide should help you understand how to make code changes to Fluent Bit.
If you have some knowledge of C, this guide should help you understand how to make code changes to Fluent Bit.

## Libraries

Most external libraries are embedded in the project in the [/lib](https://github.com/fluent/fluent-bit/tree/master/lib) folder. To keep its footprint low and make cross-platform builds simple, Fluent Bit attempts keep its dependency graph small.
Most external libraries are embedded in the project in the [/lib](https://github.com/fluent/fluent-bit/tree/master/lib) folder. To keep its footprint low and maximize compatibility in cross-platform builds, Fluent Bit attempts to keep its dependency graph small.

The external library you are mostly likely to interact with is [msgpack](https://github.com/msgpack/msgpack-c).
The external library that you're mostly likely to interact with is [MessagePack](https://github.com/msgpack/msgpack-c).

For cryptographic support, Fluent Bit uses the system installed version of OpenSSL.
Please make sure to install openssl libraries and headers before building Fluent Bit.
For cryptographic support, Fluent Bit uses the system installed version of OpenSSL. You must install OpenSSL libraries and headers before Building Fluent Bit.

### Memory Management
### Memory management

When you write Fluent Bit code, you will use Fluent Bit's versions of the standard C functions for working with memory:
When you write Fluent Bit code, you'll use the Fluent Bit versions of the standard C functions for working with memory:

* [`flb_malloc()`](https://github.com/fluent/fluent-bit/blob/master/include/fluent-bit/flb_mem.h) - equivalent to `malloc`, allocates memory.
* [`flb_calloc()`](https://github.com/fluent/fluent-bit/blob/master/include/fluent-bit/flb_mem.h) - equivalent to `calloc`, allocates memory and initializes it to zero.
* [`flb_realloc()`](https://github.com/fluent/fluent-bit/blob/master/include/fluent-bit/flb_mem.h) - equivalent to `realloc`.
* [`flb_free()`](https://github.com/fluent/fluent-bit/blob/master/include/fluent-bit/flb_mem.h) - equivalent to `free`, releases allocated memory.
* [`flb_malloc()`](https://github.com/fluent/fluent-bit/blob/master/include/fluent-bit/flb_mem.h): Equivalent to `malloc`, allocates memory.
* [`flb_calloc()`](https://github.com/fluent/fluent-bit/blob/master/include/fluent-bit/flb_mem.h): Equivalent to `calloc`, allocates memory and initializes it to zero.
* [`flb_realloc()`](https://github.com/fluent/fluent-bit/blob/master/include/fluent-bit/flb_mem.h): Equivalent to `realloc`.
* [`flb_free()`](https://github.com/fluent/fluent-bit/blob/master/include/fluent-bit/flb_mem.h): Equivalent to `free`, releases allocated memory.

Note that many types have a specialized create and destroy function. For example, [`flb_sds_create()` and `flb_sds_destroy()`](https://github.com/fluent/fluent-bit/blob/master/include/fluent-bit/flb_sds.h) \(more about this in the next section\).
{% hint style="info" %}
Many types have specialized create and destroy functions, like [`flb_sds_create()` and `flb_sds_destroy()`](https://github.com/fluent/fluent-bit/blob/master/include/fluent-bit/flb_sds.h).
{% endhint %}

### Strings

Fluent Bit has a stripped down version of the popular [SDS](https://github.com/antirez/sds) string library. See [flb\_sds.h](https://github.com/fluent/fluent-bit/blob/master/include/fluent-bit/flb_sds.h) for the API.
Fluent Bit has a stripped down version of the popular [SDS](https://github.com/antirez/sds) string library. See [`flb_sds.h`](https://github.com/fluent/fluent-bit/blob/master/include/fluent-bit/flb_sds.h) for the API.

In general, you should use SDS strings in any string processing code. SDS strings are fully compatible with any C function that accepts a null-terminated sequence of characters; to understand how they work, see the [explanation on Github](https://github.com/antirez/sds#how-sds-strings-work).
In general, you should use SDS strings in any string processing code. SDS strings are fully compatible with any C function that accepts a null-terminated sequence of characters. To understand how they work, see the [explanation on Github](https://github.com/antirez/sds#how-sds-strings-work).

### HTTP Client
### HTTP client

Fluent Bit has its own network connection library. The key types and functions are defined in the following header files:

* [flb\_upstream.h](https://github.com/fluent/fluent-bit/blob/master/include/fluent-bit/flb_upstream.h)
* [flb\_http\_client.h](https://github.com/fluent/fluent-bit/blob/master/include/fluent-bit/flb_http_client.h)
* [flb\_io.h](https://github.com/fluent/fluent-bit/blob/master/include/fluent-bit/flb_io.h)
* [`flb_upstream.h`](https://github.com/fluent/fluent-bit/blob/master/include/fluent-bit/flb_upstream.h)
* [`flb_http_client.h`](https://github.com/fluent/fluent-bit/blob/master/include/fluent-bit/flb_http_client.h)
* [`flb_io.h`](https://github.com/fluent/fluent-bit/blob/master/include/fluent-bit/flb_io.h)

The following code demonstrates making an HTTP request in Fluent Bit:
The following code demonstrates an HTTP request in Fluent Bit:

```c
#include <fluent-bit/flb_upstream.h>
Expand Down Expand Up @@ -113,13 +114,13 @@ static flb_sds_t make_request(struct flb_config *config)
}
```

An `flb_upstream` structure represents a host/endpoint that you want to call. Normally, you'd store this structure somewhere so that it can be re-used. An `flb_upstream_conn` represents a connection to that host for a single HTTP request. The connection structure should not be used for more than one request.
The `flb_upstream` structure represents the host/endpoint that you want to call. Normally, you'd store this structure somewhere so that it can be reused. An `flb_upstream_conn` represents a connection to that host for a single HTTP request. This connection structure shouldn't be used for more than one request.

### Linked Lists
### Linked lists

Fluent Bit contains a library for constructing linked lists- [cfl\_list](https://github.com/fluent/fluent-bit/blob/master/lib/cfl/include/cfl/cfl_list.h). The type stores data as a circular linked list.
Fluent Bit contains a library for constructing linked lists: [`cfl_list`](https://github.com/fluent/fluent-bit/blob/master/lib/cfl/include/cfl/cfl_list.h). The type stores data as a circular linked list.

The [`cfl_list.h`](https://github.com/fluent/fluent-bit/blob/master/lib/cfl/include/cfl/cfl_list.h) header file contains several macros and functions for use with the lists. The example below shows how to create a list, iterate through it, and delete an element.
The [`cfl_list.h`](https://github.com/fluent/fluent-bit/blob/master/lib/cfl/include/cfl/cfl_list.h) header file contains several macros and functions for use with the lists. The following example shows how to create a list, iterate through it, and delete an element.

```c
#include <cfl/cfl.h>
Expand Down Expand Up @@ -174,11 +175,11 @@ static int example()
}
```

### Message Pack
### MessagePack

Fluent Bit uses [msgpack](https://msgpack.org/index.html) to internally store data. If you write code for Fluent Bit, it is almost certain that you will interact with msgpack.
Fluent Bit uses [MessagePack](https://msgpack.org/index.html) to internally store data. If you write code for Fluent Bit, it's almost certain that you will interact with MessagePack.

Fluent Bit embeds the [msgpack-c](https://github.com/msgpack/msgpack-c) library. The example below shows manipulating message pack to add a new key-value pair to a record. In Fluent Bit, the [filter\_record\_modifier](https://github.com/fluent/fluent-bit/tree/master/plugins/filter_record_modifier) plugin adds or deletes keys from records. See its code for more.
Fluent Bit embeds the [`msgpack-c`](https://github.com/msgpack/msgpack-c) library. The following example shows how to manipulate message pack to add a new key/value pair to a record. In Fluent Bit, the [`filter_record_modifier`](https://github.com/fluent/fluent-bit/tree/master/plugins/filter_record_modifier) plugin adds or deletes keys from records. See its code for more.

```c
#define A_NEW_KEY "key"
Expand Down Expand Up @@ -262,19 +263,19 @@ static int cb_filter(const void *data, size_t bytes,
return FLB_FILTER_MODIFIED;
```

Please also check out the message pack examples on the [msgpack-c GitHub repo](https://github.com/msgpack/msgpack-c).
For more info, see the MessagePack examples in the [msgpack-c GitHub repository](https://github.com/msgpack/msgpack-c).

## Plugin API

Each plugin is a shared object which is [loaded into Fluent Bit](https://github.com/fluent/fluent-bit/blob/1.3/src/flb_plugin.c#L70) using dlopen and dlsym.
Each plugin is a shared object which is [loaded into Fluent Bit](https://github.com/fluent/fluent-bit/blob/1.3/src/flb_plugin.c#L70) using `dlopen`and `dlsym`.

### Input

The input plugin structure is defined in [flb\_input.h](https://github.com/fluent/fluent-bit/blob/master/include/fluent-bit/flb_input.h#L62). There are a number of functions which a plugin can implement, most only implement `cb_init`, `cb_collect`, and `cb_exit`.
The input plugin structure is defined in [`flb_input.h`](https://github.com/fluent/fluent-bit/blob/master/include/fluent-bit/flb_input.h#L62). There are a number of functions which a plugin can implement, but most only implement `cb_init`, `cb_collect`, and `cb_exit`.

The [`"dummy"` input plugin](https://github.com/fluent/fluent-bit/tree/master/plugins/in_dummy) is very simple and is an excellent example to review to understand more.
The [`"dummy"` input plugin](https://github.com/fluent/fluent-bit/tree/master/plugins/in_dummy) is very basic and is an excellent example to review to understand more.

Note that input plugins can use threaded mode if the flag `FLB_INPUT_THREADED` is provided.
Input plugins can use threaded mode if the flag `FLB_INPUT_THREADED` is provided.
To enable threading in your plugin, add the `FLB_INPUT_THREADED` to the set of `flags` when registering:

```c
Expand All @@ -295,11 +296,11 @@ struct flb_input_plugin in_your_example_plugin = {

### Filter

The structure for filter plugins is defined in [flb\_filter.h](https://github.com/fluent/fluent-bit/blob/master/include/fluent-bit/flb_filter.h#L44). Each plugin must implement `cb_init`, `cb_filter`, and `cb_exit`.
The structure for filter plugins is defined in [`flb_filter.h`](https://github.com/fluent/fluent-bit/blob/master/include/fluent-bit/flb_filter.h#L44). Each plugin must implement `cb_init`, `cb_filter`, and `cb_exit`.

The [filter\_record\_modifier](https://github.com/fluent/fluent-bit/tree/master/plugins/filter_record_modifier) is a good example of a filter plugin.
The [`filter_record_modifier`](https://github.com/fluent/fluent-bit/tree/master/plugins/filter_record_modifier) is a good example of a filter plugin.

Note that filter plugins can not asynchronously make HTTP requests. If your plugin needs to make a request, add the following code when you initialize your `flb_upstream`:
Filter plugins can not asynchronously make HTTP requests. If your plugin needs to make a request, add the following code when you initialize your `flb_upstream`:

```c
/* Remove async flag from upstream */
Expand All @@ -308,17 +309,15 @@ upstream->flags &= ~(FLB_IO_ASYNC);

### Output

Output plugins are defined in [flb\_output.h](https://github.com/fluent/fluent-bit/blob/master/include/fluent-bit/flb_output.h#L57). Each plugin must implement `cb_init`, `cb_flush`, and `cb_exit`.
Output plugins are defined in [`flb_output.h`](https://github.com/fluent/fluent-bit/blob/master/include/fluent-bit/flb_output.h#L57). Each plugin must implement `cb_init`, `cb_flush`, and `cb_exit`.

The [stdout plugin](https://github.com/fluent/fluent-bit/tree/master/plugins/out_stdout) is very simple; review its code to understand how output plugins work.
The [stdout plugin](https://github.com/fluent/fluent-bit/tree/master/plugins/out_stdout) is very basic; review its code to understand how output plugins work.

## Development Environment
## Development environment

Fluent Bit provides a standalone environment for development.
Developers who use different OS or distributions can develop on a simple, common stack.
The development environment provides the required libraries and tools for you.
Fluent Bit provides a standalone environment for development. Developers who use different operating systems or distributions can develop on a basic, common stack. The development environment provides the required libraries and tools for you.

Development environments provided for
Development environments are provided for:
- [Devcontainer](https://github.com/fluent/fluent-bit/blob/master/DEVELOPER_GUIDE.md#devcontainer)
- [Vagrant](https://github.com/fluent/fluent-bit/blob/master/DEVELOPER_GUIDE.md#vagrant).

Expand All @@ -332,9 +331,11 @@ cmake -DFLB_DEV=On ../
make
```

Note that Fluent Bit uses Cmake 3 and on some systems you may need to invoke it as `cmake3`.
{% hint style="info" %}
Fluent Bit uses CMake 3. On some systems you might need to invoke it as `cmake3`.
{% endhint %}

To enable the unit tests run:
To enable the unit tests, run the following command:

```text
cmake -DFLB_DEV=On -DFLB_TESTS_RUNTIME=On -DFLB_TESTS_INTERNAL=On ../
Expand All @@ -343,16 +344,11 @@ make

Internal tests are for the internal libraries of Fluent Bit. Runtime tests are for the plugins.

You can run the unit tests with `make test`, however, this is inconvenient in practice. Each test file will create an executable in the `build/bin` directory which you can run directly. For example, if you want to run the SDS tests, you can invoke them as follows:
You can run the unit tests with `make test`. However, this is inconvenient in practice. Each test file will create an executable in the `build/bin` directory, which you can run directly. For example, if you want to run the SDS tests, you can invoke them as follows:

```text
$ ./bin/flb-it-sds
Test sds_usage... [ OK ]
Test sds_printf... [ OK ]
SUCCESS: All unit tests have passed.
```

## Need more help?

The best way to learn how Fluent Bit code works is to read it. If you need help understanding the code, reach out to the community, or open a PR with changes that are a work in progress.

1 change: 1 addition & 0 deletions vale-styles/FluentBit/Acronyms.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -71,6 +71,7 @@ exceptions:
- SCM
- SCSS
- SDK
- SDS
- SIEM
- SLA
- SQL
Expand Down