Facilitate the management of Fluent Bit and Fluentd in Kubernetes
Fluent Operator provides great flexibility in building a logging layer based on Fluent Bit and Fluentd.
Once installed, the Fluent Operator provides the following features:
- Fluent Bit Management: Deploy and destroy Fluent Bit DaemonSet automatically.
- Fluentd Management: Deploy and destroy Fluentd StatefulSet automatically.
- Custom Configuration: Select input/filter/output plugins via labels.
- Dynamic Reloading: Update configuration without rebooting Fluent Bit and Fluentd pods.
- Multi-tenant log isolation: Fluentd supports multi-tenant log isolation through
label_router
plugin. - Pluggable deployment components: Either Fluent Bit or Fluentd can be deployed separately.
- Fluent Operator
Although both Fluent Bit and Fluentd can collect, process(parse and filter) and then forward log to the final destinations, still they have strengths in different aspects.
Fluent Bit is a good choice as a logging agent because of its lightweight and efficiency, while Fluentd is more powerful to perform advanced processing on logs because of its rich plugins.
- Fluent Bit only mode: If you just need to collect logs and send logs to the final destinations, all you need is Fluent Bit.
- Fluent Bit + Fluentd mode: If you also need to perform some advanced processing on the logs collected or send to more sinks, then you also need Fluentd.
- Fluentd only mode: If you need to receive logs through networks like HTTP or Syslog and then process and send the log to the final sinks, you only need Fluentd.
Fluent Operator includes CRDs and controllers for both Fluent Bit and Fluentd which allows you to config your log processing pipelines in the 3 modes mentioned above as you wish.
Fluent Bit will be deployed as a DaemonSet while Fluentd will be deployed as a StatefulSet. The whole workflow could be described as below:
The following CRDs are defined for Fluent Bit:
FluentBit
: Defines the Fluent Bit DaemonSet and its configs. A custom Fluent Bit imageghcr.io/fluent/fluent-operator/fluent-bit
is required to work with FluentBit Operator for dynamic configuration reloading.ClusterFluentBitConfig
: Select cluster-level input/filter/output plugins and generates the final config into a Secret.FluentBitConfig
: Selects namespace-level filter/output/parser and cluster-level parser plugins and generates final config into a Secret.ClusterInput
: Defines cluster-level input config sections.Parser
: Defines namespace-level parser config sections.ClusterParser
: Defines cluster-level parser config sections.MultilineParser
: Defines namespace-level multiline-parser config sectionsClusterMultilineParser
: Defines cluster-level multiline-parser config sectionsFilter
: Defines namespace-level filter config sections.ClusterFilter
: Defines cluster-level filter config sections.Output
: Defines namespace-level output config sections.ClusterOutput
: Defines cluster-level output config sections.
Each ClusterInput
, ClusterParser
, ClusterFilter
, ClusterMultilineParser
, ClusterOutput
represents a Fluent Bit config section, which are selected by ClusterFluentBitConfig
via label selectors. Fluent Operator watches those objects, constructs the final config, and finally creates a Secret to store the config which will be mounted into the Fluent Bit DaemonSet. The entire workflow looks like below:
To enable Fluent Bit to pick up and use the latest config whenever the Fluent Bit config changes, a wrapper called Fluent Bit watcher is added to restart the Fluent Bit process as soon as Fluent Bit config changes are detected. This way, the Fluent Bit pod needn't be restarted to reload the new config. The Fluent Bit config is reloaded in this way because there is no reloading interface in Fluent Bit itself. Please refer to this known issue for more details.
The following CRDs are defined for Fluentd:
Fluentd
: Defines the Fluentd Statefulset and its configs. A custom Fluentd imagekubesphere/fluentd
is required to work with Fluentd Operator for dynamic configuration reloading.FluentdConfig
: Select cluster-level or namespace-level scope input/filter/output plugins and generates the final config into a Secret.ClusterFluentdConfig
: Select cluster-level input/filter/output plugins and generates the final config into a Secret.Filter
: Defines namespace-level filter config sections.ClusterFilter
: Defines cluster-level filter config sections.Output
: Defines namespace-level output config sections.ClusterOutput
: Defines cluster-level output config sections.
Kubernetes v1.16.13+ is necessary for running Fluent Operator.
Install the latest stable version
kubectl apply -f https://raw.githubusercontent.com/fluent/fluent-operator/release-3.2/manifests/setup/setup.yaml
# You can change the namespace in manifests/setup/kustomization.yaml in corresponding release branch
# and then use command below to install to another namespace
# kubectl kustomize manifests/setup/ | kubectl apply -f -
Install the development version
kubectl apply -f https://raw.githubusercontent.com/fluent/fluentbit-operator/master/manifests/setup/setup.yaml
# You can change the namespace in manifests/setup/kustomization.yaml
# and then use command below to install to another namespace
# kubectl kustomize manifests/setup/ | kubectl apply -f -
NOTE: For the helm based install, Helm v3.2.1 or higher is needed.
To install or upgrade Fluent Operator using Helm:
export FLUENT_OPERATOR_CONTAINER_RUNTIME="containerd" # or "cri-o", "docker" depending on the container runtime being used (see `values.yaml`)
helm repo add fluent https://fluent.github.io/helm-charts
helm upgrade --install fluent-operator fluent/fluent-operator \
--create-namespace \
--set containerRuntime=${FLUENT_OPERATOR_CONTAINER_RUNTIME}
By default, all CRDs required for Fluent Operator will be installed. To prevent helm install
from installing CRDs, you can set fluent-bit.crdsEnable
or fluentd.crdsEnable
to false
.
Helm does not manage the lifecycle of CRDs, so if the Fluent Operator CRDs already exist, subsequent chart upgrades will not add or remove CRDs even if they have changed. During upgrades, users should manually update CRDs:
wget https://github.com/fluent/fluent-operator/releases/download/<version>/fluent-operator.tgz
tar -xf fluent-operator.tgz
kubectl replace -f fluent-operator/crds
For more info on various use cases of Fluent Operator, you can refer to Fluent-Operator-Walkthrough.
This guide provisions a logging pipeline including the Fluent Bit DaemonSet and its log input/filter/output configurations to collect Kubernetes logs including container logs and kubelet logs.
Note that you need a running Elasticsearch v5+ cluster to receive log data before start. Remember to adjust output-elasticsearch.yaml to your own es setup. Kafka and Fluentd outputs are optional and are turned off by default.
kubectl apply -f manifests/logging-stack
# You can change the namespace in manifests/logging-stack/kustomization.yaml
# and then use command below to install to another namespace
# kubectl kustomize manifests/logging-stack/ | kubectl apply -f -
You can also deploy the Kubernetes logging pipeline with Helm, just need to set the Kubernetes
parameter to true
(default):
helm upgrade fluent-operator --create-namespace -n fluent charts/fluent-operator/ --set containerRuntime=docker
If you want to deploy fluentd
, just need to set the fluentd.enable
parameter to true
:
helm upgrade fluent-operator --create-namespace -n fluent charts/fluent-operator/ --set containerRuntime=docker,fluentd.enable=true
Within a couple of minutes, you should observe an index available:
$ curl localhost:9200/_cat/indices
green open ks-logstash-log-2020.04.26 uwQuoO90TwyigqYRW7MDYQ 1 1 99937 0 31.2mb 31.2mb
Success!
The Linux audit framework provides a CAPP-compliant (Controlled Access Protection Profile) auditing system that reliably collects information about any security-relevant (or non-security-relevant) event on a system. Refer to manifests/logging-stack/auditd
, it supports a method for collecting audit logs from the Linux audit framework.
kubectl apply -f manifests/logging-stack/auditd
# You can change the namespace in manifests/logging-stack/auditd/kustomization.yaml
# and then use command below to install to another namespace
# kubectl kustomize manifests/logging-stack/auditd/ | kubectl apply -f -
Within a couple of minutes, you should observe an index available:
$ curl localhost:9200/_cat/indices
green open ks-logstash-log-2021.04.06 QeI-k_LoQZ2h1z23F3XiHg 5 1 404879 0 298.4mb 149.2mb
Fluentd acts as a log forward layer that receives logs from Fluent Bit or other Apps through the network.
For more info on various use cases of Fluent Operator Fluentd CRDs, you can refer to Fluent-Operator-Walkthrough.
Here you can find supported Fluent Bit plugins.
Here you can find supported Fluentd plugins.
Here you can find the plugin group information.
Here you can use the built-in HTTP Server in Fluent Bit.
Here you can customize parser in Fluent Bit.
If you want to learn more about Fluent-Operator, please refer to the misc.
Here you can learn how to use custom plugin in FluentBit and Fluentd.
- Support containerd log format
- Add Fluentd CRDs as the log aggregation layer with group name
fluentd.fluent.io
- Add FluentBit Cluster CRDs with new group name
fluentbit.fluent.io
- Rename the entire project to Fluent Operator
- Support more Fluentd & FluentBit plugins
- golang v1.16+.requirement
- kubectl v1.19.2+.
- kubebuilder v3.1+ (the project is build with v3.1.0)
- Access to a Kubernetes cluster v1.19.2+
Make sure you have a KUBECONFIG file loaded for your local developement kubernetes cluster.
- Install CRDs:
make install
export NAMESPACE=something
to set the namespace for the operator- Run:
make run
API Doc is generated automatically. To modify it, edit the comment above struct fields, then run go run cmd/doc-gen/main.go
.
Most files under the folder manifests/setup are automatically generated from config. Don't edit them directly, run make manifests
instead, then replace these files accordingly.
Fluent Operator is sponsored and open-sourced by the KubeSphere Team and maintained by the Fluent community.
This project was accepted by the Fluent community as a sub-project in 2021 and is currently a vendor-neutral and community-driven project.
You are welcome to join us in the Fluent community to polish it together:
- Slack: Create an account in the Fluent Slack workspace and then search
#fluent-operator
to join or click #fluent-operator to join if you already have an account - Wechat: join the KubeSphere Observability SIG by following the
kubesphere
WeChat subscription - Bi-weekly Meeting: https://meeting.tencent.com/dm/mZPB9QKdtX0F (You can find the SIG-Observability meeting schedule here)
- From FluentBit Operator to Fluent Operator: The Long Way to Come - Benjamin Huo
- What is Fluent Operator and how do you use it - Henrik Rexed
- Under the Hood with Fluent Bit Operator: Kubernetes-native Log Processor - Feynman Zhou & Dhruv Kela
- Quick Start: Learn Fluent Operator in 5 minutes (中文) - Dehao Cheng