Skip to content

Files

Latest commit

 

History

History

fluentd

Folders and files

NameName
Last commit message
Last commit date

parent directory

..
 
 
 
 
 
 
 
 
 
 
 
 

Logzio-fluentd

Helm is a package management tool that uses Charts to deploy pre-configured sets of Kubernetes resources. The Logzio-fluentd chart enables you to transmit logs from your Kubernetes cluster to Logz.io via Fluentd. Fluentd's flexibility and availability of plugins allow for easy distribution of logs to various third-party services, including Logz.io.

The chart defaults to configuration for Conatinerd CRI. If your cluster uses Docker as CRI, please refer to daemonset.containerdRuntime in the configuration table.

Note: This chart is for shipping logs only. For a chart that ships all telemetry data, including logs, metrics, traces, and SPM, use our Logzio Monitoring chart.

Deploying the Chart:

1. Create a monitoring namespace

Your DaemonSet will be deployed under the namespace monitoring.

kubectl create namespace monitoring

2. Add logzio-fluentd repo to your helm repo list

helm repo add logzio-helm https://logzio.github.io/logzio-helm

3. Deploy

The following command installs the Chart using the default values. If you want to modify any settings, append the --set flag(s) to this command along with the parameters you wish to change. For more information and examples, refer to the configuration table. You can find additional details on how to customize the Chart's values here.

Replace <<LOG-SHIPPING-TOKEN>> with the token of the account you want to ship to.

Replace <<LISTENER-HOST>> with your account's listener host. You can find your listener in your manage tokens page.

helm install -n monitoring \
--set secrets.logzioShippingToken='<<LOG-SHIPPING-TOKEN>>' \
--set secrets.logzioListener='<<LISTENER-HOST>>' \
logzio-fluentd logzio-helm/logzio-fluentd

If adding the secret directly through the Helm chart is not possible or secure (for example, due to the absence of secure value-file storage), you can override the secret's name by modifying secretName in the values file. This adjustment allows you to place an external secret containing the keys logzioShippingToken and logzioListener through alternative methods.

4. Check Logz.io for your logs

Give your logs some time to get from your system to ours, and then open Logz.io.

Configuration

This table contains all the parameters in values.yaml. If you wish to change the default values, specify each parameter using the --set key=value argument to helm install in step 2. For example:

helm install -n monitoring \
  --set terminationGracePeriodSeconds=40 \
  --set daemonset.logzioLogLevel=debug \
  --set-file configmap.extraConfig=/path/to/config.yaml \
  logzio-fluentd logzio-helm/logzio-fluentd
Parameter Description Default
image The logzio-fluentd docker image. logzio/logzio-fluentd
imageTag The logzio-fluentd docker image tag. 1.4.0
nameOverride Overrides the Chart name for resources. ""
fullnameOverride Overrides the full name of the resources. ""
apiVersions.daemonset Daemonset API version. apps/v1
apiVersions.serviceAccount Service Account API version. v1
apiVersions.clusterRole Cluster Role API version. rbac.authorization.k8s.io/v1
apiVersions.clusterRoleBinding Cluster Role Binding API version. rbac.authorization.k8s.io/v1
apiVersions.configmap Configmap API version. v1
apiVersions.secret Secret API version. v1
namespace Chart's namespace. monitoring
fargateLogRouter.enabled Boolen to decide if to configure fargate log router false
env_id Add to your logs field env_id with identification of the environment you're shipping logs from. ""
isRBAC Specifies whether the Chart should be compatible to a RBAC cluster. If you're running on a non-RBAC cluster, set to false. true
isPrivileged Specifies whether to run the Damonset with priviliged security context false
serviceAccount.name Name of the service account. ""
daemonset.podSecurityContext Security context for the pod level {}
daemonset.securityContext Security context for the container level {}
daemonset.initContainerSecurityContext Security context for the init container {}
daemonset.tolerations Set tolerations for all DaemonSet pods. See values.yaml.
daemonset.nodeSelector Set nodeSelector for all DaemonSet pods. {"kubernetes.io/os": "linux"}
daemonset.affinity Set affinity rules for the scheduler to determine where all DaemonSet pods can be placed.
daemonset.fluentdSystemdConf Controls whether Fluentd system messages will be enabled. disable
daemonset.fluentdPrometheusConf Controls the launch of a prometheus plugin that monitors Fluentd. false
daemonset.includeNamespace Use if you wish to send logs from specific k8s namespaces, space delimited. Should be in the following format: kubernetes.var.log.containers.**_<<NAMESPACE-TO-INCLUDE>>_** kubernetes.var.log.containers.**_<<ANOTHER-NAMESPACE>>_**. ""
daemonset.kubernetesVerifySsl Enables to validate SSL certificates. true
daemonset.auditLogFormat Match Fluentd's format for kube-apiserver audit logs. Set to audit-json if your audit logs are in json format. audit
daemonset.containerdRuntime Deprecated from chart version 0.1.0. Determines whether to use a configuration for a Containerd runtime. Set to false if your cluster doesn't use Containerd as CRI. true
daemonset.cri Container runtime interface of the cluster. Used to determine which configuration to use when concatenating partial logs. Valid options are: docker, containerd. containerd
daemonset.LogFileRefreshInterval The interval of refreshing the list of watch file for log files.. 60s
daemonset.logzioBufferType Specifies which plugin to use as the backend. file
daemonset.logzioBufferPath Path of the buffer. /var/log/fluentd-buffers/stackdriver.buffer
daemonset.logzioOverflowAction Controls the behavior when the queue becomes full. block
daemonset.logzioChunkLimitSize Maximum size of a chunk allowed. 2M
daemonset.logzioQueueLimitLength Maximum length of the output queue. 6
daemonset.logzioFlushInterval Interval, in seconds, to wait before invoking the next buffer flush. 5s
daemonset.logzioRetryMaxInterval Maximum interval, in seconds, to wait between retries. 30
daemonset.logzioRetryForever If true, plugin will retry flushing forever true
daemonset.logzioFlushThreadCount Number of threads to flush the buffer. 2
daemonset.logzioLogLevel The log level for this container. info
daemonset.excludeFluentdPath Path to fluentd logs file, to exclude them from the logs that Fluent tails. /var/log/containers/*fluentd*.log
daemonset.extraExclude A comma-seperated list (no spaces), of more paths to exclude from the Fluentd source that tails containers logs. For example - /path/one.log,/path/two.log ""
daemonset.containersPath Path for containers logs. "/var/log/containers/*.log"
daemonset.posFile Path for containers logs pos file. "/var/log/fluentd-containers.log.pos"
daemonset.logType Set log type for the logs. "k8s"
daemonset.extraEnv If needed, more env vars can be added with this field. []
daemonset.resources Allows you to set the resources for Fluentd Daemonset. See values.yaml.
daemonset.extraVolumeMounts If needed, more volume mounts can be added with this field. []
daemonset.terminationGracePeriodSeconds Termination period (in seconds) to wait before killing Fluentd pod process on pod shutdown. 30
daemonset.extraVolumes If needed, more volumes can be added with this field. []
daemonset.init.extraVolumeMounts If needed, more volume mounts to the init container can be added with this field. []
daemonset.init.containerImage Init container image for the fluentd daemonset. busybox
daemonset.priorityClassName Set priorityClassName for all DaemonSet pods. ""
daemonset.updateStrategy Strategy to use when updating the Daemonset. {}
windowsDaemonset.enabled Enables Fluentd Daemonset for Windows. true
windowsDaemonset.kubernetesVerifySsl Enables to validate SSL certificates (windows). true
windowsDaemonset.auditLogFormat Match Fluentd's format for kube-apiserver audit logs. Set to audit-json if your audit logs are in json format. (windows) audit
windowsDaemonset.containerdRuntime Deprecated from chart version 0.1.0. Determines whether to use a configuration for a Containerd runtime. Set to false if your cluster doesn't use Containerd as CRI. (windows) true
windowsDaemonset.cri Container runtime interface of the cluster. Used to determine which configuration to use when concatenating partial logs (windows). Valid options are: docker, containerd. containerd
windowsDaemonset.LogFileRefreshInterval The interval of refreshing the list of watch file for log files.. 60s
windowsDaemonset.logzioBufferType Specifies which plugin to use as the backend. file
windowsDaemonset.logzioBufferPath Path of the buffer. (windows) /var/log/fluentd-buffers/stackdriver.buffer
windowsDaemonset.logzioOverflowAction Controls the behavior when the queue becomes full. (windows) block
windowsDaemonset.logzioChunkLimitSize Maximum size of a chunk allowed. (windows) 2M
windowsDaemonset.logzioQueueLimitLength Maximum length of the output queue. (windows) 6
windowsDaemonset.logzioFlushInterval Interval, in seconds, to wait before invoking the next buffer flush. (windows) 5s
windowsDaemonset.logzioRetryMaxInterval Maximum interval, in seconds, to wait between retries. (windows) 30
windowsDaemonset.logzioRetryForever If true, plugin will retry flushing forever (windows) true
windowsDaemonset.logzioFlushThreadCount Number of threads to flush the buffer. (windows) 2
windowsDaemonset.logzioLogLevel The log level for this container. (windows) info
windowsDaemonset.excludeFluentdPath Path to fluentd logs file, to exclude them from the logs that Fluent tails. /var/log/containers/*fluentd*.log
windowsDaemonset.extraExclude A comma-seperated list (no spaces), of more paths to exclude from the Fluentd source that tails containers logs. For example - /path/one.log,/path/two.log ""
windowsDaemonset.containersPath Path for containers logs. "/var/log/containers/*.log"
windowsDaemonset.extraEnv If needed, more env vars can be added with this field. (windows) []
windowsDaemonset.resources Allows you to set the resources for Fluentd Daemonset. (windows) See values.yaml.
windowsDaemonset.extraVolumeMounts If needed, more volume mounts can be added with this field. (windows) []
daemonset.terminationGracePeriodSeconds Termination period (in seconds) to wait before killing Fluentd pod process on pod shutdown. 30
windowsDaemonset.extraVolumes If needed, more volumes can be added with this field. (windows) []
windowsDaemonset.priorityClassName Set priorityClassName for all DaemonSet pods. (windows) ""
windowsDaemonset.updateStrategy Strategy to use when updating the Daemonset. {}
windowsDaemonset.nodeSelector Set nodeSelector for all DaemonSet pods. {"kubernetes.io/os": "windows"}
clusterRole.rules Configurable cluster role rules that Fluentd uses to access Kubernetes resources. See values.yaml.
secrets.logzioShippingToken Secret with your logzio shipping token. ""
secrets.logzioListener Secret with your logzio listener host. listener.logz.io. " "
secrets.customEndpoint Secret with your custom endpoint, for example:http://endpoint:8080. Overrides secrets.logzioListener ""
secrets.enabled When true, the logzio secret will be created and managed by this Chart. If you're managing the logzio secret by yourself, set to false. true
secretName Name of the secret in case it's placed from an external source. logzio-logs-secret
configMapIncludes Initial includes for fluent.conf. See values.yaml.
configmap.extraConfig If needed, more Fluentd configuration can be added with this field. {}
configmap.fluent Configuration for fluent.conf. See values.yaml.
configmap.kubernetes Configuration for kubernetes.conf. See values.yaml.
configmap.system Configuration for system.conf. See values.yaml.
configmap.systemd Configuration for systemd.conf. See values.yaml.
configmap.kubernetesContainerd Deprecated from chart version 0.1.0. Configuration for kubernetes-containerd.conf. This is the configuration that's being used when daemonset.containerdRuntime is set to true See values.yaml.
configmap.partialDocker Configuration for partial-docker.conf. Used to concatenate partial logs that split due to large size, for docker cri. See values.yaml.
configmap.partialContainerd Configuration for partial-containerd.conf. Used to concatenate partial logs that split due to large size, for containerd cri. See values.yaml.
configmap.audit Configuration for audit.conf. See values.yaml.
configmap.auditJson Configuration for audit-json.conf. This is the configuration that's being used when daemonset.auditLogFormat is set to audit-json See values.yaml.
configmap.envId Config snippet for adding env_id field to logs See values.yaml.
configmap.customSources Add sources to the Fluentd configuration ""
configmap.customFilters Add filters to the Fluentd configuration ""
configmap.customFilterAfter Add filters to the Fluentd configuration, after default filters ""
logLevelFilter Add log level filter. Regex of the log level(s) you want to ship. For example, if you want to ship warning and error logs, use WARNING|ERROR. Possible levels are: DEBUG, INFO, WARNING, ERROR, TRACE. ""

Note: If you're adding your own configuration file via configmap.extraConfig:

  • Add a --set-file flag to your helm install command, as seen in the example above.
  • Make sure that the yaml file with your configuration is in the following format:
my-custom-conf-name.conf: |-
   # .....
   # your config
   # .....
my-custom-conf-name2.conf: |-
   # .....
   # your config
   # .....

Handling image pull rate limit

In some cases (i.e spot clusters) where the pods/nodes are replaced frequently, the pull rate limit for images pulled from dockerhub might be reached, with an error: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limits. In these cases we can use the following --set command to use an alternative image repository:

--set image=public.ecr.aws/logzio/logzio-fluentd
--set daemonset.init.containerImage=public.ecr.aws/docker/library/busybox

Adding a custom log_type field from attribute

To add a log_type field with a custom value to each log, you can use the annotation key log_type with a custom value. The annotation will be automatically parsed into a log_type field with the provided value. e.g:

...
  metadata:
    annotations:
      log_type: "my_type"

Will result with the following log (json):

{
...
,"log_type": "my_type"
...
}

Uninstalling the Chart

The command removes all the k8s components associated with the chart and deletes the release.

To uninstall the logzio-fluentd deployment:

helm uninstall -n monitoring logzio-fluentd

Configuring Fluentd to concatenate multiline logs using a plugin

Fluentd splits multiline logs by default. If your original logs span multiple lines, you may find that they arrive in your Logz.io account split into several partial logs.

The Logz.io Docker image comes with a pre-built Fluentd filter plug-in that can be used to concatenate multiline logs. The plug-in is named fluent-plugin-concat and you can view the full list of configuration options in the GitHub project.

Example

The following is an example of a multiline log sent from a deployment on a k8s cluster:

2021-02-08 09:37:51,031 - errorLogger - ERROR - Traceback (most recent call last):
File "./code.py", line 25, in my_func
1/0
ZeroDivisionError: division by zero

Fluentd's default configuration will split the above log into 4 logs, 1 for each line of the original log. In other words, each line break (\n) causes a split.

To avoid this, you can use the fluent-plugin-concat and customize the configuration to meet your needs. The additional configuration is added to:

  • kubernetes.conf for RBAC/non-RBAC DaemonSet
  • kubernetes-containerd.conf for Containerd DaemonSet

For the above example, we could use the following regex expressions to demarcate the start and end of our example log:

<filter **>
  @type concat
  key message # The key for part of multiline log
  multiline_start_regexp /^[0-9]{4}-[0-9]{2}-[0-9]{2}/ # This regex expression identifies line starts.
</filter>

Sending logs from nodes with taints

If you want to ship logs from any of the nodes that have a taint, make sure that the taint key values are listed in your in your daemonset configuration as follows:

tolerations:
- key: 
  operator: 
  value: 
  effect: 

To determine if a node uses taints as well as to display the taint keys, run:

kubectl get nodes -o json | jq ".items[]|{name:.metadata.name, taints:.spec.taints}"

Sending logs from eks on fargate

If you want to ship logs from pods that are running on fargate set the fargateLogRouter.enabled value to true, the follwing will deploy a dedicated aws-observability namespace and a configmap for fargate log router. More information about eks fargate logging can be found here

helm install \
--set fargateLogRouter.enabled=true \
--set secrets.logzioShippingToken='<<LOG-SHIPPING-TOKEN>>' \
--set secrets.logzioListener='<<LISTENER-HOST>>' \
logzio-fluentd logzio-helm/logzio-fluentd

Monitoring fluentd with prometheus

In order to monitor fluentd and collect input & output metrics. You can enable prometheus configuration with the daemonset.fluentdPrometheusConf and windowsDaemonset.fluentdPrometheusConf parameter (default to false). When enabling promehteus configuration, the pod collects and exposes fluentd metrics on port 24231, /metrics endpoint. The templates contains annotations to easly ship when using promehteus shipper or logzio-telemetry chart. Monitoring Windows fluentd is not supported.

Fluentd images for windows server

By default the fluentd image for windows-server supports windows server 2019. If needed, the fluentd image can be changed to support windows server 2022 with the following commands:

--set windowsImage=logzio/fluentd-windows-2022 \
--set windowsImageTag=0.0.5

Change log

  • 0.30.6:
    • Upgrade fluentd version to 1.18.0
  • 0.30.5:
    • Upgrade fluentd version to 1.17.1
  • 0.30.4:
    • Fix nodeSelector indentation
  • 0.30.3:
    • Resolve nodeSelector bug
  • 0.30.2:
  • Remove default resources limits
  • 0.30.1:
  • Handle empty etcd log key, populated based on message key.
  • 0.30.0:
  • Upgrade fluentd version to 1.16.5
  • Fix bug of env-id.conf
Expand to check old versions
  • 0.29.2:
  • Enhanced env_id handling to support both numeric and string formats.
  • 0.29.1:
    • Added enabled value, to conditianly control the deployment of this chart by a parent chart.
    • Added daemonset.LogFileRefreshInterval and windowsDaemonset.LogFileRefreshInterval values, to control list of watched log files refresh interval.
  • 0.29.0:
    • EKS Fargate logging:
    • Send logs to port 8070 in logzio listener (instead of port 5050)
  • 0.28.1:
    • Added windowsDaemonset.enabled customization.
  • 0.28.0:
    • Added daemonset.initContainerSecurityContext customization.
    • Added daemonset.updateStrategy customization.
  • 0.27.0:
    • Added daemonset.podSecurityContext, daemonset.securityContext customization.
  • 0.26.0:
    • Bump docker image to 1.5.1.
    • Add ability to configure pos file for containers logs.
  • 0.25.0:
    • Add parameter isPrivileged to allow running Daemonset with priviliged security context.
    • Bug fix: Fix template for fluentd.serviceAccount, and fix use of template in service account.
  • 0.24.0:
    • Add parameter configmap.customFilterAfter that allows adding filters AFTER built-in filter configuration.
    • Added daemonset.init.containerImage customization.
    • Added fluentd image for windows server 2022.
  • 0.23.0:
    • Allow filtering logs by log level with logLevelFilter.
  • 0.22.0:
    • Add custom endpoint option with secrets.customEndpoint.
  • 0.21.0:
  • Bump docker image to 1.5.0:
    • Upgrade fluentd to 1.16.
    • Upgrade gem fluent-plugin-logzio to 0.2.2:
      • Do not retry on 400 and 401. For 400 - try to fix log and resend.
      • Generate a metric (logzio_status_codes) for response codes from Logz.io.
  • 0.20.3:
    • ezKonnect support: Added logz.io/application_type to type annotation check .
  • 0.20.2:
    • Upgrade docker image logzio/logzio-fluentd to 1.4.0:
      • Use fluentd's retry instead of retry in code (raise exception on non-2xx response).
  • 0.20.1:
    • Added log level detection for fargate log router
    • Remove namespace value, replaced by Realese.namespace in all templates
  • 0.20.0:
    • Upgraded windows image to logzio/windows:0.0.2:
      • Added prometheus monitor plugin
      • Added dedot plugin
    • Updated windowsDaemonset.fluentdPrometheusConf - now controls prometheus config for collecting and exposing fluentd metrics.
  • 0.19.0:
    • Upgraded image to logzio/logzio-fluentd:1.3.1:
      • Added prometheus monitor plugin
    • Updated daemonset.fluentdPrometheusConf - now controls prometheus config for collecting and exposing fluentd metrics.
  • 0.18.0:
    • Added log_level detection for "warn" level.
  • 0.17.0:
    • Add secrets.enabled to control secret creation and management. (#194)
  • 0.16.0:
    • Increased memory request and limit to 500Mi, cpu request to 200m.
  • 0.15.0:
    • Added dedot processor - auto replace . in log field to _.
  • 0.14.0:
    • Fix typo in fargateLogRouter
  • 0.13.0:
    • Removal of field log_type. Auto populating type instead.
  • 0.12.0:
    • Added auto detection for log_level field.
  • 0.11.0:
    • Upgrade image logzio/logzio-fluentd:1.2.0:
      • Upgrade to fluentd 1.15.
      • Upgrade plugin fluent-plugin-kubernetes_metadata_filter to 3.1.2.
  • 0.10.0:
    • Added an option to parse log_type annotation into log_type field.
  • 0.9.0:
    • Added a default value for env_id field.
  • 0.8.0:
    • Add ability to add environment id with env_id field.
  • 0.7.0:
    • Add ability to change the secret name with secretName. #133
  • 0.6.1:
    • Fix bug for extraConfig (#114).
  • 0.6.0:
    • Added daemonset.priorityClassName and windowsDaemonset.priorityClassName.
  • 0.5.0:
    • Add support for daemonset.affinity value.
    • Add support for fargate logging.
  • 0.4.1:
    • Upgrade default image version to logzio/logzio-fluentd:1.1.1.
  • 0.4.0:
    • Allow dynamically set the log type for the logs.
  • 0.3.0:
    • Added new value fields: daemonset.excludeFluentdPath, daemonset.extraExclude, daemonset.containersPath, configmap.customSources, configmap.customFilters.
    • Added support for windows containers.
  • 0.2.0:
    • Added daemonset.nodeSelector.
  • 0.1.0:
    • Upgrade default image version to logzio/logzio-fluentd:1.0.2 which also supports ARM architecture.
    • Deprecated variables: daemonset.containerdRuntime, configmap.kubernetesContainerd.
    • Added configmap.partialDocker, configmap.partialContainerd that concatenate logs that split due to large size (over 16k). To learn more go to the configuration table.
    • Added daemonset.cri to match the partial log config to the cluster's CRI. To learn more go to the configuration table.
  • 0.0.4:
    • Refactor configmaps
  • 0.0.3:
    • Edit configmap template name
  • 0.0.2:
    • Fix templates name - allow dyncmically change it.
  • 0.0.1:
    • Initial release.