Skip to content

Latest commit

 

History

History
148 lines (117 loc) · 11.5 KB

README.md

File metadata and controls

148 lines (117 loc) · 11.5 KB

Elastic Distribution for OpenTelemetry Collector

This is an Elastic supported distribution of the OpenTelemetry Collector.

Running the Elastic Distribution for OpenTelemetry Collector

To run the Elastic Distribution for OpenTelemetry Collector you can use Elastic-Agent binary downloaded for your OS and architecture. Running command

./elastic-agent otel --config otel.yml

from unpacked Elastic Agent package will run Elastic-Agent as an OpenTelemetry Collector. The --config flag needs to point to OpenTelemetry Collector Configuration file. OTel mode is available only using otel subcommand. Elastic Agent will not do any autodetection of configuration file passed when used without otel subcommand and will try to run normally.

To validate OTel configuration run otel validate subcommand:

./elastic-agent otel validate --config otel.yml

Use the components command to get the list of components included in the binary:

./elastic-agent otel components

feature gates are supported using --feature-gates flag.

Components

This section provides a summary of components included in the Elastic Distribution for OpenTelemetry Collector.

Receivers

Component Version
filelogreceiver v0.119.0
hostmetricsreceiver v0.119.0
httpcheckreceiver v0.119.0
jaegerreceiver v0.119.0
jmxreceiver v0.119.0
k8sclusterreceiver v0.119.0
k8sobjectsreceiver v0.119.0
kafkareceiver v0.119.0
kubeletstatsreceiver v0.119.0
nginxreceiver v0.119.0
nopreceiver v0.119.0
otlpreceiver v0.119.0
prometheusreceiver v0.119.0
receivercreator v0.119.0
redisreceiver v0.119.0
zipkinreceiver v0.119.0

Exporters

Component Version
debugexporter v0.119.0
elasticsearchexporter v0.119.0
fileexporter v0.119.0
kafkaexporter v0.119.0
loadbalancingexporter v0.119.0
otlpexporter v0.119.0
otlphttpexporter v0.119.0

Processors

Component Version
attributesprocessor v0.119.0
batchprocessor v0.119.0
elasticinframetricsprocessor v0.13.0
elastictraceprocessor v0.3.0
filterprocessor v0.119.0
geoipprocessor v0.119.0
k8sattributesprocessor v0.119.0
lsmintervalprocessor v0.4.0
memorylimiterprocessor v0.119.0
resourcedetectionprocessor v0.119.0
resourceprocessor v0.119.0
transformprocessor v0.119.0

Extensions

Component Version
filestorage v0.119.0
healthcheckextension v0.119.0
k8sobserver v0.119.0
memorylimiterextension v0.119.0
pprofextension v0.119.0

Connectors

Component Version
routingconnector v0.119.0
signaltometricsconnector v0.3.0
spanmetricsconnector v0.119.0

Persistence in OpenTelemetry Collector

By default, the OpenTelemetry Collector is stateless, which means it doesn't store offsets on disk while reading files. As a result, if you restart the collector, it won't retain the last read offset, potentially leading to data duplication or loss. However, we have configured persistence in the settings provided with the Elastic Agent package.

To enable persistence for the filelogreceiver, we add the file_storage extension and activate it for filelog. Execute export STATE_PATH=/path/to/store/otel/offsets and use the following configuration to enable persistence:

receivers:
  filelog/platformlogs:
    include: [ /var/log/system.log ]
    start_at: beginning
    storage: file_storage/filelogreceiver
extensions:
  file_storage/filelogreceiver:
    directory: ${env:STATE_PATH}
    create_directory: true
exporters:
  ...
processors:
  ...
service:
  extensions: [file_storage]
  pipelines:
    logs/platformlogs:
      receivers: [filelog/platformlogs]
      processors: [...]
      exporters: [...]

Warning

Removing the storage key from the filelog section will disable persistence, which will lead to data duplication or loss when the collector restarts.

Important

If you remove the create_directory: true option, you'll need to manually create a directory to store the data. You can ignore this option if the directory already exists.

Persistence in standalone Docker mode

By default, when running Elastic Distribution for OpenTelemetry Collector in Docker, checkpoints are stored in /usr/share/elastic-agent/otel_registry by default. To ensure data persists across container restarts, you can use the following command:

docker run --rm -ti --entrypoint="elastic-agent" --mount type=bind,source=/path/on/host,target=/usr/share/elastic-agent/otel_registry  docker.elastic.co/elastic-agent/elastic-agent:9.0.0-SNAPSHOT otel

Known issues:

  • You face following failed to build extensions: failed to create extension "file_storage/filelogreceiver": mkdir ...: permission denied error while running the otel mode
    • Cause: This issue is likely because the user running the executable lacks sufficient permissions to create the directory.
    • Resolution: You can either create the directory manually or specify a path with necessary permissions.