OpenTelemetry instrumentation

[kalaspuff]

Using OpenTelemetry instrumentation

Install tomodachi using the opentelemetry extras to enable instrumentation for OpenTelemetry. In addition, install with the opentelemetry-exporter-prometheus extras to use Prometheus exporter metrics.

local ~$ pip install tomodachi[opentelemetry]
local ~$ pip install tomodachi[opentelemetry,opentelemetry-exporter-prometheus]

When added as a Poetry dependency the opentelemetry extras can be enabled by adding tomodachi = {extras = ["opentelemetry"]} to the pyproject.toml file, and when added to a requiements.txt file the opentelemetry extras can be enabled by adding tomodachi[opentelemetry] to the file.

Auto instrumentation: tomodachi --opentelemetry-instrument

Passing the --opentelemetry-instrument argument to tomodachi run will automatically instrument the service with the appropriate exporters and configuration according to the set OTEL_* environment variables.

If tomodachi is installed in the environment, using tomodachi --opentelemetry-instrument service.py is mostly equivalent to running opentelemetry-instrument tomodachi run service.py and will load distros, configurators and instrumentors automatically in the same way as the opentelemetry-instrument CLI would do.

local ~$ OTEL_LOGS_EXPORTER=console \
    OTEL_TRACES_EXPORTER=console \
    OTEL_METRICS_EXPORTER=console \
    OTEL_SERVICE_NAME=example-service \
    tomodachi --opentelemetry-instrument run service/app.py

The environment variable TOMODACHI_OPENTELEMETRY_INSTRUMENT if set will also enable auto instrumentation in the same way.

local ~$ OTEL_LOGS_EXPORTER=console \
    OTEL_TRACES_EXPORTER=console \
    OTEL_METRICS_EXPORTER=console \
    OTEL_SERVICE_NAME=example-service \
    TOMODACHI_OPENTELEMETRY_INSTRUMENT=1 \
    tomodachi run service/app.py

Auto instrumentation using the opentelemetry-instrument CLI

Auto instrumentation using opentelemetry can then be activated by starting services using opentelemetry-instrument [otel-options] tomodachi run [options] <service.py ...>.

# either define the OTEL_* environment variables to specify instrumentation specification
local ~$ OTEL_LOGS_EXPORTER=console \
    OTEL_TRACES_EXPORTER=console \
    OTEL_METRICS_EXPORTER=console \
    OTEL_SERVICE_NAME=example-service \
    opentelemetry-instrument tomodachi run service/app.py

# or use the arguments passed to the opentelemetry-instrument command
local ~$ opentelemetry-instrument \
    --logs_exporter console \
    --traces_exporter console \
    --metrics_exporter console \
    --service_name example-service \
    tomodachi run service/app.py

Manual instrumentation

Auto instrumentation using either tomodachi --opentelemetry-instrument, setting the TOMODACHI_OPENTELEMETRY_INSTRUMENT=1 env value or using the opentelemetry-instrument CLI are the recommended ways of instrumenting services, as they will automatically instrument the service (and libs with instrumentors installed) with the appropriate exporters and configuration.

However, instrumentation can also be enabled by importing the TomodachiInstrumentor instrumentation class and calling its' instrument function.

import tomodachi
from tomodachi.opentelemetry import TomodachiInstrumentor

TomodachiInstrumentor().instrument()

class Service(tomodachi.Service):
    name = "example-service"

    @tomodachi.http(GET, r"/example")
    async def example(self, request):
        return 200, "hello world"

Starting such a service with the appropriate OTEL_* environment variables would properly instrument traces, logs and metrics for the service without the need to use the opentelemetry-instrument CLI.

local ~$ OTEL_LOGS_EXPORTER=console \
    OTEL_TRACES_EXPORTER=console \
    OTEL_METRICS_EXPORTER=console \
    OTEL_SERVICE_NAME=example-service \
    tomodachi run service/app.py

Service name dynamically set if missing OTEL_SERVICE_NAME value

If the OTEL_SERVICE_NAME environment variable value (or --service_name argument to opentelemetry-instrument) is not set, the resource' service.name will instead be set to the name attribute of the service class. In case the service class uses the default generic names (service or app), the resource' service.name will instead be set to the default as specified in https://github.com/open-telemetry/semantic-conventions/tree/main/docs/resource#service.

Note that instrumentation for logging will currently primarily use the OTEL_SERVICE_NAME, and if it's missing then use the name from the first instrumented service class.

Exclude lists to exclude certain URLs from traces and metrics

To exclude certain URLs from tracking, set the environment variable OTEL_PYTHON_TOMODACHI_EXCLUDED_URLS (or OTEL_PYTHON_EXCLUDED_URLS to cover all instrumentations) to a string of comma delimited regexes that match the URLs.

Regexes from the OTEL_PYTHON_AIOHTTP_EXCLUDED_URLS environment variable will also be excluded.

For example,

export OTEL_PYTHON_TOMODACHI_EXCLUDED_URLS="client/.*/info,healthcheck"

will exclude requests such as https://site/client/123/info and https://site/xyz/healthcheck.

You can also pass comma delimited regexes directly to the instrument method:

TomodachiInstrumentor().instrument(excluded_urls="client/.*/info,healthcheck")

Prometheus meter provider (experimental)

The tomodachi.opentelemetry module also provides a Prometheus meter provider that can be used to export metrics to Prometheus. Run opentelemetry-instrument with the --meter_provider tomodachi_prometheus argument (or set OTEL_PYTHON_METER_PROVIDER=tomodachi_prometheus environment value) to enable the Prometheus meter provider.

local ~$ OTEL_PYTHON_METER_PROVIDER=tomodachi_prometheus \
    opentelemetry-instrument tomodachi run service/app.py

Environment variables to configure Prometheus meter provider

• OTEL_PYTHON_TOMODACHI_PROMETHEUS_ADDRESS specifies the host address the Prometheus export server should listen on. (default: "localhost")
• OTEL_PYTHON_TOMODACHI_PROMETHEUS_PORT specifies the port the Prometheus export server should listen on. (default: 9464)
• OTEL_PYTHON_TOMODACHI_PROMETHEUS_INCLUDE_SCOPE_INFO specifies whether to include scope information as otel_scope_info value. (default: true)
• OTEL_PYTHON_TOMODACHI_PROMETHEUS_INCLUDE_TARGET_INFO specifies whether to include resource attributes as target_info value. (default: true)
• OTEL_PYTHON_TOMODACHI_PROMETHEUS_EXEMPLARS_ENABLED specifies whether exemplars (experimental) should be collected and used in Prometheus export. (default: false)
• OTEL_PYTHON_TOMODACHI_PROMETHEUS_NAMESPACE_PREFIX specifies the namespace prefix for Prometheus metrics. A final underscore is automatically added if prefix is used. (default: "")

Dependency requirement for Prometheus meter provider

The tomodachi_prometheus meter provider requires that the opentelemetry-exporter-prometheusand prometheus_client packages package are installed.

Use tomodachi extras opentelemetry-exporter-prometheus to automatically include a compatible version of the exporter.

OpenMetrics output from Prometheus with exemplars enabled

With exemplars enabled, make sure to call the Prometheus client with the accept header application/openmetrics-text to ensure exemplars are included in the response.

curl http://localhost:9464/metrics -H "Accept: application/openmetrics-text"

Note that if the accept header application/openmetrics-text is missing from the request, exemplars will be excluded from the response.

Example: starting a service with instrumentation

This example will start and instrument a service with OTLP exported traces sent to the endpoint otelcol:4317 and metrics that can be scraped by Prometheus from port 9464. All metrics except for target_info and otel_scope_inf will be prefixed with "tomodachi_". Additionally exemplars will be added to the Prometheus collected metrics that includes sample exemplars with trace_id and span_id labels.

local ~$ TOMODACHI_OPENTELEMETRY_INSTRUMENT=1 \
    OTEL_TRACES_EXPORTER=otlp \
    OTEL_EXPORTER_OTLP_ENDPOINT=otelcol:4317 \
    OTEL_PYTHON_METER_PROVIDER=tomodachi_prometheus \
    OTEL_PYTHON_TOMODACHI_PROMETHEUS_EXEMPLARS_ENABLED=true \
    OTEL_PYTHON_TOMODACHI_PROMETHEUS_ADDRESS=0.0.0.0 \
    OTEL_PYTHON_TOMODACHI_PROMETHEUS_PORT=9464 \
    OTEL_PYTHON_TOMODACHI_PROMETHEUS_NAMESPACE_PREFIX=tomodachi \
    tomodachi run service/app.py

[codecov]

Codecov Report

Patch coverage: 76.06% and project coverage change: +0.67% 🎉

Comparison is base (deca849) 72.92% compared to head (058fbec) 73.59%.

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #1790      +/-   ##
==========================================
+ Coverage   72.92%   73.59%   +0.67%     
==========================================
  Files          43       52       +9     
  Lines        6193     7646    +1453     
==========================================
+ Hits         4516     5627    +1111     
- Misses       1677     2019     +342     

Files Changed	Coverage Δ
tomodachi/helpers/safe_modules.py	100.00% <ø> (ø)
tomodachi/launcher.py	50.00% <0.00%> (-4.23%)	⬇️
tomodachi/opentelemetry/prometheus.py	47.36% <47.36%> (ø)
tomodachi/opentelemetry/exemplars.py	48.68% <48.68%> (ø)
tomodachi/container.py	67.15% <52.63%> (-0.96%)	⬇️
tomodachi/opentelemetry/auto_instrumentation.py	62.50% <62.50%> (ø)
tomodachi/importer.py	69.72% <66.66%> (-0.76%)	⬇️
tomodachi/transport/amqp.py	79.47% <75.00%> (-0.41%)	⬇️
tomodachi/transport/http.py	65.69% <80.00%> (+0.15%)	⬆️
tomodachi/opentelemetry/distro.py	82.78% <82.78%> (ø)
... and 11 more

... and 2 files with indirect coverage changes

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.