Add logs to spans using OTEL instrumentation with Jaegar backend

Current Situation:

At the time of writing, in August 2021, OTEL spans have no mechanism to add logs as found in implementations such as Jaegar.

Workaround:

As we saw here, jaegar backend interprets OTEL exceptions in way where the contents of the exception are put in as Logs in the associated span.

Now, exceptions are a form of events, and it seems jaegar backend interprets events as Logs. So we can replicate this behavior by:

1. Creating a custom log appender
2. make it create an OTEL event populate logging details in it.
3. add the event to the current span.

This span will be interpreted by jaegar backend in a way where all the events are put in as individual log items in that span.

Custom Log Appender

Below is a basic LogAppender i wrote based on SpanLogsAppender.java from the spring-cloud project.

import ch.qos.logback.classic.Level;
import ch.qos.logback.classic.spi.ILoggingEvent;
import ch.qos.logback.classic.spi.IThrowableProxy;
import ch.qos.logback.classic.spi.ThrowableProxy;
import ch.qos.logback.core.AppenderBase;
import io.opentelemetry.api.common.Attributes;
import io.opentelemetry.api.common.AttributesBuilder;
import io.opentelemetry.api.trace.Span;
import io.opentelemetry.api.trace.StatusCode;


public class SpanLogsAppender extends AppenderBase<ILoggingEvent> {

    /**
     * This is called only for configured levels.
     * It will not be executed for DEBUG level if root logger is INFO.
     */
    @Override
    protected void append(ILoggingEvent event) {
        final Span currentSpan = Span.current();
        AttributesBuilder builder = Attributes.builder();

        if (currentSpan != null) {
            builder.put("logger", event.getLoggerName())
                    .put("level", event.getLevel().toString())
                    .put("message", event.getFormattedMessage());

            currentSpan.addEvent("LogEvent", builder.build());

            if (Level.ERROR.equals(event.getLevel())) {
                currentSpan.setStatus(StatusCode.ERROR);
            }

            IThrowableProxy throwableProxy = event.getThrowableProxy();
            if (throwableProxy instanceof ThrowableProxy) {
                Throwable throwable = ((ThrowableProxy)throwableProxy).getThrowable();
                if (throwable != null) {
                    currentSpan.recordException(throwable);
                }
            }
        }
    }
}

My local versions:

• spring boot : 2.5.1
• io.opentelemetry.opentelemetry-bom : 1.2.0
• jaegar backend: 1.18 (windows)