Spring Cloud Sleuth 3.0 Migration Guide

Spring Cloud Sleuth 3.0 Migration Guide

** This document is a work in progress **

New Features

Most new features in Sleuth 3.0 are around Baggage.

There's now a nicer way to manipulate baggage values, or add them as tags:

// assuming country-code is setup in sleuth.baggage.remote-keys
BaggageField COUNTRY_CODE = BaggageField.create("country-code");

COUNTRY_CODE.updateValue(span.context(), "FO");
String countryCode = COUNTRY_CODE.get(span.context());
Tags.BAGGAGE_FIELD.tag(COUNTRY_CODE, span);

You can also change baggage configuration with Java config.

For example, you can now configure fields to flush immediately to MDC. This is helpful for functions or message processors who learn a value late.

BaggageField BUSINESS_PROCESS = BaggageField.create("bp");

@SendTo(SourceChannels.OUTPUT)
public void timerMessageSource() {
  BUSINESS_PROCESS.updateValue("accounting");
  // You want the expression %{bp} to show "accounting" in businessCode()
  businessCode();
}

Using Java config, you can setup very smart handling without affecting other properties.

@Configuration
class BusinessProcessBaggageConfiguration {
  BaggageField BUSINESS_PROCESS = BaggageField.create("bp");

  /** {@link #BUSINESS_PROCESS} will not be sent as a header */
  @Bean
  BaggagePropagationCustomizer propagateBusinessProcessLocally() {
    return fb -> fb.add(SingleBaggageField.local(BUSINESS_PROCESS));
  }

  /** {@link BaggageField#updateValue(TraceContext, String)} now flushes to MDC */
  @Bean
  CorrelationScopeCustomizer flushBusinessProcessToMDCOnUpdate() {
    return b -> b.add(
        SingleCorrelationField.newBuilder(BUSINESS_PROCESS).flushOnUpdate().build()
    );
  }

  /** {@link #BUSINESS_PROCESS} is added as a tag only in the first span. */
  @Bean
  FinishedSpanHandler tagBusinessProcessOncePerProcess() {
    return new FinishedSpanHandler() {
      @Override public boolean handle(TraceContext context, MutableSpan span) {
        if (context.isLocalRoot()) {
          Tags.BAGGAGE_FIELD.tag(BUSINESS_PROCESS, context, span);
        }
        return true;
      }
    };
  }
}

Migrations

Support for Ribbon is removed

TODO:

Support for Hystrix is removed

TODO:

Support for Eureka is removed

TODO:

ExtraFieldPropagation is deprecated

ExtraFieldPropagation.Factory -> BaggagePropagation.Factory

If you were overriding ExtraFieldPropagation.Factory to control the underlying trace format, do the same with BaggagePropagation.Factory now.

@Bean
BaggagePropagation.FactoryBuilder baggagePropagationFactoryBuilder() {
	return BaggagePropagation.newFactoryBuilder(StackdriverTracePropagation.FACTORY);
}

ExtraFieldPropagation.get,set -> BaggageField

BaggageField should be used to manipulate current baggage.

Ex.

// assuming country-code is setup in sleuth.baggage.remote-keys
BaggageField COUNTRY_CODE = BaggageField.create("country-code");

COUNTRY_CODE.updateValue(span.context(), "FO");
String countryCode = COUNTRY_CODE.get(span.context());
Tags.BAGGAGE_FIELD.tag(COUNTRY_CODE, span);

https://github.com/openzipkin/brave/tree/master/brave#using-baggagefield

SLF4J (MDC)

Property names

Properties controlling SLF4J are now under "spring.sleuth.baggage", but with the same meaning.

Those coming from 2.x should migrate with the following:

• spring.sleuth.log.slf4j.whitelisted-mdc-keys -> spring.sleuth.baggage.correlation-fields
• spring.sleuth.log.slf4j.enabled -> spring.sleuth.baggage.correlation-enabled

The former property names are still read, but will be removed at some point.

parentId and sampled (spanExportable) MDC Fields names are no longer set

For performance reasons, we no longer set the following fields by default:

• parentId
• spanExportable

Sleuth 3.0 implicitly sets below, if you haven't overridden it with "logging.pattern.level"

logging.pattern.level=%5p [${spring.zipkin.service.name:${spring.application.name:}},%X{traceId:-},%X{spanId:-}]

WARNING: This no longer includes the spanExportable field. If you are using Grok/Logstash, and didn't set "logging.pattern.level" in your 2.2.x application, a change to the implicit pattern can break your parsing.

If this applies to you, before you upgrade, set "logging.pattern.level" in your 2.2.x app explicitly to either the above pattern, or the below including spanExportable if your Grok configuration uses it:

logging.pattern.level=%5p [${spring.zipkin.service.name:${spring.application.name:}},%X{traceId:-},%X{spanId:-},%X{spanExportable:-}]

For those using spanExportable, when upgrading to 3.0, add the following to your Java config to avoid the field always showing up as '-'.

@Bean CorrelationScopeCustomizer addSampled() {
  return b -> b.add(SingleCorrelationField.create(BaggageFields.SAMPLED));
}

X-B3-* MDC Fields names are no longer set

The Sleuth 1.x MDC field names are no longer set in correlation config. Their replacements exist since Sleuth 2.0, noted here for completion:

• X-B3-TraceId -> traceId
• X-B3-ParentSpanId -> parentId
• X-B3-SpanId -> spanId
• X-Span-Export -> sampled

If you are still using the deprecated X-B3- names in your logstash configuration, update to the current ones before you upgrade to 3.0.

If for some reason, you need to add the deprecated ones even in 3.0, you can add an additional scope decorator like so, accepting the duplicate overhead of doing so:

@Bean ScopeDecorator legacyIds() {
  return MDCScopeDecorator.newBuilder()
                         .clear()
                         .add(SingleCorrelationField.newBuilder(BaggageFields.TRACE_ID)
                                                    .name("X-B3-TraceId").build())
                         .add(SingleCorrelationField.newBuilder(BaggageFields.PARENT_ID)
                                                    .name("X-B3-ParentSpanId").build())
                         .add(SingleCorrelationField.newBuilder(BaggageFields.SPAN_ID)
                                                    .name("X-B3-SpanId").build())
                         .add(SingleCorrelationField.newBuilder(BaggageFields.SAMPLED)
                                                    .name("X-Span-Export").build())
                         .build();
}

MDC Fields no longer default to "dirty"

Before, all (whitelisted) correlation fields were treated as dirty. Regardless of if values were the same when entering a scope, each value was reverted. This caused extreme overhead in defense of an edge case where someone uses MDC.put() on the same span, and wants it cleaned up for them. We no longer clean up out-of-band changes by default.

If you want to have sleuth clean up ad-hoc use of MDC.put(), you can mark all fields dirty like this, or do similar for the field you want to control:

@Bean
CorrelationScopeCustomizer makeCorrelationFieldsDirty() {
	return b -> {
		Set<CorrelationScopeConfig> configs = b.configs();
		b.clear();

		for (CorrelationScopeConfig config : configs) {
			if (config instanceof SingleCorrelationField) {
				SingleCorrelationField field = (SingleCorrelationField) config;
				if (!field.readOnly()) {
					config = field.toBuilder().dirty().build();
				}
			}
			b.add(config);
		}
	};
}

Baggage

Property names

Properties controlling Baggage are now under "spring.sleuth.baggage", but with the same meaning.

Those coming from 2.x should migrate with the following:

• spring.sleuth.baggage-keys -> custom BaggagePropagationCustomizer described later
• spring.sleuth.local-keys -> spring.sleuth.baggage.local-fields
• spring.sleuth.propagation-keys -> spring.sleuth.baggage.remote-fields
• spring.sleuth.propagation.tag.whitelisted-keys -> spring.sleuth.baggage.tag-fields

The former property names are still read, but will be removed at some point.

spring.sleuth.baggage-keys (prefixing) is deprecated

The "spring.sleuth.baggage-keys" property assigned two headers for each field. This causes unnecessary overhead and can be accomplished in another way now:

@Configuration
static class CustomBaggageConfiguration {
  @Bean BaggagePropagationCustomizer countryCodeBaggageConfig() {
    return fb -> fb.add(SingleBaggageField.newBuilder(BaggageField.create("country-code"))
                           .addKeyName("baggage-country-code")
                           .addKeyName("baggage_country-code")
                           .build());
  }
}

Spring Messaging

Sleuth 1.x Trace ID Headers are deprecated for "b3"

The following custom spring-messaging headers added in Sleuth 1.0 are no longer sent, and a log warning is issued once if they are by outside code.

• spanId
• spanSampled
• spanParentSpanId
• spanTraceId
• spanFlags

Sending the above headers actually increases the headers by up to 10 because they are duplicated in the "native" part of messages. This overhead is extreme especially if messages never leave the process.

The solution is to only send b3 single format, which has been in sleuth since 2.0 and is compatible with JMS. The B3 single format is always parsed and takes precedence, even if multiple headers are sent, so this is a safe change.

Note: Unlike RPC, messaging spans never join with their parent. Better performance is achieved by not propagating the producer's parentId downstream.