Brave v4.7

Brave 4.7 includes enhancements from our community members and the ability to send data to Zipkin in v2 json format.

@hypnoce noticed the service graph wasn't accurate when tracing multiple database connections with our JDBC tracer. You can now control the remote service name by adding a DB property like below:

jdbc:mysql://127.0.0.1:3306/mydatabase?zipkinServiceName=myServiceName

@jorgheymans also helped in the same JDBC trace instrumentation by fixing an unrelated race condition.

@jcchavezs raised his first java language pull request, ensuring when the tracing component is in noop mode, no data collection happens.

Zipkin v2 format

(note this is updated for version 4.8)

Brave uses the zipkin-reporter project to send data to Zipkin collectors. Our version 2 json format is smaller and measurably more efficient.

Once you've upgraded your Zipkin servers, opt-into the version 2 format by using the dependency io.zipkin.reporter2:zipkin-reporter and your favorite sender ex io.zipkin.reporter2:zipkin-sender-okhttp3
Ex:

   /** Configuration for how to send spans to Zipkin */
   @Bean Sender sender() {
-    return OkHttpSender.create("http://your_host:9411/api/v1/spans");
+    return OkHttpSender.create("http://your_host:9411/api/v2/spans");
   }
 
   /** Configuration for how to buffer spans into messages for Zipkin */
-  @Bean Reporter<Span> reporter() {
-    return AsyncReporter.builder(sender()).build();
+  @Bean Reporter<Span> spanReporter() {
+    return AsyncReporter.create(sender()).build();
   }

Stick the v2 reporter here in Brave:

     return Tracing.newBuilder()
-        .reporter(reporter()).build();
+        .spanReporter(spanReporter()).build();

If you are using Spring XML, the related change looks like this:

-  <bean id="sender" class="zipkin.reporter.okhttp3.OkHttpSender" factory-method="create"
+  <bean id="sender" class="zipkin.reporter2.okhttp3.OkHttpSender" factory-method="create"
       destroy-method="close">
-    <constructor-arg type="String" value="http://localhost:9411/api/v1/spans"/>
+    <constructor-arg type="String" value="http://localhost:9411/api/v2/spans"/>
   </bean>
 
   <bean id="tracing" class="brave.spring.beans.TracingFactoryBean">
     <property name="reporter">
       <bean class="brave.spring.beans.AsyncReporterFactoryBean">
+        <property name="encoder" value="JSON_V2"/>

Thanks for all the efforts. Very likely, the next release of Brave will include the ability to propagate data besides trace identifiers. If interested, watch the issue or chat with us on gitter.

Brave 4.6

Brave 4.6 adds Kafka 0.11.0.0 trace instrumentation and lets you be more flexible with thread locals.

Kafka Tracing

Message Tracing is routinely asked for by users who want to see relationships between producers and consumers.

With significant effort by @ImFlog and supported by review by @dgrabows and @devinsba, you can now trace Kafka message flows with Brave. Here's the README, if you'd like to jump right in.

KafkaTracing offers decorators for your producers and consumers, doing the heavy lifting for you.

To use the producer simply wrap it like this :

Producer<K, V> producer = new KafkaProducer<>(settings);
TracingProducer<K, V> tracingProducer = kafkaTracing.producer(producer);
tracingProducer.send(new ProducerRecord<K, V>("my-topic", key, value));

Same goes for the consumer :

Consumer<K, V> consumer = new KafkaConsumer<>(settings);
TracingConsumer<K, V> tracingConsumer = kafkaTracing.consumer(consumer);
tracingConsumer.poll(10);

The above will report data to Zipkin when messages are sent or consumed (even in bulk). Unlike RPC, message processing is decoupled from consumption. For example, messages are often consumed in bulk and processing may happen later or never.

When you are ready to process a message, you can continue the trace like this:

Span forProcessor = kafkaTracing.joinSpan(record);

Non-inheriting default context

Those using CurrentTraceContext.Default, may be aware that it uses an inheritable thread local.
This exists for historical reasons around convenience. However, there are problems with it, such as the inability to cleanup data. You can create CurrentTraceContext.Default without inheritance, and without writing your own.

Brave v4.5

Brave 4.5 adds an end user friendly span customizer and fixes a couple glitches.

End user friendly span customizer

@SirTyro put SpanCustomizer to practice at Tyro and found a common implementation case: lookup the current span or drop input if one isn't in progress. Remember, SpanCustomizer is intended for user code, something easy to mock, unlikely to break compat, etc. If using this implied a hidden dependency on a larger type like Tracer to lookup things, its value disappears.

Poking around, we found @reta handled exactly the same concern in CXF code for JAX-RS @Context parameters. Realizing how important this is to get right, we merged @SirTyro's CurrentSpanCustomizer to provide a smart default out-of-the-box.

Here's an example of what usage might look like

// Some DI configuration wires up the current span customizer
@Bean SpanCustomizer currentSpanCustomizer(Tracing tracing) {
  return CurrentSpanCustomizer.create(tracing);
}

// user code can then inject this without a chance of it being null.
@Inject SpanCustomizer span;

void userCode() {
  span.annotate("tx.started"); // user doesn't care if there is a trace is in progress
  ...
}

Glitch fixes

• @jorgheymans fixed a doc bug on p6spy configuration, which referenced an incorrect parameter. This will help the next person from having to figure out what went wrong. Thanks!
• @kameshsampath fixed our provided dep on jsr305 so that it doesn't break OSGi containers or integrations like brave-karaf

Brave v4.4

Brave v4.4 offers new sampling libraries and a kill switch.

Thanks very much to Tyro, notably @SirTyro for championing much of this work including design reviews, code and practice on all of these features. Thanks also to @reta @jplock and @llinder for design comments needed to ensure other frameworks can reuse these tools. Thanks also to @felixbarny for offering de#sight as many of these features already exist in StageMonitor.

Declarative sampling

Frameworks like JAX-RS allow us to route requests to certain java methods based on annotations present on it. It makes a lot of sense to help these frameworks make tracing declarative, too. We now have DeclarativeSampler, which allows framework extenders an easy path to weave in tracing.

Here's an example from our feature test

    @GET
    @Path("bar")
    @Traced(enabled = false) // overrides path-based decision to trace /bar
    public String bar() {
      return "";
    }

Note: We explicitly decided to not add a brave-specific tracing annotation. The above is just an example. However, we did improve our JAX-RS library, notably exposing ContainerAdapter, such that the above is easier to implement.

Parameterized sampling

Exposing an http sampling api in previous versions of Brave quickly led to requests for a rule-based implementation. Users want to define something simple, like a method, path and rate as opposed to coding an implementation. They also want to define parameterized rules similarly for non-http. For example, a rule with a routing key and queue name for RabbitMQ tracing. Brave v4.4 adds ParameterizedSampler and a default implementation: HttpRuleSampler

Ex. Here's a sampler that traces 80% requests to /foo and 10% of POST
requests to /bar. This doesn't start new traces for requests to favicon
(which many browsers automatically fetch). Other requests will use a
global rate provided by the tracing component.

httpTracingBuilder.serverSampler(HttpRuleSampler.newBuilder()
  .addRule(null, "/favicon", 0.0f)
  .addRule(null, "/foo", 0.8f)
  .addRule("POST", "/bar", 0.1f)
  .build());

Abandoning a span

One thing many noticed is that it is at times easy to create junk within a trace. Redundant or irrelevant data can end up confusing people looking at a trace. In abuse cases or bugs, you can might end up with traces so large they cannot render. Tuning traces will be an ongoing effort. In Brave 4.3 you can now use Span.abandon() to drop a span you know to be unhelpful rather than reporting to Zipkin. Thanks to @SirTyro for his first pull request!

Disabling tracing at runtime

The risk team at Tyro needs the ability to disable all tracing activity at runtime, for example to mitigate tracing abuse cases. As Brave already includes as noop spans, we settled on Tracing.setNoop to disable tracing at runtime. This not only will make all spans noop, but it will also drop any in-flight spans until noop mode is unset.

Brave v4.3

Brave v4.3 includes new instrumentation for common libraries like servlet and JDBC.

We've had high demand to redo our instrumentation over Brave's new apis like brave.Tracer. It took time to put these together the right way, with buy-in from users. Thanks to input and contributions from @jplock @pavolloffay @llinder @devinsba @reta @hyleung @felixbarny @SirTyro and @garyd203, the following are now in place.

Instrumentation

The following decorate commonly used libraries such that trace data end up in Zipkin. Notably, http libraries have portable configuration. This lets you do things like map a custom header as a span tag, and have that work for all libraries.

• grpc - Tracing client and server interceptors for grpc
• httpasyncclient - Tracing decorator for Apache HttpClient 4.0+
• httpclient - Tracing decorator for Apache HttpClient 4.3+
• jaxrs2 - Tracing filters and a feature to automatically configure them
• mysql - Tracing MySQL statement interceptor
• okhttp3 - Tracing decorators for OkHttp 3.x
• p6spy - Tracing event listener for P6Spy (a proxy for calls to your JDBC driver)
• servlet - Tracing filter for Servlet 2.5+ (including Async)
• sparkjava - Tracing filters and exception handlers for SparkJava
• spring-web - Tracing interceptor for Spring RestTemplate
• spring-webmvc - Tracing interceptor for Spring WebMVC

Note: While many of the above now do new things like async tracing, we still have users who trace legacy apps. Notably, our Servlet and Spring instrumentation remain compatible with JRE 6 and Servlet 2.5. Our example shows how to trace an XML-configured app.

Configuration

• brave-context-log4j12 - adds trace and span IDs to the Log4J v1.2 Mapped Diagnostic Context (MDC)
• brave-context-log4j2 - adds trace and span IDs to the Log4J v2 Thread Context
• brave-context-slf4j - adds trace and span IDs to the SLF4J Mapped Diagnostic Context (MDC)
• brave-spring-beans - This allows you to setup tracing with XML instead of custom code.

See also

While there are a significant amount of instrumentation here, many exist in community repos. Some have already or are in the process of moving to Brave v4 apis. For example, watch out for ratpack, play and cassandra. Regardless, if you can help with things we are missing, please raise a pull request or let us know what you are working on via gitter.

Note

Brave 4.3.1 is the minimum version you should use, as this includes adjustments from early users of Brave 4.3.0.

Brave v4.2

Brave 4.2 introduces Tracing: an object graph that holds common things you need.

Tracing Component

When instrumenting RPC calls, you often need more than just the Tracer. For example, you need means to propagate a trace context over headers. You may also need to wrap an executor service. Brave 4.2 simplifies configuration via a component (object graph) that exposes these utilities.

You build it the same way you did before, via a builder.

Tracing tracing = Tracing.newBuilder()
      .localServiceName("my-service")
      ...
      .build();

Now you can compose other things using it. Maybe like this:

okhttpTracing = OkHttpTracing.create(tracing);
callFactory = okhttpTracing.callFactory(okhttpClient);

This is a nice way to pass around an object graph and eliminates the temptation of tacking convenience methods onto Tracer! Thanks to @bogdandrutu and @llinder for discussions leading to this design.

Static initialization

When instrumenting things that load via system properties or JDBC, you may find yourself in a bad spot. This often presents itself as sharing things via static variables or injection.. yuk! When you get into this bind, you can use Tracing.current() or Tracing.currentTracer() to see if tracing is available.

Here's an example of a MySQL interceptor using this.

public ResultSetInternalMethods preProcess(String sql, Statement interceptedStatement,
     Connection connection) throws SQLException {
   Tracer tracer = Tracing.currentTracer();
   if (tracer == null) return null; // not initialized yet!

Remember, only use Tracing.current() when you have no means to get a reference to a tracing component. Thanks to @ewhauser for the initial design work.

CXF is upstream!

Apache CXF tracing is now upstream, so no longer released here. Please see the migration notes @reta made for details.

Brave v4.1

Brave v4.1 introduces SparkJava and deprecates Apache CXF instrumentation. It also adds a new api to replace the ThreadBinder apis present in prior versions.

SparkJava

SparkJava integration has been requested multiple times over the last year. Thanks to effort by @songxin1990, you can setup tracing in only a few lines.

BraveTracing tracing = BraveTracing.create(brave);
Spark.before(tracing.before());
Spark.exception(Exception.class, tracing.exception(new ExceptionHandlerImpl()));
Spark.afterAfter(tracing.afterAfter());

CXF

Apache CXF started in Brave, but thanks to @reta is now in the upstream project. Please stop using brave-cxf3 in favor of CXF's built-in Brave tracing integration.

Current Span

Many of you started using the new Brave 4 Tracing apis. One element missing from this was how to propagate a trace context across threads. Starting in Brave 4.1, there are utilities like Tracer.currentSpan() and Tracer.nextSpan() which rely on a plugin called CurrentTraceContext.

https://github.com/openzipkin/brave/tree/master/brave#current-span

CurrentTraceContext can be customized to integrate with SLF4J or even share data with other tracing systems like Finagle. Check out our feature tests for how this works.

This feature was a long time coming and is a key dependency of Brave 4.x modernization which should be feature complete by v4.2. Many thanks to those who participated in the design and code review, particularly @bogdandrutu @cschneider @devinsba @eirslett @kmaci3k @llinder @reta @smaldini and @tramchamploo

Brave v4.0.2

Brave v4.0.2 fixes a bug where the client address of servlet or grpc instrumentation mistook localhost for the IPv4 address 0.0.0.1 (#337)

Brave 4.0.1

Brave 4.0.1 includes a couple small changes noticed by people using the new api.

• @lijunyong noticed there was no integration from Brave 4's tracer with Brave 3's server tracer. This was an oversight, and now added to TracerAdapter and documented (#334)
• @vlsi we accidentally made animal sniffer annotations a compile dependency. This is also fixed (#336)

Thanks for trying out Brave and reporting glitches!

Brave v4

Brave 4 is a rewrite of the core tracing apis, designed to live alongside existing code. All you need to do is use TracerAdapter to create a (Brave 3) .. Brave. You don't have to change anything else.

Tracer brave4 = Tracer.newBuilder()...build();
Brave brave3 = TracerAdapter.newBrave(brave4);
// use what you already use today

Brave 4's api is made for today's problems, including arbitrarily nested, potentially asynchronous tasks. The new Span type is used for all types of operations: client, server, one-way, local.

Here's an example of modeling a nested task:

try (Span root = tracer.newTrace().name("2pc").start()) {
  try (Span child = tracer.newChild(root.context()).name("prepare").start()) {
    prepare();
  }
  try (Span child = tracer.newChild(root.context()).name("commit").start()) {
    commit();
  }
}

For more, take a look at the comprehensive README or learning tests for new features. Future versions of Brave will improve in-process propagation and start porting instrumentation appropriately.

Thanks to the reviewers @robinst @jplock @devinsba and @raphw, as well the prior art that inspired the new design.