Skip to content

kscaldef/summingbird-hybrid-example

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

1 Commit
 
 
 
 
 
 
 
 
 
 

Repository files navigation

This is a modification of the summingbird-example provided by Twitter,
using a Kafka source for events and implementing the hybrid batch-realtime
approach.

PREREQUISITES
-------------

You need to have memcached, zookeeper, and kafka installed and running
on your machine on the usual ports.


RUNNING
---------------------------------

To demo the hybrid mode:

  sbt 'run-main summingbird.proto.RunHybrid'

You'll see a bunch of start-up messages, and then it should settle into regularly
spitting out some "sanity check" messages once a minute, both a detailed view of
what's been computed on a single key, and aggregate data about how many messages
have been produced and seen at various stages.  (Some small lag is expected, but
should remain fairly steady over time.)

Once every 5 minutes, the demo rolls over a new log file and kicks off a batch
computation to update the "offline" store.


Example output after running for a while:

14/03/27 13:31:30 INFO example.HybridRunner$: Offline: Some((BatchID.4653174,296))
14/03/27 13:31:30 INFO example.HybridRunner$: Online: (BatchID.4653174,Some(10))
14/03/27 13:31:30 INFO example.HybridRunner$: Hybrid: Some(306)
14/03/27 13:31:30 INFO example.RunHybrid$: Events Produced: 2875
14/03/27 13:31:30 INFO example.RunHybrid$: Events Ingested: 2875
14/03/27 13:31:30 INFO example.RunHybrid$: Events Counted (offline): 2784
14/03/27 13:31:30 INFO example.RunHybrid$: Events Counted (online): 81
14/03/27 13:31:30 INFO example.RunHybrid$: Events Counted (hybrid): 2865

About

A self-contained example of summingbird running in hybrid mode

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages