Skip to content
This repository has been archived by the owner on Jan 20, 2022. It is now read-only.

Latest commit

 

History

History
306 lines (266 loc) · 19.8 KB

CHANGES.md

File metadata and controls

306 lines (266 loc) · 19.8 KB
Raw

Summingbird

0.9.1

  • Fixes a bug in the LTuple2 class. Add a test for it: #632

0.9.0

  • Removing internal config setup from scalding platform: #629
  • Remove store that no one seems to use and has no tests: #630
  • Ianoc/optimization changes: #628
  • Fixes tests: #626
  • resilient to all stores being complete: #627
  • Resilient to a store being ahead: #625
  • move to scala test: #621
  • Killing off Summingbird Java: #624
  • Kill summingbird spark: #622
  • Bumping sbt versions: #623
  • Fixed erroneous comment in ClientStore's merge description: #620

0.8.0

  • bumping Scalding to 0.15.0, algebird to 0.10.1

0.7.0

  • Registering summingbird counters with tormenta spouts #553
  • Add counters to default summers #571
  • Fixes the types off the summingbird java storm api #601
  • Adding explicit hadoop deps; dfs-datastores to 1.3.6 #592
  • Replace deprecated ClassManifest and erasure #608
  • Spark monad #607
  • replace deprecated erasure by runtimeClass #609
  • Revert "Remove StripNameNodes" #610
  • refactored core-tests into separate sub-project #593
  • Fix checking delta batch in readDeltaTimestamps in batched store when merging #612
  • ensure at least one batch before merging #613
  • MergeableStoreFactory extends java.io.Serializable #616
  • Spout storm metrics foreach #617
  • upgrade scalacheck #615
  • Use latest bijection, algebird, scalding, storehaus, chill, and tormenta

0.6.0

  • Use latest bijection, algebird, scalding, storehaus, chill, and tormenta #565
  • Remove scala 2.9 support #565
  • Add counters to default summers #571
  • Registering summingbird counters with tormenta spouts #553
  • Remove StripNameNodes: #587
  • Revert unneeded wrapper in DagOptimizer: #584
  • Make some classes public so we can make custom platforms: #564
  • IdentityKeyedProducer in ValueFlatMapToFlatMap Dag Optimizer rule: #580
  • Lazy store init for CombinedServiceStoreFactory: #582
  • Pass the reducers option down to leftJoin: #578
  • Rename readBatched to readAfterLastBatch: #577
  • Fixes BatchedStore interval calculation: #573
  • Support for creating a store-service for Storm platform: #563
  • Support leftJoin against a store in offline platform: #557
  • Counters for ConcurrentMemory platform: #550
  • Summingbird Storm/Online refactor: #544
  • Memory platform counters: #548
  • Example of using DagOptimizer: #538

0.5.1

  • Change the javac options for doc generation in java to be successful. (Blocks maven publishing)

0.5.0

  • Bumps version of other projects. Changes to compile with latest scalding...: #533
  • Safely checking for the internal storm Implementation detail: #528
  • add implicit Successible and Predecessible for Timestamp: #530
  • Removes redundant Future.unit call: #526
  • Optimize CalendarBatcher a bit: #527
  • Fixes java publishing to not error in the doc generation/publishing -- i...: #519
  • Fix online plan bug with multiple serial summers and certain no-ops: #524
  • Fixing tormentaSpout.registerMetric call in scheduleSpout: #521
  • Use scalariform: #520
  • Adds a test for a bug if a flatMap immediately before a leftJoin produce...: #518
  • Fixed Storm stats bug with Promise: #514
  • Platform-independent stats storm scalding: #503
  • Experimental Java API and example: #499
  • First pass at a spark platform for summingbird: #502
  • Add summingbird logo: #504
  • expose flow object to mutate function: #501
  • Support java6 with our one java file: #498
  • New async summers from algebird: #497
  • Feature/stat store: #495
  • Add some option documentation: #491
  • Yet another cache: #470
  • Monoid to semigroup: #487

0.4.2

  • Use scalding version 0.9.0rc17 which has some bug fixes
  • Add fully satisfiable flag to UniqueKeyedService: #489
  • Mark the scalding platform as serializable, but also don't grab the platform in the scalding env: #485
  • smoketest displaying error running example: #483

0.4.1

  • Release tool failure fix from 0.4.0 release.

0.4.0

  • Add/use the multiplier option: #481
  • add variations on pipeFactory to allow a map function before time extrac...: #482
  • Make HDFSState public: #478
  • Feature/ianoc frozen keys: #477
  • Piping prune function through the VersionedStore factory method: #476
  • Remove deprecation indirections: #472
  • fix unidoc target: #469
  • Add hadoop2 compression options: #466
  • Few more profiler noticed changes, mostly Future.collect indexedSeq's. O...: #465
  • Add deprecation: #463
  • Vagrant folder added to .gitignore: #464
  • Manually block up sections of output from caches into lists to avoid flo...: #453
  • Fixes bug that meant storm tests weren't running: #460
  • Factor batch from online: #455
  • This moves from using the completeTopology approach to running it normal...: #458
  • Code is called heavily, and in tests where we do it 100's of times it wa...: #457
  • Clean up a little bit of this code. Uses seq/traversable once in some ap...: #456
  • This refactors time out of several places inside the online code. Allows...: #454
  • Move the hadoop defaults into scalding platform: #452
  • Feature/add option to tune output sizes online: #451
  • Move to using Mergable from storehaus instead of mergable store.: #450
  • Feature/state deprecated ptrs: #449
  • Adds BatchID interval to TS interval: #443
  • Adds a scalding-test project: #447
  • Make Commutativity settings a case object: #448
  • Splits the storm test code out into its own package to help end users be...: #442
  • Add toString to InitialBatchedStore: #444
  • Feature/long to timestamp offline: #439
  • Feature/add pruning support: #435
  • Split the scalding platform out into packages: #434
  • Add mima command support: #427
  • Add PipeFactoryOps, with mapElements, flatMapElements and mapPipe: #432
  • Feature/move code from scalding: #424

0.3.3

  • Depend on proper Tormenta version (fix semver hiccup)

0.3.2

  • We don't want to hit runningState.fail twice: #425
  • also tail producer support: #423
  • fix merge planner bug: #417
  • shard fm: #416
  • Add sinking to stores: #408
  • Add NormalScaldingStore which contains only (K, V) pairs: #407

0.3.1

  • Bump the chill version - makes storm provided: #406
  • sort by batch if monoid is commutative: #411
  • Fix README to properly run Storm locally: #412
  • Feature/async summer store: #400

0.3.0

  • Add serializers for BatchID, Timestamp: #350
  • ADd a means to set defaults outside the planner: #351
  • fix bugs hdfs: #353
  • add default option setting: #356
  • Add options tests: #335
  • Add some hadoop defaults to the builder API: #359
  • Cleanup the Storm API a bit: #360
  • Put all Collector actions in one execute: #365
  • Producer API executors for storm and scalding: #367
  • standardize get or else scalding: #363
  • Fixes ticks: #368
  • spout metrics: #344
  • Add the summingbird prefix to the graphs: #370
  • add localOrShuffle: #371
  • Make the StormMetric Covariant: #373
  • Fix depreciation to be from sink: #375
  • add options to cfg: #376
  • Null out Tuple.values for GC: #374
  • Add timestamps to hadoop configs: https://github.com/twitter/summingbird/pull/377/files
  • add Options.toString: #378
  • Fix out of sync versions to keep: #379
  • Feature fix ffm: #381
  • Add an await timeout: #380
  • Add local or shuffle option: #384
  • Add dequeueAll to Queue: #386
  • Add in log4j and slf4j to the example: #392
  • Fixes naming in bolts for options to work right after plan: #391
  • Add error checking to the cache size: #393
  • Add more storm tests, add some new test graphs: #394
  • Proxy the select correctly on initialbatch: #389
  • Refactors the bolts from storm to be independent workers in online: #395
  • Pluggable caches in FinalFlatMap: #396
  • Uses multimerge: #399
  • ack on entry option: #398

0.2.5

  • Fix issue with externalizer: #316
  • Initial implementation of checkpoint state: #315
  • Move to sbt 0.13: #322
  • Move Dependants common code to graph: #321
  • CheckpointState -> HDFSState: #318
  • Change depthFirstOf to use List instead of Vector: #323
  • Add an optional parameter to set default parallelism for a storm source: #319
  • Add a method to prepare for optimizing FlatMapKeys: #305
  • fix the increased output issue: #327
  • Optimize iterator sums: #326
  • Make sure to register BatchID and Timespan: #324
  • Sum by key test: #330
  • Add slf4j logging to scalding: #331
  • Move the versioned store to scalding package: #332
  • storm logging: #333
  • Adds/extends some storm option tests: #334
  • Make VersionedState fail-safe: #337
  • Throw on Storm Submission w/ no online store: #338
  • General graph support online: #340
  • Summingbird config: #339

0.2.4

  • Fix off-by-one bug in scalding batched store: #311

0.2.3

  • Add convenience methods to Producer: #307
  • Massively optimize Scalding merge/sumByKey: #303
  • Improve the WaitingState state machine: #302
  • Move code for common realtime planners (storm + akka): #299
  • Fix an issue with source minify in scalding: #298
  • Improve scalding tests: #296
  • Improve storm tests: #292
  • Use an AnyVal-like class rather than java.util.Date: #295
  • improve Batcher documentation: #293

0.2.2

  • Use pipes instead of parentheses when naming online physical nodes: #273
  • Add Alsos in Producer2, OptProd2: #275
  • Multiple disjoint summers: #274
  • Make Tail sealed: #276
  • Scalding Laws not to use scalacheck: #278
  • Pull Online planner common code into core: #281
  • Use externalizer instead of meatlocker: #282
  • Standardize SourceBuilder Naming: #285
  • Add TailProducer, for graph ending point: #286
  • MemoryLaws not use scalacheck: #287
  • Added flatMapKeys to all platforms: #288
  • Added flatMapKeys to Builder API: #290
  • Add better type safety to Storm Spouts: #289
  • Add a streaming left join: #291

0.2.1

  • Add support for map only jobs in producer api #269
  • Fixes naming of nodes, options are picked up correctly #267
  • Fix missing elements in case statements for applying online graph to storm #272

0.2.0

  • Bump the version numbers of the dependencies #260
  • Upgrade to Tormenta 0.5.2, Scalding 0.9.0, Bijection 0.5.3 betas: #191
  • Feature/storm new planner: #250
  • Feature/move dag to core: #255

0.1.5

  • Control how futures are collected in Client Store: #254
  • Producer[Platform, T] is covariant on T: #251
  • Improve testing generators: #249
  • Remove manifests from the core API: #247
  • Dot graphs of our storm plan (what SB nodes go to physical nodes): #236

0.1.4

  • Fix Storm Tests: #227
  • Custom --name flag for job name in builder api: #226
  • Remove SINK_ID reference from SourceBuilder: #225
  • Add Dot Graph Generation from SB Producer: #223
  • Push filters down to storm spout: #224
  • Revert StoreIntermediate to old approach: #219
  • Add in old SummingJoin: #218

0.1.3

  • Fix bug in storm planner's forking: #217
  • Don't send empty multiGets to online service: #209

0.1.2

  • UTC Calendar Batcher: #195
  • Register injections in StormEnv: #198
  • Fix Forking bug in Storm planner: #197
  • Tidying of StormPlatform: #199
  • Add AnchorTuples option to Storm: #200
  • Allow NamedProducer after Summer in storm: #202
  • Optimization when using BijectedMonoid in a Scalding job: #203
  • Remove need for --initial-run (--start-time suffices): #204

0.1.1

  • Add --scalding.nothrowplan to not rethrow/exit with error on a flow plan error

0.1.0

  • new, producer-based API
  • Split Storm and Scalding platforms
  • Added in-memory platform
  • Complete DSL rebuild in anticipation of release.

0.0.4

  • fix compiler-induced bug in BatchAggregatorJob.

0.0.3

  • Ability to tee out data.

0.0.2

  • PresentingStore
  • Ability to join a Summingbird job against another SB job.

0.0.1

  • Initial code push.