Skip to content
This repository has been archived by the owner on Jan 20, 2022. It is now read-only.
Permalink

Comparing changes

Choose two branches to see what’s changed or to start a new pull request. If you need to, you can also or learn more about diff comparisons.

Open a pull request

Create a new pull request by comparing changes across two branches. If you need to, you can also . Learn more about diff comparisons here.
base repository: twitter/summingbird
Failed to load repositories. Confirm that selected base ref is valid, then try again.
Loading
base: 0.6.0
Choose a base ref
...
head repository: twitter/summingbird
Failed to load repositories. Confirm that selected head ref is valid, then try again.
Loading
compare: develop
Choose a head ref
Loading
Showing with 6,942 additions and 6,812 deletions.
  1. +1 −0 .gitignore
  2. +12 −0 .jvmopts
  3. +39 −5 .travis.yml
  4. +36 −0 CHANGES.md
  5. +19 −0 COMMITTERS.md
  6. +28 −11 README.md
  7. +444 −0 build.sbt
  8. +0 −408 project/Build.scala
  9. +0 −44 project/DocGen.scala
  10. +0 −54 project/Unidoc.scala
  11. +1 −1 project/build.properties
  12. +9 −12 project/plugins.sbt
  13. +7 −3 project/travis-log4j.properties
  14. +282 −222 sbt
  15. +0 −35 summingbird-batch-hadoop/src/main/scala/com/twitter/summingbird/batch/BatchConfig.scala
  16. +2 −3 ...ingbird-batch-hadoop/src/main/scala/com/twitter/summingbird/batch/state/FileVersionTracking.scala
  17. +2 −1 summingbird-batch-hadoop/src/main/scala/com/twitter/summingbird/batch/state/HDFSState.scala
  18. +6 −7 summingbird-batch-hadoop/src/main/scala/com/twitter/summingbird/batch/store/HDFSMetadata.scala
  19. +11 −6 summingbird-batch-hadoop/src/test/scala/com/twitter/summingbird/batch/HDFSStateLaws.scala
  20. +36 −19 summingbird-batch/src/main/scala/com/twitter/summingbird/batch/BatchID.scala
  21. +13 −9 summingbird-batch/src/main/scala/com/twitter/summingbird/batch/Batcher.scala
  22. +1 −1 summingbird-batch/src/main/scala/com/twitter/summingbird/batch/CombinedBatcher.scala
  23. +2 −9 summingbird-batch/src/main/scala/com/twitter/summingbird/batch/MillisecondBatcher.scala
  24. +29 −0 summingbird-batch/src/main/scala/com/twitter/summingbird/batch/OrderedFromOrderingExt.scala
  25. +8 −6 summingbird-batch/src/main/scala/com/twitter/summingbird/batch/{TimeStamp.scala → Timestamp.scala}
  26. +32 −10 summingbird-batch/src/test/scala/com/twitter/summingbird/batch/BatchLaws.scala
  27. +10 −13 summingbird-batch/src/test/scala/com/twitter/summingbird/batch/BatcherLaws.scala
  28. +29 −7 summingbird-batch/src/test/scala/com/twitter/summingbird/batch/BatcherSpec.scala
  29. +2 −2 summingbird-builder/src/main/scala/com/twitter/summingbird/Env.scala
  30. +3 −7 summingbird-builder/src/main/scala/com/twitter/summingbird/builder/CompletedBuilder.scala
  31. +6 −8 summingbird-builder/src/main/scala/com/twitter/summingbird/builder/SourceBuilder.scala
  32. +4 −4 summingbird-builder/src/main/scala/com/twitter/summingbird/scalding/ScaldingEnv.scala
  33. +1 −1 summingbird-builder/src/main/scala/com/twitter/summingbird/scalding/VersionedState.scala
  34. +1 −3 summingbird-builder/src/main/scala/com/twitter/summingbird/scalding/source/RangedSource.scala
  35. +0 −1 summingbird-builder/src/main/scala/com/twitter/summingbird/sink/Sink.scala
  36. +0 −7 summingbird-builder/src/main/scala/com/twitter/summingbird/source/OfflineSource.scala
  37. +0 −1 summingbird-builder/src/main/scala/com/twitter/summingbird/store/CompoundStore.scala
  38. +1 −1 summingbird-builder/src/main/scala/com/twitter/summingbird/storm/StormEnv.scala
  39. +9 −8 summingbird-builder/src/test/scala/com/twitter/summingbird/builder/BuilderJobTest.scala
  40. +10 −13 summingbird-builder/src/test/scala/com/twitter/summingbird/builder/OptionsTest.scala
  41. +8 −8 summingbird-builder/src/test/scala/com/twitter/summingbird/builder/SourceBuilderTest.scala
  42. +3 −2 summingbird-chill/src/main/scala/com/twitter/summingbird/chill/SBChillRegistrar.scala
  43. +0 −1 summingbird-chill/src/test/scala/com/twitter/summingbird/chill/SerializerLaws.scala
  44. +120 −0 summingbird-client/src/main/scala/com/twitter/summingbird/store/ClientMergeable.scala
  45. +54 −15 summingbird-client/src/main/scala/com/twitter/summingbird/store/ClientStore.scala
  46. +3 −3 summingbird-client/src/main/scala/com/twitter/summingbird/store/MergeOperations.scala
  47. +0 −1 summingbird-client/src/main/scala/com/twitter/summingbird/store/PresentingStore.scala
  48. +138 −0 summingbird-client/src/test/scala/com/twitter/summingbird/store/ClientMergeableLaws.scala
  49. +22 −5 summingbird-client/src/test/scala/com/twitter/summingbird/store/ClientStoreLaws.scala
  50. +0 −21 summingbird-core-java/src/main/java/com/twitter/summingbird/javaapi/Buffer.java
  51. +0 −15 summingbird-core-java/src/main/java/com/twitter/summingbird/javaapi/Function.java
  52. +0 −108 summingbird-core-java/src/main/java/com/twitter/summingbird/javaapi/JKeyedProducer.java
  53. +0 −95 summingbird-core-java/src/main/java/com/twitter/summingbird/javaapi/JProducer.java
  54. +0 −74 summingbird-core-java/src/main/java/com/twitter/summingbird/javaapi/JProducers.java
  55. +0 −28 summingbird-core-java/src/main/java/com/twitter/summingbird/javaapi/JSummer.java
  56. +0 −35 summingbird-core-java/src/main/java/com/twitter/summingbird/javaapi/JTailProducer.java
  57. +0 −13 summingbird-core-java/src/main/java/com/twitter/summingbird/javaapi/Predicate.java
  58. +0 −21 summingbird-core-java/src/main/java/com/twitter/summingbird/javaapi/Service.java
  59. +0 −19 summingbird-core-java/src/main/java/com/twitter/summingbird/javaapi/Sink.java
  60. +0 −20 summingbird-core-java/src/main/java/com/twitter/summingbird/javaapi/Source.java
  61. +0 −21 summingbird-core-java/src/main/java/com/twitter/summingbird/javaapi/Store.java
  62. +0 −20 summingbird-core-java/src/main/java/com/twitter/summingbird/javaapi/Wrapper.java
  63. +0 −102 summingbird-core-java/src/main/java/com/twitter/summingbird/javaapi/impl/JKeyedProducerImpl.java
  64. +0 −133 summingbird-core-java/src/main/java/com/twitter/summingbird/javaapi/impl/JProducerImpl.java
  65. +0 −43 summingbird-core-java/src/main/java/com/twitter/summingbird/javaapi/impl/JSummerImpl.java
  66. +0 −49 summingbird-core-java/src/main/java/com/twitter/summingbird/javaapi/impl/JTailProducerImpl.java
  67. +0 −94 summingbird-core-java/src/main/java/com/twitter/summingbird/memory/javaapi/JMemory.java
  68. +0 −18 summingbird-core-java/src/main/java/com/twitter/summingbird/memory/javaapi/JMemoryService.java
  69. +0 −15 summingbird-core-java/src/main/java/com/twitter/summingbird/memory/javaapi/JSink.java
  70. +0 −238 summingbird-core-java/src/test/java/com/twitter/summingbird/memory/javaapi/TestJMemory.java
  71. +26 −0 summingbird-core-test/src/main/scala/com/twitter/summingbird/ArbitraryWorkaround.scala
  72. +39 −0 summingbird-core-test/src/main/scala/com/twitter/summingbird/MemoryArbitraries.scala
  73. +49 −0 summingbird-core-test/src/main/scala/com/twitter/summingbird/MemoryTestExecutor.scala
  74. +83 −0 summingbird-core-test/src/main/scala/com/twitter/summingbird/PlatformTransformer.scala
  75. +12 −5 ...rc/test → summingbird-core-test/src/main}/scala/com/twitter/summingbird/TestGraphGenerators.scala
  76. +22 −12 ...rd-core/src/test → summingbird-core-test/src/main}/scala/com/twitter/summingbird/TestGraphs.scala
  77. +39 −0 summingbird-core-test/src/main/scala/com/twitter/summingbird/TestPlatform.scala
  78. +65 −14 ...ngbird-core → summingbird-core-test}/src/test/scala/com/twitter/summingbird/DependantsTests.scala
  79. +60 −26 ...→ summingbird-core-test}/src/test/scala/com/twitter/summingbird/memory/ConcurrentMemoryLaws.scala
  80. +61 −92 ...bird-core → summingbird-core-test}/src/test/scala/com/twitter/summingbird/memory/MemoryLaws.scala
  81. +6 −4 summingbird-core/src/main/scala/com/twitter/summingbird/Dependants.scala
  82. +0 −3 summingbird-core/src/main/scala/com/twitter/summingbird/ExecutionConfig.scala
  83. +0 −36 summingbird-core/src/main/scala/com/twitter/summingbird/Irreducible.scala
  84. +4 −0 summingbird-core/src/main/scala/com/twitter/summingbird/OptionalPlatform2.scala
  85. +20 −3 summingbird-core/src/main/scala/com/twitter/summingbird/Options.scala
  86. +4 −0 summingbird-core/src/main/scala/com/twitter/summingbird/Platform2.scala
  87. +97 −30 summingbird-core/src/main/scala/com/twitter/summingbird/Producer.scala
  88. +0 −2 summingbird-core/src/main/scala/com/twitter/summingbird/Stats.scala
  89. +0 −84 summingbird-core/src/main/scala/com/twitter/summingbird/graph/Expr.scala
  90. +0 −368 summingbird-core/src/main/scala/com/twitter/summingbird/graph/ExpressionDag.scala
  91. +0 −122 summingbird-core/src/main/scala/com/twitter/summingbird/graph/HMap.scala
  92. +9 −8 summingbird-core/src/main/scala/com/twitter/summingbird/memory/ConcurrentMemory.scala
  93. +52 −14 summingbird-core/src/main/scala/com/twitter/summingbird/memory/Memory.scala
  94. +55 −12 summingbird-core/src/main/scala/com/twitter/summingbird/planner/ComposedFunctions.scala
  95. +183 −217 summingbird-core/src/main/scala/com/twitter/summingbird/planner/DagOptimizer.scala
  96. +0 −43 summingbird-core/src/main/scala/com/twitter/summingbird/planner/FunctionContainer.scala
  97. +67 −7 summingbird-core/src/main/scala/com/twitter/summingbird/planner/Node.scala
  98. +1 −1 summingbird-core/src/main/scala/com/twitter/summingbird/viz/Viz.scala
  99. +0 −205 summingbird-core/src/test/scala/com/twitter/summingbird/graph/ExpressionDagTests.scala
  100. +0 −107 summingbird-core/src/test/scala/com/twitter/summingbird/graph/HMapTests.scala
  101. +0 −68 summingbird-core/src/test/scala/com/twitter/summingbird/graph/LiteralTests.scala
  102. +19 −0 summingbird-core/src/test/scala/com/twitter/summingbird/planner/ComposedFunctionsTest.scala
  103. +0 −62 summingbird-example/src/main/java/com/twitter/summingbird/example/javaapi/WordCount.java
  104. +0 −4 summingbird-example/src/main/scala/com/twitter/summingbird/example/ExampleJob.scala
  105. +7 −5 summingbird-example/src/main/scala/com/twitter/summingbird/example/Storage.scala
  106. +1 −1 summingbird-example/src/main/scala/com/twitter/summingbird/example/StormRunner.scala
  107. +3 −3 summingbird-example/src/test/scala/com/twitter/summingbird/example/SmokeTest.scala
  108. +7 −10 summingbird-online/src/main/scala/com/twitter/summingbird/online/CombinedServiceStoreFactory.scala
  109. +1 −2 summingbird-online/src/main/scala/com/twitter/summingbird/online/FlatMapOperation.scala
  110. +172 −0 summingbird-online/src/main/scala/com/twitter/summingbird/online/FutureQueue.scala
  111. +16 −12 summingbird-online/src/main/scala/com/twitter/summingbird/online/MergeableStoreFactory.scala
  112. +4 −2 summingbird-online/src/main/scala/com/twitter/summingbird/online/OnlineDefaultConstants.scala
  113. +1 −1 summingbird-online/src/main/scala/com/twitter/summingbird/online/OnlineServiceFactory.scala
  114. +1 −3 summingbird-online/src/main/scala/com/twitter/summingbird/online/Queue.scala
  115. +1 −20 .../com/twitter/summingbird/online/{MergeableStoreFactoryAlgebra.scala → WrappedTSInMergeable.scala}
  116. +26 −73 summingbird-online/src/main/scala/com/twitter/summingbird/online/executor/AsyncBase.scala
  117. +25 −28 summingbird-online/src/main/scala/com/twitter/summingbird/online/executor/FinalFlatMap.scala
  118. +1 −1 summingbird-online/src/main/scala/com/twitter/summingbird/online/executor/InputState.scala
  119. +6 −11 summingbird-online/src/main/scala/com/twitter/summingbird/online/executor/IntermediateFlatMap.scala
  120. +9 −10 summingbird-online/src/main/scala/com/twitter/summingbird/online/executor/OperationContainer.scala
  121. +33 −33 summingbird-online/src/main/scala/com/twitter/summingbird/online/executor/Summer.scala
  122. +53 −2 summingbird-online/src/main/scala/com/twitter/summingbird/online/option/AllOpts.scala
  123. +105 −0 summingbird-online/src/main/scala/com/twitter/summingbird/online/option/Summers.scala
  124. +38 −8 summingbird-online/src/main/scala/com/twitter/summingbird/planner/OnlinePlan.scala
  125. +67 −18 summingbird-online/src/main/scala/com/twitter/summingbird/planner/StripNamedNodes.scala
  126. +143 −0 summingbird-online/src/test/scala/com/twitter/summingbird/online/FutureQueueLaws.scala
  127. +101 −27 summingbird-online/src/test/scala/com/twitter/summingbird/online/PlannerSpec.scala
  128. +168 −0 summingbird-online/src/test/scala/com/twitter/summingbird/online/StripNameTest.scala
  129. +2 −3 summingbird-online/src/test/scala/com/twitter/summingbird/online/TopologyPlannerLaws.scala
  130. +136 −0 summingbird-online/src/test/scala/com/twitter/summingbird/online/executor/AsyncBaseSpec.scala
  131. +3 −1 summingbird-scalding-test/src/main/scala/com/twitter/summingbird/scalding/LoopState.scala
  132. +24 −7 summingbird-scalding-test/src/main/scala/com/twitter/summingbird/scalding/MockMappable.scala
  133. +35 −2 summingbird-scalding-test/src/main/scala/com/twitter/summingbird/scalding/TestService.scala
  134. +0 −4 summingbird-scalding-test/src/main/scala/com/twitter/summingbird/scalding/TestSink.scala
  135. +1 −5 summingbird-scalding-test/src/main/scala/com/twitter/summingbird/scalding/TestSource.scala
  136. +3 −6 summingbird-scalding-test/src/main/scala/com/twitter/summingbird/scalding/TestStore.scala
  137. +32 −10 summingbird-scalding-test/src/main/scala/com/twitter/summingbird/scalding/TestUtil.scala
  138. +191 −0 ...ngbird-scalding-test/src/test/scala/com/twitter/summingbird/scalding/BatchedStoreProperties.scala
  139. +0 −198 summingbird-scalding-test/src/test/scala/com/twitter/summingbird/scalding/LookupJoinTest.scala
  140. +90 −84 summingbird-scalding-test/src/test/scala/com/twitter/summingbird/scalding/ScaldingLaws.scala
  141. +8 −29 ...ird-scalding-test/src/test/scala/com/twitter/summingbird/scalding/ScaldingSerializationSpec.scala
  142. +0 −2 summingbird-scalding-test/src/test/scala/com/twitter/summingbird/scalding/TimePathedSourceTest.scala
  143. +4 −4 ...ngbird-scalding-test/src/test/scala/com/twitter/summingbird/scalding/UniqueKeyedServiceSpec.scala
  144. +7 −32 summingbird-scalding-test/src/test/scala/com/twitter/summingbird/scalding/VersionBatchLaws.scala
  145. +151 −0 ...gbird-scalding-test/src/test/scala/com/twitter/summingbird/scalding/VersionBatchedStoreTest.scala
  146. +80 −0 summingbird-scalding-test/src/test/scala/com/twitter/summingbird/scalding/batch/LTuple2Test.scala
  147. +1 −1 summingbird-scalding/src/main/scala/com/twitter/summingbird/scalding/Executor.scala
  148. +0 −186 summingbird-scalding/src/main/scala/com/twitter/summingbird/scalding/LookupJoin.scala
  149. +181 −130 summingbird-scalding/src/main/scala/com/twitter/summingbird/scalding/ScaldingPlatform.scala
  150. +3 −2 summingbird-scalding/src/main/scala/com/twitter/summingbird/scalding/ScaldingStatProvider.scala
  151. +2 −2 summingbird-scalding/src/main/scala/com/twitter/summingbird/scalding/Service.scala
  152. +0 −6 summingbird-scalding/src/main/scala/com/twitter/summingbird/scalding/Store.scala
  153. +6 −9 summingbird-scalding/src/main/scala/com/twitter/summingbird/scalding/batch/BatchedOperations.scala
  154. +1 −0 summingbird-scalding/src/main/scala/com/twitter/summingbird/scalding/batch/BatchedService.scala
  155. +1 −1 summingbird-scalding/src/main/scala/com/twitter/summingbird/scalding/batch/BatchedSink.scala
  156. +68 −33 summingbird-scalding/src/main/scala/com/twitter/summingbird/scalding/batch/BatchedStore.scala
  157. +0 −4 summingbird-scalding/src/main/scala/com/twitter/summingbird/scalding/package.scala
  158. +2 −3 ...ngbird-scalding/src/main/scala/com/twitter/summingbird/scalding/service/BatchedDeltaService.scala
  159. +3 −4 ...gbird-scalding/src/main/scala/com/twitter/summingbird/scalding/service/BatchedWindowService.scala
  160. +1 −1 summingbird-scalding/src/main/scala/com/twitter/summingbird/scalding/service/SimpleService.scala
  161. +2 −2 ...ngbird-scalding/src/main/scala/com/twitter/summingbird/scalding/service/SimpleWindowService.scala
  162. +0 −112 ...ngbird-scalding/src/main/scala/com/twitter/summingbird/scalding/store/DirectoryBatchedStore.scala
  163. +2 −4 summingbird-scalding/src/main/scala/com/twitter/summingbird/scalding/store/InitialBatchedStore.scala
  164. +27 −91 summingbird-scalding/src/main/scala/com/twitter/summingbird/scalding/store/VersionedBatchStore.scala
  165. +1 −5 summingbird-scalding/src/main/scala/com/twitter/summingbird/scalding/store/VersionedStore.scala
  166. +0 −20 summingbird-spark/README.md
  167. +0 −117 summingbird-spark/src/main/scala/com/twitter/summingbird/spark/CoreTypes.scala
  168. +0 −93 summingbird-spark/src/main/scala/com/twitter/summingbird/spark/PlatformPlanner.scala
  169. +0 −183 summingbird-spark/src/main/scala/com/twitter/summingbird/spark/SparkPlatform.scala
  170. +0 −383 summingbird-spark/src/test/scala/com/twitter/summingbird/spark/SparkLaws.scala
  171. +0 −164 summingbird-storm-java/src/main/java/com/twitter/summingbird/storm/javaapi/JStorm.java
  172. +0 −67 summingbird-storm-java/src/test/java/com/twitter/summingbird/storm/javaapi/TestJStorm.java
  173. +15 −18 summingbird-storm-test/src/main/scala/com/twitter/summingbird/storm/StormTestRun.scala
  174. +2 −3 summingbird-storm-test/src/main/scala/com/twitter/summingbird/storm/TestStore.scala
  175. +163 −223 summingbird-storm-test/src/test/scala/com/twitter/summingbird/storm/StormLaws.scala
  176. +0 −3 summingbird-storm-test/src/test/scala/com/twitter/summingbird/storm/StormPlanTopology.scala
  177. +65 −0 summingbird-storm-test/src/test/scala/com/twitter/summingbird/storm/StormTestExecutor.scala
  178. +109 −0 summingbird-storm-test/src/test/scala/com/twitter/summingbird/storm/StormTopologyBuilderTests.scala
  179. +153 −38 summingbird-storm-test/src/test/scala/com/twitter/summingbird/storm/TopologyTests.scala
  180. +8 −9 summingbird-storm-test/src/test/scala/com/twitter/summingbird/storm/{ → builder}/InjectionLaws.scala
  181. +11 −9 summingbird-storm/src/main/java/com/twitter/summingbird/storm/spout/FixedTupleSpout.java
  182. +43 −98 summingbird-storm/src/main/scala/com/twitter/summingbird/storm/BuildSummer.scala
  183. +39 −0 summingbird-storm/src/main/scala/com/twitter/summingbird/storm/ComponentProvider.scala
  184. +0 −8 summingbird-storm/src/main/scala/com/twitter/summingbird/storm/Constants.scala
  185. +160 −0 summingbird-storm/src/main/scala/com/twitter/summingbird/storm/Edges.scala
  186. +0 −1 summingbird-storm/src/main/scala/com/twitter/summingbird/storm/Executor.scala
  187. +68 −88 summingbird-storm/src/main/scala/com/twitter/summingbird/storm/FlatMapBoltProvider.scala
  188. +8 −1 summingbird-storm/src/main/scala/com/twitter/summingbird/storm/Producer2FlatMapOperation.scala
  189. +109 −0 summingbird-storm/src/main/scala/com/twitter/summingbird/storm/SpoutProvider.scala
  190. +3 −4 summingbird-storm/src/main/scala/com/twitter/summingbird/storm/StoreStatReporter.scala
  191. +1 −1 summingbird-storm/src/main/scala/com/twitter/summingbird/storm/StormConfig.scala
  192. +2 −5 summingbird-storm/src/main/scala/com/twitter/summingbird/storm/StormMetric.scala
  193. +11 −201 summingbird-storm/src/main/scala/com/twitter/summingbird/storm/StormPlatform.scala
  194. +2 −3 summingbird-storm/src/main/scala/com/twitter/summingbird/storm/StormStatProvider.scala
  195. +317 −0 summingbird-storm/src/main/scala/com/twitter/summingbird/storm/StormTopologyBuilder.scala
  196. +109 −0 summingbird-storm/src/main/scala/com/twitter/summingbird/storm/SummerBoltProvider.scala
  197. +0 −52 summingbird-storm/src/main/scala/com/twitter/summingbird/storm/TupleInjections.scala
  198. +132 −0 ...ngbird-storm/src/main/scala/com/twitter/summingbird/storm/builder/AggregatorOutputCollector.scala
  199. +72 −47 summingbird-storm/src/main/scala/com/twitter/summingbird/storm/{ → builder}/BaseBolt.scala
  200. +31 −0 summingbird-storm/src/main/scala/com/twitter/summingbird/storm/builder/EdgeGrouping.scala
  201. +27 −0 summingbird-storm/src/main/scala/com/twitter/summingbird/storm/builder/FormattedScheme.scala
  202. +62 −0 summingbird-storm/src/main/scala/com/twitter/summingbird/storm/builder/FormattedSpout.scala
  203. +83 −0 summingbird-storm/src/main/scala/com/twitter/summingbird/storm/builder/KeyValueSpout.scala
  204. +29 −0 summingbird-storm/src/main/scala/com/twitter/summingbird/storm/builder/OutputFormat.scala
  205. +108 −0 summingbird-storm/src/main/scala/com/twitter/summingbird/storm/builder/SpoutBuilder.scala
  206. +220 −0 summingbird-storm/src/main/scala/com/twitter/summingbird/storm/builder/Topology.scala
  207. +1 −1 summingbird-storm/src/main/scala/com/twitter/summingbird/storm/option/FlatMapOptions.scala
  208. +0 −2 summingbird-storm/src/main/scala/com/twitter/summingbird/storm/option/SummerOptions.scala
  209. +0 −2 summingbird-storm/src/main/scala/com/twitter/summingbird/storm/package.scala
  210. +3 −7 summingbird-storm/src/main/scala/com/twitter/summingbird/storm/spout/TraversableSpout.scala
  211. +77 −0 ...rd-storm/src/test/scala/com/twitter/summingbird/storm/builder/AggregatorOutputCollectorTest.scala
  212. +18 −0 summingbird-storm/src/test/scala/com/twitter/summingbird/storm/builder/Counter.scala
  213. +58 −0 ...ird-storm/src/test/scala/com/twitter/summingbird/storm/builder/TestAggregateOutputCollector.scala
  214. +148 −0 summingbird-storm/src/test/scala/com/twitter/summingbird/storm/builder/TestKeyValueSpout.scala
  215. +1 −0 version.sbt
1 change: 1 addition & 0 deletions .gitignore
Original file line number Diff line number Diff line change
@@ -14,4 +14,5 @@ project/plugins/src_managed/
*/.settings
*/.cache
*/.classpath
sonatype.sbt
vagrant/
12 changes: 12 additions & 0 deletions .jvmopts
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
-Dfile.encoding=UTF8
-Xms1G
-Xmx3G
-Xss2m
-XX:MaxMetaspaceSize=384M
-XX:ReservedCodeCacheSize=128M
-XX:+TieredCompilation
-XX:-UseGCOverheadLimit
# effectively adds GC to Perm space
-XX:+CMSClassUnloadingEnabled
# must be enabled for CMSClassUnloadingEnabled to work
-XX:+UseConcMarkSweepGC
44 changes: 39 additions & 5 deletions .travis.yml
Original file line number Diff line number Diff line change
@@ -1,6 +1,40 @@
language: scala
scala:
- 2.10.4
script:
- "sbt -Dlog4j.configuration=file://$TRAVIS_BUILD_DIR/project/travis-log4j.properties ++$TRAVIS_SCALA_VERSION test"
- "sbt -Dlog4j.configuration=file://$TRAVIS_BUILD_DIR/project/travis-log4j.properties ++$TRAVIS_SCALA_VERSION publish-local"
sudo: false
before_install: umask 0022
env:
global:
- LOG4J=file://$TRAVIS_BUILD_DIR/project/travis-log4j.properties

matrix:
include:
- scala: 2.11.12
jdk: oraclejdk8
script: ./sbt -Dlog4j.configuration=$LOG4J -DsequentialExecution=true ++$TRAVIS_SCALA_VERSION clean coverage test coverageReport mimaReportBinaryIssues
after_success:
- bash <(curl -s https://codecov.io/bash)

- scala: 2.12.4
jdk: oraclejdk8
script: ./sbt -Dlog4j.configuration=$LOG4J -DsequentialExecution=true ++$TRAVIS_SCALA_VERSION clean test

cache:
directories:
- $HOME/.sbt/0.13/dependency
- $HOME/.sbt/boot/scala*
- $HOME/.sbt/launchers
- $HOME/.ivy2/cache
- $HOME/.nvm

before_cache:
- du -h -d 1 $HOME/.ivy2/cache
- du -h -d 2 $HOME/.sbt/
- find $HOME/.sbt -name "*.lock" -type f -delete
- find $HOME/.ivy2/cache -name "ivydata-*.properties" -type f -delete

notifications:
webhooks:
urls:
- https://webhooks.gitter.im/e/535b6396650b3a9c54a7
on_success: change
on_failure: always
on_start: never
36 changes: 36 additions & 0 deletions CHANGES.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,41 @@
# Summingbird #

## 0.9.1
* Fixes a bug in the LTuple2 class. Add a test for it: https://github.com/twitter/summingbird/pull/632

## 0.9.0
* Removing internal config setup from scalding platform: https://github.com/twitter/summingbird/pull/629
* Remove store that no one seems to use and has no tests: https://github.com/twitter/summingbird/pull/630
* Ianoc/optimization changes: https://github.com/twitter/summingbird/pull/628
* Fixes tests: https://github.com/twitter/summingbird/pull/626
* resilient to all stores being complete: https://github.com/twitter/summingbird/pull/627
* Resilient to a store being ahead: https://github.com/twitter/summingbird/pull/625
* move to scala test: https://github.com/twitter/summingbird/pull/621
* Killing off Summingbird Java: https://github.com/twitter/summingbird/pull/624
* Kill summingbird spark: https://github.com/twitter/summingbird/pull/622
* Bumping sbt versions: https://github.com/twitter/summingbird/pull/623
* Fixed erroneous comment in ClientStore&#39;s merge description: https://github.com/twitter/summingbird/pull/620

## 0.8.0
* bumping Scalding to 0.15.0, algebird to 0.10.1

## 0.7.0
* Registering summingbird counters with tormenta spouts #553
* Add counters to default summers #571
* Fixes the types off the summingbird java storm api #601
* Adding explicit hadoop deps; dfs-datastores to 1.3.6 #592
* Replace deprecated ClassManifest and erasure #608
* Spark monad #607
* replace deprecated erasure by runtimeClass #609
* Revert "Remove StripNameNodes" #610
* refactored core-tests into separate sub-project #593
* Fix checking delta batch in readDeltaTimestamps in batched store when merging #612
* ensure at least one batch before merging #613
* MergeableStoreFactory extends java.io.Serializable #616
* Spout storm metrics foreach #617
* upgrade scalacheck #615
* Use latest bijection, algebird, scalding, storehaus, chill, and tormenta

## 0.6.0
* Use latest bijection, algebird, scalding, storehaus, chill, and tormenta https://github.com/twitter/summingbird/pull/565
* Remove scala 2.9 support https://github.com/twitter/summingbird/pull/565
19 changes: 19 additions & 0 deletions COMMITTERS.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@
# Committers

Please see our [Project Governance](https://github.com/twitter/analytics-infra-governance) page for more details.

## Active

| Name | Handle |
|------------------------|-----------------------------------------------------------|
| Alex Levenson | [@isnotinvain](https://github.com/isnotinvain) |
| Ben Pence | [@benpence](https://github.com/benpence) |
| Ian O'Connell | [@ianoc](https://github.com/ianoc) |
| Joe Nievelt | [@jnievelt](https://github.com/jnievelt) |
| Oscar Boykin | [@johnynek](https://github.com/johnynek) |
| Pankaj Gupta | [@pankajroark](https://github.com/pankajroark) |
| Piyush Narang | [@piyushnarang](https://github.com/piyushnarang) |
| Sam Ritchie | [@sritchie](https://github.com/sritchie) |
| Sriram Krishnan | [@sriramkrishnan](https://github.com/sriramkrishnan) |

##Emeritus
39 changes: 28 additions & 11 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,4 +1,10 @@
## Summingbird [![Build Status](https://secure.travis-ci.org/twitter/summingbird.png)](http://travis-ci.org/twitter/summingbird)
## Summingbird

[![status: retired](https://opensource.twitter.dev/status/retired.svg)](https://opensource.twitter.dev/status/#retired)
[![Build Status](https://secure.travis-ci.org/twitter/summingbird.png)](http://travis-ci.org/twitter/summingbird)
[![Codecov branch](https://img.shields.io/codecov/c/github/twitter/summingbird/develop.svg?maxAge=3600)](https://codecov.io/github/twitter/summingbird)
[![Latest version](https://index.scala-lang.org/twitter/summingbird/summingbird-core/latest.svg?color=orange)](https://index.scala-lang.org/twitter/summingbird/summingbird-core)
[![Chat](https://badges.gitter.im/twitter/summingbird.svg)](https://gitter.im/twitter/summingbird?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge&utm_content=badge)

Summingbird is a library that lets you write MapReduce programs that look like native Scala or Java collection transformations and execute them on a number of well-known distributed MapReduce platforms, including [Storm](https://github.com/nathanmarz/storm) and [Scalding](https://github.com/twitter/scalding).

@@ -90,14 +96,14 @@ Boom. Counts for the word `"i"` are growing in realtime.

See the [wiki page](https://github.com/twitter/summingbird/wiki/Getting-started-with-summingbird-example) for a more detailed explanation of the configuration required to get this job up and running and some ideas for where to go next.

## Community and Documentation

This, and all [github.com/twitter](https://github.com/twitter) projects, are under the [Twitter Open Source Code of Conduct](https://engineering.twitter.com/opensource/code-of-conduct). Additionally, see the [Typelevel Code of Conduct](http://typelevel.org/conduct) for specific examples of harassing behavior that are not tolerated.
## Documentation

To learn more and find links to tutorials and information around the web, check out the [Summingbird Wiki](https://github.com/twitter/summingbird/wiki).

The latest ScalaDocs are hosted on Summingbird's [Github Project Page](http://twitter.github.io/summingbird).

## Contact

Discussion occurs primarily on the [Summingbird mailing list](https://groups.google.com/forum/#!forum/summingbird). Issues should be reported on the GitHub issue tracker. Simpler issues appropriate for first-time contributors looking to help out are tagged "newbie".

IRC: freenode channel #summingbird
@@ -106,23 +112,34 @@ Follow [@summingbird](https://twitter.com/summingbird) on Twitter for updates.

Please feel free to use the beautiful [Summingbird logo](https://drive.google.com/folderview?id=0B3i3pDi3yVgNMHV0TXVkTGZteWM&usp=sharing) artwork anywhere.

## Get Involved + Code of Conduct
Pull requests and bug reports are always welcome!

We use a lightweight form of project governence inspired by the one used by Apache projects.
Please see [Contributing and Committership](https://github.com/twitter/analytics-infra-governance#contributing-and-committership) for our code of conduct and our pull request review process.
The TL;DR is send us a pull request, iterate on the feedback + discussion, and get a +1 from a [Committer](COMMITTERS.md) in order to get your PR accepted.

The current list of active committers (who can +1 a pull request) can be found here: [Committers](COMMITTERS.md)

A list of contributors to the project can be found here: [Contributors](https://github.com/twitter/summingbird/graphs/contributors)

## Maven

Summingbird modules are published on maven central. The current groupid and version for all modules is, respectively, `"com.twitter"` and `0.4.2`.
Summingbird modules are published on maven central. The current groupid and version for all modules is, respectively, `"com.twitter"` and `0.9.1`.

Current published artifacts are

* `summingbird-core_2.9.3`
* `summingbird-core_2.11`
* `summingbird-core_2.10`
* `summingbird-batch_2.9.3`
* `summingbird-batch_2.11`
* `summingbird-batch_2.10`
* `summingbird-client_2.9.3`
* `summingbird-client_2.11`
* `summingbird-client_2.10`
* `summingbird-storm_2.9.3`
* `summingbird-storm_2.11`
* `summingbird-storm_2.10`
* `summingbird-scalding_2.9.3`
* `summingbird-scalding_2.11`
* `summingbird-scalding_2.10`
* `summingbird-builder_2.9.3`
* `summingbird-builder_2.11`
* `summingbird-builder_2.10`

The suffix denotes the scala version.
Loading