Register for free to
view the full innovation profile.
New breed of technologies support real-time data processing in a fault tolerant, scalable and cost effective way which is helping to drive it’s adoption.
About
A Comparative Performance Evaluation of Storm, Storm Trident and Spark Streaming Overview Real-time analytics, which is the ability to perform analysis and produce insights over live data in near real-time is gaining wider interest as the benefits of greater agility through real-time insights and decision making for businesses is becoming apparent. In response, a new breed of technologies and tools is becoming available that support real-time data processing in a fault tolerant, scalable and cost effective way which is helping to drive it’s adoption. This CeADAR report provides a state-of-the-art review of popular real-time data stream processing technologies along with the results of a comparative evaluation of three commonly used open source stream processing platforms namely Storm, Storm Trident and Spark Streaming on a continuous metrics computation task. The comparative evaluation was based on Apache Spark version 1.4 and Apache Storm version 0.9.3. How It Works The Table of Contents for this 26 page CeADAR report is as follows; 1 Executive Summary 2 Introduction 3 Platforms and Configuration 3.1 Stream Processing Platforms 3.1.1 Spark Streaming, 3.1.2 Storm, 3.1.3 Storm Trident 3.2 Statistical Metric Computation 3.2.1 SlidingWindows, 3.2.2 Metrics 3.3 Datasets 3.4 Hardware and Software configuration 4 Evaluations 4.1 Correctness 4.2 Spark Streaming Checkpoint Interval 4.3 Sliding Interval 4.4 Number of Keys 4.5 Window Size 4.6 Performance using Identical Configurations 4.7 Performance using the Best Configurations 4.8 Continuous Data Streams 5 Conclusions Benefits CeADAR’s ContinuousMetrics project evaluates the general task of computing such statistical metrics over streaming data using three popular open-source stream processing platforms: Apache Spark Streaming1, Apache Storm2, and Apache Storm Trident3, and this document presents the comparative performance results as well as the conclusions and learnings gained from the work. Inventors Oisin Boydell David Haughton Brian Mac Namee Guangyu Wu