Sketching Algorithms: Big Data

About

Challenge Working with big data in environments where computational resources such as memory, processing power, and energy consumption are limited can be difficult. As we try to apply machine learning, computer vision, signal processing, and other techniques in these computing environments we are reminded of these limitations. The limited resources require using a type of streaming-based data transformation method in order to accommodate the large arrays of incoming data in an efficient manner. So-called sketching algorithms can be utilized to take account of the data structure and the platform constraints. Solution SSketch is a novel approach consisting of an automated framework for efficient analysis of dynamic big data. It is targeted towards streaming applications where each data sample can be processed only once and storage space is limited. SSketch adaptively learns from the input data and translates it to a sketch matrix or a corresponding set of lower dimensional data structures. This unique approach makes the process suitable for hardware-based acceleration performed by reconfigurable platforms like field programmable gate arrays (FPGAs) and a speed increase of up to 200 fold has been demonstrated. Given a set of user-defined constraints such as runtime, power, or available memory, SSketch can create the most accurate data sketch using its optimized automated methodology. Benefits and Features   Novel streaming-based data transformation method for reconfigurable platforms like FPGAs Customizable Application Programming Interface for rapid prototyping of arbitrary matrix-based data analysis Automated optimized method to compute the most accurate sketch matrix given a set of constraints Market Potential / Applications This new framework for processing of streaming data should find applications on reconfigurable platforms like field programmable gate arrays (FPGAs). It will be particularly valuable in situations where constraints on storage space, processing power, available memory, and energy consumption exist. SSketch can be utilized to more efficiently process streaming data in these constrained environments, enabling speed increases of up to 200 fold.