Scientific journal

ISSN 1814-2400

INFORMATION SCIENCE AND CONTROL SYSTEMS

Grigor’ev Yu.A., Ermakov O. Yu.

QUERY PROCESSING AT SPEED LAYER OF LAMBDA ARCHITECTURE SYSTEMS

The analysis of the streaming data processing at the speed layer has been carried out, which included data collection stage, message queue stage, analysis stage, data storage in the memory and data access stage. The Count-Min Sketch algorithm for calculating the frequency and sum of the values of an element in a stream has been considered. It is shown that the use of the Sketch leads to a large error in re-establishing accumulated values under sufficiently large number of elements in the stream. The implementation of the analysis stage at the speed layer in lambda architecture systems with a floating window is proposed. The stage includes a matrix of vectors (one-dimensional numeric arrays) instead of sketches. This allows reading the accumulated values directly from matrix vectors.

Keywords: lambda-architecture, streaming processing, speed layer, sketch, vector