ROME: All Overlays Lead to Aggregation, but Some Are Faster than Others
Additional information
Authors
Blöcher M.,
Coppa E.,
Kleber P.,
Eugster P. T.,
Culhane W.,
Ardekani M. S.
Type
Journal Article
Year
2021
Language
English
Abstract
Aggregation is common in data analytics and crucial to distilling information from large datasets, but current data analytics frameworks do not fully exploit the potential for optimization in such phases. The lack of optimization is particularly notable in current “online” approaches that store data in main memory across nodes, shifting the bottleneck away from disk I/O toward network and compute resources, thus increasing the relative performance impact of distributed aggregation phases.
We present ROME, an aggregation system for use within data analytics frameworks or in isolation. ROME uses a set of novel heuristics based primarily on basic knowledge of aggregation functions combined with deployment constraints to efficiently aggregate results from computations performed on individual data subsets across nodes (e.g., merging sorted lists resulting from top-
k
). The user can either provide minimal information that allows our heuristics to be applied directly, or ROME can autodetect the relevant information at little cost. We integrated ROME as a subsystem into the Spark and Flink data analytics frameworks. We use real-world data to experimentally demonstrate speedups up to 3× over single-level aggregation overlays, up to 21% over other multi-level overlays, and 50% for iterative algorithms like gradient descent at 100 iterations.
Journal
ACM Transactions on Computer Systems
Volume
39
Number ( Month )
1-4
Pages (or article number)
1-33
ISSN
0734-2071, 1557-7333
DOI
Diffusion
License
License undefined
Visibility
Private
