EUROPE - BRAZIL COLLABORATION OF BIG DATA SCIENTIFIC RESEARCH THROUGH CLOUD-CENTRIC APPLICATIONS

Diagnosing Performance Bottlenecks in Massive Data Parallel Programs

Vinícius  Dias,  Ruens  Moreira,  Wagner  Meira  Jr.,  and  Dorgival  Guedes

In: 16th  IEEE/ACM  International  Symposium  on Cluster, Cloud and Grid  Computing (CCGrid)


Abstract:

The increasing amount of data being stored and the variety of applications being proposed recently to make use of those data enabled a whole new generation of parallel programming environments and paradigms. Although most of these novel environments provide abstract programming interfaces and embed several run-time strategies that simplify several typical tasks in parallel and distributed systems, achieving good performance is still a challenge. In this paper we identify some common sources of performance degradation in the Spark programming environment and discuss some diagnosis dimensions that can be used to better understand such degradation. We then describe our experience in the use of those dimensions to drive the identification performance problems, and suggest how their impact may be minimized considering real applications.

http://ieeexplore.ieee.org/document/7515698/

Categories: