Administrators & cloud service providers
Open Source communities & application developers
Data Analytic development framework
QoS Cloud services
This asset is based on the combination of three key components of the EUBra-BIGSEA framework: 1) EC3 which automates the deployment and the initial configuration of a Big Data application and provides also mechanisms for runtime re-configuration (developed by UPV – a separate description is provided), 2) a rule-based module for pro-active run-time policies specification and execution (developed by UFCG), 3) a module implementing optimization based policies able to identify the deployment configuration of minimum costs that provides also performance guarantees (e.g., jobs are executed within a time limit or data streams are executed with no loss at a given rate, developed by POLIMI and UFMG).
The asset is targeted to cloud providers and virtual infrastructure managers, and it is especially tailored for Spark Workflows (although it also supports COMPSs applications).
Cloud providers can fully benefit from its capabilities, as it is necessary to have access to the infrastructure resources for the vertical elasticity at CPU CAP level. However, a user of public clouds can also benefit from the resource optimization, and horizontal elasticity at framework level.
In a single sentence, BIGSEAQoS can be used to provide an estimation of the resources needed for a previously profiled application to complete an execution on a given deadline, dynamically adjusting the resources by a monitoring-actuator system to correct the errors of the prediction system. Shortly, it can define and adjust the cloud resources to ensure that a user obtains the results on a given time. Moreover, prediction accuracy service improves its accuracy as the system runs.
Two scenarios should be separated here:
The system is on prototype version, demonstrated under lab conditions. The links are:
All the components are modular and can be used separately on top of the deployment. Documentation is embedded within the repositories and also available at D3.5: http://www.eubra-bigsea.eu/sites/default/files/D3.5%20EUBra-BIGSEA%20QoS%20infrastructure%20services%20final%20version.pdf
Two scenarios should be separated here:
No license costs are applied, as they are licensed as Open Source under Apache 2.0 (as well as all their dependencies). However, it requires some operational costs for the profiling of the applications, which are provided in a separate asset (dagSIM). Modelling an application requires to perform two or tree runs of the application with different configurations and processing the logs (through another component SparkLogParser). In the context of the project, it required two days per application.
IPRs are shared among the four institutions.
For more information get in touch at firstname.lastname@example.org
View related publications
--> Danilo Ardagna, Enrico Barbierato, Athanasia Evangelinou, Eugenio Gianniti, Marco Gribaudo, Túlio B. M. Pinto, Anna Guimarães, Ana Paula Couto da Silva, and Jussara M. Almeida. 2018. Performance Prediction of Cloud-Based Big Data Applications. In Proceedings of the 2018 ACM/SPEC International Conference on Performance Engineering (ICPE '18). ACM, New York, NY, USA, 192-199. DOI: https://doi.org/10.1145/3184407.3184420.
--> Athanasia Evangelinou, Michele Ciavotta, Danilo Ardagna, Aliki Kopaneli, George Kousiouris, Theodora Varvarigou, Enterprise applications cloud rightsizing through a joint benchmarking and optimization approach, Future Generation Computer Systems, Volume 78, Part 1, 2018, Pages 102-114, ISSN 0167-739X, https://doi.org/10.1016/j.future.2016.11.002.
--> Eugenio Gianniti, Alessandro Maria Rizzi, Enrico Barbierato, Marco Gribaudo, and Danilo Ardagna. 2017. Fluid Petri Nets for the Performance Evaluation of MapReduce and Spark Applications. SIGMETRICS Perform. Eval. Rev. 44, 4 (May 2017), 23-36. DOI: https://doi.org/10.1145/3092819.3092824.
--> Sergio Lopez-Huguet, Igor Natanael, Andrey Brito, Ignacio Blanquer, Vertical Elasticity on Marathon and Chronos Mesos frameworks, Special Issue on Parallel/Distributed Computing and Optimization of the Journal of Parallel and Distributed Computing, under evaluation.