EUROPE - BRAZIL COLLABORATION OF BIG DATA SCIENTIFIC RESEARCH THROUGH CLOUD-CENTRIC APPLICATIONS

Deliverable 4.4 QoS extensions for the big and fast data eco-system

This deliverable describes the design and the implementation details regarding the big and fast data ecosystem extensions and adaptations for Quality of Service (QoS) guarantees. It focuses on the main technologies involved in the eco-system (i.e., Ophidia and Spark) and the extensions and implementations to support automated deployment, job monitoring, elasticity, as well as the integration activities of these technologies with the QoS infrastructure developed at the level of WP3. The extensions support data processing applications with QoS deadlines by dynamically adjusting the resources available for the job execution.

The solutions presented in the document capitalize on the plugin-based architecture implemented by WP3 to provide a reactive and dynamic PaaS environment over cloud-based resources, capable of handling Big Data applications with QoS constraints, minimizing resource usage at the same time.

Moreover, the document provides some usage examples and relevant code snippets related to these extensions, along with a brief evaluation of a couple of WP4 implementations built on top of the adapted big data technologies to highlight how the resource availability can impact on the execution time and, in turn, affect QoS constraints.

 

Categories: