Cloud computing is becoming a key solution to provide the underlying infrastructure to run big data analysis in a pay-per-use basis without up-front investment, especially beneficial for SMEs. These business opportunities for SMEs are strategic for Europe & Brazil.
Data-driven decision processes guide many sectors of our economy. In complex systems that do not lend themselves to intuitive models (e.g, natural sciences, social and engineered systems), data-driven modelling hypothesis generation and high performance computing scenario-based simulations have a key role to understanding system behaviour and interactions.
Running effectively big data analysis systems is challenging.The execution time of a complex job is generally unknown in advance. Thus, determining the optimal number of nodes in a cluster, shared among multiple users performing heterogeneous tasks, is an important problem.The incoming workload might change, run time management policies need react to workload fluctuations and application mutual runtime interferences.
EUBra-BIGSEA is developing a framework for ensuring the Quality of Service (e.g., guaranteeing that jobs are completed before a given deadline) of data analytics services on top of cloud computing infrastructures. The project provides:
- Smart policies for vertical and horizontal elastic adjustment of resources allocated to meet deadlines and dynamic adjustment of workloads at the level of Virtual Machines, containers and physical servers.
- A rich and comprehensive API to enable Virtual Appliances to monitor performance indicators ranging from the low-level to the application-specific, spanning different resources.
- The support of advanced business models (price-based, dynamic re-scheduling of data analyses to exploit the best use of infrastructural resources, minimising costs) for on-premise or public infrastructures.
- The usage of standards for the definition of the software appliances.
All the components will be platform-agnostic and will work on different Cloud Management Platforms (CMPs) avoiding changes to the application code thanks to a vendor agnostic implementation of the monitoring platform and configuration and adaptation mechanisms. The EUBra-BIGSEA solution will include configuration and contextualisation services, a repository for deployment recipes and Virtual Machines, a monitoring system using Service Level Agreements and a testbed. Finally, the use of standards for managing VMs and CMPs (OCCI), and Virtual Appliance software specifications (TOSCA) will be fostered.