Uma plataforma de rede definida por software para ambientes de computação paralela e distribuída

Abstract

The growth in the volume and diversity of data caused by the Big Data phenomenon has revolutionized business and science, at the same time as it demands ever-increasing computational resources. High-performance computing (HPC) platforms tailored to massively parallel numerical simulations offer computational capacity that can be leveraged by Big Data Analytics solutions. Nevertheless, the confluence of Big Data and HPC, although it appears to be natural, should be examined in several aspects, which involves the suitability of several of its elements. In particular, the network infrastructure needs to be efficient and flexible to fit the very distinct demands of typical applications of such parallel and distributed computing environments. The software-defined network paradigm (SDN) may favor this integration, thanks to its global view and its higher level of programmability, which simplify network management and make it more adaptive and effective. In this context, this work presents an SDN platform capable of supplying the network performance requirements of Big Data and HPC applications. The platform seeks to optimize data communication by identifying network traffic through an API and dynamically applying the most appropriate routing mechanisms to each traffic profile. This approach evidences an application-aware network model that allows the decrease in the execution time of applications. Evaluations through simulations in specific scenarios demonstrate the feasibility and applicability of the platform, by reducing the average execution time of MPI applications by about 11%, and of Hadoop applications by around 6%.