A framework for test case prioritization in the continuous software engineering

Abstract

Regression tests are executed after every change in software. In a software development environment that adopts Continuous Software Engineering practices such as Continuous Integration, software is changed, built and tested many times in a short period. Each execution can take hours to finish, delaying feedback about failures to the developer. To prevent this, regression test optimization techniques are used. One such technique is test case prioritization (TCP), which reorder the execution of the test cases according to some goal. The most common goal is fault detection, in which test cases are ordered so that those that have higher probability of detecting faults are executed first. One problem with this approach is that there are lots of different available techniques in the literature, but the amount of evidence of its use is low. Furthermore, there is almost no infrastructure support to adopt those techniques at the industry context. The goal of this work is to design and implement a framework that allows the use, experimentation and implementation of TCP techniques. We hope that this will help practitioners on adopting these techniques at the industry context, more specifically, in the continuous software engineering environment. We also hope that creating this infrastructure will encourage researchers on performing more empirical studies regarding test case prioritization techniques effectiveness. In order to show the feasibility of the proposed framework, we perform an empirical study with 16 different TCP techniques executed on a total of 22 versions of 2 different open source projects. Results suggest that using those TCP techniques result in faster feedback about the existence of failures in the projects, possibly resulting in shorter development cycles.