Research on software testing has resulted in many innovative technologies, for example automating tedious tasks such as test case generation for object-oriented software. However, demonstrating scientific advance is not necessarily the same as demonstrating practical value: A technology that works well on small, artificial problems as often used in research papers might not scale up to the complexity of real systems. It is hence essential to use large “real-world” case studies to minimize the threats to external validity when evaluating research tools. However, neither choosing such case studies nor applying research prototypes to them are trivial tasks. We created a large statistically sound case study on open source software, and applied the EVOSUITE test generation tool on it in order to shed light on three important questions: First, what is the state of the art in test generation? Second, what are the challenges in applying research tools to real-world code? Third, what are the problems in selecting representative case studies? Our case study is the SF110 corpus: a collection of 100 Java projects randomly selected from SourceForge plus the 10 most popular projects on SourceForge, totaling up to 19,509 classes with more than 700 thousand bytecode level branches. Although EVOSUITE achieved an average bytecode branch coverage of 60% on this case study, the main challenges in covering code do unexpectedly lie in limitations that have been largely ignored in the literature: e.g., handling of files and network sockets. This large and statistically valid corpus lets us analyze statistical correlations between the performance of a testing tool and the chosen case study. One such (counterintuitive) correlation result is that the larger an application is, the easier it is to automatically generate unit tests for its classes.
Source : Internet
