Software systems are large and complex nowadays whose thorough manual testing requires a lot of resources. That is why automated software testing is getting significant attention in both industry and academia. Several methods and tools have been already published that do not only automate the testing process, but are also able to automatically generate test inputs and test cases. There is an active research in solutions using symbolic execution, which choose test inputs based on the structure of the software.
In the last decade several tools have been created, which are able to generate test inputs using algorithms based on symbolic execution. However, experience indicates that each tool has its strengths and weaknesses: they cannot often cope with different complicated cases (for example strings or complex objects). The developers of the tools use different programs in their case studies and the existing surveys concentrated on experiments with complex software systems. Thus, currently there is no accurate feedback available about the language elements and program structures that a given tool cannot cope with.
The purpose of the research presented in this thesis was to define a language-independent evaluation methodology with which the test generation tools based on symbolic execution can be evaluated and compared to each other and then to compare the tools of one concrete platform. Previously no such comparison has been made, but this would be beneficial for both the tool developers and tool users. The developers would receive feedback for their tool and the users would receive guidance to choose the most suitable tool to solve their problem. The contributions of the paper can be summarized as follows:
- I have determined an evaluation methodology to examine the abilities of test generation tools using symbolic execution.
- It has been implemented for Java and I have created an automated test execution framework for it.
- With the test suite I have analysed several Java test generation tools and I have evaluated and compared the results.