Regression testing is an essential component in software development, aimed at verifying that recent code changes do not negatively impact existing functionalities. A prevalent challenge in this area is the occurrence of flaky tests, which exhibit inconsistent outcomes—passing or failing without any code alterations. Such tests erode the reliability of automated testing frameworks, leading to increased debugging efforts and diminished confidence in software stability. This research addresses the challenge of flaky tests by exploring machine learning and prediction models, specifically focusing on test smells and four principal root causes of flakiness: Async Wait, Concurrency, Test Order Dependency, and Resource Leak. While existing methods that analyze test case vocabulary often encounter issues such as elevated operational costs, tendencies to overfit, and context sensitivity, our study enhances the accuracy of identifying flaky tests by incorporating these four additional determinants into the test smell-based approach. We conducted a thorough evaluation of this augmented model's predictive capabilities in a cross-project context and analyzed the contribution of each newly added flakiness factor in terms of information gain. Our findings indicate that Async Wait and Concurrency, in particular, show the highest information gain, highlighting their significant role in flakiness prediction. Moreover, we compared the performance of our hybrid feature based approach against both the vocabulary-based approach and the traditional test-smell based approach, focusing on the improved accuracy, precision, recall in predicting flaky tests.
Golisano College of Computing and Information Sciences
Mohamed Wiem Mkaouer
Ali ben Mrad
Bodke, Saurabh, "Hybrid Test-Smell Based Approach for Prediction of Flaky Tests" (2023). Thesis. Rochester Institute of Technology. Accessed from
RIT – Main Campus