Abstract

Regression testing is an essential component in software development, aimed at verifying that recent code changes do not negatively impact existing functionalities. A prevalent challenge in this area is the occurrence of flaky tests, which exhibit inconsistent outcomes—passing or failing without any code alterations. Such tests erode the reliability of automated testing frameworks, leading to increased debugging efforts and diminished confidence in software stability. This research addresses the challenge of flaky tests by exploring machine learning and prediction models, specifically focusing on test smells and four principal root causes of flakiness: Async Wait, Concurrency, Test Order Dependency, and Resource Leak. While existing methods that analyze test case vocabulary often encounter issues such as elevated operational costs, tendencies to overfit, and context sensitivity, our study enhances the accuracy of identifying flaky tests by incorporating these four additional determinants into the test smell-based approach. We conducted a thorough evaluation of this augmented model's predictive capabilities in a cross-project context and analyzed the contribution of each newly added flakiness factor in terms of information gain. Our findings indicate that Async Wait and Concurrency, in particular, show the highest information gain, highlighting their significant role in flakiness prediction. Moreover, we compared the performance of our hybrid feature based approach against both the vocabulary-based approach and the traditional test-smell based approach, focusing on the improved accuracy, precision, recall in predicting flaky tests.

Library of Congress Subject Headings

Computer software--Testing; Computer software--Reliability; Debugging in computer science; Regression analysis; Machine learning

Publication Date

12-12-2023

Document Type

Thesis

Student Type

Graduate

College

Golisano College of Computing and Information Sciences

Advisor

Mohamed Wiem Mkaouer

Advisor/Committee Member

Ali ben Mrad

Advisor/Committee Member

Makram Soui

Campus

RIT – Main Campus

Plan Codes

SOFTENG-MS

Download

Share

COinS