Abstract

Regression testing, a critical process in software development, ensures that the recent code changes have not adversely affected existing functionality. A significant challenge in this domain is the existence of flaky tests—tests that inconsistently pass or fail without any changes to the code. These flaky tests undermine the reliability of automated testing and can lead to increased debugging and decreased confidence in software stability. To tackle the issue of flaky tests, there exists approaches that make use of machine learning and prediction models. However, approaches that rely on analyzing test case vocabulary typically face challenges like higher costs to run the tests, a tendency to overfit, and sensitivity to different contexts. In this research, we conduct an in-depth study on employing test smells and four key flakiness root causes—Async Wait, Concurrency, Test Order Dependency, and Resource Leak—as predictive features for identifying flaky tests. Building upon existing research that validates the use of test smells as flakiness indicators, our study focuses on enhancing the accuracy of the test smell-based model by integrating these four additional flakiness determinants. We rigorously assessed the augmented model's ability to predict flaky tests in a cross-project context and analyzed the information gain contributed by each newly added flakiness root cause. Our findings revealed that particularly the Async Wait and Concurrency categories demonstrated the highest information gain, underscoring their pivotal role in predicting test flakiness. Furthermore, we benchmarked the enhanced test smell-based model against the conventional vocabulary-based approach and the original test smell model, specifically examining improvements in prediction accuracy for flaky tests.

Library of Congress Subject Headings

Computer software--Testing; Computer software--Reliability; Debugging in computer science; Regression analysis; Machine learning

Publication Date

12-12-2023

Document Type

Thesis

Student Type

Graduate

Degree Name

Software Engineering (MS)

Department, Program, or Center

Software Engineering, Department of

College

Golisano College of Computing and Information Sciences

Advisor

Mohamed W. Mkaouer

Advisor/Committee Member

Christian Newman

Advisor/Committee Member

Ikram Chaabane

Campus

RIT – Main Campus

Plan Codes

SOFTENG-MS

Download

Share

COinS