Automated Bug Severity Prediction using Source Code Metrics, Static Analysis, and Code Representation
Date
2022-09-12
Authors
Journal Title
Journal ISSN
Volume Title
Publisher
Abstract
In the past couple of decades, significant research efforts are devoted to the prediction of software bugs. However, most existing work in this domain treats all bugs the same, which is not the case in practice. It is important for a defect prediction method to estimate the severity of the identified bugs so that the higher severity ones get immediate attention. In this thesis, we provide a quantitative and qualitative study on two popular datasets (Defects4J and Bugs.jar), using 10 common source code metrics, and also two popular static analysis tools (SpotBugs and Infer) for analyzing their capability in predicting defects and their severity. We studied 3,358 buggy methods with different severity labels from 19 Java open-source projects. Results show that although code metrics are powerful in predicting buggy code, they cannot estimate the severity level of the bugs. In addition, we observed that static analysis tools have weak performance in both predicting bugs (F1 score range of 3.1%-7.1%) and their severity label (F1 score under 2%). We also manually studied the characteristics of the severe bugs to identify possible reasons behind the weak performance of code metrics and static analysis tools. Also, our categorization shows that Security bugs have high severity in most cases while Edge/Boundary faults have low severity. Furthermore, we show that code metrics and static analysis methods can be complementary in terms of estimating bug severity. For finding the effectiveness of machine learning models in predicting bug severity, we train 8 different models on code metrics only as a baseline and evaluate them based on different evaluation metrics. The overall result was not promising, but the Decision Tree and Random Forest models have better results. Then, we leveraged the pre-trained CodeBERT model to use code representation by feeding the source code input only, and the results improved significantly in the range of 29%-140% for different metrics. We also integrated code metrics into the CodeBERT model by providing two architectures named ConcatInline and ConcatCLS which enhance the CodeBERT model efficacy.
Description
Keywords
Citation
Mashhadi, E. (2022). Automated Bug Severity Prediction using Source Code Metrics, Static Analysis, and Code Representation (Master's thesis, University of Calgary, Calgary, Canada). Retrieved from https://prism.ucalgary.ca.