A comprehensive evaluation of SZZ variants through a developer-informed oracle
Informazioni aggiuntive
Tipo
Articolo pubblicato in rivista scientifica
Anno
2023
Lingua
Inglese
Sommario
Automatically linking bug-fixing changes to bug-inducing ones (BICs) is one of the key data-extraction steps behind several empirical studies in software engineering. The SZZ algorithm is the de facto standard to achieve this goal, with several improvements proposed over time. Evaluating the performance of SZZ implementations is, however, far from trivial. In previous works, researchers (i) manually assessed whether the BICs identified by the SZZ implementation were correct or not, or (ii) defined oracles in which they manually determined BICs from bug-fixing commits. However, ideally, the original developers should be involved in defining a labeled dataset to evaluate SZZ implementations. We propose a methodology to define a “developer-informed” oracle for evaluating SZZ implementations, without requiring a manual inspection from the original developers. We use Natural Language Processing (NLP) to identify bug-fixing commits in which developers explicitly reference the commit(s) that introduced the fixed bug. We use the built oracle to extensively evaluate existing SZZ variants defined in the literature. We also introduce and evaluate two new variants aimed at addressing two weaknesses we observed in state-of-the-art implementations (i.e., processing added lines and handling of revert commits).
Parole chiave
SZZ, Defect prediction, Empirical study
Periodico
Journal of systems and software
Volume
202
Pagine (o numero dell’articolo)
111729
Diffusione
Licenza
CC BY-NC-ND
Visibilità
Pubblico
Status open access
Green
