Giulia Sellitto, Filomena Ferrucci
The Impact of Release-based Training on Software Vulnerability Prediction Models
8^th ACM Celebration of Women in Computing - WomENcourage 2021

Abstract  Software vulnerability prediction models have been an object of study for several years. The ability to predict which portions of code are more prone to contain vulnerabilities leads to focus testing efforts, potentially increasing code quality and reducing security threats. Most of the proposed models have been evaluated on various datasets using the k-fold cross-validation method, which rotates the data to use in several rounds. However, in a real-case scenario, one is interested in training the model using data related to prior releases of software, and obtaining predictions on the current code to be released. Due to the difference between the in vitro validation and the supposed operational context, there is a gap between the performance observed in research studies and those that would be obtained in a real environment. We want to bridge this gap between researchers and practitioners, by evaluating how some popular vulnerability prediction models would perform in a realistic scenario. We follow the suggestions made in recent works and evaluate the models with a release-based validation approach. Our initial findings reveal that this approach leads to lower performance, so there is the need to come up with innovative solutions that can be effectively exploited in real software developing and testing scenarios.