Model vs system level testing of autonomous driving systems : a replication and extension study
Informazioni aggiuntive
Autori
Stocco A.,
Pulfer B.,
Tonella P.
Tipo
Articolo pubblicato in rivista scientifica
Anno
2023
Lingua
Inglese
Sommario
Offline model-level testing of autonomous driving software is much cheaper, faster, and diversified than in-field, online system-level testing. Hence, researchers have compared empirically model-level vs system-level testing using driving simulators. They reported the general usefulness of simulators at reproducing the same conditions experienced in-field, but also some inadequacy of model-level testing at exposing failures that are observable only in online mode. In this work, we replicate the reference study on model vs system-level testing of autonomous vehicles while acknowledging several assumptions that we had reconsidered. These assumptions are related to several threats to validity affecting the original study that motivated additional analysis and the development of techniques to mitigate them. Moreover, we also extend the replicated study by evaluating the original findings when considering a physical, radio-controlled autonomous vehicle. Our results show that simulator-based testing of autonomous driving systems yields predictions that are close to the ones of real-world datasets when using neural-based translation to mitigate the reality gap induced by the simulation platform. On the other hand, model-level testing failures are in line with those experienced at the system level, both in simulated and physical environments, when considering the pre-failure site, similar-looking images, and accurate labels.
Parole chiave
Autonomous driving, Model testing, System testing, DNN testing, Deep neural networks
Periodico
Empirical Software Engineering
Volume
28
Numero ( Mese )
3
Pagine (o numero dell’articolo)
73
ISSN
1382-3256, 1573-7616
Diffusione
Licenza
CC BY
Visibilità
Pubblico
Status open access
Hybrid
