My current research interest is testing deep learning systems. More specifically, I am working on providing explanations for misbehaviours of deep learning systems. Deep learning systems are becoming increasingly prevalent in various applications, including safety-critical ones such as self-driving cars. This has made it increasingly important to guarantee the quality and safety of these solutions. Testing techniques should effectively address the unique challenges posed by DL systems. To ensure that deep learning systems behave as expected, it is crucial to be able to explain their behavior. In particular, (1) I have explored the feature space of DL systems through illumination search and provided human-interpretable explanations for DL systems, (2) I have created a search-based focused test generator to generate inputs in predefined feature space areas, (3) I have empirically compared our explanations with the state-of-the-art, (4) I am implementing fully automated feature-based explanations for DL systems.