A-Test Autonomic Software Testing

In this project we address the problem of software failures that emerge in production. We will define, develop and validate a holistic approach to automatically generate test cases for exercising software systems with scenarios from production, and revealing bugs that emerge in production, before they lead to system failures.

Software failures in production environments, that is, when the software system runs in the field, are unavoidable, and may have severe consequences. Testing software in testbed environments, before deploying the software in the field, cannot reveal all failures, due to both the combinatorial explosion of execution scenarios and execution conditions that emerge only in production. Field testing approaches test software systems with data from the field to reveal failures that escape testing in testbed: Ex-vivo testing approaches deal with applications that process data difficult to build artificially; In-vivo approaches exploit pre-designed test cases in the field. Field testing approaches address only few classes of failures that emerge in production, while missing many relevant cases. Machine learning and autonomous system testing addresses the specific issues of testing the behavior of machine learning and autonomous systems. They deal with built-in learning and adaptive behavior, not with failures that emerge in production.

In this project, we address the problem of efficiently and effectively revealing general failures that emerge in production: We will define an efficient approach to test scenarios that emerge in production by (i) generating test templates and general oracles from widely accessible information (code structure, natural language comments, execution traces, reviews, legacy test suites), to capture the relevant in- formation for testing the target system, (ii) triggering scenarios that emerge in production and have not been fully tested yet, to frame scenarios that may nurse emerging bugs, (iii) instantiating test templates with the identified conditions, to generate production test cases, (iv) executing production test cases with sandboxes and digital twins, to reveal emerging bugs, (v) generating test suites from the test templates, to minimize the overall effort for software testing.

We will exploit (i) semantic parsing to generate prefixes for test templates from source code and execution traces, (ii) sequence-to-sequence learning to generate general assertions for test oracles from natural language comments and contextual information, (iii) genetic programming to improve test oracles, (iv) deep learning to classify emerging scenarios and trigger new test cases from monitored execution states and traces, (v) code generation and software engineering to instantiate and execute test cases from test templates and triggers.

We will validate the approach on retail, autonomous traffic, and learning management systems on both synthetic and real production scenarios, to demonstrate the general applicability and the maturity of the approach to test large software systems in production.

This project will impact both research and production. It will will impact research in software engineering with (i) approaches to automatically generate test cases, (ii) approaches to automatically generate general test oracles, (iii) methodologies to test software systems for field failures, (iv) homogeneous environments for seamlessly testing software systems from testbed to production. It will impact on production of software systems with an approach to reduce field failures, recognized as one of the main costs of software systems.