Autonomous and automated systems are increasingly being used in IT such as finance, but also transport, medical surgery and industry automation. Yet, the distrust in their reliability is growing. This presentation introduces the validation of autonomous systems. We evaluate in practical situations such as automatic driving and autonomous robots different validation methods. The conclusion: Classic methods are relevant for coverage in defined situations but must be supplemented with cognitive test methods and scenario-based testing.

Target Audience: Testers, Quality assurance, Architects, Requirements Engineers, Product Owners, Software Engineers
Prerequisites: Testing basic know-how
Level: Advanced

Extended Abstract:
Autonomous and automated systems are increasingly being used in IT such as finance, but also transport, medical surgery and industry automation. Yet, the distrust in their reliability is growing. There are many open questions about the validation of autonomous systems: How to define reliability? How to trace back decision making and judge afterwards about it? How to supervise? Or, how to define liability in the event of failure? The underlying algorithms are difficult to understand and thus intransparent. Traditional validations are complex, expensive and therefore expensive. In addition, no repeatable effective coverage for regression strategies for upgrades and updates is available, thus limiting OTA and continuous deployment.

With artificial intelligence and machine learning, we need to satisfy algorithmic transparency. For instance, what are the rule in an obviously not anymore algorithmically tangible neural network to determine who gets a credit or how an autonomous vehicle might react with several hazards at the same time? Classic traceability and regression testing will certainly not work. Rather, future verification and validation methods and tools will include more intelligence based on big data exploits, business intelligence, and their own learning, to learn and improve about software quality in a dynamic way.

Verification and validation depend on many factors. Every organization implements its own methodology and development environment, based on a combination of several of the tools presented in this presentation. It is however relevant to not only deploy tools, but also build the necessary verification and validation competences. Too often we see solid tool chains, but no tangible test strategy. To mitigate these pure human risks, software must increasingly be capable to automatically detect its own defects and failure points.

Various new intelligent validation methods and tools are evolving which can assist in a smart validation of autonomous systems.

This presentation introduces the validation of autonomous systems. We evaluate in practical situations such as automatic driving and autonomous robots different validation methods. The conclusion: Classic methods are relevant for coverage in defined situations but must be supplemented with cognitive test methods and scenario-based testing.