At first glance, testing AI systems seems very different from testing “conventional” systems. However, many standard testing activities can be preserved as they are or only need small extensions.

In this talk, we give an overview of topics that will help you test AI systems: Attributes of training/testing/validation data, model performance metrics, and the statistical nature of AI systems. We will then relate these to testing tasks and show you how to integrate them.

Target Audience: Developers, Testers, Architects
Prerequisites: Basic knowledge of testing
Level: Basic

Extended Abstract:
From a testing perspective, systems that include AI components seem like a nightmare at first glance. How can you test a system that contains enough math to fill several textbooks and changes its behavior on the whims of its input data? How can you test what even its creators don’t fully understand?

Keep calm, grab a towel and carry on - what you have already been doing is still applicable, and most of the new things you should know are not as arcane as they might seem. Granted, some dimensions of AI systems like bias or explainability will likely not be able to be tested for in all cases. However, this complexity has been around for decades even in systems without any AI whatsoever. Additionally, you will have allies: Data scientists love talking about testing data.

In this talk, we give an overview of topics that will help you test AI systems: Attributes of training/testing/validation data, model performance metrics, and the statistical nature of AI systems. We will then relate these to testing tasks and show you how to integrate them.