While AI systems differ in some points from "traditional" systems, testing them does not have to be more difficult - knowing the right questions to ask will go a long way. In this talk we will:

• Arm you with a checklist of questions to ask when preparing to test an AI system
• Show you that testers and data scientist have common ground when testing AI systems

Keep calm and test on - AI systems are not that different from "normal" systems.

Target Audience: Testers, Data Scientists, Developers, Product Owners, Architects
Prerequisites: Basic knowledge of software testing
Level: Advanced

Extended Abstract:
If you're a tester about to test your first AI system, or wanting to move into that area, you're probably wondering how you can prepare for the role. While we usually do not deal with complexity in the magnitude of Large Language models like chatGPT, AI systems still seemingly offer different challenges than "traditional" systems.
You're not the first person to deal with these questions. In fact, a group of us got together to explore it in more detail. Is there a general framework of questions that testers can use to help develop a quality strategy for systems that use AI? We wanted to see if we could design one. To this end, we got together a group with diverse roles: tester, test architect, data scientist, project lead and CEO.
Join us in this talk to hear how we approached the task and what our results are, including an example of using our checklist of questions to analyse a system that uses AI. Along the way we also addressed questions like "What is the role of a tester in such projects?" and "How much math do I need?" - we'll talk about those discussions, too. We encourage participants to use our checklist and give us feedback on it!