5 Questions Before You Trust an AI System

When a new AI project lands on our desk for review, people expect us to open with questions about the model: which one, how big, how clever. We don't. We open with five questions that sound almost embarrassingly basic.

In our experience, when a project falls apart later, it's almost never the model's fault. It failed one of these five.

1. Source: where does the truth actually live?

Forget the database. Forget the embeddings. We want to know where the real information lives — the source everything else is copied from.

Is it trustworthy? Is it current? And the question that ends most conversations fastest: once this system is live, whose job is it to keep that source accurate?

Silence usually follows. Silence means there isn't an AI project yet — there's a data problem wearing a nicer outfit.

2. Fallback: what happens the moment it's wrong?

It will be wrong eventually. No system is right every time.

Play it forward: the model gives a confidently wrong answer. Does anyone notice? Does the mistake flow quietly into the next step and become three mistakes? Or does a person catch it before it reaches a customer?

Without a clear answer here, the system is one bad day away from causing real damage.

3. Trace: can you replay a single answer?

A customer points at one response and says it's wrong. Can you replay it end to end — what got retrieved, what the prompt looked like, what the model returned, what happened to that answer before anyone saw it?

If you can walk it back, you can fix it. If you can't, you're not fixing anything. You're guessing, and guessing isn't a strategy for a production system.

4. Test: does your eval look like real life?

Leaderboards are fun. They don't pay anyone's bills.

What matters is a set of test cases that look like the messy, specific things real users actually ask, each with a known right answer, checked automatically every time something changes.

Skip this and quality erodes quietly. The system gets worse over time, and nobody notices until a customer does.

5. Kill switch: who can turn it off tonight?

Eventually something goes sideways: a model updates without warning, a source gets poisoned, someone finds a crack and slips through it.

When that happens, there needs to be one name. One person authorized to shut the system down, who knows how to do it, and who is reachable at 2am. If the answer is no one, that's the first thing to fix, before anything else on this list.

None of these five questions is about artificial intelligence. They would apply equally to a payment system or a power grid.

We ask them of AI projects specifically because AI is the one place teams convince themselves they can skip this step. The model is impressive enough in a demo that everything around it starts to feel like background noise.

It isn't background noise. It's the whole product. The model gets the applause; the unglamorous infrastructure around it decides whether the system is still standing a year from now.

The five, at a glance

• Source — Where does the truth live, and whose job is it to keep that accurate?
• Fallback — What happens the instant the system is confidently wrong?
• Trace — Can you replay any single answer, end to end?
• Test — Does your eval set look like real usage, or just a leaderboard?
• Kill switch — Who can shut it down tonight, and are they reachable?