White paper · 10 November 2026

Explainable AI for medical imaging: a checklist for people who build and buy

Key points

  • An explanation is evidence about a model, not proof that the model is right.
  • Check the data before the model: an explanation of a model trained on a biased set explains the bias.
  • Test the explanation itself: does it change when the model changes, and does it match what experts look at?
  • State what the model is for and where it should not be used.
  • Keep records that someone else can audit.

Why this checklist exists

Imaging models are now accurate enough to be useful and opaque enough to be risky. A heat map over a slide or a scan looks like an explanation, and people trust it for that reason. Whether it deserves that trust is a separate question, and it can be tested. The five questions below are the ones we ask in our own work, and they are equally useful when you are buying a system.

1. What data was the model built on?

Ask for the size and source of the training and test sets, how labels were produced and by whom, and how many cases came from each site, scanner or stain. Ask whether the test set is independent of the training set at patient level, not only at image level. A model that learned a scanner signature will give you a convincing explanation of the wrong thing.

2. What kind of explanation is it?

Methods differ. Some show where in the image the model looked, some show which features or concepts drove a result, and some give a similar example. Each answers a different question. Ask which question the explanation is meant to answer, for whom, and what it cannot tell you.

3. Has the explanation been tested?

Two checks are simple and revealing. First, randomise the model's weights or the labels: if the explanation barely changes, it is describing the image, not the model. Second, compare it with what experts say they rely on, for example the regions a pathologist or radiologist marks as decisive. Agreement is encouraging, and disagreement is information, not necessarily a fault.

4. What is the model for, and where does it stop?

A clear intended use names the clinical question, the population, the image types and the user. It also names what is out of scope. Ask for performance in subgroups and for a description of the failures that matter most clinically.

5. What is recorded, and who can check it?

Look for versioned data and code, a log of which model version produced which result, and a documented way to review and correct explanations. Regulators in the UK and EU are moving towards lifecycle oversight, so records you keep from the start are cheaper than records reconstructed later.

How to use this

If you are building: answer all five in writing before the first demonstration. If you are buying: ask the supplier the same five and note which answers are missing. A short, honest answer with stated limits is a better sign than a confident one without them.

This paper gives general guidance. It does not report results from any client project, and it is not legal or regulatory advice.

All white papers