News · 30 January 2025

Reasoning traces are not explanations: what DeepSeek-R1 does and does not tell us

In January 2025 DeepSeek released R1, an open-weight reasoning model that writes out a long chain of thought before giving an answer. The release attracted attention for its cost and performance. For people who care about explainability it raised a quieter question: is a model's visible reasoning an explanation of its answer?

The intuition and the problem

It feels natural to read a step-by-step trace and conclude, "so that is why it said that." But a trace is itself generated text. It is produced by the same process as the answer, and nothing guarantees that it faithfully describes the computation that led there.

This is not a hypothetical worry. Research such as Turpin and colleagues (2023), "Language Models Don't Always Say What They Think", showed that chain-of-thought explanations can be systematically unfaithful: a model can be nudged by a biasing feature in the prompt and then produce a plausible reasoning chain that never mentions it.

What traces are good for

None of this makes reasoning traces useless.

  • Debugging. A trace shows where a model went off track, even if it is not a complete account.
  • Better answers. Spending more computation on intermediate steps often improves accuracy on hard problems.
  • Review. A human can check whether the steps are valid, which is itself a form of verification.

The distinction is between a trace as a useful artefact and a trace as evidence about the model's internals. We should use the first without assuming the second.

What a defensible explanation looks like

For a high-stakes system, such as one assisting a diagnosis or a risk assessment, an explanation should be:

  1. Faithful: it reflects what actually influenced the output, which can be tested by perturbing the input and checking that the explanation changes accordingly.
  2. Stable: similar inputs give similar explanations.
  3. Comprehensible: expressed in terms the user can act on.
  4. Validated with users: someone has checked that experts reach better decisions with it than without it.

Attribution methods, example-based explanations and concept-level analyses can each meet some of these criteria for some model types, and each has known failure modes.

The practical conclusion

Use reasoning models where they help, and treat their visible reasoning as one signal among several. If a decision must be explainable to a regulator or a patient, build the explanation layer deliberately and test it, rather than trusting that fluent text equals faithful insight.

Written in January 2025; details reflect the situation at that time.

All news