News · 16 April 2024

The xz backdoor: what it teaches us about trusting what we depend on

At the end of March 2024 a Microsoft engineer, Andres Freund, noticed that SSH logins on a test machine were a few hundred milliseconds slower than usual and were using more CPU than expected. Following that thread, he found a backdoor in the xz compression library, versions 5.6.0 and 5.6.1, tracked as CVE-2024-3094.

Why this was unusual

Most supply-chain attacks are blunt: a stolen credential, a typosquatted package. This one was patient. An account that had contributed to the project for a long time gained maintainer trust and then shipped obfuscated code that only appeared in release tarballs, not in the visible source repository. The target was not xz itself but sshd, which on some distributions loads the library indirectly.

It was caught largely by luck and by a person who cared about a tiny performance regression.

Three lessons

1. Monitoring finds what review misses. Code review did not catch this. A latency anomaly did. Systems that record baseline behaviour (login time, CPU, outbound connections) give an engineer something to notice. In our monitoring work we stress that dashboards are not decoration: they are the mechanism by which a strange 500 ms becomes a question.

2. Dependencies are people. An open-source library is maintained by humans who are often unpaid and under pressure. Treat critical dependencies the way you treat critical suppliers: know who maintains them, pin versions deliberately and have a plan if one goes bad.

3. Build what you ship, and compare it. The malicious code lived in the build artefacts. Reproducible builds, checking release tarballs against source and generating a software bill of materials (SBOM) all make that class of trick harder.

What a small team can do this month

  • Produce an inventory of what runs on your servers and where each component comes from.
  • Subscribe to security advisories for your base operating system and critical libraries.
  • Alert on unexpected changes in authentication latency and outbound connections from servers that should be quiet.
  • Include your supply chain in penetration-test scope. Attackers do not respect the boundary of "our code".

No process would have guaranteed detection here. But organisations that watch their own systems closely have far better odds than those that do not.

Written in April 2024; details reflect the situation at that time.

All news