Software Supply Chain After xz

Supply-Chain Security laid out the defenses — SBOMs to know what’s inside an artifact, provenance to know where it came from, signing to know it wasn’t tampered with — and Supply-Chain Attacks showed the mechanisms those defenses exist to stop. This page is about the single event that turned all of that from “good hygiene most teams skip” into “the thing the whole industry suddenly took seriously”: the xz-utils backdoor, disclosed in late March 2024 as CVE-2024-3094. It is worth a page of its own because of how it happened — not a stolen key, not a typo’d package name, but a human being who spent years becoming a trusted maintainer in order to plant a backdoor by hand.

The wake-up call: a backdoor years in the making

xz (and its library liblzma) is a compression tool so ubiquitous it is effectively invisible — it ships in nearly every Linux distribution and is pulled in, directly or transitively, by an enormous amount of software. Like a great deal of critical open-source infrastructure, it was maintained by essentially one volunteer, doing unpaid work in his spare time.

Over roughly two to three years leading up to 2024, an account using the name “Jia Tan” built a track record of helpful contributions, gradually earned the original maintainer’s trust, and was granted co-maintainer rights. This was not a smash-and-grab. It was a slow, patient social-engineering takeover of a position of trust — including, by accounts at the time, sock-puppet accounts pressuring the overworked maintainer to hand off more control. Once trusted, the attacker shipped the payload in releases 5.6.0 and 5.6.1.

The backdoor’s target was sshd, the OpenSSH server — the front door to basically every Linux machine on the internet. On distributions that linked sshd against liblzma (via a systemd dependency), the backdoor hooked into the authentication path so that an attacker holding a specific private key could gain remote code execution. A backdoor in xz was, in effect, a master key to a large fraction of the world’s servers.

By the numbers — how close this came

• CVE: CVE-2024-3094. Backdoored versions: xz / liblzma 5.6.0 and 5.6.1.
• Disclosure: late March 2024 (≈29 March), by Andres Freund, a Microsoft engineer and PostgreSQL developer who was investigating something unrelated.
• The tell: ssh logins to an affected machine ran roughly half a second slower than expected, and sshd was burning unexpected CPU. That ~0.5 s anomaly — plus some Valgrind errors — is what pulled the thread.
• The near-miss: the malicious releases had reached rolling / testing distributions (e.g. Debian sid, Fedora Rawhide / 40 beta, openSUSE Tumbleweed, Kali, Arch) but had not yet landed in the major stable releases. Days or weeks more and it would have been everywhere.

The honest summary: the entire internet was saved a catastrophe largely by one engineer’s curiosity about a half-second of latency. That is not a control. That is luck.

Under the hood — where the backdoor hid

The cleverest part of xz was placement. The malicious code was not in the obvious git source that a casual reviewer would read. It was hidden in the release tarballs — the packaged downloads that distributions actually build from, which are generated by the maintainer and can differ from the git tree.

The payload was disguised inside the build tooling and the test fixtures: a modified m4 build script (build-to-host.m4) and binary blobs checked in as “corrupt” test data for the compressor. During the build, the script would notice it was building a .deb/.rpm for the right architecture, extract and de-obfuscate the payload from those “test” files, and patch it into liblzma. At runtime the library used an IFUNC resolver (a legitimate glibc mechanism for choosing an optimized function at load time) to hijack a symbol on the OpenSSH authentication path. So:

   git source (clean)  ──► humans review this ──► looks fine ✓
        │
        │  but distros build from THIS, not git:
        ▼
   release tarball  ──► hidden payload in build script + "test" blobs
        │
        ▼
   build patches liblzma  ──► sshd links liblzma  ──► backdoored authentication

This is the exact nightmare the supply-chain pages warned about, made real: the source humans read was clean; the artifact that shipped was not. Source review caught nothing because there was nothing in the source to catch.

Why the old defenses didn’t fire

It is tempting to ask “where was the signing? the SBOM? the scanner?” — but notice that none of them would have stopped this. The releases were signed by the legitimate maintainer, because the attacker was a legitimate maintainer. An SBOM would have faithfully listed liblzma 5.6.1 — a real version from the real project. A vulnerability scanner had nothing to match, because this was a zero-day nobody had catalogued. The compromise was at the human trust layer, which sits upstream of every cryptographic control. xz proved that the soft underbelly of the supply chain isn’t always a key or a registry — it can be a person, and the funding-and-burnout reality that leaves critical infrastructure in the hands of a single exhausted volunteer.

It actually happened — the maintainer was the attacker

The uncomfortable lesson of CVE-2024-3094 is the one the incident-response page makes generally: human “error” is a system symptom. Here the “system” was an ecosystem that runs trillions of dollars of commerce on libraries maintained, for free, by individuals nobody pays and nobody backs up. A patient adversary exploited exactly that: an overworked solo maintainer, social pressure to share the load, and a tarball-vs-git gap few people audit. No amount of cosign verify fixes a maintainer you trusted who turned out to be hostile. That is why the response below is not only cryptographic.

The response — defense in depth, including the human layer

xz didn’t introduce new defenses so much as it moved their adoption deadline forward. The technical layers are the ones from Supply-Chain Security, now with a concrete reason to actually deploy them:

Layer	What it does	What it would (and wouldn’t) have done for xz
SLSA provenance	Signed record of what built this, from what source, how	Forces builds from a known, isolated pipeline — exposing the tarball-vs-git gap the attacker hid in
Sigstore / cosign keyless signing	Ties signatures to short-lived OIDC identities + a public log (Rekor)	Wouldn’t have caught a trusted signer, but makes signer identity auditable and revocable
SBOMs	Machine-readable inventory of every component	Wouldn’t detect the backdoor, but the day it broke, “are we running 5.6.0/5.6.1?” became a query, not a panic
Dependency pinning	Install exactly the bytes you vetted, by hash	Stops silent version drift into a poisoned release; you upgrade deliberately, not by surprise
Maintainer-trust / funding	Pay and back up critical maintainers; require multi-maintainer review	Attacks the root cause — the single, unsupported, pressurable human

Read the table as two halves. The cryptographic half — SLSA provenance levels (build from a trusted, reproducible pipeline so the artifact provably matches the source), keyless signing so identities are short-lived and logged, SBOMs so you can answer “are we exposed?” in minutes, and pinning so you never drift silently into a bad release — narrows the gap between the code humans reviewed and the bytes that ran. That gap is precisely where xz lived.

The other half is the part xz forced the industry to say out loud: the human supply chain is part of the supply chain. Initiatives that fund critical maintainers, require more than one trusted reviewer on a sensitive merge, and treat a “tiny” dependency under one person’s control as the systemic risk it is, are all responses to the social attack, not the technical one. Reproducible builds — where anyone can rebuild the artifact from source and get bit-for-bit identical output — are the strongest structural answer, because they collapse the tarball-vs-git gap entirely.

Reproducible builds close the exact door xz used

If the published artifact is bit-for-bit reproducible from the public source, then a malicious tarball that differs from git is detectable by anyone, automatically. The attacker’s whole trick — ship a release that doesn’t match the source humans reviewed — stops working. Reproducibility turns “trust the maintainer’s tarball” into “verify the bytes against the source,” which is this book’s thread applied to the build itself.

The thread: from trusting a person to verifying the bytes

The manual, error-prone step the post-xz response removes is extending trust to a human and then never re-checking the artifact that trust produced — assuming that because the source is open and the maintainer is known, the bytes that ship must match the code you read. xz proved that assumption is a single point of catastrophic failure. Provenance and reproducible builds make the artifact provably match the source; pinning makes upgrades deliberate; SBOMs make exposure answerable in minutes; and funding plus multi-maintainer review attacks the lonely, pressurable human at the root.

Production gets safer because the trust boundary stops resting on one person’s good faith and starts resting on verifiable properties of the artifact — properties an attacker can’t forge even after they’ve charmed their way into commit rights. The cost is real: reproducible builds are hard, provenance is pipeline work, and paying maintainers requires someone to actually pay them. But xz set the price of not doing it: a backdoor in sshd on a large fraction of the internet, caught by a half-second of luck. For the attacker’s-eye view of the broader category, see Supply-Chain Attacks; for where this and the other 2020s lessons point the field, see Where DevOps Is Going.

Check your understanding

1. Why is xz described as a social-engineering attack rather than a technical exploit, and why does that make it different from typosquatting or a stolen signing key?
2. The malicious code “was in the release tarballs, not the git source.” Explain why that placement defeated source review, and what reproducible builds would have done about it.
3. The releases were correctly signed and an SBOM would have correctly listed liblzma 5.6.1. So which defenses failed to fire, and why — and what would each still have been useful for?
4. How was the backdoor actually discovered, and why is “we were saved by one engineer noticing ~0.5 s of latency” an indictment rather than a reassurance?
5. Using the book’s thread, what manual leap of faith does the post-xz response remove, and which single defense most directly closes the specific gap the attacker exploited?

Show answers

1. The attacker didn’t break a control — they became a trusted maintainer over two-to-three years and used that legitimate authority to plant the backdoor. Typosquatting relies on a developer’s typo and a stolen key relies on cryptographic theft; xz subverted the human trust layer that sits upstream of every cryptographic control, so the malicious release was signed by a genuinely authorized person.
2. Distributions build from the maintainer-generated tarball, which can differ from the public git tree; the payload was hidden there (in a build m4 script and “corrupt” test blobs), so reviewers reading git saw clean code. Reproducible builds would expose it: if the artifact must rebuild bit-for-bit from public source, a tarball that differs from git is automatically detectable by anyone.
3. Signing failed to help because the signer was legitimate (the attacker was the maintainer); scanning failed because it was an uncatalogued zero-day; an SBOM wouldn’t flag a real version of a real project. They were still useful: signing keeps signer identity auditable/revocable, and the SBOM turned “are we running 5.6.0/5.6.1?” into a fast query the day it broke instead of a frantic audit.
4. Andres Freund investigated sshd using unexpected CPU and ssh logins running ~0.5 s slow (plus Valgrind errors) and traced it to the backdoor — before it reached most stable distros. It’s an indictment because no control caught it; it was one engineer’s curiosity about a tiny latency anomaly. Luck is not a security strategy.
5. It removes trusting a human and never re-verifying the artifact that trust produced — assuming open source plus a known maintainer guarantees the shipped bytes match the reviewed code. Reproducible builds (backed by SLSA provenance) most directly close the gap, because they collapse the tarball-vs-git difference the attacker hid in.