How One Engineer's Curiosity Stopped a Two-Year Spy Operation Hidden in Plain Sight
The story of how a 500-millisecond anomaly almost changed the internet forever.
It started with lag.
Not the catastrophic, headline-grabbing kind. Just... slightly slower SSH logins. A few hundred milliseconds more than usual. The kind of thing most engineers would shrug off, blame on network congestion, and forget about over their morning coffee.
Andres Freund didn’t shrug it off.
That decision, that one act of refusing to accept “good enough”, is the reason millions of Linux servers around the world weren’t silently compromised in 2024. It’s the reason your bank’s servers, hospital infrastructure, and government systems didn’t wake up one morning with a hidden backdoor installed deep in their bones.
But to understand what Freund actually stopped, we need to go back two years. To a GitHub account. To a name no one had heard of. To a patience that was almost inhuman.
Act I: The Ghost Who Wanted to Help
In 2021, a developer calling themselves “Jia Tan” appeared out of nowhere on the XZ Utils project.
If you’ve never heard of XZ Utils, that’s fine, most people haven’t. It’s a compression library, a piece of invisible plumbing deep inside almost every major Linux distribution in the world (and mind you, almost every server in the world uses a linux based distribution). When your Linux system compresses or decompresses files, XZ Utils is often doing the quiet heavy lifting.
Jia Tan started small. Submitted a patch here. Fixed a bug there. Nothing suspicious. Just good, clean, competent code contributions from a helpful stranger on the internet.
“Jia Tan” didn’t just submit code. They invested in relationships. They played the long game. They were patient in a way that only a very motivated, well-resourced actor could be.
Over the next two years, they built trust. They made themselves useful. They pushed gently, persistently, to be given more responsibility. They complained that the project’s maintainer, Lasse Collin, was overwhelmed and slow to review contributions. (He was. Open source maintainers almost always are.)
Eventually, Jia Tan was granted commit access to the XZ Utils repository.
That’s when things got dark.
Act II: The Backdoor
In early 2024, Jia Tan introduced versions 5.6.0 and 5.6.1 of XZ Utils. On the surface, they looked fine. Routine updates. The code changes were reviewed, kind of. But buried inside was something extraordinary.
The backdoor wasn’t just malicious code slapped into a function. It was a masterpiece of deception. Here’s what made it terrifying:
It was hidden in a binary test file, not in the C source code itself
It only activated in very specific conditions, on systemd-based Linux, during the build process
It targeted SSH, the protocol used to securely log into remote servers
It allowed the attacker to bypass RSA authentication entirely and execute arbitrary code
Let that last point sink in. SSH (Secure Shell) is the lock on the front door of essentially every Linux server on the internet. It’s how system administrators log in remotely. It’s how deployments happen. It’s how critical infrastructure gets managed.
This wasn’t just a backdoor. It was a skeleton key to the internet’s back office — crafted by someone who knew exactly where to hide it.
The attack used a technique called “function interposition” — hijacking a system’s dynamic linker to replace a legitimate authentication function with a poisoned one. When the compromised library loaded, it replaced OpenSSH’s RSA key validation with its own version. That version would check if an incoming connection carried a specific cryptographic signature, and if it did, skip authentication entirely and execute whatever code the attacker sent.
In plain English: if you had the right private key, you could silently log into any server running the compromised library. No password. No trace.
The versions were already making their way into Debian and Fedora testing repositories. In a matter of weeks, they would have landed in stable releases. From there, they would propagate to every major cloud server, data center, and enterprise machine running Linux.
Millions of servers. Silently compromised. Waiting.
Act III: The 500-Millisecond Anomaly
Andres Freund is a software engineer at Microsoft. On a regular workday in March 2024, he was benchmarking some performance work on his Debian Linux machine. And something nagged at him.
His SSH logins were taking about 500ms longer than they should. Not a lot. Not something most people would notice, let alone investigate. But Freund noticed. And he investigated.
$ time ssh user@localhost
real 0m0.891s # This should be ~0.3s
He ran CPU profiling tools. He traced system calls. He dug into which libraries were consuming unusual CPU time during the login process.
And there it was: liblzma, the library at the core of XZ Utils, was doing a significant amount of work during SSH authentication. That made absolutely no sense. SSH has nothing to do with data compression. Why was XZ Utils even being loaded by the SSH daemon?
“I was doing performance work and noticed some strange behaviour that I investigated further. It turned out to be from the XZ/liblzma library.” — Andres Freund, posting to the OSS-Security mailing list, March 29, 2024
He followed the trail. He found that the compromised library was being injected into the SSH process via a systemd hook. He found the obfuscated binary test files in the repository. He reverse-engineered what they were doing. He checked the version history and found Jia Tan’s fingerprints everywhere.
On March 29, 2024, he posted to the Open Source Security mailing list.
The internet held its breath.
Act IV: Who Was Jia Tan?
Nobody knows.
That’s the terrifying postscript to this story. Despite exhaustive investigation by security researchers worldwide, the real identity of “Jia Tan” has never been confirmed. The GitHub account was almost certainly a persona, a carefully constructed fictional developer with a two-year commit history built specifically to earn trust.
The clues point toward a nation-state operation. The sophistication of the attack is simply beyond what an individual hacker would typically invest. The two-year timeline, the patience, the deep knowledge of build systems, linker mechanics, and SSH internals, this reads like a professional operation with significant resources behind it.
The working theory among security researchers is that this was a state-sponsored operation, possibly Russian intelligence (the FSB or SVR), given geopolitical context, but this has never been definitively proven.
Two years of effort. Hundreds of legitimate commits. A perfectly crafted persona. All of it, dismantled by a developer who was annoyed that his terminal was slow.
The metadata tells part of the story. Jia Tan’s commits consistently happened during business hours in a timezone consistent with Eastern Europe or Central Asia. The commit activity dropped off during Chinese public holidays, possibly a deliberate misdirection, possibly a genuine clue.
We may never know who they really were.
Why This Should Terrify You (And Why You Should Care)
The XZ Utils backdoor is more than a great spy thriller. It reveals something uncomfortable about the infrastructure the modern world runs on.
Open source software powers everything. Your smartphone, your bank, your hospital, your government’s digital systems, all of it runs, in part, on code maintained by volunteers. People like Lasse Collin, who was genuinely burned out, genuinely overwhelmed, and genuinely grateful when a helpful stranger named Jia Tan showed up to help carry the load.
There is no corporate security team behind XZ Utils. No legal department. No red team. Just a lone maintainer doing his best.
The question the security community is now asking is uncomfortable: how many Jia Tans are still out there, still waiting, still building trust, not yet caught?
The answer, almost certainly, is: more than one.
The Accidental Hero
Andres Freund has been appropriately celebrated since his discovery. But he’s been somewhat self-effacing about the whole thing — noting, correctly, that he got lucky. That the right person happened to be doing performance work on the right machine at the right time.
That’s true. But luck favors the prepared mind. Freund noticed because he was the kind of engineer who investigates anomalies instead of dismissing them. He had the skills to trace the problem from a millisecond-level performance blip all the way down to a malicious binary hidden in a test suite. Not everyone would have. Very few people would have.
The most sophisticated supply chain attack in the history of open source software was stopped not by a security tool, an automated scanner, or a corporate red team. It was stopped by one engineer’s refusal to ignore something slightly weird.
There’s a lesson in that. Not just for security engineers, but for all of us who work with complex systems: weirdness is signal. Anomalies are worth your time. The thing that doesn’t quite make sense deserves five more minutes.
Andres Freund gave it five more minutes. And the internet is better for it.
Epilogue
The compromised versions of XZ Utils (5.6.0 and 5.6.1) were quickly pulled. Patches were issued. The major distributions that had included them rolled back. The window of exposure was short — the versions had only reached testing and bleeding-edge repositories, not stable releases.
Lasse Collin, the maintainer who was unknowingly targeted, wrote a brief, shaken post acknowledging the compromise. The GitHub account for Jia Tan was suspended. The carefully constructed two-year persona evaporated.
The open source community has since begun serious conversations about sustainability, funding, and security review for critical infrastructure packages. The OpenSSF (Open Source Security Foundation) has ramped up efforts to identify and protect critical projects.
And somewhere out there, whoever was behind this is still out there. Patient. Methodical. Waiting to try again.
Next time, we might not get lucky.
Further Reading
• Andres Freund’s original disclosure on the oss-security mailing list (March 29, 2024)
• CVE-2024-3094 — the official vulnerability record
• Openwall mailing list archives for the full technical breakdown
• Rhea Karty & Simon Henniger’s deep technical analysis of the backdoor mechanism