Try reloading the player or open this episode directly on YouTube.
This episode of Ship It Weekly discusses the recent hijacking of LiteLLM's release pipeline via stolen GitHub tokens, leading to a malicious release. It also covers GitHub's new package firewall, critical PHP2Shell vulnerabilities, and the compromise of eMudhra's CA keys.
Almost nothing in this week's episode broke inside anyone's own code.
LiteLLM's source on GitHub was clean. The PHP that got patched wasn't anybody's app, it was the runtime underneath it. And eMudhra's problem wasn't a break-in at all. It was a key ceremony from April 2025 that nobody had looked at closely enough.
Every main story this week came through something you depend on but don't run. A library's release pipeline. The registry your builds pull from. The runtime under your app. The trust store baked into your images.
That's where I want to spend this one.
Start with LiteLLM.
The part that stuck with me isn't the payload, even though the payload is bad. It's how the release got out.
According to LiteLLM, the attacker had stolen GitHub CLI tokens from three of their engineers. One account changed the release pipeline and kicked off the release. A second one approved it. A third deleted the workflow records afterwards.
So they had a two-person approval rule, and one attacker satisfied it with two stolen identities.
I don't think that makes two-person approval useless. It means it's a control on people, and the attacker wasn't using people. They were using tokens. If any valid session for an approver can complete the approval, then your real control is how well you protect those sessions: how long the tokens live, where they end up logged, and whether anything notices when an approver's token starts doing things it has never done before.
LiteLLM thinks those tokens leaked through logs on an internal deployment. That's about the most ordinary failure there is. Every one of us has found a token in a log.
Last week I said that if you lock a compromised version, you're just consistently compromised. This is the other side of that. The tag looked right. The artifact wasn't. Pinning keeps you on the version you reviewed. It can't help when the version you're moving to was built by someone else.
And if 1.107.0 landed anywhere in your world, a laptop, a CI runner, an image that pulled latest, updating isn't the fix. The payload ran every time Python started in that environment, even when you ran pip. Treat every credential that environment could see as stolen, rotate it, and rebuild from 1.106.2. Wiz saw the stolen AWS keys being tested within hours.
GitHub's package firewall landed the same week, and it's the other half of that conversation.
It sits between your machines and the public registry. Coding agents, CI hosts and internal registries pull through it, and it checks every request against your policies before anything installs: malware and vulnerability advisories, supply chain red flags like a release that has lost the attestation earlier versions had, and a cooldown.
The cooldown is the one I keep thinking about. Don't install anything published in the last however many days.
It sounds too simple to matter. But the bad LiteLLM release was on PyPI for under two hours. Even a short cooldown means somebody else finds out first.
To be clear, it wouldn't have stopped LiteLLM. It's npm only for now, and it's for GitHub Enterprise Cloud. But the question behind it applies to everyone: where do your builds pull packages from, and is there anything in that path that can say no?
PHP2Shell is a different kind of dependency.
Two critical bugs in PHP itself. Not WordPress, not a plugin, not a framework. One is memory corruption when PHP runs out of memory while parsing a multipart request, which in practice means a file upload. The other is a use-after-free, where state from a request that hit a fatal error survives into the next request on the same worker. Both rated 9.2, and no login needed.
Amazon's honeypots saw an exploitation attempt on October 17th, four days before disclosure. To be fair, reliable exploitation is still hard on a lot of setups, and most of the public proofs of concept don't work. But a four-day head start should change how we read "nothing in the wild yet." And this isn't only about your own servers: Palo Alto has confirmed PAN-OS and Panorama are affected, with hotfixes still on the way.
My honest take: patching PHP is the easy part. Finding it is the hard part.
The old internal tool. The WordPress box marketing spun up. The appliance that has PHP inside and never mentions it in the docs. If you can't answer "where is PHP running?" today, this is a good week to start that inventory. And when you patch, restart the FPM pools and web server workers, or the old binary keeps answering requests.
eMudhra is the strangest story of the week.
Nobody stole those keys. A flaw in a key generation ceremony back in April 2025 produced keys with too little randomness, so the private keys for eight intermediate certificates could be worked out from the public certificates. Mozilla recovered them within two hours of the report. Scott Helme got all eight in under a second.
The browsers handled it well. Chrome blocked the six TLS intermediates in under two hours, Firefox followed that night, and Apple, Android and Microsoft followed within three days. eMudhra says only one of the eight was in production and none had issued certificates, and Chrome found no logged TLS certificates under them.
So the browsers are fine. The interesting question is everything that isn't a browser.
A Go binary. Python's certifi bundle. The trust store baked into a container image. Those keep trusting whatever they shipped with until someone updates them and rebuilds. Go has fixed releases out, but a fixed Go release doesn't fix the binary you built last month.
Your trust store is a dependency. It's in every image you build, and most of us never look at it. If you already rebuild your base images on a schedule, this is the kind of week that schedule exists for. If you don't, eMudhra is a pretty good argument for starting.
The human closer tied it together in a way I didn't expect.
Brent Chapman's piece is about paging, not supply chains. His point is that we page on what's easy to collect, CPU, memory and disk, and those tell you how the machine feels. They don't tell you whether a user can log in. A service at 95% CPU that's meeting its latency targets isn't an incident. A service at 5% CPU that's failing half its requests is.
Back in September I said a disk alert at 99% isn't really an alert, it's a notification. Chapman's version is cleaner. A disk that fills in a week is a ticket for business hours. A disk that fills in an hour is a page, because users are about to feel it.
Two ideas from his piece I'd steal tomorrow. At Google, teams reviewed every alert from the past week and asked whether it was actionable or should be downgraded to a ticket. And at Slack, a spike in visits to the public status page often showed a customer problem before any internal alert fired.
Now run this week's stories through that lens.
A credential stealer doesn't spike your CPU. A distrusted certificate shows up as users who can't connect. A zone losing power shows up as people who can't pay. None of it shows up as how the machine feels.
That's where all of this landed for me.
The things most likely to hurt you this year probably aren't in your own code, and they probably won't show up on your dashboards first. They're in the pipelines, registries, runtimes and trust stores you depend on but don't run.
The defense is mostly boring. Know what you depend on. Give yourself a way to say no. Rotate and rebuild when it goes wrong. And alert on what your users feel.
Your users don't care how your machines feel.
They care whether they can log in.
📝 Notes
Show Notes
This week on Ship It Weekly, LiteLLM's release pipeline was hijacked again: stolen GitHub CLI tokens from three engineers changed the pipeline, kicked off the release and approved it, and a credential-stealing 1.107.0 went out to PyPI and Docker Hub. GitHub put a package firewall into public preview, with advisory and supply chain checks and a cooldown for new releases. PHP fixed PHP2Shell, two critical core runtime bugs (CVE-2026-109618 and CVE-2026-109619) that someone tried to exploit four days before disclosure. And eMudhra, a certificate authority in India, found out the private keys for eight intermediate certificates could be worked out from its own public certificates.
The thread: most of this came through things you depend on but don't run. A library's release pipeline, the registry you pull from, the runtime under your app, and the trust store in your images.
In the lightning round: another GitLab CVSS 10 (CVE-2026-110264), Google's report on the europe-west2 London outage, AWS taking its European Sovereign Cloud off the global backbone on October 24, and a GitHub audit log change that can quietly break SIEM rules.
The human closer, from Brent Chapman: page on what users actually do. Your users don't care how your machines feel. They care whether they can log in.
On Monday, Jinger Meilani joins me to talk about running retros like a tabletop RPG.
Highlights
• LiteLLM 1.107.0: two stolen identities passed a two-person approval
• GitHub package firewall: advisories, attestations and a cooldown
• PHP2Shell: an exploit attempt four days before disclosure
• Patching PHP is easy, finding all of it is hard
• eMudhra: CA private keys worked out from public certificates
• Page on what users feel, not how the machine feels
The Things You Depend On But Don't Run
Almost nothing in this week's episode broke inside anyone's own code.
LiteLLM's source on GitHub was clean. The PHP that got patched wasn't anybody's app, it was the runtime underneath it. And eMudhra's problem wasn't a break-in at all. It was a key ceremony from April 2025 that nobody had looked at closely enough.
Every main story this week came through something you depend on but don't run. A library's release pipeline. The registry your builds pull from. The runtime under your app. The trust store baked into your images.
That's where I want to spend this one.
Start with LiteLLM.
The part that stuck with me isn't the payload, even though the payload is bad. It's how the release got out.
According to LiteLLM, the attacker had stolen GitHub CLI tokens from three of their engineers. One account changed the release pipeline and kicked off the release. A second one approved it. A third deleted the workflow records afterwards.
So they had a two-person approval rule, and one attacker satisfied it with two stolen identities.
I don't think that makes two-person approval useless. It means it's a control on people, and the attacker wasn't using people. They were using tokens. If any valid session for an approver can complete the approval, then your real control is how well you protect those sessions: how long the tokens live, where they end up logged, and whether anything notices when an approver's token starts doing things it has never done before.
LiteLLM thinks those tokens leaked through logs on an internal deployment. That's about the most ordinary failure there is. Every one of us has found a token in a log.
Last week I said that if you lock a compromised version, you're just consistently compromised. This is the other side of that. The tag looked right. The artifact wasn't. Pinning keeps you on the version you reviewed. It can't help when the version you're moving to was built by someone else.
And if 1.107.0 landed anywhere in your world, a laptop, a CI runner, an image that pulled latest, updating isn't the fix. The payload ran every time Python started in that environment, even when you ran pip. Treat every credential that environment could see as stolen, rotate it, and rebuild from 1.106.2. Wiz saw the stolen AWS keys being tested within hours.
GitHub's package firewall landed the same week, and it's the other half of that conversation.
It sits between your machines and the public registry. Coding agents, CI hosts and internal registries pull through it, and it checks every request against your policies before anything installs: malware and vulnerability advisories, supply chain red flags like a release that has lost the attestation earlier versions had, and a cooldown.
The cooldown is the one I keep thinking about. Don't install anything published in the last however many days.
It sounds too simple to matter. But the bad LiteLLM release was on PyPI for under two hours. Even a short cooldown means somebody else finds out first.
To be clear, it wouldn't have stopped LiteLLM. It's npm only for now, and it's for GitHub Enterprise Cloud. But the question behind it applies to everyone: where do your builds pull packages from, and is there anything in that path that can say no?
PHP2Shell is a different kind of dependency.
Two critical bugs in PHP itself. Not WordPress, not a plugin, not a framework. One is memory corruption when PHP runs out of memory while parsing a multipart request, which in practice means a file upload. The other is a use-after-free, where state from a request that hit a fatal error survives into the next request on the same worker. Both rated 9.2, and no login needed.
Amazon's honeypots saw an exploitation attempt on October 17th, four days before disclosure. To be fair, reliable exploitation is still hard on a lot of setups, and most of the public proofs of concept don't work. But a four-day head start should change how we read "nothing in the wild yet." And this isn't only about your own servers: Palo Alto has confirmed PAN-OS and Panorama are affected, with hotfixes still on the way.
My honest take: patching PHP is the easy part. Finding it is the hard part.
The old internal tool. The WordPress box marketing spun up. The appliance that has PHP inside and never mentions it in the docs. If you can't answer "where is PHP running?" today, this is a good week to start that inventory. And when you patch, restart the FPM pools and web server workers, or the old binary keeps answering requests.
eMudhra is the strangest story of the week.
Nobody stole those keys. A flaw in a key generation ceremony back in April 2025 produced keys with too little randomness, so the private keys for eight intermediate certificates could be worked out from the public certificates. Mozilla recovered them within two hours of the report. Scott Helme got all eight in under a second.
The browsers handled it well. Chrome blocked the six TLS intermediates in under two hours, Firefox followed that night, and Apple, Android and Microsoft followed within three days. eMudhra says only one of the eight was in production and none had issued certificates, and Chrome found no logged TLS certificates under them.
So the browsers are fine. The interesting question is everything that isn't a browser.
A Go binary. Python's certifi bundle. The trust store baked into a container image. Those keep trusting whatever they shipped with until someone updates them and rebuilds. Go has fixed releases out, but a fixed Go release doesn't fix the binary you built last month.
Your trust store is a dependency. It's in every image you build, and most of us never look at it. If you already rebuild your base images on a schedule, this is the kind of week that schedule exists for. If you don't, eMudhra is a pretty good argument for starting.
The human closer tied it together in a way I didn't expect.
Brent Chapman's piece is about paging, not supply chains. His point is that we page on what's easy to collect, CPU, memory and disk, and those tell you how the machine feels. They don't tell you whether a user can log in. A service at 95% CPU that's meeting its latency targets isn't an incident. A service at 5% CPU that's failing half its requests is.
Back in September I said a disk alert at 99% isn't really an alert, it's a notification. Chapman's version is cleaner. A disk that fills in a week is a ticket for business hours. A disk that fills in an hour is a page, because users are about to feel it.
Two ideas from his piece I'd steal tomorrow. At Google, teams reviewed every alert from the past week and asked whether it was actionable or should be downgraded to a ticket. And at Slack, a spike in visits to the public status page often showed a customer problem before any internal alert fired.
Now run this week's stories through that lens.
A credential stealer doesn't spike your CPU. A distrusted certificate shows up as users who can't connect. A zone losing power shows up as people who can't pay. None of it shows up as how the machine feels.
That's where all of this landed for me.
The things most likely to hurt you this year probably aren't in your own code, and they probably won't show up on your dashboards first. They're in the pipelines, registries, runtimes and trust stores you depend on but don't run.
The defense is mostly boring. Know what you depend on. Give yourself a way to say no. Rotate and rebuild when it goes wrong. And alert on what your users feel.
Your users don't care how your machines feel.
They care whether they can log in.