Ship It Weekly Host Commentaries

The Operator Is No Longer Just a Person at a Terminal

One thread runs through almost every story in this week’s episode: more and more of the work inside our systems is being performed by something other than the engineer who originally asked for it.

Sometimes that is a managed AWS service. Sometimes it is a Kubernetes controller. Sometimes it is a library buried underneath an application. Increasingly, it is an AI agent interacting with infrastructure tooling.

That changes how I think about operations because the action you request and the action that eventually happens are not always the same thing.

The Kubernetes vulnerability is probably the cleanest example.

CVE-2026-2270 is not interesting because somebody suddenly discovered that Kubernetes RBAC is useless. The attacker still needs meaningful permissions, and exploitation requires a fairly specific set of conditions.

What makes it interesting is the role of the controller.

A user has permission to manipulate one resource. The StatefulSet controller sees that resource and then performs another operation using the controller’s own authority. Under the vulnerable conditions, that can result in a pod being created in another namespace.

The user did not have permission to directly perform that operation.

The controller did.

That distinction matters.

When we review access, we naturally focus on the principal making the request. Can this service account create pods? Can this engineer modify deployments? Can this CI job access secrets?

But modern platforms contain a lot of intermediaries.

Controllers reconcile state. Operators provision resources. CI systems assume roles. GitHub Apps receive permissions. Cloud services react to events. Infrastructure automation creates other infrastructure.

You are often granting somebody permission to ask a more privileged system to do something on their behalf.

That is not inherently bad. It is basically how platforms work.

But it means authorization analysis cannot always stop at the API request.

Sometimes you also need to ask what is going to react to that request afterward.

The Undici vulnerability has a similar shape, although at a completely different layer.

The Node.js application is the client. Normally, that sounds like the safer side of the connection.

We spend enormous amounts of time thinking about protecting servers from hostile clients. Validate the request. Sanitize the payload. Limit the body size. Authenticate the caller.

But clients consume data too.

In this case, a malicious WebSocket server can send compressed data that triggers an error inside Undici’s decompression path. The failure happens below the application's normal WebSocket error handling, and the Node process can terminate.

Your code initiated the connection.

Your process still consumed untrusted input.

That is an easy assumption to miss because developers often think about trust according to direction.

Incoming traffic is dangerous.

Outgoing traffic is something we chose.

Except the fact that your application chose to connect somewhere does not guarantee that the thing on the other end will always behave correctly.

The endpoint could be compromised.

DNS could change.

A third-party service could return malformed data.

Or there could simply be a parser bug in your own dependency.

That does not mean we should treat every outbound connection like an active attacker. It means the trust boundary is not automatically located at the ingress controller.

Sometimes it is inside the HTTP client.

Sometimes it is inside decompression.

Sometimes it is inside a parser nobody on the application team even knew they were running.

The AWS DevOps Guru retirement adds another version of the same problem.

When we consume a managed service, we give up a lot of operational responsibility intentionally.

That is the product.

You do not run DevOps Guru. AWS does.

You integrate with the API, define some resources, consume the output, and build processes around it.

That can make the service feel like infrastructure in the same way S3 or CloudWatch feels like infrastructure. It exists. You use it. You stop thinking very much about the software underneath it.

Until the provider puts an end date on it.

AWS is giving customers plenty of notice here. This is not a shutdown happening next week.

But the migration still illustrates something worth remembering about managed services.

You did not eliminate the lifecycle.

You transferred ownership of part of it.

AWS decides when the service changes. AWS decides when the API stops receiving features. AWS decides when the resource types disappear.

Your side of that contract is dealing with the dependencies you created around it.

That might include CloudFormation resources.

It might include Terraform.

It might include automation consuming insights.

It might include operational processes where somebody expects DevOps Guru to produce a particular signal during an incident.

The migration is not simply:

“Turn off DevOps Guru and turn on DevOps Agent.”

AWS itself is pointing different pieces of the old capability toward different destinations.

Monitoring and anomaly detection may belong in CloudWatch.

AI-assisted investigation may belong in DevOps Agent.

If your team used the service heavily, you first have to understand what job it was actually doing for you.

That sounds obvious, but I have seen enough infrastructure migrations where the first conversation is about replacing a product instead of replacing a capability.

Those are not always the same project.

Then there is Cloudflare.

The number that makes that story worth paying attention to is 48 percent.

Cloudflare says AI agents now account for almost half of Wrangler usage.

That is a much more concrete signal than somebody predicting that agents are eventually going to change software development.

They already changed the usage pattern enough that Cloudflare is changing the tooling.

Wrangler exposes a few hundred operations. Cloudflare has thousands of API operations. Maintaining a manually designed CLI command for everything becomes a very different problem when software is consuming the interface at scale.

So Cloudflare built cf closer to the API itself and open-sourced Forge, the generation system behind it.

I think this is where the agent conversation becomes much more interesting for platform engineers.

The first wave was mostly about putting an agent in front of tools designed for humans.

Let it type shell commands.

Let it call Terraform.

Let it inspect Kubernetes.

Let it operate the same CLI somebody would have used manually.

The next wave is going to involve changing those interfaces because machines use them differently.

Humans care about discoverability in a command hierarchy.

Humans tolerate interactive questions.

Humans read formatted tables.

Humans remember weird command names because they have used the tool for years.

An agent benefits from consistency, complete coverage, predictable schemas, structured output, and fewer special cases.

If a meaningful percentage of the consumers of your developer platform become software agents, that absolutely can influence API and CLI design.

That does not necessarily mean we need a separate “AI version” of every tool.

In fact, the better outcome may be the opposite.

Design interfaces that are predictable enough that both humans and automation can use them without maintaining two completely different worlds.

There is also an authorization question hiding behind all of this.

If an agent is operating infrastructure on behalf of an engineer, whose authority is it using?

What can it do?

What can it discover?

What happens when it chains several individually reasonable operations together?

The Kubernetes story is a useful reminder here.

Security is not always about whether the original actor has direct permission to perform the final action.

Sometimes the important question is what another component will do after receiving the request.

That becomes even more important when the requester is software capable of generating hundreds of those requests very quickly.

The human closer from Lorin Hochstein fits into this better than I initially expected.

His argument is that some categories of availability risk are always going to be present.

Resources are finite.

Networks fail.

Security controls can affect availability.

Production systems have to change.

And the mechanisms we add to improve reliability create additional states of their own.

Controllers are one of those mechanisms.

Managed services are one.

Retries are one.

Failover is one.

Automation is one.

Agents are going to be one too.

We add these things because manually operating everything would be slower, less reliable, and often impossible at the scale we run today.

But every layer capable of making a decision also becomes another place where the system can behave differently than the person at the keyboard expected.

That is why I do not think reliability work eventually converges on a world where enough automation eliminates operational surprises.

The automation gets better.

The platforms get better.

The interfaces get better.

The failure modes change.

Then somebody still gets paged at three in the morning because the system entered a state nobody anticipated.

The job is not to avoid automation or abstraction.

We could not operate modern systems without them.

The job is to understand where authority lives, what can act on our behalf, and how much visibility we have when those actors make a decision.

The operator is no longer just the engineer sitting at the terminal.

It is the controller reconciling the object.

It is the managed service behind the API.

It is the library parsing the response.

It is the CI runner assuming a role.

And increasingly, it is an agent issuing commands for us.

We still own what happens next.

The thing that stuck with me most this week is how many of these stories are really about trust. Not necessarily whether you trust AWS, GitHub, Microsoft, or a particular open-source project, but what happens after your systems have decided something is trusted.

That is where a lot of the interesting infrastructure and security problems seem to live now.

Take the CrowdSec incident. The initial compromise was in the software supply chain, but that was not really the end result the attacker cared about. The valuable part was the credential they were able to get from it. Once that OAuth token existed outside the environment where it belonged, fixing the original package did not fix the larger problem.

According to CrowdSec, that stolen token was eventually used to copy roughly 170 private GitHub repositories in about nine minutes.

Nine minutes.

That is not much of an incident-response window.

By the time somebody realizes something is wrong, starts a Slack thread, pulls in security, checks the logs, figures out which account is involved, and starts revoking access, the actual data movement may have been over for hours, days, or longer.

And I think that changes how we should think about containment.

We spend a lot of time talking about initial access. Patch the vulnerability. Remove the malicious package. Reimage the developer laptop. Kill the process.

Obviously you need to do all of those things.

But the question immediately after that should be: what could this thing authenticate to?

If it was a developer workstation, that answer might be GitHub, AWS, Kubernetes clusters, package registries, CI systems, internal tooling, databases, artifact repositories, or half a dozen other things.

Then you have another question: what credentials were actually present?

That sounds simple until you try answering it during a real incident.

Some credentials are short-lived. Some are not. Some are sitting in environment variables. Some are managed by credential helpers. Some are PATs somebody created two years ago. Some exist in CI. Some belong to GitHub Apps. Some are SSH keys. Some are sessions that are already authenticated.

That is why the GitHub credential inventory announcement in the lightning round is more useful than it might initially sound.

Being able to ask, “Show me the credentials capable of accessing this enterprise, who owns them, what they can access, when they expire, and when they were last used,” is useful on a normal Tuesday.

During an incident, it can become one of the first questions you need answered.

The EvilTokens story approaches the same problem from a different direction.

The part that interests me is not really “hackers are using AI.”

We have heard some version of that story enough times already.

The more interesting part is what AI becomes useful for once an attacker already has access.

A compromised mailbox can contain years of information. Thousands or tens of thousands of emails. Vendor conversations. Invoices. Org charts. Travel schedules. Internal projects. Who approves payments. Who reports to whom. Which vendors regularly ask for money. How the CFO writes. How the CEO writes. Which conversations are already happening that an attacker might be able to insert themselves into.

Historically, that is incredibly valuable information, but there is a human cost to going through all of it.

Now you can classify it.

Instead of an attacker manually reading ten thousand messages trying to understand the company, a model can help answer much narrower questions.

Who has financial authority?

Which conversations involve payments?

Who does this person regularly trust?

Which identity would be useful to impersonate?

That does not fundamentally reinvent business email compromise. It makes the information gathered after compromise much easier to use.

The device-code phishing component is interesting for a similar reason because the victim can actually be interacting with Microsoft’s legitimate authentication infrastructure.

They are not necessarily typing their password into some terrible copy of a Microsoft login page hosted on a random domain.

They can authenticate normally. They can complete MFA. Microsoft can successfully verify that they are exactly who they claim to be.

The problem is what they authorized.

That distinction matters.

We sometimes reduce authentication security to “Do we have MFA?” when the actual system is more complicated than that. MFA can verify the identity of the person completing an authentication flow. It does not automatically mean that person understands what application, device, or session they are authorizing.

Again, the system established trust. The problem happened after that.

The Next.js vulnerability is a little different, but I think there is a related engineering lesson there too.

ImageResponse sounds harmless.

If somebody told me they were investigating a critical remote code execution vulnerability and then said the affected functionality generates Open Graph images, that would not be the first place I expected them to go.

But the server does not care whether we mentally categorize something as “just an image.”

It is still executing code.

It is still processing data.

And if attacker-controlled data reaches that processing path, it is part of your attack surface.

That is also why vulnerability scanners can only take you so far.

Knowing that you have an affected version of Next.js is important. Knowing whether your application actually uses the vulnerable Node.js ImageResponse implementation, where the input comes from, and whether an attacker can control that input is what tells you how the vulnerability relates to your environment.

The same principle applies well beyond Next.js.

A dependency inventory tells you what software exists.

It does not necessarily tell you how that software is being used.

Then there is Elastic Beanstalk Cluster Mode, which is almost the infrastructure version of the same abstraction problem.

AWS is taking something developers already understand as a PaaS experience and putting shared EKS infrastructure underneath it.

I actually like that direction.

I have spent enough of my career working with Kubernetes to know that most developers should not need to understand every detail of Kubernetes just to deploy an application.

A good platform should remove unnecessary complexity.

But abstraction does not make the underlying infrastructure stop existing.

If multiple applications are now sharing infrastructure, somebody still needs to understand what that means for isolation, resource contention, upgrades, capacity, and blast radius.

The developer experience can be:

“Here is my application. Please run it.”

That is great.

The platform engineering experience still has to include:

“What happens when the thing underneath all of these applications has a bad day?”

And that is probably the thread connecting most of this week’s stories for me.

We keep building better abstractions around infrastructure, authentication, dependencies, and developer workflows. That is generally a good thing. Engineers should not have to manually reason about every implementation detail every time they deploy an application or authenticate to a service.

But abstractions create boundaries where we can forget what is happening underneath them.

A valid token does not mean the person using it should still have it.

A successful MFA challenge does not mean the user intended to authorize that session.

An image-generation endpoint is not automatically harmless because its output is a PNG.

A managed platform running on Kubernetes does not eliminate Kubernetes failure modes.

And removing the thing that originally compromised you does not revoke everything the attacker may have already taken.

The abstraction can simplify the normal path.

You still need to understand the trust underneath it when the normal path breaks.

That is usually where the incident starts getting interesting.

The thing I keep coming back to this week is how much of our infrastructure is allowed to change without a commit ever landing in our repository.

The ubuntu-latest story is probably the cleanest example. You can have a pipeline that has been green for six months. Nobody changes the YAML. Nobody updates a dependency. Nobody merges anything. Then one morning it fails because ubuntu-latest now points at a different operating system. From Git’s perspective, nothing changed. From the system’s perspective, something pretty fundamental changed.

And I don’t think the lesson is that latest is bad. There are good reasons to let things move. If I pin a runner image forever, eventually I’m the person running a four-year-old environment because everybody is afraid to touch it. I’ve seen the same thing happen with Terraform providers, Kubernetes versions, base images, Helm, language runtimes, and basically every other dependency we use. Pinning can make today’s build reproducible, but it can also turn tomorrow’s upgrade into a much bigger project.

The better question is whether the movement is intentional.

If I’m using ubuntu-latest, I should know that I’m opting into GitHub moving that environment forward. If I’m using a loose Terraform provider constraint, I should understand what versions that constraint allows. If I’m pulling a container tag that can be overwritten, I should know that the same tag might not mean the same image tomorrow. None of those decisions are automatically wrong. They become a problem when the team thinks something is fixed when it actually isn’t.

Helm 3 is almost the opposite version of the same problem. Nothing is suddenly going to stop working when support ends in February. Your charts aren’t going to look at the calendar and refuse to deploy. That’s actually what makes end-of-life dates so easy to ignore.

The change isn’t necessarily in the software. The change is in the support around it.

Before the deadline, a security issue can get a Helm 3 patch. After the deadline, it doesn’t. Kubernetes continues moving. Client libraries continue moving. The ecosystem moves even if your Helm binary doesn’t. Eventually the gap between the environment you froze and the environment around it gets large enough that somebody has to deal with all of it at once.

That’s why I like doing these migrations when they’re still boring.

Try Helm 4 in a non-production pipeline now. Run the charts. Find the plugin somebody forgot existed. Find the script using an old flag. Find the CI image with Helm 3 baked into it. Those are cheap discoveries in September. They’re much more annoying discoveries when a security issue forces the migration later.

The GitHub workflow execution story gets at another part of this that I think platform teams are still figuring out.

We’ve spent years getting better at controlling what code can do after it starts running. We have IAM policies, short-lived credentials, protected environments, secret scanning, branch protection, OIDC, and increasingly granular permissions inside CI.

But there’s an earlier question: should this automation have started at all?

That sounds obvious, but CI systems grew up as developer tooling. You push code, the thing runs. You open a pull request, the thing runs. A bot makes a change, the thing runs.

That was fine when the pipeline compiled some code and ran unit tests.

Modern CI can assume an AWS role, push an image, sign an artifact, publish a package, modify infrastructure, or deploy production. At that point, triggering a workflow is itself a privileged operation.

That’s why I think GitHub’s execution protections are more interesting than just another Actions setting. I can say this workflow is fine for contributors to trigger, but this one isn’t. This bot can start these workflows, but not that deployment workflow. And I can evaluate the policy before enforcing it and breaking everything.

We touched pull_request_target last week, so I don’t think there’s much value in beating that particular example to death again. What matters is the direction. Last week GitHub gave us more control over what lower-trust workflows can do with caches. This week they’re giving us more control over whether workflows execute in the first place.

That’s a healthier model than treating every workflow in a repository as if it has the same trust requirements.

Then there’s the Cisco story, which is a different kind of dependency problem.

Security appliances are easy to mentally put into a separate category from the rest of our software. The email gateway is the thing protecting us. The firewall is the thing protecting us. The VPN appliance is the thing protecting us.

But they’re still software.

And in some ways they have one of the worst jobs in the environment because we’re deliberately feeding them hostile input all day.

An email gateway exists to parse email coming from people you don’t trust. A firewall processes traffic you don’t trust. A VPN concentrator accepts connections from networks you don’t control. They sit exactly where an attacker wants to interact with them.

So when one of those systems has a remotely exploitable vulnerability, the fact that it’s a security product doesn’t reduce the urgency. It can increase it.

The operational problem is that these systems also tend to become infrastructure furniture. Nobody thinks about the email gateway when it’s working. Nobody gets promoted because they upgraded the VPN appliance without incident. But everybody notices when an upgrade takes email down for an hour.

That creates a really predictable incentive to leave critical infrastructure alone.

And “don’t touch it because it works” is a strategy that works right up until it doesn’t.

I think that’s the thread connecting most of this week’s stories. Infrastructure isn’t a static collection of things we built. It’s a collection of things moving at different speeds.

GitHub changes the runner image.

Helm moves a major version forward and eventually stops maintaining the previous one.

GitHub changes what CI events and identities are allowed to execute.

Cisco releases a fix because attackers found a way through software sitting at the edge of the environment.

Your application can be completely unchanged through all of that.

That’s why I don’t really trust the phrase “nothing changed” during an incident.

I understand what people mean when they say it. Usually they mean nobody deployed the application. And that’s useful information.

But the next question should be: what else could have changed?

Did the runner change?

Did DNS change?

Did a certificate expire?

Did a secret rotate?

Did a package update?

Did an external API change?

Did a cloud provider change something underneath us?

Did a floating dependency resolve differently?

Did something reach end-of-life?

Did somebody change policy in a completely different system?

Sometimes the fastest way to get unstuck during an incident is to stop asking who changed the application and start asking what the application depends on.

And I think there’s a practical platform-engineering lesson in that too.

Know which parts of your environment are pinned.

Know which parts intentionally float.

Know which parts are controlled by somebody else.

And for the things that are supposed to move, give yourself a way to discover the change before production discovers it for you.

That might mean testing the next GitHub runner image before latest moves. It might mean automated dependency updates. It might mean tracking end-of-life dates. It might just mean having a staging environment that actually experiences upgrades instead of being pinned to exactly the same old versions as production.

The goal isn’t to stop change.

That’s impossible, and trying usually just stores up more change for later.

The goal is to make change visible enough that when something breaks at two in the morning, “nothing changed” isn’t where the investigation stops.

It’s where it starts.

Related Ship It Weekly Episodes

Episode 66Sep 11, 2026 14:47Amazon Linux 2027, GitHub Actions Cache Security, Secret-Scanning Merge Blocks, N-central CVSS 10 RCE, Karmada Graduation, ShieldCrash, CodeQL ARM64 & When Observability Fails TooEpisode: Amazon Linux 2027, GitHub Actions Cache Security, Secret-Scanning Merge Blocks, N-central CVSS 10 RCE, Karmada Graduation, ShieldCrash, CodeQL ARM64 & When Observability Fails Too
Episode 65Sep 4, 2026 17:26AWS GWLB TCP Reset, Azure DevOps Live Migrations to GitHub, GitHub Runner Enforcement, Docker Root Risk, Lambda IAM Updates, PostgreSQL Upgrade Traps, SonicWall Zero-Days & Better Incident ReviewsEpisode: AWS GWLB TCP Reset, Azure DevOps Live Migrations to GitHub, GitHub Runner Enforcement, Docker Root Risk, Lambda IAM Updates, PostgreSQL Upgrade Traps, SonicWall Zero-Days & Better Incident Reviews
Episode 64Aug 28, 2026 16:48Cloudflare Saves 100TB of RAM, AI Drives Server Prices Up, AWS Adds a Fourth London AZ, Route 53 DNS Self-Service, AKS eBPF Routing, Go 1.27, and the Danger of Hidden Infrastructure AssumptionsEpisode: Cloudflare Saves 100TB of RAM, AI Drives Server Prices Up, AWS Adds a Fourth London AZ, Route 53 DNS Self-Service, AKS eBPF Routing, Go 1.27, and the Danger of Hidden Infrastructure Assumptions

The thing that stood out to me this week is how much of infrastructure security is really about making boundaries explicit before somebody finds out the hard way where they actually are.

The GitHub cache change is probably my favorite example.

Most of us think about CI caches as a performance optimization. You cache dependencies or build artifacts so the next job runs faster. Pretty boring.

But if an untrusted pull request can write something into a cache, and a privileged workflow restores that cache later, the cache just became a security boundary.

The attacker does not necessarily need your production credentials. They need control over something that eventually gets trusted by a job that does have those credentials.

That is why I like GitHub adding explicit read, write, write-only, and no-access modes. It is not some huge new security product. It is taking an existing capability and finally letting us describe who should actually be able to do what with it.

And I think there are probably a lot of things in CI/CD that we still treat as plumbing that should really be treated as security controls.

Artifacts are one.

Caches are one.

Runner selection is another.

Reusable workflows definitely are.

If your deployment process consumes something created earlier in the pipeline, you should probably know who was allowed to create it.

The secret-scanning change is similar.

I have always preferred controls that happen as close as possible to the developer doing the work.

Finding a leaked credential in a security dashboard three days later is useful.

Stopping the pull request from merging is much better.

Push protection is better still if you can stop it before the secret ever gets committed.

But none of those controls have to be mutually exclusive.

People bypass things.

Detection rules change.

Secrets get introduced in weird ways.

Having another enforcement point at merge gives you another chance to catch it.

And importantly, it does not depend on somebody remembering to go check another dashboard.

The platform just says no.

Fix this first.

That is what good guardrails should do.

The Amazon Linux story is a little different, but I think there is a similar operational lesson.

Nobody gets particularly excited about testing a new base operating system.

But Amazon Linux 2027 changing things like the kernel, DNF, crypto libraries, CPU baseline, and SELinux defaults is exactly the kind of change that exposes assumptions you forgot were assumptions.

Maybe your monitoring agent expects something that is no longer there.

Maybe some bootstrap script needs filesystem access SELinux now blocks.

Maybe an old binary does not like the new CPU baseline.

Maybe your patching process depends on tooling that does not support the preview yet.

None of these are particularly interesting problems.

They are just really annoying problems when you discover them during a migration.

That is why previews are useful.

You can take your existing image pipeline, swap the base image in a test environment, and see what catches fire.

If nothing does, great.

If something does, you have time to figure out why without anybody waiting for production to come back.

The N-central story is the one I would take most seriously this week.

A CVSS 10 pre-auth RCE under active exploitation is already bad.

Put it in an RMM platform and it gets considerably worse.

These tools exist specifically to control other machines.

They install software.

Run commands.

Manage endpoints.

Usually with pretty significant privileges.

That is the whole reason you bought the thing.

It is also exactly why an attacker wants it.

We sometimes talk about protecting the control plane in Kubernetes or cloud infrastructure like it is a special architectural concept.

But there are control planes all over an organization.

Your identity provider is a control plane.

Your CI/CD platform is one.

Your endpoint-management system is one.

Your virtualization management is one.

Your backup infrastructure can be one.

And your RMM absolutely is one.

If compromising one system gives somebody legitimate mechanisms to control a thousand other systems, that one system should probably not have the same security posture as a random application server.

That sounds obvious when you say it that way.

In practice, those management systems sometimes become the oldest things in the environment because everyone is afraid to touch them.

Which is exactly the opposite of what you want.

Then there is the disk failure from the closer.

I like that story because there is nothing exotic about it.

The disk filled.

That is probably one of the oldest infrastructure failures there is.

The interesting part is that the failure started taking observability with it.

And I have seen versions of that problem plenty of times.

You are investigating why something is broken, but the logs you need are on the machine that is broken.

Or the monitoring agent is starved for the same CPU or memory as the application.

Or the network path you use to collect telemetry is the network path that just failed.

Suddenly the incident is not just, “Why is production down?”

It is, “Why is production down, and why did all my graphs stop updating at exactly the moment I needed them?”

That is a much harder problem.

I do not think the answer is to build some completely separate, bulletproof observability environment that shares nothing with production. You could spend an absurd amount of money trying to eliminate every shared failure domain.

The useful question is simpler.

What failures can blind me?

If this node dies, do I still know it died?

If this disk fills, did the logs already leave the machine?

If this cluster goes away, can something outside the cluster tell me?

If this AWS account has a problem, is every tool I need to diagnose it inside that same account?

You do not need independence everywhere.

You need enough independence that your most important failure modes still leave evidence behind.

The disk story also gets into something I think we occasionally get wrong with alerting.

An alert that fires before the outage is not automatically a good alert.

Timing matters.

If the disk hits 90 percent and you have six hours before it fills, that is useful.

If it hits 99 percent and you have forty-five seconds, technically the monitoring system caught it before the outage.

Congratulations.

Nobody could do anything about it.

The useful metric is runway.

How long do I have before this becomes a problem?

Is that enough time for automation to handle it?

Is it enough time for a human to respond?

If not, move the threshold.

That is probably the connection I see across most of this week’s stories.

A lot of infrastructure work is deciding where you want to find out something is wrong.

I would rather find out an Amazon Linux dependency is broken in a preview environment.

I would rather find out a cache permission is wrong before an untrusted workflow writes to it.

I would rather find a secret while the pull request is still open.

I definitely want to find an N-central vulnerability before somebody is actively using my management plane against me.

And I want to know a disk is filling while I still have enough disk left to do something about it.

None of that is particularly flashy engineering.

It is mostly moving discovery earlier.

But moving discovery earlier is often the difference between fixing something during normal working hours and explaining at three in the morning why all the dashboards went blank at the same time production went down.

The thing that stood out to me most this week is how many infrastructure problems are not really caused by complete failure. They are caused by systems failing slowly, quietly, or in ways that look healthy from the outside.

The Gateway Load Balancer story is probably the clearest example. If an inline firewall or network appliance dies, the backend application might still be perfectly healthy. The problem is the path between the client and that application. Existing TCP connections can sit there for minutes waiting to time out, which means the system is technically broken but nothing has failed clearly enough for the rest of the stack to react.

That is why I like the TCP Reset feature.

Sometimes the best recovery mechanism is just telling the truth faster.

This connection is dead. Stop waiting. Try again somewhere else.

We spend a lot of time designing systems to avoid failures, but failures are inevitable. What matters just as much is how obvious the failure is when it happens. A clean error can trigger retries, failover, alerts, or fallback behavior. A request that hangs for three minutes just consumes time, connections, threads, and patience.

Fast failure sounds harsh, but operationally it can be much kinder.

The Azure DevOps migration story is another version of that same idea, just applied to change instead of failure.

Moving repositories sounds simple until you have actually been involved in a large migration. The Git objects are usually the easy part. Everything surrounding them is where the real work lives.

Branch protections.

Pipelines.

Webhooks.

Service accounts.

Secrets.

Release automation.

Permissions.

Bots.

Scripts with hardcoded repository URLs that somebody wrote six years ago and nobody remembers.

The fact that Microsoft is building live synchronization into the migration process is important because it recognizes that engineering cannot just stop while the platform team moves everything around.

A good migration should feel boring to the people using the system.

That usually means doing more work behind the scenes so everybody else can keep working.

But live migration does not remove risk. It changes where the risk is.

Instead of one giant cutover, you now have a period where two systems exist at the same time. That means you need to understand synchronization, identity, policy conversion, and exactly what happens during the final handoff.

It is still a production migration.

The tooling just gives you a better way to control the blast radius.

The GitHub runner story is probably the most immediately actionable one this week.

Self-hosted runners are really easy to forget about.

They run.

Jobs pass.

Nobody touches them.

Eventually they become part of the furniture.

But those runners are executing code with access to repositories, credentials, internal networks, artifact stores, cloud accounts, and deployment systems.

They are not just build machines.

They are part of your security boundary.

So GitHub enforcing minimum versions makes sense.

And if that enforcement breaks your pipeline, the real problem probably is not GitHub enforcing an update.

The problem is that your runner maintenance process depended on nobody noticing they were old.

That is where ephemeral infrastructure helps.

If runners are created from a maintained image, execute a job, and disappear, then replacing them becomes normal instead of disruptive.

You still have to maintain the image.

You still need patching.

You still need testing.

But the lifecycle becomes something you expect instead of something you postpone.

That is a pattern I think applies to a lot more than CI runners.

Infrastructure that is easy to replace tends to be infrastructure that is easier to maintain.

The Docker story is the one that I think developers should pay the most attention to.

Adding yourself to the Docker group has been normal advice for years.

Nobody wants to type sudo every time they run a container.

The problem is that Docker is not just another CLI.

If your user can control a root-owned Docker daemon, your user can effectively become root.

That has always been true.

What has changed is how crowded developer machines have become.

A modern development workstation might have browser extensions, IDE plugins, package managers, coding agents, local automation, dev containers, AI tools, shell plugins, and who knows what else running in the same user session.

Every additional process expands the amount of software you are trusting.

So when we say, "the developer has access to Docker," what we may really be saying is, "anything running as the developer has access to something that can become root."

Those are not quite the same statement.

And with more autonomous tooling running locally, I think that distinction matters more now than it did a few years ago.

This is not really an argument against Docker.

It is an argument for being accurate about the privilege you are granting.

Convenience is fine.

Just do not call something low privilege when it is not.

Even the smaller stories this week fit into the same theme.

Lambda getting fuller resource-based IAM policies is mostly about giving operators better ways to express access intentionally.

The PostgreSQL upgrade issue is a reminder that configuration relationships that work perfectly for years can suddenly matter during a version boundary.

The CrowdStrike research is a good example of why security reporting needs some patience. A public PoC is interesting, but interesting is not the same thing as confirmed impact across every environment.

And SonicWall is the opposite situation. Active exploitation against Internet-facing remote access infrastructure is exactly the kind of thing where waiting for more discussion is probably not the right move.

Knowing the difference between "watch this" and "patch this now" is part of the job.

The incident review story probably ties everything together best.

I like the idea of async-first incident reviews because the first job after an incident should be figuring out what actually happened.

Not what everybody remembers happening.

Not what the loudest person in the meeting thinks happened.

What the evidence says happened.

Logs.

Metrics.

Deployments.

Alerts.

Code changes.

Timelines.

Async work gives people time to go find those things.

It also lets somebody come back and say, "Actually, that event happened four minutes later," without interrupting somebody halfway through a conference-room explanation.

But async communication has limits too.

You can collect facts asynchronously.

It is harder to resolve disagreement asynchronously.

A twenty-comment thread arguing about whether a database issue caused an API failure or the API failure overloaded the database is probably a sign that three people need to talk for fifteen minutes.

That is why I like the async-first, live-later approach.

Do not schedule twelve people for an hour just to reconstruct a timeline that could have been built in a document.

Build the timeline first.

Gather the evidence.

Identify what is actually disputed.

Then get the right people together and resolve those specific questions.

And most importantly, leave with owners.

Because a beautifully written incident review with no one responsible for the follow-up work is just documentation.

The common thread across all of these stories is that reliable systems tend to make reality explicit.

A dead connection should look dead.

A migration should have a defined cutover.

A runner should have a lifecycle.

A privileged user should be treated as privileged.

A security claim should be separated from a confirmed vulnerability.

And an incident review should separate what the evidence shows from what people assume happened.

A lot of operational pain comes from ambiguity.

The system is sort of working.

The migration is mostly done.

The runner is probably current.

The Docker group is basically harmless.

We think this caused the outage.

Those are comfortable statements right up until they are not.

Good engineering is often just removing enough ambiguity that when something changes, the system and the people operating it can react quickly.