Ship It Weekly Transcripts

AWS is retiring DevOps Guru and pointing customers toward a new generation of AI-assisted operations. A Kubernetes controller vulnerability can let namespace-scoped permissions create a pod somewhere they should not be able to. And Cloudflare says AI agents now account for almost half of the usage of its Wrangler CLI. I’m Brian Teller from Teller's Tech, and this is Ship It Weekly.

Welcome back to Ship It Weekly, the show about the DevOps, SRE, cloud, platform, and security stories that matter when you are the person keeping the thing running at three in the morning. For the weekly story list and source links, check out OnCallBrief.com. For past episodes and show notes, head over to ShipItWeekly.fm. This is episode 69. Nice. Anyway, quick programming note before we get into it.

I’ll be at AWS headquarters in Arlington on Friday for AWS DMV Community Day. I’m speaking and recording some interviews while I’m there, so you’ll see some of those conversations coming to the show soon. This week, AWS is officially putting an end date on DevOps Guru and recommending customers move toward CloudWatch and the newer Amazon DevOps Agent.

Kubernetes disclosed a controller vulnerability that can cross a namespace boundary under some specific permissions. We have a Node.js and Undici WebSocket issue where a malicious server can crash the client process. And Cloudflare has launched a new CLI as AI agents become major consumers of its developer tooling.

Then we have a quick lightning round and a human closer from SRE Weekly about why some categories of production failure may never go away. Let's get into it. First up, AWS is retiring Amazon DevOps Guru. The service will stop accepting new customers on October 29, 2026, and reach end of support on September 30, 2027. Existing customers can continue using it until then, but there will be no new features.

After end of support, the console and APIs go away and previously generated insights will no longer be retrievable. DevOps Guru was AWS’s earlier generation of AI for operations. It used machine learning to analyze operational data, identify abnormal behavior, correlate signals, and help engineers investigate problems. AWS is now telling customers to split those capabilities into two directions.

For monitoring, alerting, and anomaly detection, move toward CloudWatch. For AI-assisted investigation, AWS recommends looking at Amazon DevOps Agent. So if you use DevOps Guru today, this is more than swapping one AWS service for another. You need to figure out which capabilities you are actually using and where each one moves. There is also an infrastructure-as-code problem to deal with.

If you have DevOps Guru resources declared in CloudFormation, CDK, Terraform, or automation built around its API, you cannot leave those definitions sitting around after the service disappears. AWS specifically warns that CloudFormation operations involving withdrawn resource types can fail after end of support. You have a year, so this is not an emergency. But managed services are still dependencies.

You may not maintain the software underneath them, but you still build infrastructure, automation, and operational processes around their APIs. Eventually, some of those APIs disappear. Next, Kubernetes disclosed CVE-2026-2270. This is a vulnerability involving the StatefulSet controller and ControllerRevisions.

Under specific conditions, somebody with namespace-scoped write permissions for StatefulSets and ControllerRevisions can cause the controller to create a pod in another namespace. That sounds particularly bad because namespace-scoped permissions are supposed to stay inside the namespace. There are some important limitations.

The resulting pod will normally be deleted by Kubernetes garbage collection unless the attacker can construct a valid StatefulSet owner reference. To do that, they need the UID of an existing StatefulSet in the target namespace. The vulnerability is rated Medium with a CVSS score of 5.9 and requires substantial existing permissions. So this is not somebody outside the cluster suddenly getting arbitrary pod creation.

Kubernetes describes it as a confused deputy attack. The user does not directly have permission to create the pod in the target namespace. The controller does. The user manipulates an object they are allowed to control, and the controller performs the more privileged operation for them. That is worth considering when reviewing Kubernetes RBAC. Permissions on an object do not always end with that object.

Controllers react to resources and perform additional operations using their own authority. Sometimes, the important question is not just, “What can this identity create?” It is, “What can this identity convince a controller to create?” Third, we have a denial-of-service vulnerability in Undici.

Undici is the HTTP client used by Node.js, and its WebSocket implementation backs Node's built -in globalThis.WebSocket. CVE-2026-85024 involves WebSocket compression. A malicious or compromised WebSocket server can send a specially constructed compressed message that exceeds the decompressed payload limit and contains malformed DEFLATE data.

Inside Undici, cleanup for the size limit removes an error listener from an internal InflateRaw stream. That stream can still generate an error. When it does, Node sees an unhandled error event and terminates the process. Adding an error handler to your application's WebSocket does not fix this.

The failure occurs inside the decompression object, so the public WebSocket error and close handlers never get an opportunity to catch it. The vulnerability is rated Medium with a CVSS score of 5.9. The advisory says that roughly 130 kilobytes sent over the network can expand beyond the internal limit and terminate the process. If the application automatically reconnects, it could potentially end up in a crash loop.

Affected Undici branches have fixes in 6.28.1, 7.29.1, and 8.10.2. The operational point here is pretty simple. Your application, being the client, does not make the server trusted. Clients still parse, decompress, deserialize, and process data coming from the other side. If you use WebSockets from Node, especially against endpoints outside your control, check which version of Undici you are running.

Fourth, Cloudflare has launched a new CLI called `cf`. Normally, another CLI would not make the main story list. The numbers behind this one are why it did. Cloudflare says AI agents accounted for about 25 percent of Wrangler usage in March. Last week, that reached 48 percent. Agents also use almost twice as many distinct commands per day and are nearly four times as likely to use six or more commands.

Wrangler was primarily built for people. It exposes around 280 operations, while Cloudflare has thousands of API operations. Instead of manually adding commands for all of those, Cloudflare built `cf` around its API definitions. Cloudflare also open-sourced Forge, the generation system behind the tooling. We have spent a lot of time talking about agents using infrastructure tools.

Now we are starting to see infrastructure tools change because agents are using them. If almost half of your CLI usage is coming from software, things like consistent commands, structured output, discoverability, and complete API coverage start becoming more important. The CLI is no longer only an interface for the engineer sitting at the terminal.

It is also becoming an interface for software operating on the engineer's behalf. Quick lightning round. First, Kubernetes disclosed another vulnerability affecting Windows nodes. CVE-2026-76654 involves a pod volume mount using `subPath` with a symbolic link pointing to an attacker-controlled network share. The Windows kubelet can follow that link to a UNC path and attempt NTLM authentication to the remote share.

That can expose the NetNTLMv2 hash of the account running the kubelet, which could potentially be cracked or relayed if the node is domain joined. It is rated Medium with a CVSS score of 5.8. Second, GitHub now lets repository administrators configure custom runners for Dependabot version and security updates. You can select the runner type, a custom label, and an optional runner group.

That is useful when Dependabot needs access to private package registries or other resources only reachable from your own environment. And third, GitHub has added external custom properties for repositories. An external system of record like a CMDB or internal developer portal, can now push information like ownership, service tier, lifecycle stage, or compliance status into GitHub.

Those values remain read-only in GitHub while the external system stays the source of truth. Because custom properties can also be used with filtering and rulesets. Service catalog metadata can now feed directly into repository governance. The human closer this week comes from Lorin Hochstein and a piece featured in SRE Weekly called Omnipresent Availability Risks in Cloud Software.

The argument is that there are some categories of production incidents we probably are not going to engineer away. Take saturation. Every resource is finite. CPU, memory, disk, database connections, queue depth, network capacity. Eventually, something can hit a limit. Then there is networking. Cloud software is distributed software, and distributed software depends on networks.

Networking failures can also have enormous blast radius. Security creates another conflict. Availability says legitimate users should be able to access something. Security says illegitimate users should not. Sometimes the security control doing exactly what it was designed to do can cause the outage. An expired certificate is probably the simplest example. Then there are changes you cannot simply stop making.

Sometimes you have to migrate the database. Sometimes you have to replace infrastructure. And sometimes production is already broken and making another change is the only way to recover it. Even the systems we build to improve reliability add complexity. Retries, failover, replication, health checks, circuit breakers. They help us recover from failures, but they also create more states and interactions that can fail.

That changes how I think about one of the questions that comes up after almost every incident. How do we make sure this never happens again? For a specific preventable failure, absolutely. Fix it. Add the guardrails. Improve the alert. Remove the dangerous manual step. But we are not going to eliminate finite resources. We are not going to eliminate networks. We are not going to eliminate security boundaries.

And we are definitely not going to eliminate change. Incident response itself has to be part of reliability engineering. How quickly do we notice the problem? How quickly can we understand the blast radius? Can we get the right people involved? Can we safely make changes while the system is already behaving unpredictably? And can we learn from what happened afterward?

Sometimes improving reliability means preventing the next incident. Sometimes it means getting much better at handling the incidents you cannot prevent. That is it for this week's Ship It Weekly. We covered AWS retiring DevOps Guru and pointing customers toward CloudWatch and DevOps Agent. A Kubernetes vulnerability that can cross namespace boundaries. A WebSocket bug that can crash a Node.js process.

And Cloudflare redesigning its CLI as AI agents become a major part of its usage. Plus, another Kubernetes security issue on Windows nodes, custom Dependabot runners, and GitHub pulling service-catalog metadata into repository governance. Follow or subscribe wherever you are watching or listening.

You can find the weekly story list and source links at OnCallBrief .com and past episodes and show notes at ShipItWeekly .fm. I’m Brian Teller from Teller's Tech. Thanks for listening. And remember, you cannot engineer away every failure, but you can get a lot better at what happens next.

AWS is putting Elastic Beanstalk on top of shared Kubernetes infrastructure. A compromised developer token helped attackers copy roughly 170 private CrowdSec repositories. And Microsoft just disrupted a cybercrime platform using AI to figure out not just what to say, but who inside a company was worth impersonating. I'm Brian Teller from Teller's Tech, and this is Ship It Weekly.

Welcome back to Ship It Weekly, the show about the DevOps, SRE, Cloud, platform and security stories that matter when you are the person keeping the thing running at three in the morning. For the weekly story list and source links, check out OnCallBrief.com. For past episodes and show notes, head over to ShipItWeekly.fm. This week, AWS is rebuilding the Elastic Beanstalk model around shared EKS infrastructure.

We have a software supply-chain compromise that eventually led to CrowdSec losing private source code. There is a critical remote code execution vulnerability in Next.js. And Microsoft has disrupted EvilTokens, a cybercrime platform that compromised more than 12,000 inboxes. Then we have a quick lightning round and a human closer about why fixing the thing that got compromised does not necessarily end the incident.

Let's get into it. First up, AWS has introduced cluster mode for Elastic Beanstalk. And this is an interesting direction for a service that has been around since 2011. Traditional Elastic Beanstalk environments generally give an application its own infrastructure. Cluster mode changes that. Multiple applications can now run on pooled infrastructure powered by EKS.

You can give Beanstalk source code, a Dockerfile, or an image from ECR, and AWS handles the infrastructure underneath it. That includes deploying and scaling the application, patching the environment, and managing the EKS infrastructure. It also includes event-driven autoscaling, OpenTelemetry-based observability, Secrets Manager integration, and HTTPS through ACM.

There is no separate charge for cluster mode, but you are still paying for the EKS cluster, EKS Auto Mode, and the resources your applications consume. I think the interesting part here is the abstraction. For years, a lot of platform engineering has basically been we standardize on Kubernetes. Now, how do we stop every developer from needing to understand Kubernetes? This is another answer to that question.

Developers get something closer to a platform as a service. AWS gets Kubernetes underneath it. But shared infrastructure also means you should understand what is actually shared. Cost efficiency is one side of pooling workloads. Isolation, resource contention, upgrades, and blast radius are the other side. You may not need to operate the cluster anymore.

You should still understand what happens when the cluster has a bad day. Next, CrowdSec published an analysis of how attackers were able to copy roughly 170 of its private GitHub repositories. And the interesting part is how far back the story starts. In May, attackers compromised packages associated with TanStack and used malicious code designed to harvest credentials and tokens.

According to CrowdSec's investigation, a stolen OAuth token was later used to download its private repositories. The repositories were copied in roughly nine minutes. CrowdSec says the token no longer existed by the time it became aware of the leak, and it could not find the token in the audit data it had available during the investigation.

That is a much more useful supply-chain story than simply saying an npm package got compromised. Because the package compromise was only the beginning. The useful thing the attacker got was access. Once credentials leave the machine, patching the original package does not magically pull them back. That is where incident response can get uncomfortable. What credentials existed on that machine? What could they access?

Are they still valid? Were they copied somewhere else? Did the attacker create another way back in? And what evidence do you actually have available to answer those questions? Removing the malware matters. Rotating and tracing the credentials it may have touched matters just as much. Third, there is a critical remote code execution vulnerability affecting Next.js.

CVE-2026-94545 affects versions 16.2.0 through 16.3.5 and has been patched in 16.3.6. The issue is in the Node.js implementation of ImageResponse from next/og. If an application passes attacker-controlled data into SVG content, attributes, or styles during image generation that can lead to remote code execution. The Edge implementation is not affected.

And applications that do not pass attacker -controlled values into that SVG generation path are not affected either. What I like about this story is how harmless the vulnerable functionality sounds. Image response is commonly used to generate things like Open Graph images. You publish a page. Somebody shares the link. Your application dynamically generates the social preview image.

It is easy to mentally put that in the same category as resizing an image or generating a thumbnail. Except this is server-side rendering logic processing input. And if some of that input comes from a URL parameter, database field, user profile, CMS, or somewhere else an attacker can influence, it is no longer just presentation. It is untrusted data entering a server-side rendering path.

That is the part that I would check in addition to just checking the version number. Where are we using ImageResponse? What data gets passed into it? Where did that data originate? And can someone outside the application control any of it? That is also why dependency vulnerabilities can be hard to triage from a scanner alone. The scanner can tell you you have a vulnerable version of Next.js.

It cannot always tell you whether your application actually exposes the vulnerable code path. You still need to understand how the software is being used. If you are running an affected Next.js release and using the Node .js ImageResponse implementation, update to 16.3.6. And if you cannot update immediately, the workaround is straightforward.

Do not pass attacker-controlled values into SVG content, attributes, or styles rendered through that implementation. Fourth, Microsoft says it has disrupted Evil Tokens. EvilTokens was a cybercrime platform that combined phishing, account compromise, mailbox analysis, and AI-assisted fraud.

Since launching in February, Microsoft says it was linked to more than 12,000 compromised inboxes across more than 10,000 organizations. Microsoft seized 50 websites and disabled more than 150 additional domains connected to the operation. What makes EvilTokens different from the usual AI phishing story is what happened after an account was compromised.

The platform could analyze a victim's mailbox and identify trusted relationships, payment conversations, financial responsibilities, and people worth impersonating. So instead of just, "Write me a convincing phishing email," it could help answer, "Who does this person trust? Who approves payments?

Which conversation can I insert myself into?" And, "Who should I pretend to be?" Microsoft says EvilTokens also used device code phishing. And this is worth explaining because it can look a little different from the phishing flow people expect. Device code authentication exists for devices or applications where normal interactive login is inconvenient. The application gives you a code.

You go to a legitimate Microsoft login page on another device. Enter the code, authenticate, and the original application gets authorized. The problem is that an attacker can initiate that flow too. They get the device code and convince the victim to enter it. The victim may be on Microsoft's real login page. They may complete their normal authentication. They may even complete MFA.

But the session being authorized belongs to the attacker. So MFA did its job. It verified the user. The user was just tricked into authorizing the wrong session. Once Evil Tokens had mailbox access, Microsoft says it could analyze the contents and help identify useful relationships and financial conversations. Stolen access could also be maintained with malicious inbox rules that hid communications.

That makes this less about AI magically creating a new kind of attack. Account takeover and business email compromise are not new. What changes is how quickly an attacker can make sense of everything they stole. 10,000 emails used to be a lot of information for a person to work through. For a model, that becomes data to classify. The defensive side still looks pretty familiar.

Control authentication flows, protect tokens, watch for suspicious inbox rules, and treat a valid authenticated session as something that can still be malicious. Quick lightning round. First, F5 has a critical big IP APM vulnerability under active exploitation.

CVE-2026-94127 is a heap-based buffer overflow affecting certain BIG-IP APM configurations where an access policy and OAuth profile are configured on the same virtual server. Successful exploitation can result in unauthenticated remote code execution. If that configuration applies to you, this is a patch now item.

Second, GitHub Enterprise Cloud can now export an inventory of every credential capable of accessing the enterprise. That includes SSH keys, classic and fine-grained PATs, OAuth tokens, and GitHub App credentials. The inventory includes things like ownership, scopes, expiration, last use, and which organizations or repositories the credential can access.

There is also a REST API, which makes this much more useful for automated credential hygiene. Third, Zyxel's GS1900 series has a vulnerability that can allow an unauthenticated attacker on the LAN to execute operating system commands using a crafted HTTP request. Patches are available for supported models.

And fourth, Veeam Agent for Microsoft Windows has a privilege escalation vulnerability that can let a low-privileged local user gain SYSTEM access. Public proof-of-concept code is available. So if you are running affected version 13 builds, get them updated. The human closer this week comes back to the CrowdSec story.

One of the easiest mistakes during incident response is assuming the incident ends when you fix the thing that was originally compromised. A malicious package gets removed. A vulnerable server gets patched. A laptop gets reimaged. Problem solved. Except the attacker may have already gotten what they needed. A token. An SSH key. A session. Cloud credentials. A package registry credential.

Something sitting in a config file. Something sitting in memory. The original compromise can last five minutes. The credential stolen during those five minutes might work for months. That is why containment has to extend beyond the thing that got compromised. If a developer workstation gets hit, I want to know what that workstation could authenticate to.

GitHub, AWS, Kubernetes, package registries, CI systems, internal services. And then I want to know which of those credentials still exist. This gets harder when your logs do not go back far enough. The token has already disappeared, or you do not have a good inventory of credentials in the first place, which is actually why that GitHub credential inventory item in the lightning round matters.

During an incident, "What credentials can access this environment?" Is a question you want to answer immediately, not after three hours of spreadsheets and Slack messages. Fixing the original vulnerability stops one path in. It does not tell you what walked out before you closed the door. That's it for this week's Ship It Weekly.

We covered Elastic Beanstalk running applications on shared EKS infrastructure, the CrowdSec supply-chain compromise, a critical Next.js remote code execution vulnerability, and Microsoft's disruption of EvilTokens. Plus an actively exploited F5 vulnerability, GitHub credential inventory, Zyxel switch patches, and a Veeam privilege-escalation issue. Follow or subscribe wherever you are watching or listening.

You can find the weekly story list and source links at OnCallBrief.com and past episodes and show notes at ShipItWeekly.fm. I'm Brian Teller from Teller's Tech. Thanks for listening. And remember, fixing how they got in does not tell you what they took on the way out.

GitHub is adding new controls around who can actually execute your Actions workflows. Helm 3 has reached its final minor release and is officially on the road to end of life. And if your builds use ubuntu-latest, the operating system underneath them is about to change whether you touch the workflow or not. I'm Brian Teller from Teller's Tech, and this is Ship It Weekly.

Welcome back to Ship It Weekly, the show about the DevOps, SRE, cloud, platform, and security stories that matter when you are the person keeping the thing running at three in the morning. For the weekly story list and source links, check out OnCallBrief.com. For past episodes and show notes, head over to ShipItWeekly.fm. This week, GitHub Actions gets new workflow execution protections.

Cisco has a critical Secure Email Gateway vulnerability under active exploitation. Helm 3 has reached its final minor release. And GitHub's Ubuntu latest runner is moving from Ubuntu 24.04 to 26.04. Then we have a quick lightning round and a human closer about one of my favorite phrases during an incident. But nothing changed. Let's get into it.

First up, GitHub Actions workflow execution protections are now generally available. Last week, we talked about GitHub tightening cache permissions. This week, they are moving one step earlier in the process. Who should be allowed to start the workflow in the first place? Execution protections let organizations create rules around both the actor and the event triggering a workflow.

Normal CI might be available to everybody, while a manually triggered deployment is limited to maintainers. Dependabot might be allowed to trigger one workflow, but not another. And your production deployment workflow can have completely different rules from your normal test workflow. The protections can target individual workflow files too.

So deploy.yml can have stricter rules without applying those restrictions to everything else. There is also an evaluate mode. That means that you can see what the policy would block before actually enforcing it. For platform teams managing hundreds of repositories, that is important. You can find legitimate exceptions before turning on a security control and breaking a bunch of pipelines.

There is also a REST API, so this can be managed programmatically. And then there is pull_request_target. We talked about that last week because GitHub's new cache permissions specifically restrict cache access for lower-trust events like pull_request_target. GitHub is going further now.

Starting November 2nd, affected public repositories without their own applicable event policy will have pull request target disabled by default. If you legitimately need it, you can explicitly allow it. So the important change here is not another explanation of pull_request_target. It is that GitHub is continuing to make CI trust boundaries explicit. Last week, what can the workflow access?

This week should the workflow execute at all. Your CI system may have cloud credentials, registry access, signing keys, or deployment permissions. Being allowed to contribute code should not automatically mean being allowed to execute every piece of automation attached to it. Next, Cisco has released security updates for Secure Email Gateway and Secure Email and Web Manager.

There are multiple vulnerabilities involved, including several critical issues. The one getting the most attention is CVE-2026-76461. It is a SQL injection vulnerability in the email parsing logic of Cisco secure email gateway. An unauthenticated attacker can exploit it by sending a specially crafted email through an affected device. Successful exploitation can lead to arbitrary commands running as root.

And Cisco says the vulnerability is already being actively exploited. There are no workarounds. You need to upgrade to a fixed release. What makes this interesting is where the vulnerability lives. Email security gateways exist specifically to process untrusted content coming from the internet. Attachments, headers, URLs, message bodies, all of that parsing creates attack surface.

The same is true for firewalls, VPN concentrators, proxies, and other security appliances. The thing protecting the boundary is also sitting directly on the boundary. And these systems have a habit of becoming infrastructure furniture. They are critical, they work, and nobody wants to touch them. So upgrades get delayed.

Eventually, the security appliance protecting everything else becomes one of the oldest systems in the environment. If you run Cisco Secure Email Gateway, check your version and get onto a fixed release. Security software is still software. And sometimes, the thing inspecting the attack becomes the attack path. Third, Helm 3 has reached its final minor release.

From here, Helm 3 is basically in security maintenance mode. Helm 4 has been available since November, and Helm 3 continues receiving security fixes through February 10, 2027. After that, support ends. That does not mean that everybody needs to migrate this weekend. But if Helm is deeply embedded in your platform, the clock has officially started.

The good news is that Helm 4 maintains compatibility with most Helm 3 charts and existing releases. But there are still differences worth testing. Some CLI flags changed. Plugin behavior changed. Registry authentication should be checked. And Helm 4 uses server-side apply by default for new releases. The individual changes are not particularly scary.

The question is how many places in your environment quietly assume Helm 3 behavior? CI images, Argo CD, developer machines, deployment scripts, custom plugins, anything using the Helm SDK. This is exactly the kind of migration that I would rather start while nobody cares about the deadline. Put Helm 4 into a non-production pipeline. Run your existing charts. Find the weird assumptions now.

February is plenty of time if you start in September. It becomes considerably less time if you start on February 9th. Fourth, Ubuntu 26.04 is now generally available for GitHub Actions runners on both x64 and ARM64. But the bigger change is coming next month. GitHub is moving Ubuntu latest from Ubuntu 24.04 to Ubuntu 26.04 between October 19th and November 19th.

So if your workflow says runs-on: ubuntu-latest, you have effectively told GitHub, you choose my operating system. And GitHub is about to choose a different one. Ubuntu 26.04 has newer tools, different versions, and some software that is no longer preinstalled. If your pipeline explicitly installs everything it needs, you are probably in decent shape.

If it works because some binary happens to exist on the runner image, you might discover a dependency you did not know you had. GitHub recommends testing against ubuntu-26.04 now. That is easy enough. Change the runner label in a branch and see what happens. If you are not ready, pin it to Ubuntu 24 .04. Neither approach is inherently wrong. Pinning gives you stability, but upgrades become your responsibility.

Using latest gives you automatic upgrades, but compatibility becomes your responsibility. The problem is using a moving target while assuming it never moves. Because your repository can remain completely unchanged while the environment executing it changes underneath you. Quick lightning round. First, GitHub Actions retention changes October 1st.

Checks, workflow runs, and statuses will start following your existing Actions retention settings, which default to 90 days. If you rely on older CI history for auditing or debugging, check that setting before GitHub starts cleaning things up. Second, GitHub Advanced Security configurations can now be enforced at the enterprise level.

That means that organizations and repository administrators cannot override centrally enforced security settings. Third, GitHub has added an API showing when self-hosted Actions runner versions lose registration and runtime support. That is particularly timely because full minimum version enforcement for GitHub Enterprise Cloud begins September 25th.

If you run a large self -hosted fleet, runner lifecycle is something you need to actively manage. And fourth, GitHub's AI Scan for pull requests no longer requires CodeQL default setup. It is still in public preview for GitHub Advanced Security customers, but it expands where GitHub's AI-powered vulnerability detection can be used.

The human closer this week starts with a sentence almost every infrastructure engineer has heard during an incident. But nothing changed. The application worked yesterday. Nobody merged anything. Nobody deployed anything. The Terraform plan is empty. So why is the build broken? Because “nothing changed” usually means nothing changed in the place you were looking. Maybe the runner image changed.

Maybe a package updated, maybe a container tag moved, maybe an action changed, maybe a certificate expired, maybe DNS changed, maybe an external API changed, or maybe somebody changed the environment that your completely unchanged code runs inside. That is what makes the ubuntu-latest story interesting.

You can make zero changes to your repository and still have a different operating system execute your workflow next month. Latest is a dependency. It is just an intentionally moving dependency. The same applies elsewhere. From Ubuntu colon latest is a dependency. A loose Terraform provider constraint is a dependency. Downloading the newest version of a tool during every CI run is a dependency.

An external API is a dependency. Even time can effectively be a dependency. Certificates expire. Tokens expire. Support windows expire. Helm 3 can be completely unchanged and still go from supported software to unsupported software next February. The answer is not to pin absolutely everything forever because then nothing changes for three years until you eventually have to absorb three years of changes at once.

The better approach is deciding intentionally what is allowed to move. If you want your CI runner to float forward, great. Test it. If a production dependency needs to be deterministic, pin it. If you want dependencies continuously updated, automate those updates in controlled increments. What gets dangerous is accidental mutability. Things that can change when nobody on the team realizes they can change.

So the next time that someone says, but nothing changed, they might be completely right about your application. Start looking at everything around it. Something changed. You just have not found it yet. That's it for this week of Ship It Weekly.

We covered GitHub Actions Workflow execution protections, the actively exploited Cisco Secure Email Gateway vulnerability, Helm 3 entering its final support window, and Ubuntu latest preparing to move to Ubuntu 26.04. Plus, GitHub Actions retention changes, stronger enterprise security enforcement, self -hosted runner lifecycle management, and broader access to AI-powered code scanning.

Follow or subscribe wherever you are watching or listening. You can find the weekly story list and source links at OnCallBrief .com and past episodes and show notes at ShipItWeekly .fm. I'm Brian Teller from Teller's Tech. Thanks for listening. And remember, if nothing changed, you probably just haven't found the thing that changed yet.

Amazon Linux 2027 is now in public preview with SELinux enforcing by default DNF5 and a newer kernel. GitHub Actions now lets you explicitly control cache permissions to reduce cache-poisoning risk. And N-able N-central has a CVSS 10 pre- auth remote code execution flaw that is already being exploited. I'm Brian Teller from Teller's Tech, and this is Ship It Weekly.

Welcome back to Ship It Weekly, the show about the DevOps, SRE, cloud, platform, and security stories that matter when you are the person keeping the thing running at three in the morning. For the weekly story list and source links, check out OnCallBrief.com. For past episodes and show notes, head over to ShipItWeekly.fm.

This week, Amazon Linux 2027 enters public preview with some changes that are going to matter for compatibility. GitHub tightens Actions cache permissions. GitHub can now block pull requests that introduce exposed secrets. And N-able N-central has a critical pre-auth RCE under active exploitation.

Then we have a quick lightning round and a human closer about a disk filling up at 2:47 in the morning and taking observability down with it. Let's get into it. First up, Amazon Linux 2027 is now in public preview. This is more than a routine version refresh. You get a newer kernel, DNF5, newer language runtimes, updated Graviton support, AWS-LC, and an x86-64-v3 baseline. SELinux is also enforcing by default.

That one is probably the first thing I would test. If you have custom agents, bootstrap scripts, monitoring software, or anything that expects broad filesystem access, this is where assumptions can start surfacing. And that is really the point of the preview. Operating system upgrades are rarely hard because of the OS itself. They are hard because of everything that quietly depended on the old one.

Custom AMIs, package installs, systemd units, cloud- init, security agents, image pipelines. AWS is explicitly telling teams to test their automation now. Terraform, CloudFormation, CI/CD, custom AMIs. There are also preview limitations. SSM Patch Manager does not currently support Amazon Linux 2027 preview. That is not a reason to avoid testing it. It is exactly why you test it.

The x86-64-v3 baseline is also worth paying attention to if you have older hardware assumptions or odd build dependencies. The OS is part of your platform whether developers think about it or not. So I would treat this preview as a compatibility test opportunity. Not a migration event yet. Next, GitHub Actions now lets you explicitly control cache permissions.

And this is a better security change than it probably looks like at first glance. Build caches feel like a performance feature. Dependencies. Compiled artifacts, package metadata, but once caches cross workflow boundaries, they become part of the trust model. If a low- trust workflow can write into a cache that a higher-trust workflow later restores, you have a cache-poisoning path.

An untrusted pull request may not have production secrets or deployment access, but if it can poison a shared cache, that a privileged workflow later trusts, it has still crossed the boundary indirectly. GitHub's new cache-mode controls make that explicit. Jobs can have read access, write access, write-only access, or no cache access. Low-trust events like pull_request_target default to read-only.

Reusable workflows also cannot gain more cache permission than the caller. That is the right model. If a job only needs to consume a cache, make it read-only. If it does not need caching, give it none. CI platforms have identities, credentials, artifacts, network access, and privilege boundaries. Caches belong in that same conversation.

GitHub can now block pull requests from merging when secret scanning finds exposed credentials. This one is pretty straightforward, but useful. GitHub already has push protection, which is still the best place to stop a secret if you can catch it there. But workflows are messy. Maybe the secret was already in history. Maybe a custom detection rule was added later. Maybe somebody bypassed push protection.

The new ruleset integration gives you another checkpoint. If secret scanning finds an exposed credential in the pull request, the PR can be blocked until that alert is resolved. That can include generic secrets and custom patterns depending on configuration. What I like is where the control lives, right in the pull request. The engineer is already there. The reviewer is already there.

The status checks are already there. Security becomes one more merge condition. That is much stronger than telling engineers to remember to check a separate dashboard. Fourth, N-able N-central has a critical pre-authorization remote code execution vulnerability with a CVSS score of 10. And it is already being exploited.

N-central is remote monitoring and management software used by MSPs and IT teams to manage large numbers of endpoints. So what the product does matters as much as the vulnerability itself. If you compromise a normal server, you get that server. If you compromise a management system, that can execute commands across hundreds or thousands of machines? The blast radius is very different.

The vulnerability does not require authentication. It allows remote code execution. CISA has added it to its Known Exploited Vulnerabilities catalog. And exploitation has already been observed. N-able has released updated hotfixes with Hotfix 4, build 2026.3.1.14, superseding earlier fixes. If you run N-central, patch it. The broader point is simple.

RMM platforms, CI systems, identity providers, endpoint management, and backup infrastructure are all control planes. Anything that can control other systems deserves stronger isolation and patch discipline than a normal workload. Because if an attacker gets the control plane, they do not need to compromise endpoints one at a time. Quick lightning round. First, Karmada has graduated from the CNCF.

The graduation itself is mostly a maturity signal. The more interesting part is where multi-cluster orchestration is going, especially with distributed AI and GPU workloads. Second, there is new research called ShieldCrash involving Microsoft Defender. The claim is SYSTEM- level access on fully patched Windows machines and possible bypasses of earlier mitigations.

This one is still developing, so I would watch it, not panic over it yet. Third, CodeQL now supports Linux ARM64. Not huge by itself, but useful if ARM runners are becoming part of your CI fleet. And fourth, Dependabot can now access private GitHub Packages without needing a separate personal access token when the repository already has the right Actions access. Less long-lived credential plumbing is usually a win.

The human closer this week starts with a disk filling up at 2:47 in the morning. That is already bad. The bigger problem was that the failure also started taking out the systems that could explain what was happening. Logs stopped writing. Monitoring data started disappearing. Queries stopped working. Dashboards became less useful. Now the team had the original failure plus a visibility problem.

That is where incidents get ugly. Observability has dependencies too. Logs need storage. Metrics need CPU and memory. Collectors need network access. Dashboards have backends. If all of that shares the same failure domain as the application, one failure can blind you. The obvious version is local logs on the same disk that is filling up. But it can happen at other layers too. Monitoring on the same node.

Alerting in the same cluster. Incident tooling in the same cloud account. Status checks running from inside the same network. You do not need your entire observability stack completely isolated, but you should know what can blind you. And for important failures, you want at least one signal that survives.

Remote log shipping, external synthetic checks, independent monitoring, out-of-band alerts, or simply disk alerts with enough lead time to matter. That last one is important. A disk hitting 100% is not when the incident started. The incident started while the disk was filling. If your alert fires at 99% and the disk grows 1% every 30 seconds, that is not really an alert. It is a notification.

Good capacity alerting is about runway. How long until this becomes an outage? Can automation fix it? Can a human react in time? If the answer is no, the threshold is wrong. And that connects pretty cleanly to the rest of the episode. Amazon Linux 2027 is about testing compatibility before the platform changes underneath you. GitHub cache permissions are about defining trust before data crosses workflow boundaries.

Secret scanning rulesets are about enforcing policy before a bad change gets merged. And N-central is another reminder that the systems controlling everything else need to be treated like control planes. All of these are easier problems before the incident. That is usually the boring part of infrastructure work. Testing, reviewing permissions, adding merge rules, separating monitoring from the thing it watches.

None of it looks dramatic until it is the reason the incident lasts 10 minutes instead of three hours. That's it for this week's Ship It Weekly. We covered Amazon Linux 2027, GitHub Actions cache permissions, pull request secret scanning enforcement, and the actively exploited N-able N-central vulnerability.

Plus, Karmada graduating from the CNCF, ShieldCrash, CodeQL on Linux ARM64, and simpler Dependabot access to private GitHub Packages. Follow or subscribe wherever you are watching or listening. You can find the weekly story list and source links at OnCallBrief.com and past episodes and show notes at ShipItWeekly.fm. I'm Brian Teller from Teller's Tech. Thanks for listening.

And remember, monitoring does not help much if it dies with the thing it was supposed to monitor.

AWS can now reset failed Gateway Load Balancer connections instead of letting them hang for minutes. GitHub is getting ready to stop running jobs on stale self-hosted Actions runners. And a Linux desktop setup accidentally gave basically every user-space process a path to root through Docker. This week is mostly about failure modes hiding inside things we already trust.

I'm Brian Teller from Teller's Tech, and this is Ship It Weekly. Welcome back to Ship It Weekly, the show about the DevOps, SRE, cloud, platform, and security stories that matter when you are the person keeping the thing running at three in the morning. For the weekly story list and source links, check out OnCallBrief.com. For past episodes and show notes, head over to ShipItWeekly.fm.

This week, AWS Gateway Load Balancer gets TCP Reset to shorten recovery when a firewall appliance dies. Azure DevOps gets live migration tooling for moving enterprises to GitHub. GitHub is starting enforcement against stale self-hosted actions runners. And Omarchy shipped with a Docker configuration that effectively made normal desktop processes root-capable.

Then we have a quick lightning round and a human closer about whether incident reviews should happen asynchronously first, live later. Let's get into it. First up, AWS Gateway Load Balancer now supports TCP Reset. This is one of those small-looking features that solves a pretty ugly failure mode.

Gateway Load Balancer is commonly used to put firewalls, intrusion detection systems, and other network appliances inline with application traffic. When one of those appliances fail, new connections can usually move to a healthy target. Existing TCP connections are harder. They may keep trying to use the failed path until the connection times out.

Depending on the application and TCP settings, that can mean 30 seconds. It can also mean several minutes. From the application's point of view, the network is not obviously broken. It is just hanging. With TCP Reset enabled, Gateway Load Balancer can actively terminate those affected connections. The client gets a reset immediately and can reconnect through a healthy appliance.

That turns an ambiguous failure into an explicit one. And I think that is the useful lesson here. Fast failure is often better than slow uncertainty. A failed request gives the application something that it can react to. A connection sitting there doing nothing is much harder. Retries do not happen yet. Timeout budgets get consumed. Users wait. Threads stay blocked.

And everything upstream looks degraded instead of clearly failed. We talk a lot about graceful failover, but sometimes graceful means being very direct. This path is dead. Start over somewhere healthy. This is especially important around stateful middleboxes because the backend application may be perfectly fine while the network path in front of it is not.

The shorter you can make that ambiguity window, the faster the rest of the system can recover. Next, Microsoft has put Enterprise Live Migrations from Azure DevOps to GitHub into public preview. This is aimed at large migrations where shutting down engineering for a weekend is not really an option.

The idea is that repositories remain writable in Azure DevOps while changes continuously synchronize into GitHub Enterprise Cloud. Teams keep working during most of the migration. Then, when it is time for the final cutover, you stop writes, sync the remaining changes, and switch over. Microsoft says the final downtime can typically stay under about 30 minutes.

The migration tooling can also convert Azure DevOps branch policies into GitHub rulesets. And importantly, Azure pipelines can continue running while repositories move. That matters because repository migration is rarely just copying Git objects. The hard part is everything attached to the repository. Permissions, branch protections, CI/CD.

Webhooks, secrets, bots, service connections, release workflows, and all of the little assumptions around URLs and identities that accumulate over 10 years. A migration can be technically successful and still be operationally terrible if engineers lose half a day every time you move another batch of repositories. Live synchronization changes that equation.

It lets the migration behave more like a controlled transition instead of a hard stop. But I would still treat this like any other production migration. Inventory what depends on Azure DevOps before you move anything. Test policy conversions. Validate pipelines. Know what happens to automation that references old URLs. And have a rollback path for the cutover itself. The goal is not just moving the code.

The goal is moving the development system around the code without everybody discovering the missing pieces on Monday morning. Third, GitHub is getting serious about stale self-hosted Actions runners. This one is worth checking if you manage your own runner fleet. GitHub is rolling out minimum version enforcement. Self -hosted runners need to be at least version 2.329.0 to register.

And runners also need to stay within 30 days of current releases to continue executing jobs. GitHub is starting runtime brownouts on September 9th. Full enforcement is planned for September 25th. The obvious action is to update your runners. The more interesting question is why they are stale in the first place. A lot of teams treat self-hosted runners like infrastructure appliances. Build the image.

Deploy the runner. Forget about it until something breaks. But the runner is executing arbitrary CI workloads with access to repositories, credentials, networks, artifact stores, and often cloud permissions. It is part of your security boundary. It is also a client of GitHub's service. And that client has to evolve with the platform.

If your upgrade process requires somebody remembering to rebuild an AMI every few months, enforcement like this is eventually going to catch you. This is where ephemeral runners are really attractive. Start from a maintained image. Run the job. Destroy the environment. Regularly rebuild the base image and roll forward automatically. That does not eliminate maintenance.

It makes maintenance part of the lifecycle instead of an exception to it. And if you cannot automatically replace a runner fleet today, this brownout is probably a good excuse to figure out why. Because September 25th is a much nicer time to discover that problem than during a production deployment. Fourth, Omarchy fixed a configuration that effectively gave user-space applications a path to root through Docker.

The default user was part of the Docker group. That is common on Linux systems because it lets you run Docker commands without sudo. It is also basically root access. The Docker daemon runs as root. If your user can talk to the Docker socket, you can generally start a privileged container, mount the host filesystem, and modify the machine.

So anything running as that user inherits a very powerful capability, a browser extension, an IDE plugin, a compromised development tool, a coding agent, malware running in the session. If it can reach the Docker socket, the jump from normal user to root may be trivial. Omarchy fixed this in version 4.0.1. But the bigger point is not really about Omarchy.

The Docker group has been a known privilege boundary for a long time. What has changed is how much software we now run inside developer sessions. IDE extensions, package managers, AI coding tools, local agents, browser automation, dev containers, CLI plugins. A modern workstation is a pretty busy environment.

Giving every one of those processes indirect root access because Docker is more convenient without sudo is worth thinking about. Convenience is fine. Just name the privilege correctly. If membership in a group means the user can become root, treat that group like root access. Because an attacker absolutely will. Quick lightning round. First, AWS Lambda now supports fuller IAM resource-based policies.

You can define multiple principals and actions and use normal IAM condition keys for cross-account access. That should make larger Lambda authorization models cleaner than some of the older one permission at a time workflows. Second, AWS is warning about PostgreSQL upgrades that can fail because of circular role memberships.

RDS and Aurora upgrades from PostgreSQL 14 or earlier to 15 and newer can hit problems if roles like rds_ superuser and the built-in read-all or write- all roles end up referencing each other. AWS has a pre-upgrade query to find the problem. Not exciting. Very useful before your maintenance window. Third, a researcher released a proof of concept called FalconFlank, claiming a CrowdStrike privilege escalation path.

Right now, I would treat this one carefully. There is public research and a PoC, but no CVE, affected version matrix, or CrowdStrike advisory yet. Interesting enough to watch though. Not enough evidence to panic. And fourth, SonicWall SMA1000 appliances have zero-day vulnerabilities being actively exploited. If those appliances are in your environment, this is firmly in the patch-now category.

Internet- facing remote access infrastructure is not where I want to wait and see how exploitation develops. The human closer this week is about incident reviews, specifically whether distributed teams should start them asynchronously instead of immediately scheduling another meeting. One SRE team wrote about reconstructing a 37-minute outage almost entirely async.

People added logs, timelines, screenshots, metrics, code references, and corrections directly into the review document. That part worked really well. It gave people time to check evidence instead of relying on memory in a meeting. Someone could make a claim, and another engineer could verify it.

The timeline became much more precise, and people in different time zones could contribute without everybody needing to be online at once. But there was a downside. Disagreements became long comment threads. Questions about causality got buried. And action-item ownership became fuzzy. That sounds pretty familiar. Async communication is really good at collecting information.

It is not always good at resolving ambiguity. A document can tell you what happened. A comment thread can show that three people disagree about why it happened. At some point, putting those three people in the same conversation is probably faster. The model they landed on was basically async first, live later. Use the document to gather evidence and build the timeline.

Then hold a short live review only for the things that actually need discussion. Disputed causes, tradeoffs, decisions, and ownership. I like that a lot because the goal of a postmortem is not having a postmortem meeting. The goal is understanding the incident well enough that the organization learns something from it.

Sometimes meetings help with that, but sometimes they just force 12 people to watch one person scroll through Datadog. Async evidence gathering also changes who gets heard. In a live review, the people who remember the incident fastest or speak the most confidently can shape the narrative early. In a shared document, somebody can go back, check the logs and say, actually, that happened four minutes later.

Or this alert fired before the deployment. Or the database was already degraded before the API started failing. That is useful. But the async part still needs structure. Someone owns the timeline. Someone identifies unresolved questions. Someone turns the discussion into actions. And someone actually owns those actions afterward. Otherwise, you just traded a bad meeting for a very detailed document nobody closes.

The thing I like most here is that it treats communication style as part of incident response design. Distributed teams do not have to recreate the same meeting- heavy process companies used when everybody sat in one office. Use async for the things async does well. Use live conversation for the things humans resolve better together. And keep both focused on learning, not blame.

Because the best incident review is not the one with the most complete document. It is the one that actually changes how the next incident goes. That's it for this week's Ship It Weekly.

We covered Gateway Load Balancer TCP Reset, Azure DevOps Live Migrations to GitHub, GitHub's self-hosted Actions runner enforcement, and the Omarchy Docker privilege issue, plus Lambda IAM policy improvements, Postgres SQL upgrade traps, the FalconFlank research, and SonicWall zero-days. Follow or subscribe wherever you are watching or listening. You can find the weekly story list and source links at OnCallBrief.com.

And past episodes and show notes at ShipItWeekly.fm. I'm Brian Teller from Teller's Tech. Thanks for listening. And remember, recovery gets easier when systems fail clearly, migrations happen deliberately, and the humans reviewing incidents have enough space to figure out what actually happened.