Ship It Weekly Transcripts

Cloudflare shaved about 100 terabytes of memory from the cache behind 1.1.1.1. OVHcloud says AI demand has helped push memory costs up roughly six-fold. And AWS added a fourth Availability zone in London, exposing infrastructure that quietly assumed there would always be three. This week is mostly about assumptions that work fine until scale or reality changes underneath them.

I'm Brian Teller from Teller's Tech, and this is Ship It Weekly. Welcome back to Ship It Weekly, the show about the DevOps, SRE, cloud, platform, and security stories that matter when you're the person that keeps the thing running at three in the morning. For the weekly story list and source links, check out OnCallBrief.com. For past episodes and show notes, head over to ShipItWeekly.fm.

This week, Cloudflare shows what saving a few bytes means. When you have more than 250 billion DNS cache entries. OVHcloud is raising infrastructure prices as AI changes the memory supply chain. AWS added a fourth Availability Zone to London. And Route 53 Global Resolver is getting a better model for cross-account DNS self-service.

Then we have four quick lightning round stories and a human closer about something that almost every Kubernetes engineer eventually worries about. Running the right command against the wrong cluster. Let's get into it. First up, Cloudflare published a really good engineering deep dive into the cache behind 1.1.1.1. Cloudflare's DNS services hold more than 250 billion cached DNS entries at any given time.

At that scale, one wasted byte per entry is more than 250 gigabytes of memory across the fleet. So Cloudflare went looking for bytes, not more servers, not a new caching layer, bytes. They made five changes to how cache entries are represented and stored in memory. Together, those changes cut the memory footprint of each entry by more than half.

Across the fleet, Cloudflare says that that freed roughly 100 terabytes of RAM. And performance improved too. Insert throughput went up 43% and lookup latency dropped 19%. That is what makes this one interesting. We tend to think of optimization as a trade. Spend more CPU to save memory. Add complexity to reduce latency. But better memory layout and fewer allocations can make the whole system faster and smaller.

It also shows how different optimization looks at scale. If your application has 10,000 objects in memory, shaving eight bytes from each one probably does not matter. If you have 250 billion of them, suddenly the exact layout of a Rust struct matters a lot. Premature optimization is still real. But once you find something that exists billions of times, tiny inefficiencies stop being tiny.

Sometimes the big infrastructure optimization is not a new architecture. It is one field in a struct. Next, OVHcloud is raising prices on some infrastructure starting in September. And the reason is memory. OVHcloud says RAM costs roughly six times more in June than it did a year earlier. Suppliers are shifting manufacturing capacity.

Toward high-bandwidth memory for AI workloads which is affecting the supply and price of more traditional RAM some newer servers are reportedly going up around 40 to 59 percent certain gaming servers are increasing even more this is a good example of the second-order effects of the ai infrastructure boom you do not need to run an LLM You do not need GPUs. You do not even need an AI project.

You can still end up paying for the demand AI creates further down the supply chain. For FinOps teams, this changes some assumptions. Right-sizing still matters. Commit discounts still matter. Reducing idle capacity still matters. But none of that completely offsets the underlying hardware becoming dramatically more expensive. It can also change whether an optimization project is worth doing.

Maybe nobody cared about squeezing another 20% out of a memory-heavy database cluster last year. A 50% infrastructure price increase can change that calculation quickly. Cloud abstracts the hardware. It does not make hardware economics disappear. Eventually, those economics show up in the invoice. Third, AWS added a fourth Availability Zone to the London region. The new zone is eu-west-2d.

More capacity and more fault isolation are obviously good things. But some customer automation immediately exposed an assumption. London had three availability zones. Maybe Terraform discovers all availability zones and creates a subnet in each one. But the module only has three CIDR blocks. Maybe there are three NAT gateways, three route tables, three entries in a configuration array.

The code was dynamic enough to discover the fourth zone. The architecture was not dynamic enough to support it. That is a subtle difference. Dynamic discovery does not automatically mean dynamic compatibility. If a module supports exactly three zones, validate that and fail clearly. If it is supposed to support any number of zones, make sure every dependent resource actually scales with that number.

Do not let a cloud provider adding capacity accidentally change your production architecture. Cloud APIs are dynamic inputs. Regions change. Services expand. Instance types appear. Availability Zones get added. Sometimes the cloud provider changing underneath you is not the outage. It is just your assumptions finally getting tested.

Fourth, AWS published a useful pattern for shared DNS views with Route 53 Global Resolver. The cross-account capability itself is not brand new. The operating model is what I like. In a typical multi-account AWS environment, a central networking team owns DNS resolution. That makes sense. You probably do not want every application team independently changing the organization-wide resolver.

But application teams still own their services. So when they create a private hosted zone, somebody has to make that zone visible through central DNS. Historically, that often becomes a ticket. App team creates the zone. Networking team gets the request. Networking team associates it. App team waits. That works with five teams. It becomes a bottleneck with 50.

Shared DNS views let the networking account keep control of the resolver, while application accounts associate their own private hosted zones through AWS Resource Access Manager. The platform team keeps the guardrails. The application team gets the self-service action it actually needs. That is the balance I like in platform engineering. Centralize the decisions that need governance.

Delegate the actions team can safely own. You are not removing control. You are removing a human scheduling dependency from a routine deployment path. Quick lightning round. First, some strange hardware security research. A project called skitter-creek-bath- salts demonstrates manipulation of memory-controller translation registers in ways that can undermine security assumptions higher in the CPU stack.

This mostly affects older AMD families and requires significant privilege in practice, so this is not a panic-and-patch story, but it is a useful reminder that higher-level security boundaries only protect what they can actually see. Second, eBPF Host Routing is now generally available in Advanced Container Networking Services for AKS.

Routing and masquerading can move away from the traditional iptables path and into eBPF, which can improve latency, throughput, and CPU usage. But host-level iptables rules are bypassed. So understand what your workloads and node tooling depend on before you flip the switch. Third, CloudFront Functions has a nice observability improvement. Functions can now add custom context directly into CloudFront access logs.

That means less work correlating separate Function logs with the request that triggered them. Small feature, much nicer debugging. And fourth, Go 1.27 is out. The headline language feature is generic methods. There are also improvements around type inference, JSON, UUID support, runtime profiling, and tooling.

If Go is a big part of your environment, this is worth actually reading the release notes instead of just bumping the version in CI. The human closer this week is about a mistake that is painfully easy to make. Running kubectl against the wrong cluster. The scenario is simple. You have multiple terminal tabs open. In one tab, you switch Kubernetes context.

Later, in another tab, you run a command believing you are still pointed at the environment you were using before. The command works perfectly. That is the problem. Kubectl context lives in the kubeconfig. It is not tied to your terminal tab. So if two shells use the same kubeconfig and one changes the current context, the other silently sees that change too. No warning. No authentication failure.

Your command just succeeds against a different cluster. Those failures are dangerous because the system behaves exactly as designed while violating the operator's mental model. If kubectl crashed, you would stop. If authentication failed, you would investigate. But if a delete command succeeds against production instead of staging, everything technically worked. There are some simple protections.

Show the current cluster and namespace in your shell prompt. Pass the context explicitly for destructive actions. Use separate kubeconfigs or session-scoped contexts. And add deliberate friction around production. I like that last one. We spend a lot of time removing friction from engineering workflows. Usually, that is good. But not all friction is waste.

Typing the production cluster name before a destructive command costs five seconds. That is cheap compared with restoring a namespace because muscle memory moved faster than your attention. And the strongest protection still belongs upstream of the laptop. RBAC. Admission policy. Scoped credentials. Guardrails that limit how much one accidental command can actually destroy.

This connects back to the AWS Availability Zone story. London has three AZs. This terminal is pointed at staging. This array will always have three entries. This kubeconfig belongs to this shell. Those assumptions can be true for years. Then one day, they are not. Good infrastructure engineering is not eliminating assumptions. That is impossible.

It is knowing which assumptions can hurt you when they stop being true, then either enforcing them or designing around them. If the module only supports three zones, validate three zones. If a destructive command must only run against staging, make staging explicit. If production deletion is dangerous, put a control somewhere stronger than someone remembering which terminal they used twenty minutes ago.

A lot of incidents happen when reality changes before our mental model does. The boring fix is making reality harder to misunderstand. That's it for this week of Ship It Weekly. We covered Cloudflare saving about 100 terabytes of memory in its DNS cache. OVHcloud's AI-driven infrastructure price increases.

AWS adding a fourth London availability zone, and cross-account DNS views with Route 53 Global Resolver, plus hardware memory-isolation research, AKS eBPF Host Routing, CloudFront Functions unified logging, and Go 1.27. Follow or subscribe wherever you are watching or listening. You can find the weekly story list and source links at OnCallBrief.com and past episodes and show notes at ShipItWeekly.fm.

I'm Brian Teller from Teller's Tech. Thanks for listening. And remember, the most dangerous assumption is usually the one that has been true for so long that nobody remembers it is an assumption.

Platform engineering usually starts with a good idea. Make things easier. Standardize the boring parts. Give developers a path that works. But eventually, every platform hits the same question. What are we actually willing to say no to? Because if every workload belongs on Kubernetes, every tool has to be centralized. And every use case has to fit the same golden path.

The platform that was supposed to accelerate teams can become the thing slowing them down. I'm Brian Teller from Teller's Tech, and this is Ship It Weekly. Welcome back to Ship It Weekly, where I filter the noise and focus on what matters when you are the one running infrastructure and owning reliability. Most weeks, it's a quick DevOps, SRE, platform, cloud, and security news recap.

In between those, I do conversation episodes with people building and operating the systems we all depend on. Today, I'm joined by Justin Garrison. Field CTO at Sidero Labs, and co-host of Fork Around and Find Out. We talk about Kubernetes moving beyond the cloud, why some companies are looking back toward bare metal and on-prem infrastructure, and where cost, sovereignty, and complexity are pushing that shift.

We also get into golden paths, why platform teams need to know what they are willing to say no to. What actually deserves to be centralized, and why Kubernetes is not automatically the right answer for every workload.

And near the end, we talk about AI, security, tooling fatigue, on-call burnout, and why chasing every new technology matters less than finding the parts of engineering you actually enjoy getting good at. All right, let's jump in. Today I'm joined by Justin Garrison. He's Field CTO at Sidero Labs, co-host of Fork Around and Find Out. And we're going to talk about Field CTO.

We're going to talk about infrastructure and whatever else comes top of mind. Justin, thank you for joining me. Yeah, thanks for having me, Brian. So explain your new title, Field CTO. Excited to hear. Yeah, it's a externally facing position at some companies. It's usually more of a tech company thing. And I'd say oftentimes it's, you know, smaller tech companies. I don't, I see it sometimes at larger ones.

But sometimes they're like segmented by region or something like that. But just like a CTO might say, this is how engineers internally should work and practices we have and resources we have. The Field CTO is kind of like an external version of that, where you're kind of helping people architect around your solutions and your software. Sometimes you're making content and you're thought leading in some cases.

But for the most part, you're just you're trying to help. People be as successful as possible with whatever it is that they're trying to use with your software. And so that's kind of a, it's a, it's a bridge between kind of sales and engineering. Sometimes there is, I do quite a bit of engineering. A lot of times it's like POCs and custom engineering for like a specific problem.

And then we see how we can fold that back into the products. But then I go and I just learn stuff from a lot of really smart people doing really hard problems and it's real fun. Awesome. Yeah, it sounds fun. I guess, what does Sidero Labs do and How are you, like, what are these POCs that you're building? Yeah, yeah. Sidero is the Greek word for iron.

And if you are familiar with infrastructure in the last 10 years, you will know that Greek words usually mean something Kubernetes and iron usually means bare metal. And so that's kind of where we fit in a lot of this is we try to focus on bare metal Kubernetes. We build an operating system called Talos Linux, which came out of a lot of frustrations with managing Kubernetes and Linux at scale.

So it stripped down as much as possible. But really, we focus on what would it take to run Kubernetes anywhere. And specifically, we like to focus on kind of the bare metal edge use cases. And where do you see, because you also worked, I guess, at Amazon on EKS. How have you seen the evolution of Kubernetes over like the last five years?

And I guess more specifically, the last year or two with everything that's happened with AI development, kind of causing everything to iterate quickly.

Yeah, I mean, Kubernetes, in many ways became productized and essentially kind of like proprietary software right and anytime you go look at what does EKS actually do what does GKE do what do these hosted services do they are all proprietary software you cannot run GKE in your data center. It is their software that you will never be able to use, right? And so it's like, oh, it's based on Kubernetes.

So it's all compatible and I can move it if I want to and portability. And it's like, eh, not really, right? Like you really have to hide all of the benefits you get out of using GKE or EKS in order to be able to do that. So that was kind of like just Kubernetes was going that route because people want the easy path. They want decisions to be made for them. They don't really want what happened with like... OpenStack.

OpenStack was every decision you had to make. It was just like, oh, well, you got to pick which version of storage do you want? Which version of networking do you want? Which version? And then it just became really hard to manage because you were piecing everything together. So proprietary stacks are great. They fit together and they're usable in certain situations. Obviously, it's still growing.

It's not slowing down. People are more and more every day. They're still going to all of those services. They're still running more Kubernetes than ever before. And then the flip side of that is people wanted to do it more places. And I'd say three or four years ago, pretty much every major vendor had some form of, you can run this on-prem, whatever they called it. I worked on EKS Anywhere.

It was kind of the last project I worked on at Amazon. And it was a, it was supposed to be a flavor of EKS that you ran in your data center. And it was nothing like what the actual backend of EKS look like. It was like, yeah, we run Kubernetes. It's like, well, what's it based on? It's like, well, we have this like fork of Kubernetes called EKS distro.

And then like, we just run that like, well, how it's like, well, it completely different, right? It's all Cluster API based. And the backend of EKS is like cube ADM and a bunch of Lambda functions. Like that's literally like, oh, you can't run that on-prem and you don't want to because. EKS does it so that they can run hundreds of thousands of them. You need 10, right?

Like you don't need the same architecture to do it that way. So same thing with like GKE had Anthos, I forget what they call it now. Azure had one, everyone had this sort of like flavor of, we want you to be able to run it yourself. And none of them were actually the same under the hood. It was always just like a Kubernetes management layer. That's most of the, almost all of them were based on Cluster API.

But that was kind of where it was heading from all, most accounts like that has failed. And almost all of them, they've all pivoted at least twice now. They've all re -architected, they've added or removed features. In many cases, they've removed headcount. So those solutions are no longer managed, as maintained as they were, right? Like, especially like Broadcom had a whole Tanzu thing and that was really cool.

And they had all these things like, and then they just fired the whole team. And they're just like, ah, we'll just keep the customers happy. And then we don't need to keep investing in that because. All of them were still venues for get people to our cloud. And we wanted to get someone to run this on-prem so that eventually they would come to us when they wanted to run it in a cloud. Makes sense.

I would say that's one of the biggest differences and one of the reasons I enjoyed Talos and Sidera was the fact that it wasn't trying to move you anywhere.

All the software was like actually you should just stay on-prem like you should just stay where you want to with the software and it does not matter we're not trying to sell you to we have a a sas version of our like management plane but like you can self -host the whole thing like we don't want you to have to move unless you want to unless you say like actually this is easier let me go ahead but like we don't have

The extra motives that a lot of those cloud providers have yeah we were talking I guess before we started as well that there's a somewhat of an insurgence again in on-prem, people moving away from cloud or moving to multi -cloud or moving cloud providers.

What do you think is the reasoning for that? There's a lot of reasons. I'd say that the two biggest ones are costs and where people are mature in their cloud environments and they're like, actually, this is really expensive.

Renting these servers it just it costs a lot of money and I I wrote the best practices that amazon for EKS the best practice for cost optimization and it basically always came down to the same thing like well you should right size your workloads and you should use reserved instances and you should try to you know like just use spot wherever you could and in those cases you know you could reduce your bill by maybe up

To like 50 60 and that's great because people like this is amazing and then some people are mature enough to get there And they've reduced their bill by 60 % and they're still like, actually, this is still more expensive than it should be when we go on -prem.

And they also have a lot of overhead for that. Autoscaling takes a lot of engineering time. It takes a lot of testing and validation. When we used to autoscale back in the day, we made a budget. We had a budget once a year. That was FinOps. We have teams of FinOps people that you're hiring to try to reduce your spend. And originally that was called a budget. And people didn't have to do this.

You didn't have to pay for extra tooling, extra people to watch it. It was just like, no, we spent the budget. And if it fit in the budget, then great. That's what we wanted. And we didn't need to spend any time on auto scaling. And all of this stuff was handled at the beginning of the year. Was it flexible? No. Was it amazing? No. But you got to ignore a lot of stuff that you can't ignore anymore. And so I see some companies going to...

On-prem or just like a simpler environment because like actually I'm auto scaling by 20 every day and it's still not saving me very much money and it's just actually causing me more pain because I have to move data whenever I auto scale and I have to like drain nodes and I have to you know all this stuff just becomes fatigue over time and so people are just like I just want like a simpler environment and then the

Other side is is honestly like political in the countries and and laws affect real people uh I'd say a lot of our customers are in Europe they're European countries or or not the united states right or just anywhere that like says like actually we're not going to use a big cloud provider from the united states because hey we we don't trust u .s companies today anymore or or we we can't because we like the size of

Just land in the world right there's different politics in different areas of land and even if if your neighboring countries are close to you and you have different rules, then you have to control the data and the services and how you use them.

And so a lot of countries that have been doing that for a very long time, like it's not something we see in the United States because we have two neighbors. And in most cases, we're fairly friendly with them. And they're also pretty far away from a lot of people in the United States. Whenever I go over to places in Europe, I'm like, wow, I've been through three countries on one train ride. And that's amazing, right? I don't get that in California.

And so just the idea of like where data lives and how it trend you know moves across wires that stuff matters a lot more and so having more control over it is important to folks and they don't want to give up the kubernetes the the nice flexibility of the system and just being able to like use modern tooling uh but they will say I'm not gonna go to amazon like amazon has like a whole like like it's a different company, right?

Like to be able to sell in the EU, like they had to spin up like these other companies. And they're like, oh, we're not Amazon, we're Amazon EU. And it's like, well, no, you're still Amazon. And so like they try to hide a lot of that stuff and say like, well, politically we're part of your country. And it's like, yeah, I get it, but I still don't trust you, right? There's still a lot of that going on.

Yeah, that's fair. So what parts of Kubernetes are essential versus what parts are self -inflicted pain? I mean, it's all a little bit self -inflicted. Anything new is going to be self -inflicted. Like you want to try when we were moving to Ansible, right? Or like config management, right? Like, did you need config management? Like, maybe, maybe not.

But in general, like throughout my career, I've seen that my own cognitive ability to like understand things breaks down around like.

Probably like 30 of the thing like it's a really low number like once I had like 30 servers it was hard for me to like go beyond that and like know what's going on right and say like oh I actually understand what these 30 servers once I'm at like 50 or 60 I'm like you know like I just need to group these into some logical grouping so that I can manage a group of them a little easier and I'd say okay well now I have

10 web servers and I have two database servers and then once I get to 30 groups then I'm like well I need to like abstract this again and in kubernetes does that really well because you can just layer on these abstractions it does get difficult to know how far down the abstractions you want to go when you're looking at processes on a server and then you say okay well I probably have thousands of these like what do I

Actually like how do I actually manage this how do I understand it but at the end of the day like people's cognitive ability is is kind of difficult to like scale beyond some limited number and so we've always been abstracting it we've always been trying to kind of how do we how do we get more I don't say value out of a person but how we let one a single person manage more things and I remember my first sysadmin job

I had a hundred linux servers and that was amazing right like that was like one person managing 100 servers and again these weren't physical services were vms because I had to logically group them into things that I could maintain and redeploy and whatnot and so but it was like that was kind of the pinnacle back in the early 2000s of like where they always thought like a you know sysadmin 100 servers per sysadmin is like the golden sort of ratio of like what you can get out of them.

And now we're looking at, okay, well now a sysadmin or platform engineer, whatever we want to call it today, doing the same sort of thing, you can probably manage, you know, maybe 10 clusters, maybe 20 Kubernetes clusters. And those will probably be in like three or four different forms of what types of workloads run on them. You're gonna have your web tier, your AI tier, your state full server tier, right?

You're gonna have these clusters that are dedicated to those things, but you're still just kind of extracting the same thing. And so in that regard, Kubernetes is great at doing that abstraction for the workloads, but it still breaks down once you say, I need to, you know, I have different versions of it, or I need, you know, I need 20 different kinds of Kubernetes clusters. Okay.

You probably need to hire someone else. Yeah. I've, I know during, in my day job, we also have the issue of everybody wanting certain teams, wanting to containerize certain applications that maybe.

Don't make sense to be containerized like long -running complicated tasks that could cause noisy neighbor issues that sort of thing that just don't make sense for kubernetes workload but it's what they're what they feel comfortable with so you know they pipe it once once you learn a tool you want to use it for everything right like people are still abusing ansible because it's just ssh in a for loop right and you're

Like actually I can just use this everywhere for everything and people were using it for application deployments was ansible ever meant for that no but It's the tool they're familiar with so they can do that.

I remember when I first was learning Go and I had some coworkers that knew a little bit of Go. And they're like, okay, well, I take my Go binary and I put it in a container. I was like, why would you do that? What is the point of putting a single, a statically compiled Go binary in a container when I can just SSH it to a server and run it?

And it took me a little while to realize that actually the benefit of these containers wasn't, it was a packaging mechanism, right? I don't care, even if it's a single binary, statically, it doesn't matter.

Just the ability to be able to, you know, push it to a registry, to replicate it a thousand times, to do all these things and to isolate it so I could run it and not have to worry about like lock files or port contention, all that stuff ends up being a really good idea, even if the thing doesn't require it.

And so like that sort of common sense around or like common knowledge on this is how this thing works and how we can use it a bunch of times ends up being pretty good for most use cases. So given that, what's your take on like golden paths? Are they useful? Like if I want to set up workloads, is there a golden path to follow? I mean, every company I've been at has their own golden path, right?

Paths end up following policies and procedures at companies. And those usually reflect some form of org chart. And so it's like, oh, depending on how your org chart looks. Depends on what your golden path is going to be. And that looks very different at a small company and a large company. And even two medium -sized companies aren't going to have the same golden path.

So it's one of the reasons I think platform engineering is so popular right now is because everyone has to have their own path. Because you can't just say, here's the path, go follow it. There are some generic tools that, hey, maybe these will help you fill in some of these gaps. But in general, you kind of have to just know.

This is what our policy is around this piece of software or when we ship things or how things get to production or the testing frameworks. And in many cases, those things also really rely on like the talent you have, right?

The people you have in charge and not just the engineering teams, but also the leadership teams being able to understand how complex something is or when you should actually send something out, right? And so it's like, if you have a really you know, forward thinking and mature engineering team.

But you have an executive who's like, no, we never ship on Friday because I've been burned too many times for shipping on Friday. Right. And that's just a thing that happens that you say, actually, our policy is we can't ship on Friday. Why? I don't know, because the exec said so. Right. It's like that. Is that part of your golden path? Sure.

Like you have a rule somewhere that there's like a, you know, you do not ship. This is Friday. You need emergency, you know, authorization from the CTO or something. Right. And like that's part of that path. And people.

Embed these things into rules and that becomes their you know how all the developers do work and those always end up being like like platforms become accelerators until they they become like balls and chains right like then they're like decelerators at some point we're just like oh we're able to move fast because everything looks the same and we can do this but then once you try to add everything into that platform you're like oh now we have too many variables and we have too many thing.

So the cognitive load of this thing is more than if we just use off the shelf parts, because all of our internal stuff isn't documented. That's how he's seen it every time. Like we've been building platforms over and over again at multiple companies and using them. And when I was at Amazon, like I, the internal tooling for Amazon, I'm sure at some point was ahead of its time.

And it took me days to get it set up to be able to do one commit into an internal repo. And I'm like, this this is garbage. Like, I wish this was just GitHub. And I understand that, like, not all those things scale to the size of something like Amazon, but also every one of those tools was proprietary and not well documented.

And I was just like, hey, this would be great if I was in an office and I sat down next to someone and they walked me through it.

But as a new remote employee trying to just understand where to even find the documentation right like that stuff all falls apart and so like those golden paths again they just they they fall apart once you get down this like too many platforms or too many options on how the platform works and any sort of like even templating right like you're like oh I'm just gonna start with like a templated file and we're just

Gonna like env you know with bash like we're just gonna like expand out the variables and at some point you look at it and you're like have logic in your templates and you're like I can't follow this anymore right and that that's around the point where you say like actually maybe we should just like fork the platform and say like this is only for this type of workload and this other one is for this type of workload

And if you can split them you'll have two platforms but they'll be simpler than a single one now that makes sense so okay we have our kubernetes cluster set up we have our golden path What, in your opinion, is the next step?

What's day two look like? What is the next most important thing when setting up an application in a cluster, having that cluster set up live, especially now with AI? Does that change? From whose perspective? Is this platform engineer perspective? Yeah. From a platform engineer, I often tell them the most important thing you can figure out is what you're going to say no to.

Figure out the things that you're like, this cluster is not for that. When I was at Disney, Disney Plus had a platform internally. And the thing we said no to was stateful workloads. And it honestly was the best decision we had ever made. Because anytime we wanted to move the cluster, it was really easy for us to do. CVEs, anything else, we're like, hey, if you need stateful stuff, you go somewhere else.

I'm sorry, this platform is not for you. And and like if you can make those decisions up front and you can say like, OK, we have what we have, you know, five monitoring systems and we know how we're going to upgrade and we know how to control like all that stuff's like just getting it set up for day one. You deploy it, it can go to production.

And at some point you're going to say, OK, what what is it that I have to say no to? And if you can figure that out for day two, you'll be in a much better place for day three, because at some point you are going to have to upgrade that cluster multiple times and you're going to have to at some again talk to.

Teams that you don't know who's running this workload and they don't care about it anymore they're off to something new right and that applies to any workload in the in the era of everyone wants to be an AI company or at least everyone wants to run AI workloads that becomes harder because AI training is very stateful and needs a lot of data and you need to figure out all of your storage your networking performance

Tuning And as soon as you get all that, you're like, actually, this is going to run for a week at a time to do some sort of like fine tuning or training or something like that.

And like now I need to figure out how to snapshot it. Right. And I was like, OK. And these are these aren't new problems. These are problems that HPC environments have had for a very long time. And so a lot of what Kubernetes was built for, built for stateless web applications originally, is now running head head into this like job management and long running jobs that it just it.

Isn't well suited for at like the lower levels and and that's why people still use a lot of Slurm and they still use a lot of custom tooling for hpc and like strictly hpc environments and in kubernetes it's getting there at some point but I don't know I don't see that It's not a perfect fit.

You're going to have to layer on some other abstraction to get the features you want because Kubernetes at the API level is like you can plug in whatever you want and then go build it. And they're kind of done putting in like core functionality into the API. So given Kubernetes adoption, given saying no to things, what's a practice outside of that you wish more teams would have adopted before they needed it?

As far as like the structure of their, from a platform engineering perspective, as far as structuring for their team. It's something that I don't think the platform teams can't give themselves, but I wish that the organization would give the platform team, the agency to have like some financial power to say, to control what's going on. Right.

So if I look at like SRE, SRE, when it came out, the Google book and all this stuff, they're like, Oh, this is, this is how you do SRE. This is, this is what S and everyone's like, Oh, well it's like engineering and production and uptime and all this stuff.

It's all true, but also the thing I took out from that book was the thing that SRE at Google did that no other SRE team ever did that I saw was the SRE teams had the ability to say no. The SRE team could say, your application sucks. It has too much technical debt. I'm not going to maintain it. And they could walk away.

And then the team was on their own to like bring it up to a level of, yeah, we have to be able to fix this now. And that was only because the SRE organization had that ability, right? The funding for SRE was taken out of engineering budget from the application team. So if you wanted an SRE on your team, you had to pay for it. And then the SRE team could say, no, that's not worth our time. Right.

We have other places that we should be spending our time. Most platforms are not funded that way. Most platform teams don't get their funding from the engineering team of the application. Right. They're getting their funding from infrastructure budgets or they're getting their funding from maybe security teams. And those are the places that are saying, hey, I'm going to pay for this team.

This is a 10 person team for building our platform. Why are you building the platform? Because we need a platform. Tell me what you're trying to do. We're trying to accelerate developers so they can have less cognitive load about, developers shouldn't learn Kubernetes, we should just subtract. I'm like, well, what are you actually trying to do here? And they're like, well, we want to centralize things.

Okay, what is the benefit of you centralizing this golden path? And from my experience, the main things that you want to centralize are security, like being able to audit your systems and say like, hey, is this going to be a problem for us? If there's a CVE, you need to be able to patch it quickly. If there's a leak or a hack or something, you need to be able to audit your systems and understand what's going on.

So things like logging, monitoring, things like just like SBOMs and like security scanning, that's something you need to centralize. And if you're in a cloud environment, you need to centralize your costs, right? Because like, that's just where are we spending more money than we should be? That's like the basics of a platform is usually those.

Two things to me if you don't need to build up the golden path for all this engineering stuff if you just give them hey everyone now gets automatic data dog logging here you go and everyone automatically is in these AWS accounts and we're going to take care of the how the costs bubble up into our orgs so we can see that at central place and everyone's going to get you have to have this s -bomb in place in order to

Ship to production right that's the basics of like a platform team and I wish more platform teams or more organizations would understand you don't want to centralize everything.

There's actually not a lot of value in putting everything in the same centralized tool, because again, that's just going to slow you down at some point. Most teams are going to say, we got to centralize CI/CD, because back in the day, Jenkins was the way to go, and it was really hard to set up Jenkins servers, and so you had to set up one.

And that one became so critical to how everything worked in your company that no one could upgrade it. And so then you had this really old Jenkins server until the next team said, you know what? I'm going to send up my own Jenkins server because the other one sucks. And then you had this like battle of Jenkins servers in most companies. And at some point, everyone's like, I like the new Jenkins server.

It has the new UI or it does the new, you know, whatever, right? Like it's we want to move to the new CI/CD. It's like, well, like. Yeah, it's a cost to maintain those sorts of things. And it is valuable to provide an optional, hey, you can use this template that we have that allows you to do something faster. But mandating centralization through some sort of platform team ends up being a hindrance.

And most organizations that I talk to and platform teams that I talk to, they don't understand that. They just say, we just have to centralize everything. We need everything to run on Kubernetes. I'm like, no, you don't. Trust me, you don't want everything.

And also like, this this keep this notion of like you need kubernetes clusters to do anything is that was like before AI right now we need AI to do anything like before anyone's gonna write any code like well if claude's down I can't write any code like what are you doing like do you do you remember how this works like you you can't is it worth your time I don't know you have to decide that right like how slow are

You going to be without it should you take a break for an hour do you need to go refill your tokens whatever right but like Back in the day, like we keep doing the same thing just with a different technology, right?

Because even like 10, 15 years ago, you put a credit card on a cloud provider and that accelerated what you could do because it took too long to get a server in the data center. And then later you're like, okay, well now I'm going to, you know, my credit card is going to go to something else, some other service. So I don't have to run this thing anymore because I just don't have the time to.

And now credit cards are on cloud. And that's how like the amount of people I know that are paying like the $200, you know, thing, just so they're like, I just need to stay like up to not up to date. And even it's just like, I need to be. As productive as the next person, I'm like, no, like, what are you doing? Like, let's just understand where the problem actually lies. And yeah, we're going to keep doing this.

There's going to be a new technology that's going to come out in five years. People are going to put their credit card on it and they're going to say, actually, I just, I have to have this to work. And I always sort of like reevaluate that. Like, what are you actually trying to get done? And when does it actually need to be done by? And then once we have that conversation, we can say, okay.

Well, then let's plan this, right? It's just like a budget again. If I know my workload is going to scale to maximum of this in the next 12 months, I can plan for that. And people are so against planning today that they're just like, I just need to spend money to solve the problems. So do you think that AI kind of causes people to do less critical thinking on their own?

And I mean, the Dora report last year says that we see this AI productivity gain. Across the industry, but we're not able to quantify it correctly or quantify it accurately in the data analytics that we get out of our actual number of commits or number of features that are released or whatever.

So do you think that there is this inherent symbiotic relationship that people have formed now with their AI as far as engineers unable to be engineers without their AI tooling? I think engineers have formed too close of... Relationships with a lot of their tools. It's not an AI problem.

Vim and Emacs, this is a problem that people have religious debates about for a very long time about pretty much any tooling because they feel attached and they identify with those things. And AI is another version of that.

I think the fact that AI is changing so frequently that people are having a hard time even even figuring out what their identity is because people that were co -pilot advocates last year right if you were out there today saying like co -pilot's the way to go everyone's gonna laugh at you like no way man why are you you gotta be on clod like what are you doing right and this was this was literally eight months ago

That people were just like this was the way to go and in microsoft just like everything's gonna be co -pilot and now it's like they kind of lost the open AI you know deals and now they're just like well now we're just gonna run whatever AI llm we think is the right one and and so like Everyone's changing their mind about this stuff.

I do think that the kind of the I guess the status quo of how people are working today where I want to review code more than write code is more common in some of these tools. I don't think that's and I do think AI has accelerated that but I don't think it's like completely new. Right. Like I always think of Ruby on Rails was amazing. The first time I ran it, where I was just like, oh yeah, make me a new website.

And like, it would template out like 20 folders and all this stuff. And it's like, you just put in these eight variables and we're ready to go. I'm like, that's amazing. Right. But like, and I didn't review the code. I didn't look at it. I just said, okay, well, yeah, Ruby on Rails, ship it. Like, you know what you're doing. And we're, we're kind of, it's, it's that, but it's.

A lot more flexible right like we're just we're doing the same sort of stuff and even like go back a long time right compilers and compilers they just take your you know take your high level cobalt and and make it machine code and that's amazing right like you can actually do that without needing to review it and at some point people move more and more to like how do this architect together where where are the

Important things for me to have more input versus just being able to know like oh I know how these pieces should fit together someone else can actually like make the glue or the templates for me So do you think that with the advent of what we have, Claude, Mythos, and Project Glasswing, do you think that that changes the prioritization now more towards cybersecurity on four platform teams and DevOps teams alike?

Or do you think that it's status quo and we just work from the outside in, make sure there's a WAF, lock everything down, and make sure our packages are up to date? And that's enough. I love how optimistic you are that people are actually using the WAFs and everything. I mean, because I look at what people are shipping today.

None of that is in place right this is just like oh yeah we're just gonna we're just gonna burn through this as fast as we can and again and that's I mean to some of the larger you know companies that have more to lose they have the policies they have the practices they're gonna say like actually no you can't ship that to production without these things and engineers and people are pretty smart they're gonna try to

Work around it and figure out how can I get my thing done to be successful my way everyone always tries to work around some of that stuff but at the end of the day like there is going to be some controls on okay Where is this actually going?

What does it have access to? I think all those attacks are going to be more sophisticated, but I don't see it being measured yet. Like, S -bombs are measured, right? Security is always reactionary in that, like, oh, there's no CVEs in my architecture. I'm like, yeah, there are. It's like, no one knows about them yet, right? And I think that that is going to be, until we can measure it, we can't manage it.

And so we have to flip this to be a little more proactive. And WAFs are one way of doing that. We're just going to have a firewall, we're going to say no DDoS, whatever. We're going to put some things in place, but once your application's online, it's fair game for anyone to start poking at it. And that is still, I see, who was it just last week? Calendly? They closed source their software.

Actually, we don't want people to see our code. We now need to close source it because being open source is a vector for... An AI bot to read our code and say, oh, I see exactly where this is. At the end of the day, it's probably going to be some subscription and it's going to be tied to some level of insurance, right?

There's cybersecurity insurance and some of those cybersecurity insurances are going to say, hey, you have to run this mythos on your code base once a month or something like that. There's going to be some requirements there just to say, we did what we could. But at the end of the day, you still have to... If you don't have a way for customers to access your stuff, it's going to be really hard to sell something.

No, that's for sure. Okay. So closing out, what's one piece of advice for platform or SRE people trying to stay sane this year with everything that's happening, all of the new features, all of the new models, how do we stay sane? I would say find the things that you're interested in, right? Like don't just keep throwing YAML at the wall, right? Like, like you actually like, Figure out, oh, you know what?

I really like networking. I think storage is cool. I want, you know, I want to understand some level of something like find the things that you're passionate about and that you think, oh, that's interesting. Whether it's a career path or not, right? Like whether it's like, you don't have to plan out. I think this is going to be big, so I'm going to go into it, right? Let's just like figure out for you.

Like when I first started Kubernetes, it was not the winner, right? Like this was, this was back in like Mezos was the thing that everyone ran. And when I was like, oh.

This kubernetes thing is kind of cool and why is it different than what Mesos offers or what Slurm offers or and and just knowing that like it doesn't always work out I have tons of projects that were just throw away that like I learned a skill deep enough that I don't need it anymore but it doesn't matter that it didn't pay out for me like I I had fun I learned the thing and in having joy in what you do and is more important than like having some form of like financial success, right?

Like the happiness, the joy, the being able to like still be yourself is a lot harder. And platform teams have a lot of breadth of things that they're going to look at. SRE teams, you're going to burn out no matter what. Like there's, oh, if you are on call, I don't care where you're on call for, that level of pressure is always going to cause some level of burnout.

So talk to your manager early about, hey, how can I make sure that like this is sustainable at a team level? Like, hey, we know we're going to lose one person a year to this. How do we handle on-call rotations when we're short people? Those are all things that you should be thinking about on making this a little more enjoyable for yourself and not just burning all your tokens and then going home and just crashing.

At some point, you're just like, what do you have fun doing? And what do you look forward to every week? If there's something at work that you look forward to, you should do more of that. Right. You should you should say, like, actually, you know, I really like this thing. When can I do that twice a week now? Can I do it? Maybe, you know, like like dedicate a week, a month or something.

That's the important stuff, because that's the stuff that will give you the energy just to be a better.

Person in general like you are more enjoyable to be around when you like what you're doing in a lot of cases and you like doing it with the same you know right people and that's not always a choice everyone can make it's not always a freedom that everyone has but when you do have those things try to do more of it and try to focus on it I 100 agree with that justin where can people find your work I'm on social media

Quite a bit I'm hanging out a lot on blue sky um justin garrison.com is my website I still blog there I've been blogging for over 20 years and over that the entire 20 years I've I've averaged one blog post a month awesome which is amazing right like I look back at it when I migrated from a bunch of different places into you know into my site and I was like wow like I have a lot of articles like that's pretty crazy

And I just added them up and I'm like actually that's that's amazing like I don't blog all the time but I when I do I go into like I'm gonna do three or four blog posts and I'll put them all out there.

And so I'm on average, I'm about one a month. And then I also, in the podcast Workaround Findouts, we're doing that once a month as well. So I kind of have those two things that I just try to keep up. And those are where I'm interested in things and when I want to talk to people and where I find joy. Awesome. Well, I'll put links for all of that in the show notes. And Justin, thank you so much for coming on.

Really appreciate it. Yeah. Thank you. That was my conversation with Justin Garrison, Field CTO at Sidero Labs. And co-host of Fork Around and Find Out. The biggest takeaway for me is that good platforms are defined as much by what they refuse to do as what they support. It is tempting to centralize everything. One Kubernetes platform. One CI/CD system. One golden path. One way every team is supposed to work.

And that can absolutely make things easier at first. But eventually... The exceptions pile up. The platform absorbs more responsibility. And abstraction that was supposed to reduce cognitive load becomes its own source of cognitive load. I liked Justin's example from Disney +, where the platform simply said no to stateful workloads. That constraint made everything else easier to operate, upgrade, move, and recover.

That is a useful way to think about platform engineering in general. Do not start with how do we support everything? Start with what problem are we actually solving? What should be centralized? And what should explicitly live somewhere else? The same idea applies to Kubernetes and AI. Once we learn a useful tool, we tend to want to use it everywhere. Kubernetes becomes the answer to every workload.

AI becomes a requirement before somebody writes a line of code. Sometimes that is useful. Sometimes we are just replacing one dependency with another. So my takeaway is to keep questioning the abstraction. Know why you are using Kubernetes. Know why you are centralizing something. Know what your golden path is optimizing for. And know when the right answer is simply this platform is not for that.

You can find Justin's work, his writing, and fork around and find out through the links in the show notes. Follow or subscribe to Ship It Weekly wherever you listen and find previous episodes at shipitweekly.fm. Thanks for listening, and I'll see you later this week.

GitHub had another widespread outage this week. Agentic browsers have a vulnerability that can turn a webpage into an execution path. And AWS is finally pushing Certificate Manager users away from email validation. This week is mostly about dependencies we treat as boring until they become the incident. I'm Brian Teller from Teller's Tech, and this is Ship It Weekly.

Welcome back to Ship It Weekly, the show about the DevOps, SRE, cloud, platform, and security stories that matter when you are the person keeping the thing running at three in the morning. For the weekly story list and source links, check out OnCallBrief.com. For past episodes and show notes, head over to ShipItWeekly.fm. This week, GitHub had a major outage.

We have a new agentic-browser vulnerability called PleaseFix. AWS Certificate Manager is ending email validation. And Cloudflare is experimenting with TypeScript-native CI workflows. Then we have a quick lightning round and a human closer about what happens when policy changes far away from engineering still land directly on the people running production. Let's get into it.

First up, GitHub had another widespread outage. The incident affected the website, API, Actions, pull requests, issues, webhooks, authentication, and Copilot. At points, web and API error rates were around 20%, with some services reportedly much worse. The obvious takeaway is that GitHub was down. The more useful one is that GitHub is not really just a developer tool anymore.

For a lot of companies, GitHub sits directly in the production path. It runs CI/CD. It holds source code. It manages pull requests and approvals. It may be part of authentication. It triggers deployments. And increasingly, AI tooling depends on it too. So when GitHub has an outage, the impact is not just that engineers cannot push code.

You can lose deployment capability, change management, incident automation, and sometimes even access to the thing you need in order to fix something else. That means GitHub belongs in dependency planning the same way any other critical SaaS provider does. Can you deploy if it is unavailable? Can you roll back? Can you access the last known-good artifact? Can you authenticate to the systems you need?

And do your incident procedures still work if the place where your runbook lives is also having an outage? There is also a difference between being unable to ship new code and being unable to recover existing code Those are not the same risk. Maybe it is completely acceptable for deployments to stop when GitHub is unavailable. In fact, that may be safer. But rollback is different.

If your rollback procedure starts by checking out a repository, running a GitHub action, or waiting for an approval inside GitHub, then your recovery path shares the same dependency as your deployment path. That is worth testing before the outage. You do not necessarily need a second source-control platform sitting around.

But keeping signed artifacts somewhere independent, documenting emergency deployment procedures, and knowing exactly which parts of your recovery path depend on GitHub can make a huge difference. Developer infrastructure is production infrastructure now. Treating it otherwise is mostly wishful thinking. Next, there is a vulnerability called PleaseFix affecting agentic browsers.

The core issue is that the browser is no longer just displaying content. An agentic browser can interpret content, make decisions, use tools, and take actions. That changes the threat model. A malicious webpage is not only trying to trick a person anymore. It may be trying to influence an autonomous system that has access to credentials, local files, cloud services, or external APIs.

That is a very different boundary. The interesting part here is not whether the model is smart enough to recognize a bad instruction. The interesting part is whether the surrounding system allows that instruction to become an action. If a webpage can indirectly cause the agent to run a command, submit data, or interact with another service, then the browser has effectively become part of your execution environment.

And that means the same controls we use everywhere else still matter. Scoped credentials. Tool-level permissions. Network restrictions, human approval for sensitive actions, and a clean separation between untrusted content and privileged operations. I also think this is where the phrase human-in-the-loop gets oversold.

A confirmation dialog is useful, but if the agent controls the text surrounding that confirmation, summarizes the action for the user, or performs enough harmless-looking steps before asking for approval, then clicking yes can become almost automatic. The better control is to make high-risk actions technically distinct. Reading a webpage should not implicitly grant the ability to upload a local file.

Summarizing an issue should not grant the ability to merge a pull request. Browsing documentation should not grant shell access. Those boundaries should exist outside the model. We keep talking about prompt injection like it is a weird AI-specific problem. A lot of the time, it is really an authorization problem wearing new clothes.

The input is untrusted, the action is privileged, and there is not enough enforcement between the two. Third, AWS Certificate Manager is ending support for email validation for new public certificates. DNS validation is becoming the path forward. That sounds boring. It is also exactly the kind of boring change that breaks things months later because nobody remembers which certificate still depends on an old workflow.

Email validation has always been awkward operationally. Someone has to receive the validation message. The right mailbox has to exist. The right person has to notice it. And renewals can depend on that process continuing to work. DNS validation is much easier to automate and much easier to make durable. This also lines up with another recurring theme from this week.

Certificate expiry is still causing real outages, which is kind of amazing in 2026. Certificates are predictable. They have expiration dates. They are machine-readable. And yet, teams still get surprised by them. Usually, the problem is not the certificate itself. It is ownership. Nobody knows who owns the domain. The renewal process depends on an old mailbox. A certificate was created manually years ago.

Or the monitoring only checks the application after the certificate has already expired. Another trap is monitoring only the certificate you think production is serving. There may be a load balancer with one certificate, a CDN with another, an internal service mesh issuing its own certificates, and some forgotten endpoint using something completely different. The useful inventory is not just what certificates exist.

It is which endpoint presents which certificate, who renews it, and what happens if renewal fails. That is a much more operational question. The fix is not complicated. Inventory the certificates. Know who owns them. Automate renewal where possible. Alert well before expiration, and test the renewal path before the deadline.

Certificate management is one of those areas where boring automation is dramatically better than heroic incident response. Fourth, Cloudflare is experimenting with CI workflows written in TypeScript. The interesting idea is that the workflow itself becomes normal application code.

You can define steps, retries, concurrency, and caching behavior in TypeScript instead of expressing everything through a large YAML configuration. There are some obvious advantages. You get normal language constructs. And you can test parts of the workflow more like software. But there is also a tradeoff.

The more programmable CI becomes, the easier it is for the pipeline to turn into another application nobody really owns. We have all seen YAML pipelines become impossible to reason about. Code can absolutely become impossible to reason about too. There is also a governance question here. One advantage of declarative CI is that the set of things a pipeline can express is intentionally constrained.

Once workflows become arbitrary code, you gain flexibility. But you also need stronger conventions around libraries, reviews, dependency management, and security. Otherwise, every team eventually invents its own tiny workflow framework. So, I do not think that the lesson is that YAML is bad and TypeScript is good. The lesson is that CI systems are software systems. They need structure. They need tests.

They need clear ownership. They need versioning. And they need enough observability that someone can understand why step seven retried four times and then deployed anyway. If CI is part of your production delivery path, treating the workflow definition like real software makes a lot of sense. Just remember that real software also comes with maintenance. Quick lightning round.

First, Dynatrace is acquiring Arize for about $915 million. That is another sign that AI observability is becoming part of the mainstream observability platform instead of living in a separate niche. Second, AWS open-sourced Dogwood for governing agent tool calls. The interesting part is policy around what an agent can do, when it can do it, and under what conditions. That fits directly with the PleaseFix story.

Third, Pulumi 3.258 adds opt-in credential encryption. Small change, but a useful reminder that IaC state and configuration often contain more sensitive material than teams realize. And fourth, one AWS credential breach was reportedly noticed because the bill changed. Unexpected egress charges became the security signal. FinOps data is not just about cost anymore. Sometimes it is telemetry for compromise.

The human closer this week comes from a story titled Mario saved the EU but broke my system. The details are pretty fun, but the part I liked is the familiar engineering experience underneath it. A policy or platform decision gets made somewhere far away from the team operating the system. The change may be reasonable. It may even be the right thing to do. But the operational consequences still land on somebody.

That somebody is usually an engineer who did not write the policy, did not choose the deadline, and still has to make production work afterward. That happens with regulations. Security requirements, browser changes, cloud provider defaults, certificate rules, and internal compliance projects. From the outside the change can look simple.

From the inside it can touch identity, networking, billing, deployment, observability, and a pile of assumptions nobody documented. I think that is one of the underrated parts of infrastructure work. You spend a lot of time translating decisions made at one layer into consequences at another. And sometimes the hardest part is not technical at all.

It is explaining that a change described as one checkbox in a compliance document might actually require three teams, a migration plan, downtime coordination, and six weeks of testing. That is not engineers being difficult. That is the difference between describing an outcome and implementing it safely.

The best platform and infrastructure teams get good at making that translation visible early, not after the deadline, not after the outage. Early enough that everyone understands what the change actually costs. And the best engineers are usually the ones who can do that without turning every change into a fight. Understand why the change exists. Figure out where the blast radius actually is.

Communicate the tradeoffs clearly. And make the transition boring. That is usually the real job. Not preventing change. Making change survivable. That's it for this week of Ship It Weekly.

We covered GitHub's outage, the PleaseFix agentic-browser vulnerability, AWS Certificate Manager moving away from email validation, and Cloudflare's TypeScript CI workflows, plus Dynatrace and Arize, AWS Dogwood, Pulumi credential encryption, and an AWS breach detected through unexpected egress costs. Follow or subscribe wherever you are watching or listening.

You can find the weekly story list and source links at OnCallBrief.com and past episodes and full show notes at ShipItWeekly.fm. I'm Brian Teller from Teller's Tech. Thanks for listening. And remember, the boring dependency is usually boring right up until it becomes the whole incident.

A lot of teams say they are doing DevOps because they have stand-ups, sprints, pipelines, and maybe a platform team. But ceremony is not the same thing as progress. And now, AI can write Terraform, scripts, and infrastructure code faster than ever without necessarily understanding whether the architecture makes sense. The tools keep changing. The harder questions do not. What are we trying to improve?

What should we automate? And does the person operating it understand what is actually underneath it? I'm Brian Teller from Teller's Tech, and this is Ship It Weekly. Welcome back to Ship It Weekly, where I filter the noise and focus on what matters when you are the one running infrastructure and owning reliability. Most weeks, it's a quick DevOps, SRE, platform, cloud, and security news recap.

In between those, I do conversation episodes with people building, teaching, and operating these systems. Today, I'm joined by Ned Bellavance.

Founder of Ned in the Cloud and host of the Day 2 DevOps podcast we talk about why teams often copy the ceremonies of devops without adopting the principles how platform teams should decide what actually matters and why basic security hygiene still beats chasing every new threat we also get into the future of infrastructure as code Terraform's scaling limits, AI-generated code, emerging tools like Swamp, and why architecture and troubleshooting skills may become more valuable as writing code gets cheaper.

All right, let's jump in. Today, I'm joined by Ned Bellavance. He's the founder of Ned in the Cloud, and he teaches this stuff for a living. So he's got a really good lens on where teams actually are. Versus where they think they are. Ned, thank you for joining me. Thank you. Thank you for having me, Brian. Really appreciate it. So give me your take.

When teams say that they're doing DevOps or they're doing platform engineering, what's usually true? What's usually hype? Jeez, that's a great question. I think a lot of the hype has to do with process and ceremony. So a lot of what you'll hear... When they're talking about practicing DevOps, is that ceremony that you put around actually getting work done.

So your daily stand-ups and embracing Agile and doing sprints and all that kind of stuff, setting up the ceremony and the look and the feel of what being truly a DevOps organization is. And a lot of the time they stop at the ceremony, but they don't actually adopt the principles that were behind DevOps to begin with. Because what was DevOps really meant to solve?

It was meant to solve a communication gap that existed between the operations and the developers. I mean, it's in the name, right? We took two different things, we shortened them, and we smooshed them together, and we got DevOps. So that was the goal. It was not to have these strict ceremonies where you had to go through this arduous process every day and...

And that would make you just by default better and more efficient as an organization. No, it was these other things. We needed to break down the walls that were between the two groups, have a common language that we could use to communicate needs, and have a feedback loop.

I mean, that's one of the main icons you see in DevOps is that nice infinity symbol of deploying it in production and then having monitoring and feedback go back to the developers. The developers can improve and go through the testing.

And the qa and then it automatically gets deployed to production ad infinitum right so that cycle is less prevalent than you might be led to believe but people still tend to adopt the ceremony instead of what actually will make them more efficient yeah so they're they're surface level saying hey this is what we're supposed to do but they're not maybe diving into what that actually means at the heart of it right and a

Lot of them have gotten mired in buzzwords And that can totally happen because there are so many buzzwords that are thrown out.

When I first started encountering DevOps, this had to be 2016, 2000. Yeah, 2016 ish. I was put on a project where it was literally an application team that was building something for a financial institution. And they had sort of some cloud adjacent folks, but not really. So they brought.

Me and some other folks in who did understand AWS in the cloud to help them build out their infrastructure as code and their pipeline. And they had heard all the buzzwords of like agile and stand up and, you know, OKRs and all that kind of stuff, but they hadn't really implemented what would make their life easier, which was an automated pipeline that actually deployed. The infrastructure and the code.

So that was what we actually focused on was getting that pipeline built out and the process in place with all the necessary checks. And they found that their development velocity increased because now when someone pushed new code, it would only take half a day for it to end up in production if it passed through all the correct gates. As opposed to before, it was probably taking a week for it to end up in production.

What does good look like without needing that giant platform team? Is it having everything in IaC? What is it? Where do you start? Well, I think the place to start for any organization is figuring out where you are today and being honest about it. So really, where are we today? I need to know that. And where are we trying to get to? What are our goals? What's actually important to the organization?

Because you could build a super streamlined pipeline that is shipping crap. And that's not actually going to improve your organization in any way. So understanding what the organization's goals are, how those goals affect what you should be doing as the DevOps team or the platform team within the organization, and then setting metrics that actually measure how effective you are at those goals.

That's where you need to start, just baseline. And then you can come up with a strategy and an actual plan. To implement and get better at the things that matter. So, okay, let's say that we have that initial setup done. We have CI/CD, we've shifted left to a degree. What would you say is next? What matters? Is it authentication? Is it IAM? Is it resilience?

Again, I know it's probably, it depends, right, for each organization. It depends. It always depends. I mean, the things that I would look for and the concerns I would have, a lot of them center around security. Because that's such a problem right now. And I'm sure you saw Project Glasswing got talked about pretty recently. Anthropic's knew, or was it Mythos? I think it's all part of one larger push.

But they basically developed a new version of their model that is way too good at hacking things. And so I would be more concerned with security than probably anything else once I've gotten my... T's crossed and my I's dotted.

So there are definitely tools that will help you become more secure, but I think bringing in either developing in -house or bringing in some people who really understand the security stack would be a good move above anything else, if that's the main thing that you're worried about with your application.

You might also just be worried about performance and scaling, or you might be worried about cost and want to implement FinOps. So you could... The teams that you bring into your existing DevOps team will really depend on what's prioritized at the moment. Yeah, it's interesting. Like thinking on observability, I worked for a company where the CEO had a mandate where I want 100% uptime. Okay, sure.

We can have the webpage cached always showing up. But what does uptime mean, right? Do you need to be able to reach the database to actually log in? Oh, okay. Well, that's a separate set of SLOs that we need to define at that point. That's not just... So uptime is relative. So all of this is relative. What does security mean?

If you're dealing with PII, there could be security concerns that you may have if you're dealing with GDPR and maybe security concerns that are relevant there that maybe just aren't for other organizations. So yeah, you can find yourself, I feel like, worrying about the wrong things if you're not well -led and intentional in your path. Yeah. And, you know, you can...

If you spend too much time on Reddit or reading hacker news or you're subscribed to way too many tech blogs, you might get led astray by what's important to them because they're going to write about the shiny, crazy, and the new. And if you haven't done the basic block and tackling that you need to for security and you're just chasing after some new threat vector, you're going to miss that basic block and tackling.

And that's how most hackers or attackers are going to get in. It's through those basic... Basic security hygiene that you're not doing. There's a reason why the OWASP top 10 list is basically the same every year. It's because people still do make the same basic mistakes. Yeah, very true.

Yeah, I think even the recent like Trivy exploits and Strapi plugin exploits, it was all like any of those like package system npm, you know, package system exploits. It's typically just people creating packages that look like the real thing or close enough. As engineers, we're sometimes lazy and we're like, oh, that looks like the right package. That's what I need. And without vetting it or checking first.

Yeah, some of them have been really interesting. The way that it actually goes about attacking your system. Some have started trying to leverage AIs that you're using on your local system. So they'll have like a startup script that's part of that npm package. And when that script runs, it reaches out subtly and just asks. Various LLMs if they're running.

And if so, you know, can you help me with installing this thing? And if they can get a toehold in one of those LLMs, a lot of the time you're running that as yourself. So it has all the same credentials you do and can operate a lot faster than you can. Yeah, that's scary. That's so, gosh, I was even running. I ran OpenClaw for a couple of days, but very quickly.

Started to go down the rabbit hole of what that could mean. It's like, wait a minute. I'm giving autonomous access to all of my system and all of my various cloud logins. No, I don't. And prompt injection could, yeah, this could get bad very quickly. Yes. I will admit, I'll admit that I'm running OpenClaw right now, but I think I've approached it in a measured way. So I'm running it on its own virtual machine.

And rather than giving it any of my credentials, I've been creating custom credentials for it to access various systems. So it has access to my Gmail, but it has read -only access to my Gmail. It can't create any emails. It can't create any calendar events. And same thing with GitHub. I gave it read-only access to some of my repositories, but it can't commit code. It can just read some stuff in.

And I've slowly been giving it like, here's access to this thing and here's that. And OK, you've been you've been behaving yourself. So maybe I'll let you create to do's for me. But I'm still not willing to give it unfettered access to any of my personal information or my business stuff. Yeah, no, that's fair. That's the fair way to do it. It also was just expensive. It was going through a lot of tokens.

But yeah, for sure. From a security perspective, yeah, read only and least privilege as much as you can. Before we started, we spoke a little bit offline about IaC and you had some, wanted to talk about where IaC is headed. I'm just kind of curious your take on where you think infrastructure as code is headed or where we're going, especially with AI. Yeah. Becoming more prevalent.

I mean, to put some like historical context on it, I've been working in the IaC space, you know, since before Terraform was a thing. 2014, 2013, maybe. And back then, infrastructure as code was CloudFormation. And it was JSON only. And it was awful. I feel bad for anybody who lived through that era before they adopted YAML to make it a little more compact and readable.

And I had to change my mindset from my scripting background, which was a lot of PowerShell and Bash, to this idea of a declarative configuration. Because I'd been running these long PowerShell scripts to deploy like AWS environments or Azure environments. And declarative was such a mind shift for me. Okay, I'm no longer telling it step by step. This is what I want you to create.

And I need to have it checked to make sure that thing doesn't already exist before it recreates it. Now I'm just telling it what I want. But the language is a little awkward because when you're declaratively creating multiple instances, it's not actually a for loop. It's like this other construct and you just have to get comfortable with it.

And I did, you know, fully embraced Terraform at a certain point because it was so much easier to use and it worked across the two major clouds that I was working and doing consulting. And that's kind of been the status quo for the last eight years is just Terraform has... Slowly been improving and adding new features and doing interesting things.

And people have tried to shake things up by making Pulumi as a good example of like, we're going to create something that snaps into your general purpose programming language. So if you're mostly a developer, maybe you'll feel right at home with this. But it's still the same sort of idea of declaring what you want and making it so and state has to live somewhere. I don't know where things are going to go from here.

Because Terraform's starting to hit some limitations on what it's able to do with the model it started with. And I think, you know, this is pure speculation on my part. So this is not information I got from HashiCorp or anything. But I feel like as they approach a potential Terraform 2.0 era, they're going to have to really overhaul some of the internals of how Terraform does its thing and break some stuff.

And then you can do that in a major revision, right? But AI kind of flooded onto the scene and said, hey, maybe there's a faster and quicker way to do things that's still reproducible. It's still code that you can check in somewhere. But now it's not necessarily a human writing any of that code. And that's where I think we're quickly approaching.

Yeah, I also think about the divergence with OpenTofu and some of the changes that they've made kind of influencing, I guess, what HashiCorp may have on their roadmap. And as that divergence grows, there's a gap there that forms that could lead to another product or a completely different way of deploying that infrastructure, maybe with AI, like you mentioned.

Yeah, and that split's not healthy for either ecosystem because it does create that, okay, well, if I want to migrate from one to the other, I have to keep an eye on what's supported across both of them. They've been doing a decent job of keeping feature parity, but the actual implementation detail is going to be different. Because they can't copy code from each other, not directly at least.

But I think just that paradigm of declarative code and then this standard workflow for deploying and then updating your infrastructure, it's a good workflow, but it definitely has its size limitations. So when your state gets too big, everything slows way, way, way down before you do any kind of comparison.

You have to refresh the contents of your local state because Terraform needs a current picture of what your actual architecture, your deployed architecture looks like. It doesn't always know. And so it tries to do a full refresh of everything. Even if you've only changed one thing in your configuration, I'm changing my EC2 instance size.

That doesn't impact the VPC or the subnets or the security groups or the S3 buckets or like the 26 other things that are in my config. It impacts one resource, but during that plan, it still refreshes all 27 resources. That model doesn't scale. And when you want to take those things and break them apart into separate states, that is a painful and arduous process.

So I think on the short term, AI is just going to make writing Terraform and doing these kind of operational tasks a bit faster. But if we're looking longer term, I think there's whole new classes of solutions that are going to come out that maintain some of the core principles of what IaC does, but do it in a different way that's a little more efficient and a little friendlier.

So I did have a company reach out to me where it was deterministic AI creating infrastructure.

I don't necessarily want to give them a shout out because I haven't used the product yet, but I'm curious if that is like this net new possible way of deploying infrastructure that will be adopted or at least I'm sure they're trying to at least make sure that it is but yeah I'd be curious to see if there's new ways of deploying infrastructure it's interesting we're talking about like those calls years ago at a

Company I worked for we were 100% in on HashiCorp this is pre-IBM 100% in on Vault, Terraform, Packer all of it and we were we had all of our GitHub repositories in Terraform as well And we quickly realized that you can hit that API limit very, very easily.

And GitHub's is pretty restrictive if you're not on like an enterprise, like a higher level enterprise program. So yeah, we were then partitioning out repo creation and branch protection rules based on teams and other arbitrary ways of our partitioning out the state files. So we didn't have to hit those resource limits just by doing a...

Making sure there was no drift changes or making sure that we were updating branch rules. Yeah, that totally makes sense. Arduous and broken and yeah, just the wrong way to do it. Yeah, if listeners aren't familiar, like when you do a whole bunch of reads against GitHub's API, that counts against your rate limit.

So if you're doing a massive, if you're managing all these repos with Terraform and you try to do a single configuration update on one repository and it's pulling. 100 repos and all the settings in those, yeah, boom, you hit that limit and your plan fails. That's hugely frustrating. So I would like to see a solution that's a little more intelligent about the graph that exists and intelligently updating that graph.

The other thing that Terraform is predicated on, and most of the other ones are as well, is this idea of CRUD, or CRUDL, really. So the standard operations for any resource that exists is you can create it, you can update it, you can read, you can delete, and you can list out resources.

And that's really well documented in the providers, and it makes the providers extremely reliable because they only have these actions, and that's it. But if you work in operations, you know that's not the end-all and be-all of infrastructure. Those are not the only operations that are permitted. There's operations like... Running a sync or resetting a virtual machine or backing up a database.

And those are actions that don't fit neatly into one of those categories. Interestingly, HashiCorp has introduced Terraform actions to sort of try to help out with those non-CRUD operations. But I've been playing around with a new tool that's being developed by System Initiative called Swamp. And, you know, I'm not going to say that Swamp is the end all and be all.

But its approach is really interesting in that it builds models in the same way that you would build resources for a Terraform provider. But it doesn't have strictly defined actions that have to exist. And it's all TypeScript based, at least at the moment. So when you build out your model, your model for a resource can have a number of different actions. And you can call them whatever you want them to be.

You work with an LLM to write the TypeScript that is going to be behind those actions. So if I was managing an Azure virtual machine, I could not only create, update, delete, etc. I could also restart it. I could kick off an extension inside of it. I could do a backup of it. I could do all these other operations. And I can also have scoped operations. I could have an operation that is strictly updating metadata tags.

And then I could potentially... Hand out permissions that that's the only thing that a specific pipeline or operation can do is update those metadata tags. So I really, I'm enjoying, I'm experimenting with it now and I'm enjoying the model and the fact that it's really easy to extend because you can download somebody else's set of models. They call it an extension.

And then if the action you're looking for or the property that you need isn't in it, you can just work with the LLM and Swamp and and say this is what I'm looking for add it to the extension and it will write the TypeScript for you and test and validate it and if everything works out you can upload a new version of the extension for other people to use interesting so okay given that model with Swamp given where

Terraform is OpenTofu Terragrunt if you want to wrap all that and make it more dry where we don't know what the next evolution is necessarily of IaC that will be adopted by everybody.

How do you recommend preparing for that future? Is it smaller state files, larger state files? How do we go about logically building out the infrastructure with IaC tools in a way that will make that adoption easier in the future? Any truly viable solution is going to have an off-ramp from Terraform, basically.

CloudFormation and ARM templates too, but any real serious solution is going to have to have a way for you to migrate off your existing infrastructure as code onto this new platform. I don't see that happening, especially at enterprises, within the next five years.

I know the speed at which enterprises tend to move, and they're not going to throw away all the effort and the skills that they've put into building up their Terraform portfolio. For some new solution that's totally unproven. So if you're looking for like, should I even bother learning Terraform? Like, yeah, you probably should.

But I think the more important thing to learn, and this applies whether you're using Terraform or CloudFormation or Ansible or Swamp or like 10 other products that are out there, none of those matter if you don't understand what you're actually deploying. All of them assume that you have some level of knowledge of...

The different services and features that exist on the different cloud platforms and how they interact and work together. AI can help you with that a bit. If you describe the shape of the application you want, it can make some educated guesses on, well, that sounds a lot like a Lambda. You should probably implement that as a Lambda.

And okay, you're going to need, you know, an elastic load balancer to provide availability for these VMs, stuff like that. It can make some pretty good educated guesses. But I think...

Our job as, I guess, system administrator turned DevOps engineer turned AI prompt engineer, our job is still to understand the fundamental architecture behind all this stuff and be able to describe it and explain why we need certain infrastructure set up in a particular way. That skill is way, way more valuable than being able to write good code in any of these different languages because...

Regardless of what tool wins out, AI is going to be writing most of the code at this point. I know that's not like a certainty, but it's like a 90% certainty that writing code now is extremely cheap. Writing really good architecture documents is still very expensive. So I would focus on learning the fundamentals of the different cloud platforms and learning the fundamentals of like networking.

Of Linux, of databases, of security, and being able to take all of those different disciplines and turn it into a design document that can then be interpreted by an LLM. Okay, so along those lines, if you were talking to someone that was new in DevOps, cloud, where would you tell them to focus first? Kind of what you already mentioned, but I mean, where should... Where should their focus be?

Should it be on prompt engineering? And should it not be that? Should it be the fundamentals first and then bring AI into the mix after the fact so you know what you're vetting when you look at the output? Yeah, I feel like my value is the 20 plus years I've been in IT and just seen some stuff. And I don't think there's a way to speed run that experience.

And being hands -on, at least for me, is the best way to get that experience. So even though... I might never have to make a Cat 5 cable myself. I've done it. I've taken the crimper, I've separated out all the pairs, I've put it into the thing and crimped it up and tested it with a cable tester. I've done that. I know how that fundamentally works.

And even though it's totally useless because I can buy a Cat 5 cable for pennies on the dollar, I still think it was useful to gain that knowledge. And so... If I could compare that to like writing code versus having AI write it for me, you might never have to write another line of Terraform code in like a year or two.

But understanding how to properly compose a module, understanding how to organize code effectively, those are the things that are going to pay dividends when something goes wrong. Because that's in the true job of the SRE. Is to be there when things go wrong and they always will. And knowing what wrong looks like and being able to sort of guess based off of prior experience what the probable cause is.

Lots of different tools can help you with that. But those like the key troubleshooting skills is something you still need to hone on your own. That's fair. So what training in your experience actually sticks for people new? Is it labs, projects, incident drills, internal docs? What do you find works the best when working with junior engineers or people fresh? What works the best is to do it wrong. Okay.

That sounds weird, right? What works the best for me, and so I'll use I statements because this is what works best for me, is I only ever read like the first three pages of the manual and then I try to do it. And it never works right. It's broken. It's misconfigured. I'm frustrated.

But the process of reading the docs, figuring out where I went wrong, and then fixing it and redoing it correctly, I'm going to remember that whole process because it truly was a process. Clicking like a let's get started script and just having it build all the background for me, that means I'm never going to truly learn those things. At least, you know, like I said, for me, there's a reason why.

Kelsey Hightower wrote Kubernetes the Hard Way. It's not because it's impossible to script the installation of Kubernetes. Clearly that's not the case, right? But if you do it the hard way, you're probably going to screw it up. So you're going to have to recreate a certificate authority and reissue certs because you got that wrong.

And then you probably spun up your node wrong or you got the kubelet misconfigured or your networking is incorrect. And you're just going to trip over all these things. And as you trip over those things, you realize where the gaps are and where things can go wrong. And so when your production cluster falls over at 3 a .m., you're going to feel like you actually have a handle on what's happening inside that cluster.

As opposed to, I've only ever used this template and deployed automatically and never had to think about what happens if the networking gets screwed up within my cluster. That's fair. I mean, I find even still, I'm learning after 25 plus years that same way, right? By breaking things.

And I mean, even OpenClaw, if I wanted to, you know, trying that, it was playing around with it and letting it break things in a controlled environment. But yeah, I'm just playing with it. I, you know, I still find that I don't know everything about Kubernetes. I don't know everything about cloud. I can't. It's impossible. But being open to that, being open to that process and being open to it.

To getting it wrong is valuable, very valuable. Yeah, I've had three pillars. Well, I established three pillars for myself a long time ago. One of them was make yourself uncomfortable. So just be uncomfortable. Put yourself in an uncomfortable, difficult situation. Not too uncomfortable, but like step outside of your comfort zone, right?

Another one was be kind because everybody else is just trying to trudge through this thing called life as well. And the third one was... Be prepared to fail and be okay with it. Because failure is one of the ways we learn. And I am failing constantly. I spent last week fighting with this Swamp thing to set up my home lab.

And I had moments where I was yelling at the monitor and hitting escape and going outside to take a walk. But after a week, I used Swamp to rebuild my entire home lab. It was a big learning experience for me on how to use LLMs more effectively. To understand Synology's weird API is a little bit better. Don't get into that if you don't have to.

And also, I learned some new ways to capture web traffic and do analysis on it. I didn't even think that would be a benefit, but it was just like in the troubleshooting, I learned a whole bunch of new things. So I think being willing to experiment and being willing to try something and fail is the mark.

Of a lifetime learner and that's what we all are when it comes to technology you have to be a lifetime learner because the technology is not going to stop for you I'd be very dubious to trust any senior or plus level engineer that's worked 10 plus years in the industry and hasn't taken down production or has that as like a statement like oh I yeah I've never well if you haven't then you what have you really been

Doing like what yeah how deep do you actually work in the system or do you know the system um because I think we've all been there right yeah totally It's an unfortunate reality, but humbling yourself and working as a team and working through it rather than just panicking is when you learn probably the most, too, is in those scenarios, in those situations, not in the building of the POC or in the building.

I mean, those can be valuable as well, but or in building of the SOP, you know, building out your standing operating procedure that you may run or run book that you may use someday, but actually doing it and being in the incident. Is where the real work and value, I think, is built for the most part. Yeah, and I have a podcast. It's called Day 2 DevOps.

And that was sort of the inspiration behind the name is because day one or that POC, when you first deploy everything and it's new and it's shiny, yeah, it's probably going to hum along pretty well. But day two, when you actually have to start operating it and updating it and real traffic starts hitting it, that's when you learn how good your architecture actually was. And that's what we like to focus on.

In the podcast is kind of more realistic implementations of technology and less the hello world and the day one of what happens when you deploy tech. Very cool. So along those lines, where can people find out more about you find out more about your podcast? Any other things that you'd like to plug or talk about? Yeah, I mean, so I have two things I'd like to plug one is that podcast day two DevOps.

So if that sounds interesting to you, if you want to hear some interesting conversations. About DevOps adjacent technologies, Day 2 DevOps, all spelled out. You can find it there at day2devops.com. And the other thing that I'm currently doing is I'm doing a live stream every Thursday at 1 o'clock Eastern to help people study for the Terraform Authoring and Operations Professional exam.

So every week I cover a few of the topics that are part of the exam. And based on my mapping, that's going to be going on for a couple months because there's a lot of topics. So if that sounds interesting to you, if you thought about sitting the exam or you just have like weird questions about Terraform, go to my YouTube channel. It's Ned in the Cloud and check it out. 1 p.m. On Thursdays. Awesome.

I'll put links for all those in the show notes as well. Ned, thank you so much for coming on. Really, really appreciate it. Thanks, Brian. That was my conversation with Ned Bellavance, founder of Ned in the Cloud. And host of Day 2 DevOps. The biggest takeaway for me is that tools and terminology can make a team look much further along than it really is.

You can have stand-ups, pipelines, infrastructure as code, and an internal platform. None of that guarantees you are improving communication, shortening feedback loops, or shipping something the business actually needs. The same applies to AI.

AI can make writing Terraform and automation code much cheaper, but it does not remove the need to understand what you are deploying, why the architecture works, what permissions it needs, or what failure looks like in production. If anything, that judgment becomes more valuable. I also liked Ned's point about learning by getting things wrong, not recklessly in production.

But in labs and controlled environments where you can break something, troubleshoot it, and understand what was hidden behind the template. You cannot completely speed run that experience. So focus less on whether your organization looks like it is doing DevOps and more on whether the practices are producing better outcomes. Use AI to move faster.

But understand what it generates learn the tools but do not confuse the tool with the architecture and keep building the troubleshooting experience you will need when the clean demo becomes a day 2 production system you can find Ned in the Cloud and the Day 2 DevOps podcast through the links in the show notes Follow or subscribe to Ship It Weekly wherever you listen and find previous episodes at shipitweekly.fm. Thanks for listening and I'll see you later this week.

Railway lost an upstream network path, recovered it, and still kept having problems because stale connections had moved onto the wrong network. Stripe says it cut database pages by about 30 % by replacing hard-coded recovery logic with graph search. And a critical Kata Containers vulnerability let guest root become host root. This week is mostly about what happens after the obvious failure.

I'm Brian Teller from Teller's Tech, and this is Ship It Weekly. Welcome back to Ship It Weekly, the show about the DevOps, SRE, Cloud, platform, and security stories that matter when you are the person keeping the thing running at three in the morning. For the weekly story list and source links, check out OnCallBrief.com For past episodes and show notes, head over to ShipItWeekly.fm.

This week, Railway published the RCA for a major U.S. East outage. Stripe shared how it uses graph search and state machines to automate database remediation. Kata Containers patched a critical guest to host escape. And DynamoDB now has native vector search.

Then we have three quick lightning stories and a closer about learning cloud-native engineering by actually working on real systems instead of just finishing another tutorial. Let's get into it. First up, Railway published the incident report for its July 2nd US East outage. The incident started with an upstream ISP problem. That alone is not especially unusual. The more interesting part is what happened next.

Railway disconnected the affected carrier. But in the process, it also removed the site's last default route. Storage traffic then fell back onto a management network that was never intended to carry that load. Eventually, the routing problem was fixed, but recovery did not mean everything immediately went back to normal.

Some long-lived connections had already established themselves over that slower management network. They stayed there. So even after the main route came back up, some traffic continued using the wrong path. Railway also described around 20,000 private network connections ending up blackholed during the incident. This is one of those failures where the original trigger matters less than the state it leaves behind.

We tend to think of recovery as a binary thing. Route is broken. Route is fixed. Incident over. But distributed systems do not always reset themselves just because the thing that caused the problem has been corrected. Connections stay open. Caches preserve bad information. Sessions remain pinned. Retries build pressure somewhere else. And the failover paths can quietly become the new steady state.

That is the lesson that I like here. When you restore the control plane, you still need to ask what data-plane state was created while things were broken. What connections need to be recycled? What caches need to expire? What sessions need to be re-established? And what fallback paths are still carrying traffic they were never supposed to handle? The first fix gets you out of the fire.

The second part is making sure the system actually returns to the architecture you intended. Next, Stripe shared how it reduced database paging by replacing a lot of hard-coded remediation logic with something much more dynamic. Stripe operates a large MongoDB fleet, and like a lot of mature infrastructure, recovery used to depend heavily on specific runbooks and rules. If this condition happens, run this action.

If that action fails, try this other one. That works. Until the number of possible states gets large enough that the runbook itself becomes a system you have to maintain. Stripe's newer approach models remediation as a graph. The current state of the database is a node. Possible recovery actions are edges. And the system can search for a safe path from the current broken state toward a healthy one.

They combine that with state machines to track what is happening, and Dijkstra's algorithm to find possible remediation paths. The result, according to Stripe, is roughly a 30% reduction in database-related pages. They also reported around 200 fewer pages per year and fewer days where shards stay unhealthy. What I like about this is that it is not really an AI story. It is an automation story.

We sometimes jump straight from manual runbooks to give an agent access to production. There is a huge middle ground between those two. You can model the problem, you can encode valid state transitions, you can define which actions are safe, and then let software calculate the recovery path without making the recovery system completely open-ended. That is a much more interesting form of automation to me.

It still gives the system flexibility, but the flexibility exists inside boundaries engineers actually defined. And just as importantly, the recovery logic becomes something you can reason about. You can inspect the graph. You can test transitions. You can understand why the system selected one path instead of another. Automation gets much more useful when it reduces toil without also reducing explainability.

Third, Kata Containers patched a critical guest -to -host escape. The vulnerability affected the runtime-rs virtio-fs path. In the affected configuration, root inside the guest could cross the isolation boundary and end up executing as root on the host. That is about as bad as container isolation bugs get. Kata exists specifically to strengthen the boundary between a workload and the machine underneath it.

Instead of relying only on normal container isolation, workloads run inside lightweight virtual machines. So when a vulnerability crosses from guest root to host root, it is attacking the exact security property people adopted Kata to get. Affected versions run through 3.30.0. The fix is in 3.31.0. The practical takeaway here is pretty simple.

If you are running Kata Containers, this is not a patch that I would leave sitting in the normal backlog. But there is a broader point too. Isolation technology changes the attack surface. It does not remove it. A VM boundary is stronger than a namespace boundary in a lot of scenarios.

But virtio, shared filesystems, device emulation, runtime components, and the glue connecting guest and host all become part of that trusted boundary. Every time that we say that something is sandboxed, the next question should be, by what? And what still crosses that boundary? Isolation is not a checkbox. It is an architecture. DynamoDB now supports native vector search.

That means that applications can store embeddings and run similarity searches without automatically reaching for a separate vector database. For teams already using DynamoDB, that can remove a surprising amount of architecture. Today, a common pattern is to store application data in one system, generate embeddings, write those embeddings somewhere else, and then keep both stores synchronized.

Now you can potentially keep the primary data and vector representation closer together. That is attractive for things like semantic search, recommendations, retrieval-augmented generation, and agent memory. But I would not read this as vector databases are dead.

A specialized vector engine may still be the right answer if vector search is the core of the workload or if you need capabilities DynamoDB does not provide. The interesting part is that vector search is becoming another normal database feature. We have already watched this happen with JSON support, full-text search, caching, analytics, and other capabilities that once demanded separate systems.

That usually leads to a good architecture question. Do I actually need another service? Every additional database creates another backup strategy, another availability model, another security boundary, another cost center, and another thing someone has to understand during an outage. Sometimes that complexity is justified. Sometimes the best infrastructure decision is deleting a box from the diagram.

Quick lightning round. First, AWS brought explicit proxy functionality into network firewall. That gives teams another way to control outbound traffic while keeping firewall rules, intrusion detection, TLS inspection, and workload -aware controls in the same general system. There is also a new no-source-preservation mode that can work with NAT Gateway.

Useful, but as always with AWS networking, look closely at the traffic path and processing costs before you decide the cleaner diagram is also the cheaper one. Second, Gateway API 1.6 makes TCPRoute and UDPRoute stable. That matters for workloads that are not HTTP. Databases, queues, game servers, DNS.

And other Layer 4 services can now use stable Gateway API resources instead of treating Gateway API as mostly an HTTP routing story. And third, containerd 2.4 is in beta. The release includes work around warm image caching, runtime introspection, snapshotters, and other internals, along with some breaking changes. If containerd sits underneath your platform, beta release notes are worth reading early.

The worst time to learn that something you depended on disappeared is during the Kubernetes upgrade that also changed six other things. The human closer this week comes from a developer writing about getting deeper into cloud-native through the LFX mentorship program. What stood out to me was not the specific technology. It was the progression. Documentation work leads to actually deploying things.

Deploying things leads to debugging OpenTelemetry and Prometheus. And debugging leads to understanding why the pieces behave the way they do. That is where most engineering knowledge really comes from. Tutorials are useful because they remove friction. Everything works in the order that the author expected. The YAML is correct. The versions match. The network behaves.

And the command in step 7 produces the output shown in the screenshot. Real systems do not do that. A real environment has the wrong version. The collector starts but exports nothing. The service account technically exists, but does not have the permission you thought it had. The dashboard is empty. The example in the documentation was written for a different release. And now you have to figure out why.

That frustrating part is where judgment starts getting built. You stop memorizing commands and start building a mental model. You learn which logs matter. You learn what assumptions to check first. You learn what healthy actually looks like. And eventually, you get faster. Not because you know every answer, but because you know how to narrow the problem.

That is one reason that I think contributing to open source mentorship programs, labs, and even small personal projects can be more valuable than stacking another 10 hours of tutorials. You need environments where things can be wrong. Because production is mostly an environment where things can be wrong. There is nothing bad about starting with the happy path. Just do not stop there.

The goal is not to finish the tutorial. The goal is to become comfortable when the tutorial stops matching reality. That's it for this week of Ship It Weekly. We covered Railway's US East outage, Stripe's graph -based database remediation, the critical Kata Containers escape, and DynamoDB native vector search. Plus, AWS Network Firewall Proxy, Gateway API 1.6, and containerd 2.4.

Follow or subscribe wherever you are watching or listening. You can find the weekly story list and source links at OnCallBrief.com and past episodes and show notes at ShipItWeekly.fm. I'm Brian Teller from Teller's Tech. Thanks for listening. And remember, recovery is not finished just because the original failure is gone.