Ship It Weekly Transcripts

AI can crank out code fast now. The part that's not speeding up is our ability to ship it safely. And if you don't have guardrails, you are basically just moving failures faster. Hey, I'm Brian. I work in DevOps and SRE, and I run Tellers Tech. Ship It Weekly is where I filter the noise and focus on what actually matters when you are the one running infrastructure and owning reliability.

Most weeks, it's a quick news recap. In between those, I do interview episodes with folks who are actually building and operating real systems. Today is one of those interviews. I'm joined by Mike Lady, a senior DevOps engineer and the creator behind Enterprise Vibe Code on YouTube.

Mike's deep in distributed systems, and we're talking about day two readiness, guardrails, and why that stuff matters even more now that AI can generate code fast. We get into what day two readiness means in plain English. The symptoms you see when teams ship without guardrails, what a real day -two audit looks like, and how Mike thinks about AI can write code, but it can't ship safely without gates.

We also talk about AI agents watching builds, branch protection, quality gates, and where the line is between AI -assisted development and AI taking actions in production. All right, let's jump in. Today, I'm joined by Mike Lady, a senior DevOps engineer from Enterprise Vibe Code on YouTube. He's focused on distributed systems.

And we're talking about day two readiness, guardrails, and why the ability to ship safely matters even more now that AI can generate code fast. Mike, thank you for joining me. Thanks. Thank you for having me. So I'm curious, give me your thesis on why do guardrails matter more than good code? I feel like it's a kind of...

Process like we we can we're constraining people and ai to create like uh code that meets the bar meets a certain standard I feel like these it's the it's basically the same thing a human generated code ai generated also all go into the same like pipeline all go into the same like build test deploy phases right so I feel like people are freaking out like making a big deal out of this when we have the same system for people as we do for AI code, right?

And ultimately it's people reviewing and approving this code anyway, right? So I feel like this is, I don't know, we're in the same thing. We're still doing the same thing and we can do it even better now in the age of AI -aided development or five code, whatever you want to call it. People like to call it different things. So for sure, yeah.

So what do you what is day two readiness mean in plain English for people out there that may not know the term or understand the meaning behind it? At least for for what how I'm talking about is like day zero is like you don't have an app. You just have an idea. Day one is you launch the app. You go from zero to one. You launch the app. Hooray. You you have something on the Internet that people can use.

And the day two is the part that comes after that. It's like. How do we maintain this long term? How do we add new features? How do we fix bugs? How do we add security updates and whatnot? Like, how do we change the app over time and maintain it? Because it's one thing to go from zero to one, make it, and that's like the first part of the work. It's a much longer, much more long tail of work after day two, basically.

Yeah, for sure. So what symptoms do you see when teams don't have guardrails, typically? They just pray. They put code into production and just hope that it works. There's some famous quote, it's like, hope is not a strategy, right? You want to be able to try to prove out that this code that you make is as correct as possible.

So if you don't have guardrails, you're going to have weird bugs, you're going to have features that just completely get deleted or something like that, or just don't work. Your users are going to be pissed. It's just not a good thing to just kind of ship, like merge directly to main and then ship that directly to prod and just blindly do that. Yeah.

So do you think that guardrails end up slowing teams down or speeding them up? It's, yeah, ultimately speeding them up. People who are like really... I don't know, early startup run and gun. They're going to feel like, oh, it feels like it feels slow. It's more process. Like we can't get things done. Like we have to fix the tests.

But like now in the age of AI driven development, vibe coding, it's like the AI can do all the hard stuff for you. Like it can like write the tests. It can fix the tests. It can. Do all the stuff that you don't want to do. And then you can focus on the stuff that you don't want to, you want to focus on with features and whatever bug fixes that you want to do and kind of product development.

We get to operate at a higher level now rather than dealing with the build breaking, right? Like we could just say, like my favorite thing to do with AI agents is to have it watch the PR build and it monitors it. It sees the, whatever, the break, like whatever. Thing it fails on and then goes and tries to fix it. And just it goes off in its own loop and just works to make the build pass.

Of course, I need to check the code to make sure it doesn't like just delete the test to make the build pass. Right. But like there's a certain like abstraction or certain like like I'm giving the agent a little bit more control autonomy. I'm delegating the boring developer task of, all right, fix the build.

Type of thing so I think I got a little off track of what you're going but no no speed up versus slow down yeah so ultimately it speeds you up because you're not dealing with all the failure like all the bugs that crop up if you don't have guardrails so yeah I I think too it goes back to like the whole philosophy between but behind iac like slowing down to speed up yep to a certain degree but even that barrier is

Gone now to a certain degree with ai but yeah I've also run into ai agents saying oh you need this test to pass no matter what okay I'll just remove the test or I'll just have it return one it passed see aren't you happy right exactly yep for sure yeah ai agents will lie cheat and steal to make their reward function happy and that that's a steve yeagy quote I'm a yeagy fan boy from uh he has a book and her book vibe

Code vibe coding highly recommend it written with gene kim gene devops god right absolutely yeah so that that's why I like inherit like why I'm working like I'm a big like stevie like beads fan and his newest thing gas town I'm a contributor on it now it's uh yeah I'm super hyped on it cool anyways awesome so walk me through what you check in like a day two readiness audit like what would be like your checklist like

Your top I don't know three things that you would check right so are you properly using source control is it like Okay, you're using Git probably because everything's on GitHub.

Hopefully. Hopefully, yeah, right? Hopefully. Okay, are you using branches? Like, are you just pushing the main? Are you using PRs on GitHub, right? Like, and are you blocking those PRs with quality gates? That's kind of the next thing. Like, do you have a... Is main protected? Branch protection. Yeah, branch protection. Make sure the agent doesn't cheat and just pushes it straight to main, right?

And you actually have to go through all the quality gates on the PR to make sure it passes and you can merge into main. So that's like source control and then quality gates. It's like, okay, does it build? Does it test different levels of tests, like your unit integration, functional? Again, all these tests are essentially free now.

Like they just cost tokens like they don't cost developer time right so it's that's why I'm so like excited about this idea is like before with when I was I'm still devops but like when I first uh started doing devops I was got into through the like testing I was kind of like a ui automator first and then they were like oh how are we going to orchestrate these ui tests it's like oh okay wow grow into a DevOps role for Mac.

It was actually iOS DevOps. So I was mobile DevOps first. And we were racking Mac minis and running these UI tests, making sure the app doesn't break. But it was a pain in the ass to maintain those tests, right? Everyone knows UI tests most brittle. The things on the screen change. You click the button. Maybe you have to click it again because the click didn't send through or whatever.

Like there's all kinds of things where UI tests are like very flaky in general. But now, Like you can just tell it to update it and it'll do it right. And it'll probably fix it in a way that's satisfactory. So that's why I'm so like hyped on this testing quality gate type of thing. Plus like you can run like test driven development.

So you can like have the AI write the test first and then it writes the implementation to pass, make the test pass. So I think it's a huge like boon to all these like methodologies that we all know we should. Have been doing like TDD this whole time. Like, like whatever, for 20 years, I don't know how long TDD has been around, but like I first learned about it in college and I've never done it until now.

Test driven development. Yeah. Yeah. Test driven development. Yes. Yeah. So, so yeah, that's like the, whatever the quality gates, the second thing, config and secrets management. Like, are you like for, for me, I have like a demo app that lives in GitHub. I store all the secrets.

Uh in the git uh not the git repo in the like configuration for the repo and it has like a little secrets place or whatever for it but for um adbs there's secrets management for all these other things have like very specific ways to handle secrets because like that you're gonna have api keys you're gonna have uh things that you need to like connect to or whatever and deploy to so we want to handle those properly so those are And I'll throw in a bonus fourth thing, like deployment model, like how, how do you deploy?

Do you like deploy to like a staging environment first? And then do you like test against that staging environment? Then once that, that test is good, like, do you deploy to prod after that? So like having some sense of like, like test deploying the thing, whether that be on a PR, whether it be on a staging environment, making sure those.

We deploy it somewhere first and ensure that everything comes up before just deploying straight to prod and then breaking prod. Yeah, I think there's certainly a case to be made for testing in prod with the right feature flags and the right segmentation where you could do 1 % or 0 .01 % of traffic, but certainly not day one, day two, day three. You need to have a mature practice before you're able to sign on to that.

Right. Yep. Exactly. Okay, so jumping, we've talked a little bit about AI. So jumping into the AI coding, guardrails. So you have a couple posts on LinkedIn where you talk about AI can write code, but it can't ship it safely without guardrails. What does that actually look like in practice? We talked a little bit about this already, but I'm just kind of curious. What's your philosophy behind that?

Do you treat AI as like a junior developer or junior engineer that you... Build gates on or what gates do you insist with that AI agents? Yeah. So it's a process. So like we have a whole, like we kind of have to force it to go down a path and you do a process. So like that's at the, like we have the same gates at the PR level for everyone, right? For human generated code or AI generated code.

But even before that, I have kind of a, kind of like a plan, like have it plan itself.

Have other agents look at the plan and comment on the plan using like stevie's beads framework but like you could probably do it with other things but and then kind of like think about the plan given those other comments like is these agents aren't even necessarily like the same it's not just claude like I use claude gemini codex whatever else is out there cursor with like rock or something like that like I I think all these different um models are kind of different perspectives.

They're all trained a little bit differently. Yeah, they may be all trained on the internet in general, but like they all have like different, a little bit different perspectives. So I treat those as like a team that kind of like review the plan. And with beads, at least like it can comment and it's non -destructively adding to the plan, like adding different perspectives to the plan.

So it's a beads is like a issue tracking, but for agents where. It's kind of like Jira, you know, like you get a Jira, you can comment on the Jira, you can like help people like work through like making the Jira better or whatever, right? And in a non -destructive way without just saying edit and whatever, completely changing the source material, right? Like without changing the description box, say.

Adding comments is kind of like a way to give your perspective, give your take without. Completely changing the original material. So there's the plan phase. This is kind of like the process guardrail. Plan phase, implementation phase, I have like a belief that you should probably use like one model or one agent as a daily driver. Like you know that agent in and out. Like mine is Claude, but it could be anything.

If you're comfortable with Cursor and you know how Cursor responds or whatever. You learn its quirks. Yeah. Yeah. You learn how it's what it tends to do when it when it tries to lie, cheat and steal. You learn when what it yeah, when it tries to take shortcuts and you can recognize that right away.

So you implement with your daily driver and then you can review, like make a PR and then have all those same agents that like did the planning with you do the PR review and they can all like I forgot a step in the.

The the planning is like you incorporate those comments from those other agents into the plan right and then in the pr review you can say all right take a look at the pr take a look at the the beads the issues and see like how well does it like implement the plan and is there anything that's missing or are there any issues like security issues or perform it like you can have it come at different angles so that's all

Like before we enter our like main pipeline So that's like a pipeline within itself, a pipeline within like the implementation.

I have implementation as like a stage name, but like in the AI agent driven development process, I guess. So I would say that that's like a guardrails, but like human, the shaping that we do, like when we're interacting with the AI. Are you setting? Yeah, that makes sense. Do you?

Are you setting specific like pre -prompt personalities, personas, or are you just relying on like Opus and Sonnet and Gemini like to have their different personality traits because of their training, their LLM training? Yeah, I'm just going with the kind of the base model. A lot of people like to play house.

This is not my quote, but like I'm taking this from, I think this dude from Human Layer, his name might be Dax or something. I saw.

He was giving like a talk youtube video if you search like human layer dude on on youtube uh you'll probably find it and he says like oh yeah people like to play house and have their their teams of agents with their their dev and their qa their manager their product manager their you're like it feels like you're doing uh I don't know like theory crafting like you're you're playing dnd you're the expert like whatever with 20 years experience it's like do you really need to hype the agent up that much?

Or is the, is the model, is the stuff already in the model? Yeah, exactly. By, by asking it, it can, you like review the security. You don't need to prompt it as you're a security researcher. You're a security engineer. Yeah. It's like, you're only, you're kind of obfuscating what it can do at that point too, because you're, you're, you're narrow cat, like narrowing what it's, what it's pulling from or what it can.

Yeah, for sure. So yeah, no, I, I think, yeah, I just use the, as vanilla as possible.

The models so okay just curious then do you notice you would mention as we use more model different models we notice that there's traits when they lie cheat and steal what's your take on obviously the landscape's always changing but what's your take on models right now like is there a specific model that oh this is well and above great for devops versus this one isn't great for devops or cic cic right yeah uh I use

Claude mostly like that that's like a good general I mean, all of them are coding models, but like it seems to be like pretty, like pretty good.

But I do like when I do Codex, Codex seems very thorough when it does its reviews. It always takes the longest. Gemini seems to like be really quick and like it gives some good feedback or whatever, but like Codex seems to take its time and really like analyze what's going on. It really goes through like the issue and usually has some pretty good feedback. So yeah. I think they're all pretty valuable.

So big tech, big AI doesn't want you to know that you can use all of them, that you can use all of them together. Like all of them just want all of your tokens, right? Quad wants all of your enterprise's tokens, like Gemini or OpenAI, whoever. They want your entire token budget. But if you can, I don't know, spare the $20 a month for multiple providers. I know it's hard out there. I know, I know.

Whatever, tough economy and whatnot, but like I view spending on AI is learning. Like you're investing in yourself. You're, you're this as a developer, this is where the industry is going. And I, I'm never going to, I'm hopefully never going to write another line of code.

I'll say that every day and I'll still end up writing a couple of lines here and there, but like it's the, the probability is becoming less and less every day. Right. So. Yeah, you're the manager or the, I guess, the senior working with the junior that's actually doing the work. So Cursor even has the ability now to run multiple models at the same time. Right.

And I think you can even do multiple iterations or runs of the same prompt and verify the output. To be fair, it's on my Cursor. I don't use it. I haven't used that yet, but I have used like the multiple models to kind of see which is best for. And it depends. I mean, it's everyone always wants to use the max model, you know, because they always think that their problem is the most important deep thinking problem.

Yeah, that's generally not the case. Yep. Generally, you can be you can do quite well with just using auto or even just using their auto and their their switcher is actually really good. They're being able to switch models and let them decide which model to use actually works out really well. I mean, they are seeing whatever. Millions of requests a day. So they probably know what they're talking about, right?

You mentioned things like CICD, branch protections, and agents .md. How do these fit together? How do they fit together? Okay. So the agents .md is kind of like your implementation. Like how is the agent on your computer going to act? How is it going to... It's kind of like the base instructions for... How you operate with it on your computer, right?

So then agents to AMD are kind of like the guardrails on your laptop, what you are trying to steer your agent to do as long as it can remember. Like when you run out of contacts, you're, yeah, it's bound to forget stuff, right? But yeah, agents to AMD is kind of like guardrails on your computer. You steer towards the problem. You're trying to solve the problem. You can have like...

There's new things out there that you can have agent orchestration where you have multiple agents running in parallel. They're all trying to like solve different problems potentially on your laptop. You open up all these different PRs and then the CICD pipeline guardrails kind of take over from there.

And then you try to make your builds pass and hopefully you don't like, but you have good enough tests, good enough like code coverage.

Too like that's another big one is like there's like no excuse to not have 80 code coverage now like it's tests are free they just cost tokens so I feel like the the cicd pipeline blocks like basically code but that doesn't work with your existing tests and you can gate the kind of like the code coverage part and say oh it has to be above 80 if you're adding a bunch of code and you gate on 80 and it goes below 80 it

Can require hey you have to add tests like a heart like A hard requirement, you're not allowed to merge until you have a certain level of testing.

And then, oh, branch protection. Yeah, so you don't, like, merge, like, commit directly in main. Like, that's, like, the worst offense is, like, from your developer laptop, you merge directly in main, you, whatever. Potentially, it goes out to production, but you probably, it probably won't because, like, if you don't have CICD pipelines, then you probably wouldn't, like, have branch protection to produce.

To begin with. I mean, I guess you could. Like, if you know CICD pipelines, I don't know why you wouldn't have branch protection, but, like, yeah. These are just all kinds of, like, multi -dimensional guardrails to, like, steer and kind of, like, put your agent on rails and go in the direction that you want it to go. Absolutely.

What do you think the line is between AI -assisted development and AI takes action in prod? Ooh, yeah. Like, so, I've done it myself for my little test app, and, like, it's... Like, I don't care. Like, I've got node users right now. Like, I can let it do stuff in my AWS account and it solves things pretty quickly. Like, it's pretty nice to have it give it, like, access to my AWS account.

It looks at all the logs and, like, honestly, I'm not that, like, I'm not great at AWS. Like, I don't know all the terminology or even whatever, but, like, I'm deploying this app with Terraform into multiple AWS accounts and I have, like, basically a...

Dev AWS account and prod AWS account and I'm kind of like segregating it in that way and yeah if I have like some issue like it didn't this was an issue that was having was it wasn't cleaning up so it deploys as a lambda it whatever tears down it wasn't tearing everything down it was the ENIs were sticking around for some reason and I was like all right just look at the AWS account and see like what why were they

Sticking around and it uh made some like github action like cleanup uh job that like ran afterwards to like make sure everything was cleaned up so uh again like I'm this is some people might scoff at that people may be like oh you don't actually know the real cause or whatever I'm like well but like the agent will know the real cause and like I'm kind of delegating that like you wouldn't know the real cause of like I

Don't know if you're a senior developer and then your junior developer on your team does something and it works, like, yeah, you can kind of see how it works, but you weren't the one writing the code, right?

Like, ultimately, we're delegating some amount of responsibility to other developers. And this is just one more step. We're delegating to a developer on our computer. So I feel like people have this.

Sense of they're losing control or they're losing like like they're almost like identity like their sense of importance as a developer of just of knowing things if they delegate it out they're like what do I do you just get up leveled right like you you're you're now a manager like you you you are now responsible you are still responsible for the quality and what comes out like you're still the one approving pr

Merging it so of course you have to like kind of know what it looks like you you have to kind of know what's going on but I don't know for my small little test app.

Like, yeah, I don't really know what's going on, but like for, so assisted development, totally. Everyone should be using it for touching prod. Yeah. Depends like maybe read only potentially.

Like if, if, if it can give you the logs and can say, given like you give it just a read only I am role say, and it can read all the logs, you read the events, it can read through the traces and like, it can come up with the answer probably much quicker than you can. Like.

Like as I was, I was doing this exercise, the other, not exercise, but like I caught myself reading through a Jenkins log and trying to like trace through like my, uh, like the team's GitHub and the shared library and like, where is this coming from? And I'm like, oh wait, I can just throw it to quad and like quad can figure it out for me like much quicker.

Whereas that would have taken me half a day potentially before. So I feel like the, the habit to build is. All right, how do I use AI first? Like with this thing, like, how do I, how do I throw clot at this thing? And then if it can't, if it's having issues or you can't figure it out, all right, maybe I need to like step in and whatever, do it the old way. But like, that's always like the fallback option. Right. So.

Yeah. I've worked at companies where you've set up MCP servers that would have read -only access to Argo CD, Cube CTL, just to be able to get a. You know, maybe Datadog logging, if there's logs there that need to be, or Sumo Logic or whatever else.

And then it just compiles all that information, looks through it, and can give you a quicker answer than you can by SSH -ing into a box and looking, jumping around, looking at it. Yeah, poking around. So yeah, I think there's 100 % merit to that. Using AI to help figure out root cause is important. For sure. And to your earlier point, you know, back in the day, if you didn't know why...

A network interface was still existing in AWS, you would probably try to read through the docs or reach out to AWS support. And all you're doing is streamlining that process. You're still, it's still the same net result, but you're, and if you do go to AWS support now, you'll probably get an AI -assisted response anyway, right? So. Yeah. Right. That's funny.

But I think the interesting thing is like people are afraid of responsibility. They're afraid, like this is a point that. That's fair.

Again, like, stevie they bring this up is like you're you're now the head their analogy in that book is you're a head chef in a kitchen full of agents like you're of agents as chefs like that you're not making every little single piece of the ingredient for the the dish but you're ultimately responsible those are your michelin stars on the line if something go bad goes out to a customer right so somehow even like

With just we have regular software development organizations, entire teams are kind of built on kind of this delegation of responsibilities, right?

The manager managing a team isn't reading, like isn't coding the code, right? Like they may be reading, they may be in the PRs, they may be like, whatever. But like the less, like I would say an ideal manager is technical, but like they're not like, they're not like a super engineer, right? Like they're not the best engineer on the team. Like they're delegating that responsibility to their team.

So, and I would say managers have been vibe coding this whole time and we're just now like ICs are now like having to deal with that kind of delegation or responsibility, like delegation of like implementation, but still having the responsibility of making sure it's good. So, yeah. Does that, does that make sense? No, it's a hundred percent. Yeah.

And I love that analogy about being like a restaurant or being chefs and then being like your sous chefs or your. Your assistants. I think that that makes a lot of sense. So embracing AI is important for a hundred percent. Now more than ever too, because if you're not, then you're behind, like you're going to get outpaced by someone else that is. Yeah. Potentially outperformed. Yeah.

Like that's where, I mean, we haven't seen that yet, but like it may be coming, like it may be the performance reviews this, if not this year, next year, like just like getting really like granular here. Like if you are, If someone on your team is using AI, they may be putting out like 10 times, potentially, these are, you know, round numbers, orders of magnitude, whatever, two, five, 10 times more work than you do.

And then come performance review time, like maybe not this year, it's kind of early still, but like next year, maybe you don't get a promotion because somebody on your, even though you may think you deserve it or whatever, someone on your team is. Using AI, embracing it. They're a year ahead of you in learning how to use it. It's going to be tough to catch up. Yeah. Anyways.

So wrapping up, given that, the talk of AI, is there any hype take about AI and engineering that you want to kill? Killing a hype take. Interesting. I mean, what haven't I, like with the raining on people's parade about playing house or whatever, but I don't know. Let's see. I think so. Okay. People.

Like to be smug and say I'm gonna end up cleaning up your slop code in whatever six months to a year or whatever I'm like bro nobody's gonna hire you if you're gonna insult like what they did like if they built if they vibe coded an entire product whatever that was successful enough to make money to then have to hire somebody to help clean up the code they're not gonna hire you yeah who's who like posting openly on

Linkedin has a whole x feed of like ai vibe coding hate they're not going to hire you and I think people this is again like the the identity thing is so like kind of wrapped up in this and I think that's kind of underneath all of this is that people want to feel smart and they and they are smart like they're for sure like they're all these people are probably well more qualified at coding than I am it's just I'm

Willing to operate at a whatever I'm willing to give that up like maybe because I'm not great.

I'm not like, I haven't, I've never been like the best developer or whatever, but I'm like, I see how these systems connect together. And like with being DevOps, we glue all these things together. We're at kind of the crux of all these systems interacting and testing and building and whatnot, deploying. I can, I'm willing to say, all right, I don't really enjoy that whole like coding process anyway.

Let me just take a step up and like kind of connect all these systems together at a.

Whatever higher level like I know that I hate that term like that makes me feel like high and mighty or whatever but like it's a level up the stack like people aren't manually coding and assembly anymore people use the high level language of C and people were like oh you're never gonna like know what the what the actual bits are and like we had this whole like that that was the same conversation happened then right So I feel like people being smug online and like having these AI wars or whatever, I take advantage of it.

I like to be a little edgelord online, being pro vibe coding or whatever. But then people, whatever, come by and be negative. And I'm like, thank you for the engagement. But yeah, people aren't going to hire the smug people online who are anti -AI, who have successfully vibe coded applications. So, okay. So wrapping up, you had mentioned the vibe coding books, Gene Kim, Steve Yegge.

Um, is there anything else you'd like to leave our listeners with? I have a YouTube channel where I live stream. It's called Enterprise Vibe Code. I live stream myself, vibe coding, just kind of building in public type of thing. I do it most mornings. I'm a morning person. I get up early, weirdly enough, and people from around the world join the stream and jump in the chat and talk.

So yeah, I ended up talking more than building, which is fine by me. Like I'm whatever, like it's just interesting to, yeah. Recently started that. And I think people are enjoying kind of like talking with other people about this thing and like trying to get a, get a grasp of what is possible because there is no manual, right?

Like nobody, like this is, this is the closest thing we have to a manual and it's not even a manual. Like there, there's no like specific, uh, way to do it or specific way to teach it other than just do it. Right. So, so I, I, I do it for other people in front of on live and hopefully people get something out of that. Yeah, it seems like it's been resonating too. I looked at your channel.

We spoke a little bit before we started recording, which is awesome. So check out Enterprise Vibe Code on YouTube, Mike Lady on LinkedIn, and I'll have all the information and everything we talked about in the show notes. Thanks for coming on, Mike. Really appreciate it. Awesome. Thanks for having me, Brian. Appreciate it. All right, that's my conversation with Mike Lady from Enterprise Vibe Code.

My biggest takeaway is his framing that guardrails aren't process for process sake. They are what makes speed real. Without them, you are just shipping faster failures. And with AI in the mix, that matters even more. AI can write code and even fix builds, but it will absolutely take shortcuts if you let it. So branch protection, CI quality gates, and sane deployment paths. Thanks for listening, and I'll see you later this week.

This week feels like the platforms are getting opinionated. Not here's a new feature, more like here's the new default way you're going to operate. And if you don't notice the default changed, you'll feel it later. Usually as surprise work, surprise risk, or surprise downtime. What's up, everybody?

I'm Brian, and this is Ship It Weekly, a short weekly show where I filter the noise and focus on what actually changes how we run infra. Quick update before we jump in. Ship It Weekly is officially a video podcast now. Every episode going forward is available on YouTube in full video format. If you're audio only, nothing changes. Same feed, same cadence, and same show. Also, OnCallBrief .com is live.

That's my system for tracking DevOps and infra news without drowning in tabs. Briefs start on Sunday to get refined throughout the week, and Thursday is the final pass. If you want to see what I'm tracking before the episode drops, that's the place. And Teller's Tech now has a sub stack too. If you want episodes and weekly briefs delivered to your inbox, that's the easiest option.

And one last thing, I'm starting another round of interviews. If you want to come on and talk a migration, a post -mortem, a weird outage, or how your team actually runs production, hit me up on shipitweekly .fm. All right, let's get into it. Five stories for today. GitHub is putting agentic workflows directly into actions that changes what CI can do without a human driving.

Gen 2 is moving away from GitHub to Codeburg. It's a reminder that the Forge is not neutral infrastructure anymore. Argo CD upgrades are forcing server -side apply in certain paths. This is one of those small line in the notes, big day in prod things. AWS Config expanded coverage again, which sounds boring until you realize governance scope can move under your feet.

And AWS enabled nested visualization on virtual EC2 instances, a capability unlock that's going to change what people attempt to run. Then we'll do a lightning round and a human closer on the gap between AI everywhere and incidents still being painful. First main story, GitHub agentic workflows in actions. GitHub dropped agentic workflows into technical preview.

The simple version is you write the intent in Markdown and an agent runs inside GitHub actions to do repo work. It's not just run these steps, it's more like here's the goal, go handle it. Think issue triage, basic repo maintenance, investigating CI failures, and proposing fixes. This is not the same thing as Agent HQ. Agent HQ was agents in the GitHub experience. This is agents inside your automation engine.

That's a very different place to add intelligence. So why does this matter? Actions is where a lot of secrets and permissions live. It's also where small automations quietly become core production workflows. When an agent can take actions, not just produce suggestions, you've created a new right path. And right paths have two roles. One, they need ownership. But two, they need constraints. Because the failure mode isn't always malicious.

The failure mode is helpful, fast, and wrong the scary version is an agent editing workflow files because workflow files are basically the keys to the kingdom or an agent doing cleanup that breaks a dependency you didn't even remember existed or an agent repeatedly retrying something until it finds a path that works but violates policy the first time an agent edits a workflow it's a different game So do this Monday.

First, inventory what actions can actually do today. Don't start with features, start with permissions. Which repos allow workflows to write back to the repo? Which repos allow modifying workflow files? Which workflows can deploy, publish artifacts, or touch environments? Second, separate read -only automation from write automation. Agents can comment and propose, sure, but merges should require a human.

That's the clean line that prevents a lot of pain. Third, lock down permissions in workflows. If you are still running broad default tokens, you need to fix that now. Treat write scopes like production credentials, because that's what they are. Fourth, add explicit approval gates for anything high leverage. Secrets, deployments, runners, workflow file changes. If an agent can do it, a human should approve it.

At least until you trust it and you've tested the blast radius. Fifth, logging. You need a record of what the agent changed and why, not just standard out from the workflow. Because the post -incident question won't be who clicked it, it'll be what sequence of actions did the agent take. Okay, that's story one. Story two is Gentoo moves off of GitHub to Codeburg.

So Gentoo is moving repositories off of GitHub and towards Codeburg. They are pretty direct about why. They're uncomfortable with the direction GitHub is going around co -pilot and AI pressure. This isn't a rage quit. It's an operational migration. Mirrors, workflow changes, community adjustments, all of the annoying real -world stuff. So why does this matter? Because GitHub isn't just a place where code lives.

It's your auth story. Your CI story, your review flow, your issue tracker, your release automation, your dependency and security alerts. So when the platform changes incentives or defaults, it's not a cosmetic change. It alters how work gets done. AntGen2 is basically saying, we want leverage and we want options. Even if you don't care about the politics, the engineering lesson is real.

If a platform becomes your default for everything, you don't notice lock - until you have to change something fast. And then you realize half your workflow isn't portable. Not the code, the workflow. Mirrors aren't boring. Mirrors are exit ramps. So do this Monday. Pick one repo that matters. Map the dependencies. What breaks if GitHub is down for a day? Not developers are annoyed. What breaks operationally?

Builds, releases, packages, security workflows, required checks, even how do we coordinate changes? Now, ask a better question. What do we need to keep shipping if the forge is degraded? Do we have mirrors? Do we have backups of more than Git objects? Issues and PR metadata matter. Release artifacts matter. Actions, config, and required checks matter. If you've never tested Restore, do a small exercise.

Pretend the repo is gone. Restore it and run the minimum path to ship. You don't need a full migration plan. You need confidence that you are not trapped. Okay, that's story two. Story 3 is Argo CD upgrades and server -side apply requirements. This one is pure operator reality.

Argo CD has an upgrade path where server -side apply is required in certain setups, especially when Argo is managing itself or when you are applying manifests directly. The reason is a Kubernetes annotation size limitation. The last applied blob can get too large and it breaks apply behavior in weird ways. So the fix becomes use SSA. Let the API server own field management. So why does this matter?

This isn't just a flag you add to a command. It changes how ownership works. SSA tracks fields differently. And that means upgrades can overwrite things you didn't realize you were relying on. This is where hidden customizations come back to haunt you. The little prod -only patches. The tolerations someone added once. The probe tweak that never got upstreamed into your real config.

Upgrades are where tribal knowledge gets erased, or worse, half -erased. And Argo is a special kind of risky upgrade because it's your deploy system. If Argo is down, you don't just lose a tool. You lose the safe path to change. Then everyone starts doing manual kubectl, and Drift shows up immediately. GitOps is calm until the GitOps system is the incident. So do this Monday. First, look at how you deploy Argo today.

Is Argo managing itself? If yes, Verify SSA is enabled in the application sync options before upgrading. Don't wait to learn this during the upgrade. Second, diff your live Argo resources against what you think you apply, and then find the hand edits. Find the temporary patches, write them down, and formalize them. Third, build an upgrade lane for Argo, even if it's small. A rehearsal environment.

Same method, same manifest, same shape. Practice upgrade and rollback, and validate can we sync a known app after upgrade. Fourth, rehearse Argo as down mode. How do you deploy without it? How do you stop it from fighting you if it's partially alive? How do you get back to a known good state? Because when Argo breaks, every minute feels expensive. And thinking clearly gets harder. Okay, that's story three.

Story four is AWS Config adding 30 new resource types. So AWS Config just added support for 30 additional resource types. Here's the key detail. If you record all resource types, Config can start tracking new types automatically. So Scope expands under you without you doing anything. This is a quiet change. But it affects inventory, governance, and sometimes cost. So why does this matter?

Most teams don't treat Config like a living system. They treat it like a checkbox. Then one day they try to get serious about governance and compliance, and they realize Config is actually foundational data. So when coverage expands, that's good. But it also means more evaluation surface. More resources showing up in aggregators. More roles being evaluated. More non -compliant noise.

And the worst failure mode is not config has data. It's config has data nobody owns. New resource types show up. Roles fire. Nobody knows who should fix it. So it becomes a platform team problem by default. And platform teams get buried in triage instead of improving systems. Inventory expansion is great until it becomes surprise accountability. So what I would do Monday, go check your config recorder settings.

Are you recording all resource types? If yes, decide if that's intentional or if it's just we clicked it once years ago.

Then check your roles which rules will start evaluating these new types tagging roles encryption rules public access rules all the stuff if new types will create noise decide routing who owns the alerts who owns remediation also baseline your config usage and costs not because this change will wreck your bill but because it changes the scope and it's easier to explain early than late Finally, tighten your ownership metadata.

If you can't answer who owns this resource, governance becomes a scavenger hunt. That's story four. Story five is EC2 nested visualization on virtual instances. AWS now supports nested virtualization on certain virtual EC2 instances. Historically, nested virtualization on AWS was usually a bare metal story. Now it's possible on virtualized instances for some families. This is a capacity unlock.

It's also a behavior unlock because the moment this exists, teams will attempt things they couldn't justify before. Full lab environments inside EC2. VM heavy testing. Security sandboxes. Let's run a hypervisor inside our runner fleet. So why does this matter? Nested virtualization sounds niche, but it's really about reproducibility and isolation.

If you've ever wanted a test environment that looks closer to prod, this helps. And if you have tooling that expects a hypervisor, this helps. And if you were paying the bare metal tax purely for nested vert, this might be a cost lever, which is going to matter for some orgs. But it's also a foot gun if you combine it with credentials and loose network controls.

Anything that starts looking like a workstation gets treated like a workstation. People install random tools. People store secrets in the wrong places. People run just this one thing. And that's why anything that looks like a workstation eventually gets treated like one. So do this Monday. If you have runner fleets, build fleets, or sandbox accounts, add this to your threat model.

Ask what changes if nested VMs become available. Then set boundaries. Which accounts allow it? Which VPCs allow it? Do you want tighter egress controls? And then document and internal stance. Even a short note helps. Like, we support this for these use cases on these instance families with these guardrails. Because if you don't write the rules, you will end up inheriting random experiments.

And then you'll learn about them during an incident or a bill review. Okay, that's story five. Okay, time for the lightning round, short and practical. GitHub updated their status page experience. There's now a 90 -day historical view and better linking between incident days and availability trends. And honestly, given GitHub's hiccups lately, having a status page that's actually useful is a welcome addition.

OpenBuild Service published a post -mortem on a disruption that came down to database migration and locking behavior. It's a good reminder that migration plan is not the same as rollback plan. And a quick reminder because we covered it already, GitHub Actions extended the self -hosted runner minimum version enforcement window. Treat that as runway, not permission to ignore it.

If you have self -hosted runners, schedule the upgrade work. Another quick GitHub one, Actions had early February updates around things like runner controls and settings that reduce surprise drift across orgs. It's not headline news, but it's the kind of incremental improvement that saves platform teams time. AWS config expanding coverage is also a reminder of a bigger pattern.

A lot of discover everything services expand under you as AWS adds new stuff. That's not bad. It just means you need ownership or the tool becomes noise. And if you are experimenting with agentic workflows, don't skip the boring part. Permissions, approval gates, and audit trails. That's the difference between useful automation and mystery automation. Okay, that's the lightning round. Time for the human closer.

There's a post called Lots of AI SRE, No AI Incident Management, and it nails something that feels obvious once you say it. Most AI tooling in ops is aimed at producing output faster. Write the YAML, draft the runbook, summarize the log, generate the postmortem doc. That's useful, but it's not the core pain during a real incident. Incidents aren't mostly writing. Incidents are uncertainty. What changed? What's real?

What's correlated versus not? And incidents are coordination. Who's driving? Who's communicating externally? Who's making the rollback call? And how do we keep the team aligned when five things are happening at once? That is still wildly human. And honestly, that's what makes on -call exhausting. Now tie this back to today's stories. We are putting more automation into the workflow.

And in some cases, we are giving it more agency. Agents in actions deploy systems that can strand you mid -upgrade. Governance tools that expand scope automatically. And all of this increases the number of things happening around incidents. So if those tools don't reduce uncertainty, they can increase chaos. The win is not faster output. The win is less uncertainty for tired humans. If AI can help, great.

But the bar is, does it help you decide what to do next safely? Does it tell you what it's unsure about? Does it show you what it tried and ruled out? Can it give you an explanation you can trust at 3 a .m., not just a confident guess? So my take this week is simple. When you evaluate tooling, don't judge it by how clever it sounds. Judge it by whether it reduces uncertainty when you are on call.

Because that's the moment that matters. That's where reliability is real. Okay, time for a recap. Today we talked about GitHub agentic workflows and actions, and how it's not just nicer CI, it's a new right path that needs guardrails. Gen 2 moving towards Codeburg, Forge choice is supply chain, governance, and leverage, not just convenience. Argo CD upgrades requiring SSA in certain paths.

Control plane upgrades deserve their own lane and rehearsals. AWS Config adding 30 new resource types. Great coverage, but scope can expand under you, so be intentional. EC2 nested virtualization on virtual instances. Capability unlock, and also a new what -will -teams -attempt -now moment. The lightning round was around some GitHub stories and OpenBuild service publishing a postmortem.

If you want the video version, full episodes are now on YouTube going forward. If you want the weekly briefs, OnCallBrief .com is live. And if you want everything delivered by email, Teller's Tech Substack is up. And lastly, if you want to come on the show for an interview, reach out at ShipItWeekly .fm. More episodes, links, and show notes are on ShipItWeekly .fm. All right, I'm Brian, and I'll catch you next week.

Picture this. You're on your laptop, coffee in hand, doing the normal start of day shuffle. Slack, email, calendar, Jira, whatever. And you click a link. Not a sketchy link, not free crypto nonsense. It's a link that looks like documentation. Or a GitHub issue.

Or a skill page or even a link your agent surfaced for you because it was helping and that one click is enough to hand over the keys to the thing you've been testing a local ai agent that can run commands read files hit apis and act on your behalf not in the cloud not in a sandbox you forgot existed on your actual machine with your actual credentials that's what we're talking about today Hey, I'm Brian from Teller's Tech, and this is Ship It Weekly.

Be sure to visit shipitweekly .fm for all of the show notes and past episodes. Today is a special. No lightning round, no normal format. This is a special episode on OpenClaw. OpenClaw isn't just another AI tool. It's a preview of what a lot of us are about to be dealing with at work, whether we like it or not. Local agents, real tools, real credentials, real consequences.

And OpenClaw is basically the clearest case study we've had so far for what happens when autonomy meets messy reality. Security boundaries, plugin ecosystems, web UIs, token handling, and humans doing human things. All right. Let's set the stage. If you somehow miss the hype, OpenClaw is an agent platform you can run locally. The pitch is simple. AI that actually does things. Not just chat, not just suggestions.

It can connect to your tools and take action. It can manage your calendar, triage email, message people, hit APIs, automate little workflows. The kind of stuff we all hack together with scripts. Except now, it's driven by a model that can reason through a task.

And the locally part is what made it explode because a bunch of people are sick of handing their entire inbox, calendar, or internal docs to some random SaaS agent. Running it on your own machine feels like control. And look, I get the appeal. It's the same reason we like local dev environments, self -hosted runners, internal tooling. You want ownership, you want knobs, and you want to see what it's doing. But here's the thing.

When you run an agent locally you're not just running software you're running an operator a little automation brain that wants to access everything files tokens browsers ssh cloud clis whatever you let it touch so the question becomes what are we actually building here because from a devops and sre lens this is not an app this is a control plane and control planes have two roles one they always end up with scary permissions because otherwise they can't do the job.

And two, they always end up becoming the target. And over the last few weeks, OpenClaw basically speed ran both of those roles. Here is the situation in one line. People installed local agents.

Gave them real access and the ecosystem immediately started getting hit like it was npm plus a browser admin panel plus remote management tool which yeah because that's what it is so you had multiple things happening at once you had a serious vulnerability story where a web ui plus token handling plus browser behavior created a one -click path to take over You had a marketplace story, where skills or extensions turned into a malware delivery channel.

You had the usual people expose local admin things to the internet story, because of course they did. And then on top of that, you now have the meta story, the creator getting hired by OpenAI, which is basically a signal flare that the big players are going all in on agents. So it's not just, wow, this one tool had issues. It's the entire category is now real. And we need to talk about how we treat it like adults.

Because if your engineers start running agents on workplace machines with access to AWS, GitHub, CLI, and maybe even payment methods, you don't have a toy problem. You have a production security problem that just moved onto laptops. Let's talk about the vulnerability angle without getting into exploity details.

The headline version is, there was a high severity issue where a crafted link could result in token leakage and then gateway compromise, leading to remote code execution. And if you are thinking, wait, how does a link do that if the agent is local? That's the important part. A lot of people hear local and their brain goes, cool, so it's behind local host, so I'm safe.

But local host is not a magical security boundary. It's a networking convenience. And the browser is the ultimate helpful idiot in security stories. It will happily make requests from your machine to other places on your behalf. So if you have a local control panel in your browser, and that control panel can talk to a privileged local service, you have tokens involved.

Congratulations, you have reinvented a whole class of web security problems. This is the part where DevOps folks sometimes roll their eyes because it sounds like front -end security. But it's not front -end security, it's admin plane security. It's the same category as an internal Kubernetes dashboard, a Jenkins UI, a self -hosted GitHub runner with a web panel, a secrets UI, or an Argo UI.

If it can trigger actions, it's an attack surface. And with OpenClaw, the core lesson isn't, wow, they had a bug. The lesson is agents collapse trust boundaries. Because the UI isn't just showing you data, it's holding the steering wheel.

So if a browser session can be tricked into handing over a token or connecting somewhere it shouldn't or trusting something it shouldn't, the impact is way bigger than someone saw a page. The impact is you just handed someone an operator account for something that can execute on your machine, which is basically the worst version of developer laptop compromise. Because now it's not even a human making the decisions.

It's an automation system that can be nudged. So practical takeaway here. If you are running OpenClaw or anything like it, you need to patch fast, obviously. But also, stop thinking local equals safe. Local just means the blast is on you first. And that sounds dramatic, but it's true.

The minute you have a privileged local service plus a browser UI plus tokens, you are in the same design space as a mini control plane. So you need to treat it that way. Now, let's talk about the part that will feel extremely familiar to anyone who has lived through supply chain headaches. OpenClaw has skills. There are extensions, add -ons, whatever you want to call them.

And there was a wave of malicious skills showing up, including hundreds flagged in reporting. This is the oldest story on the internet. A popular platform shows up, a registry shows up, a bunch of people install things because, hey, it's open source and the community is building cool stuff. And attackers go, oh, sick, an executable distribution channel with confused users. The difference here is the payload.

With normal package ecosystems, malware tends to be about stealing tokens, crypto, SSH keys, browser data, that kind of thing. With agent ecosystems, the malware doesn't just steal, it can also steer. Because skills don't just sit there. They influence what the agent can do, what it can access, and what kinds of actions it will take. And the social engineering is painfully predictable.

It's stuff like, install this prerequisite, run this command real quick, paste this into your terminal. If you've ever watched someone get popped by a fake homebrew tap or a sketchy curl pipe bash, you already know the vibe. Now, layer in the agent angle. The agent is reading markdown. The agent is summarizing pages. The agent is trying to be helpful. So the attack surface becomes any content the agent consumes.

Not just who can message it, but the content itself. That's a weird shift, and it matters for how we build controls. Because an agent can be tricked through an email it reads, a doc it summarizes, a ticket it opens, a website it fetches, a pastebin it looks at. And if the agent has tool access, the question becomes, can that untrusted content cause tool execution? If yes, congratulations.

You just made reading the internet equivalent to running code unless you build a guardrail. This is why the OpenClaw story matters to DevOps more than most AI hype. It's not about AI is coming. It's about we just added a new automation surface where content can turn into action. And that's a big deal. So this is the part I actually care about. Because tools come and go.

OpenClaw could disappear tomorrow and the core problem stays. The core problem is autonomous agents are becoming a new class of privileged workflows. Except the workload is running on somebody's laptop, or in random VMs, or in somebody's home lab, or eventually in some sanctioned internal deployment. And it has access to things we normally treat as high value.

Cloud credentials, source control, CICD, secrets managers, internal APIs, sometimes payment methods because people are wiring these agents into subscriptions or usage -based services, or yeah, even credit cards for auto purchase type stuff. So let's reframe it in SRE language. An agent is an operator that accepts untrusted input. An agent is an operator that can take actions.

And an agent is an operator that is very hard to reason about because its decision engine is not deterministic code you wrote. It's a model that can be influenced. So what do we do with operators? We reduce permissions. We isolate environments. We add approval steps for dangerous actions. We add audit logs. We set boundaries like egress controls. We separate duties. We rotate credentials. And we monitor.

We run it like production. And the reason this is tricky is because a lot of people are approaching agents like a productivity app. They are treating it like installing Notion. But it's closer to installing a junior admin who never sleeps and can be convinced by a well -written paragraph. So here's the mindset shift I want you to take away from this episode. If your agent can run tools, it is infrastructure.

If it touches credentials, it is privileged infrastructure. If it reads untrusted content, it is exposed infrastructure. And it needs controls that match that. Which leads me to the next point. Most orgs are not set up for this, culturally or technically, because we've spent years building guardrails around CI and prod.

We've spent even less time building guardrails around laptops, especially when the laptop is now running a local control plane. So we need a minimum viable safety approach. Not perfect, not academic, just don't be reckless. So let's keep this practical. If you are experimenting with OpenClaw or any local agent framework, here's the bar I'd personally want, even just for tinkering.

First, don't run it on your main machine with your main creds. I know, I know, everybody does it because it's convenient. But if the agent needs AWS access, you need to give it dedicated AWS identity that is scoped down. Separate account if you can, or at least separate role with tight permissions, short -lived tokens, and no administrator access because I'm just testing. Same idea for GitHub. Same idea for GCP.

Same idea for anything. Second, you need to separate the agent's environment from your daily environment. A VM is fine. A separate machine is better. A separate user account is better than nothing. The real point is to avoid agent compromise equals my whole dev life is compromised. Because a lot of the stuff you actually care about is sitting right there on your laptop.

SSH keys, browser sessions, cloud CLIs, kube configs, slack tokens, password manager sessions, all of it. Third, don't expose the control interface. And if you do, don't do it casually behind some reverse proxy you copied from a blog. This is the I put it behind engine X so it's fine trap. If it's an admin plane, it needs real auth, origin controls, and it should not be discoverable from the public internet. Period.

Fourth, treat untrusted content like a biohazard. If your agent is reading emails from the open internet, browsing the web, or pulling in random docs, consider a split agent approach. One agent that is read -only, whose whole job is summarizing untrusted content. Then, a second agent that has tools, but only sees the summary, not the raw content.

That might feel annoying, but it's basically the same concept as don't run customer input through the same context that can trigger production actions. Different zone, different trust level. Fifth. Approvals. If your agent can do anything destructive, require explicit approval for those actions. And I don't mean it prints a message and asks nicely.

I mean a real control point, a config that says these tools require confirmation. A workflow where send money, rotate keys, delete resources, merge PRs, apply Terraform, all require a human step. And here's the important nuance. Approvals should be tied to action classes, not tied to do I trust the agent. Because the agent can be tricked. That's the whole point. So your controls can't be vibes based.

They have to be structural. Which leads me to the next point. Observability. If an agent is acting on your behalf, you need to be able to answer basic questions later. What did it read? What tools did it invoke? What credentials did it use? What calls did it make? What files did it touch? And if the answer is, uh, it just kind of did stuff, you're going to have a terrible time the first time something goes wrong.

This is where I think DevOps people can actually contribute a lot. Because we already know how to wrap scary automation in safely. We already know how to build pipelines with audit trails. We already know how to treat privileged systems like they're hostile by default. So if your team is adopting agents, push for the boring stuff. Centralized logs for agent actions. Tool invocation logs with arguments.

A clear mapping of agent identity to credential identity. Rate limits because agents can loop. Cost controls because agents can loop and burn money. And a kill switch, always a kill switch. Because an agent that can autonomously take actions is basically a distributed failure generator if you don't contain it. And I'm not saying that to be dramatic.

I'm saying it because every SRE has seen what happens when automation goes slightly sideways. Now, imagine the automation can be socially engineered. Cool. So we need to build with that in mind. Now, the recent update that changes the vibe a little. The creator of OpenClaw got hired by OpenAI. That's not a random headline. That's a signal.

It tells you agents are not going to stay in the cool open source side project lane. The big labs want this. They want personal agents, enterprise agents, multi -agent systems, agent marketplaces, all of it. So even if OpenClaw itself fades, the pattern is here to stay. And as that happens, two things are going to be true at the same time. The tools will get way better.

And the security problems will get way more interesting. Because adoption drives attacker attention. And agents, by design, sit exactly where attackers love to be. In the middle of identity, action, and trust. So the question for us, as DevOps and SRE people, isn't should agents exist? They're going to exist. The question is, do we treat them like production systems, or do we treat them like toys until they bite us?

Because right now, a lot of orgs are about to repeat the same mistake we made with CI systems 10 years ago. Do you remember when Jenkins was just a build box and then suddenly it was the keys to prod? Yeah, agents are going to be that, except faster. Alright, let's land this. OpenClaw is the current headline. But the real story is bigger. Local autonomous agents are a new control plane.

They will end up with real access because otherwise they aren't useful. They will get targeted because that's where the value is. And local doesn't mean safe. It means the consequences start with you. So if you are experimenting with agents, awesome. Just do it like an SRE. Real credentials means real controls. And if you're leading a team, don't wait for a policy meeting in six months. Get ahead of it.

Decide what safe experimentation looks like in your org before everyone quietly installs an agent and wires it into prod stuff because it's convenient. All of the links and references for this episode and the show notes are on shipitweekly .fm. If you got something out of this, a rating or review goes a long way and it helps other folks find the show. I'm Brian from Tellers Tech and see you next time. Thanks.

This week is basically a tour of the safety system became the incident. A protection rule that made sense once, then quietly started blocking real users. A Kubernetes permission that everyone treats like read -only, but it absolutely is not. A platform example that actually got the control plane versus data plane separation right.

And compliant scope expanding in a way that is easy to underestimate until you are drowning in evidence requests. So yeah, not a week of flashy tech. More like a week of go -check -your -assumptions. Thank you. Hey, I'm Brian, and this is Ship It Weekly. If you like the show, follow or subscribe wherever you are listening. It helps a ton. Everything lives at shipitweekly .fm.

Also, I'm starting another round of interviews. If you'd like to come on and talk about real world ops, hit me up at shipitweekly .fm.

All right let's get into it four main stories for today github reworked layered abuse defenses after legacy rules blocked legitimate traffic kubernetes node proxy git the telemetry permission that can turn into cluster wide rce hcp vault and what actually stayed up during a real aws regional disruption AWS PCI DSS scope expansion and the operational reality of compliance scope changes. Then a quick lightning round.

Then a human closer on reasonable assurance turning into busy work and what to do about it. Some of GitHub's legacy defenses are blocking legitimate traffic. So what happened? GitHub had users hitting unexpected too -many -request errors. And it's not because everyone suddenly got evil, but because old abuse mitigation rules were still active long after the original incidents that created them. So they went back.

Traced it, and reworked how layered defenses are managed, including how those rules get maintained and retried. This is a February 2026 story, and it's the kind of thing every platform team recognizes instantly. So why does it matter? If you ship defensive controls and you don't give them a lifecycle, they become permanent production dependencies. And that's the sneaky part. A security layer is not just security.

It sits on the request path. It can block revenue. It can break signups. It can break API clients. And it can create phantom outages that look like the app is slow when the app is fine. The failure mode is also brutal for teams because when the source of the problem is an old mitigation rule, the people on call might not even know it exists.

There's no muscle memory, there's no runbook, there's no customer complaints, and a bunch of dashboards that lie by omission. Old mitigations are like dead code until they suddenly run your business. So what do you need to do Monday? Do a quick inventory of your traffic safety layers. That could be CDN rules, WAF rules, bot protections, rate limits, edge middleware.

App throttles, whatever sits between the user and your service. Now you need to ask yourself two annoying questions. Who owns each layer? Not the team and actual owner. And how do you disable it safely if it starts blocking legitimate traffic? If you don't have a kill switch plan, you need to make one. Even if it's ugly, even if it's toggle this feature flag and accept higher risk for an hour.

Then add one reliability metric you probably don't track today. False positives. Not in a philosophical way, in a how many legitimate requests did we reject way. Because once your defensive layer starts rejecting good traffic, it's not a security success. It's an availability incident with a security label on it. Okay, that's story one.

For our second main story today, it's Kubernetes node proxy get and the read -only permission that isn't. So what happened? There is a Kubernetes RBAC permission that a lot of orgs grant to monitoring and observability tooling. It looks harmless. It's get on nodes slash proxy. The intent is basically let this thing scrape node metrics, stats, logs, healthy endpoints, that kind of stuff.

But research and write -ups in late January 2026 show how that permission can be abused to reach kubelet permissions and turn into arbitrary command execution in pods. The punchline is simple. A permission that teams treat like read -only telemetry can collapse trust boundaries if you hand it out broadly. So why does this matter? This is exactly how clusters get popped in real life. Not always by some fancy zero day.

Sometimes by a permission that was granted for convenience. Because observability stacks are the classic just give it cluster admin snowball. You start with it just needs to scrape metrics. Then it just needs to list pods. Then it needs node stats. Then you're granting node proxy access because charts and docs tell you to. And the scary part is psychological. Permissions like this don't trigger your gut.

They don't look like exec. They don't look like secrets. They look like plumbing. So nobody thinks to threat model it. Kubernetes RBAC is a minefield because the dangerous stuff looks boring. So what do you need to do Monday? First, go find it. Search your cluster roles for nodes slash proxy. Then list every subject that binds to those roles. Don't stop at Prometheus.

Look for logging agents, APM agents, cluster UIs, anything installed by Helm charts with default RBAC. Second, force the uncomfortable conversation. Which tool truly needs this? Is it using it or is it just granted because it was in the chart? If you truly need it, scope it tighter. Separate service accounts per tool. List namespace access where possible.

And document the justification so six months from now, somebody knows why it exists. Third, add detection where it matters. A lot of teams only monitor Kubernetes API server audit logs and call it a day. But if the abuse path is Kubelet -level behavior reached through proxying, you want visibility into that access pattern too. At minimum, treat node proxy access granted as a review trigger.

That permission should not be invisible. That's story two. Story three is HCP vault resilience during a real AWS regional disruption. Okay, so what happened? HashiCorp published a write -up on how HCP Vault behaved during a real AWS US East 1 disruption. Their control plane experienced elevated HTTP 500s and intermittent panics around 7 a .m. UTC.

But they say customer HCP Vault dedicated clusters maintained 100 % uptime and kept serving workloads. So the management plane had issues while the data plane stayed up. That's the key. So why does this matter? This is the kind of architecture decision that sounds like overkill until it saves you.

Control plane outages are common, not because everyone is bad at engineering, but because control planes are complex and they're multi -tenant and often exposed to weird edge cases. But customers don't care if your admin dashboard is having a bad day. They care if secret resolution fails and their apps stop booting. So that separation matters.

If your management layer being flaky can break your production usage path, you built one shared blast radius. And you see this pattern everywhere. Terraform is down so we can't deploy is annoying. Terraform is down so production can't read config is unacceptable. If admin plane downtime breaks prod reads, you don't have separation. Okay, so what are the action items? Do this for your top three critical systems.

Write down what is control plane and what is data plane. Then answer one brutally honest question. If the control plane disappears for two hours, what still works? Can apps still authenticate? Can apps still read what they need to run? Can you still scale? Can you still recover? Also, check your runbooks. A lot of runbooks quietly assume the control plane is healthy.

They tell you to click here, or run this automation, or use this UI. If the whole point is resilience, right, the control plane is down path too. Even if it's ugly CLI, because the day you need it, you will not be in a calm and well -rested mental state. Okay, that's story three. Story four is AWS PCI DSS compliance package expansion and what it really means for teams.

So AWS announced updates to their fall 2025 PCI DSS compliance package. They added two services to the scope of their PCI DSS certification. AWS Security Incident Response, and AWS Transform. They also added the Asia Pacific Taipei region into the PSI DSS scope. On paper, that sounds like a nice checkbook update. In practice, scope changes have consequences. So why does it matter?

Compliance scope changes do not usually break production, but they absolutely create work. And they create it in the most dangerous way. Slow, distributed, easy to underestimate.

And easy to turn into chaos when audit season hits so here's the trap a scope change lands security is happy leadership is happy then six months later some poor team is asked to reprove a pile of controls except now the region lists changed the service list changed and nobody knows what in scope even means in your org that's how reasonable assurance turns into hours of evidence churn And it's why compliance can feel like it fights delivery, even when it's trying to protect the business.

Audits aren't the enemy. Recreating proof from scratch is. So if this compliance change affects you and you're in a PCI environment, do a quick boundary check. What accounts are in scope? What regions are allowed? What services are approved. Then decide how you will generate evidence, not we'll do it when asked. Pick the repeatable pieces now. A reusable evidence package is the goal.

One place that says what the control is, how it's implemented, and how you prove it. You can point to it, update it, and stop rewriting narratives every time a new spreadsheet shows up. If you want compliance to be sustainable, you have to treat evidence like a product, not like a one -off homework assignment. Okay, that's story four. Now it's time for the lightning round. Here's some quick hits.

GitHub Actions extended the timeline for self -hosted runner minimum version enforcement. Starting March 16th, 2026, older self -hosted runners get blocked if they're below the minimum version. With a brownout period between February 16th and March 16th to help you find the stragglers. This is one of those, it won't be urgent until it's urgent things.

If you run self -hosted runners, go check what versions are actually deployed. Next, Headlamp, the Kubernetes UI project, is now officially part of the Kubernetes SIG UI, and they posted a 2025 highlights recap. The reason I care about this is simple. Kubernetes UIs are finally moving beyond toy dashboard into useful day -to -day tooling.

And teams need sane, supported ways to visualize cluster state without giving everyone kubectl to prod. And last, AWS Network Firewall Active Threat Defense. AWS wrote up how it draws near real -time intelligence from MadPot, their honeypot sensor network, and uses that to detect and block threats faster. Even if you never buy the feature, the idea is worth stealing. Speed matters.

Threat intel that shows up three weeks later is mostly trivial. Okay, that's the lightning round. Time for the human closer. And this week we're going to be talking about reasonable assurance turning into busywork. There's a thread in r slash sre that's basically the most relatable sentence ever. At what point does reasonable assurance turn into busywork? And the vibe is not we hate audits.

It's more like why are we spending engineer time formatting proof instead of reducing risk? That thread pairs perfectly with the GitHub story this week. GitHub had legacy protections that stuck around. They outlived the incident that created them and started blocking real users. That's what happens when a control doesn't have an owner and a lifecycle. Compliance busywork is the same failure mode, just slower.

A control exists. Maybe it's good. Maybe it's necessary. But over time, the evidence requests multiply. The templates change. The risk language changes. And suddenly, the work has not proved this system is safe. It's translate what we already do into 12 different formats. One comment in this thread nails it. If it's consistent and repetitive, automate the documentation. Audits matter. Formatting shouldn't.

That's the move. You don't win by arguing with auditors. You win by making your proof boring. One control. One source of truth. One evidence artifact that stays alive. And then you update it when reality changes, not when somebody sends you a spreadsheet. So my Monday challenge is simple. Pick one control you get asked about constantly.

Access reviews backup validation change management whatever now measure how many engineer hours go into evidence churn for that one control in one cycle if it's painful good you just found your next high leverage reliability project because busy work is not just annoying It steals time from actually making the system safer. Okay, that's the human story. Time for a quick recap.

We talked about how GitHub reworked layered defenses after legacy mitigations started blocking legit traffic. We talked about Kubernetes node slash proxy Git and how it's a permission you should treat like a real threat boundary. We talked about HCP Vault and how it's a clean example of control plane pain not taking down data plane availability. And we talked about AWS expanding their PCI DSS scope.

And that's your reminder to productize compliance evidence before it becomes chaos. And the human story was reasonable assurance turns into busy work the moment you are just formatting proof. If you like the show, follow or subscribe wherever you're listening. And everything, including the show notes and links, will be on shipitweekly .fm.

And if you want to come on for an interview round, reach out at shipitweekly .fm. I'll catch you next week. Thanks.

This week felt like AI went from cute helper to, okay, this is now part of the platform, not just in your IDE. It's showing up in CI, incident response, and in the places where mistakes become expensive. Also, Azure had one of those outages that doesn't just break a service. It breaks your ability to operate the service. And that one always hits different. Welcome back to Ship It Weekly.

I'm Brian from Tellers Tech, and this is Ship It Weekly, the DevOps and SRE news show for people who actually get paged. All links are in the show notes on shipitweekly .fm. And a quick ask up front. If you're getting value out of the show, hit follow or subscribe wherever you listen. And if you're on Apple Podcasts or Spotify, a rating helps a lot. Even a quick review is huge for discoverability.

All right, five main stories for today. One, the Azure control plane incident and why it creates the most annoying type of failure. Two, GitHub leaning harder into multi -agent dev with Claude and Codex in the loop, plus actions getting a little less painful. Three, Claude Opus 4 .6 and the real difference between better model and new failure mode. Four.

Google talking openly about SREs using Gemini CLI during outages. Five, the connective tissue story, MCP as plumbing, and observability getting pulled into the AI platform gravity well. Then we'll do a lightning round. Then a quick follow -up on Ingress Engine X and N8n since we covered those in previous episodes.

And we'll close with a human closer around trust, guardrails, and why - why agentic work is not a free lunch. Story one, Azure incident. VM management ops gets weird and that ripples everywhere. So what happened? Azure had a platform incident that impacted VM management operations across multiple regions. So not a VM went down. More like the thing that manages VMs is degraded. Create, update, scale, start, stop.

The day two operations that every platform team builds automation around. And when that gets weird, you feel it everywhere because it's not just VEBS, it's auto -scaling, it's instance group updates, it's node pools trying to heal, it's pipeline steps that assume I can always start a node or I always roll a deployment. So why does it matter?

This is the outage category I hate the most because it creates misleading symptoms. Your app might look up. Your load balancers are still answering. Your metrics are fine until they aren't. But deploys start hanging. Scale -outs don't happen. Rollbacks don't apply cleanly. And you end up in this gray zone where everyone is arguing about whether it's our code or the cloud. And here's the key point.

Most teams build reliability thinking about the data plane, requests, latency, error rates, but control plane failures break your ability to respond. If you can't change the system, you can't mitigate. And it turns a 15 -minute incident into a two -hour incident purely because you can't operate your own platform.

And if you've ever been stuck in a loop where you're trying to fix something and all your fixing tools are failing too, that's what this is. So what do you do Monday? Go look at your automation paths and ask a boring question. What happens when the control plane is slow or degraded? Do you have retries with exponential backoff? Or do you have hammer it until it works? Do you have timeouts that fail safely?

Or do you have jobs that hang forever? Because a lot of people accidentally build infinite retry storms into their pipelines. And the control plane is exactly where that becomes self -inflicted pain. Also, do you have a decision point in your incident process that says, stop trying to change the system, stabilize, communicate, wait for the provider? That's hard to do because it feels like giving up.

But sometimes it's the correct move. And one more thing. If your response plan assumes you can always add capacity or always recreate nodes or always scale out, make sure you have a mitigation that doesn't require that. Sometimes the right move is degrade features instead of add servers. Sometimes it's shift traffic instead of rebuild. And you only learn that when you talk through it before the outage.

All right, that's story one. Story two is GitHub leaning into multi -agent dev. And there's a bunch of practical implications for platform teams hiding under the surface. So story two is GitHub multi -agent in the workflow, plus actions getting less painful. What happened? GitHub rolled out AgentHQ support for using Anthropic Cloud and OpenAI Codex in GitHub and VS Code alongside Copilot.

So instead of co -pilot only, it's more like pick the agent that's best for the job. And GitHub Actions shipped workflow updates too, including a case function in Expressions. If you've ever had a workflow where the logic is basically a pile of nested ands and pipes, you probably just felt your shoulders drop a little. So why does it matter? Two reasons. First, multi -agent is an emission of reality.

People are already switching between tools. Cursor. Copilot, Claude Code, whatever. Different models are better at different things. Some are better at refactors. Some are better at reasoning through messy requirements. Some are better at quick code completion. GitHub putting multiple models in the same surface is them trying to become the control plane for AI -assisted development.

And if you are a platform team, that matters. Because the moment these tools can open PRs and modify workflows, they are part of your software supply chain. Second, actions changes matter because CI is still where a ton of production risk lives. Workflow changes are not just developer productivity. They are permissions. They are secrets. They are build artifacts. They are deployment paths.

So if you are enabling AI agents in that world, you should treat it the way that you treat any automation. Not cool, ship it. More like cool, what's the blast radius? So what do you do Monday? If you want to test agent PRs, do it like onboarding a new engineer. Start with read -only or low -risk tasks. Have it summarize failing CI. Have it draft release notes.

Have it propose a change and explain why, but don't let it merge. And keep it out of workflow files and infra repos until you've decided what your governance is. Because workflow files are basically production code. Now, about that case function. That sounds tiny, but here's what it really buys you. It lets you reduce clever logic and replace it with readable logic. And readable CI is safer CI.

Because the number of incidents that start with why did this deploy to prod is not zero. If your CI logic is hard to reason about, you are one sleepy reviewer away from a bad day. Also, a quick mid -episode reminder. If you are enjoying the show, hit follow or subscribe. It's a small thing, but it helps a ton. That's Story 2. Story 3 is Opus 4 .6.

Story 3 is Opus 4 .6, and I want to talk about this in a way that's actually useful, not just the new model dropped. Claude Opus 4 .6, the real change is trust and longer running work. Anthropic released Claude Opus 4 .6. They are positioning it as better at agentic tasks, longer horizons, more reliable multi -step work, more give me a goal, I'll do a thing behavior. So why does it matter? Here's the trap.

When people say better model, a lot of folks hear cool, the answers will be a little smarter. But the bigger shift is behavioral. When the tool gets good enough, people start delegating bigger chunks of work. Not rewrite this function, more like implement this feature. Or refactor this module.

Or update these workflows or go fix the failing tests and that's where the failure mode changes because now it's not the assistant gave me a wrong answer it's the assistant completed a plan that was subtly wrong And it did it confidently. And it did it across 15 files. So now you've got to do a different kind of review. Not just does this compile, more like does this match intent?

And also, did it change anything surprising? Because agents are really good at making changes that look plausible. So what do you do Monday? If you are using these tools seriously, I think teams need a new habit. Spec first, even lightweight. Before you let the agent touch the repo, you want a short plan. What are you changing? Why? What does done look like? What are the risks? What test will it prove?

Because without that, the agent will happily fill in the blanks with something that feels right, but isn't. Also, decide what you consider sacred ground.

For a lot of teams that's ci workflows iam network policy and anything that changes prod deployment behavior if an agent is going to touch those that should be behind stricter review two humans minimum no auto merge and probably a show me exactly what changed and why requirement and if you are a security minded org this is the week to remind people AI capacity upgrades help defenders and attackers.

It's easier to find bugs. It's easier to exploit them, too. So the defense is still the same boring stuff. Least privilege, secrets hygiene, fast patching, and not letting random automation have right access to the keys to the kingdom. That's story three. Story four is Google talking about SREs using Gemini CLI in outages. And this is where the AI and the ops conversation gets real.

Google SREs Gemini CLI as an outage sidekick. What happened? Google published a write -up on SREs using Gemini CLI during real outages. The framing is basically AI in the terminal as an incident companion. They're using it to classify symptoms, pull context, suggest mitigations, and reduce the early incident chaos. So why does it matter?

This is the direction a lot of teams want to go, even if they're not saying it out loud. Because the hardest part of incident response is not typing commands. It's context. What changed? What's impacted? What are the known safe mitigations? What's the fastest way to reduce customer pain? And under stress, humans drop context. We forget what we already tried. We lose track of timelines.

We waste cycles writing status updates that are vague because nobody has the full picture. So the AI as scribe idea is actually super compelling. You let the tool do the relentless bookkeeping. Summarize logs, pull the last deploy, list recent config changes, draft a timeline, draft a status update. Draft a mitigation checklist based on the runbook.

If you can reliably do those things faster, you've bought the human's time to think. And time to think is basically MTTR reduction. But there's an obvious danger. The moment you let AI run commands, you have to treat it like any other automation with prod access. You don't give it a shell and vibes. You give it narrow tools. And you build guardrails. So what do you do Monday?

If you want to copy this pattern without being Google, start with the safe version. Make an incident helper that is read -only. It can query logs. It can query metrics. It can pull deployment history. It can draft comms. And it can suggest mitigations from the runbook. But a human has to execute. That alone is valuable. And it's a good stepping stone.

Because the minute you jump straight to AI executes mitigation, you are going to have trust problems. And trust problems become adoption problems. And then this whole thing turns into a failed experiment. So start small, prove it helps, and build from there.

That's story four story five is the this will matter more over time story mcp and the platform layer that's forming around agents ai is becoming platform plumbing mcp governance and observability consolidation We are seeing vendors rally around MCP, the Model Context Protocol, as a way to connect agents to tools and data sources.

Miro launched an MCP server story to connect collaboration artifacts into AI coding tools. Kong announced an MCP registry angle for governance and tool discovery. So we are moving towards a world where agents can be given tools in a standardized way.

And in parallel observability is getting pulled into the ai orbit moves like snowflake planning to acquire observe are basically saying telemetry is data and we want ai sitting on top of it so why does this matter let me translate mcp into normal person ops language it's basically a standard way to say here are the tools an agent can use here's how it calls them and here's the context it can see And if that becomes common, platform teams are going to have a new inventory problem.

What agents exist? What tools do they have access to? What credentials are behind those tools? What data sources are exposed? What's logged? What's audited? And who owns the thing when it breaks? Because if you don't handle this like a real integration layer, you will end up with shadow agents the way we ended up with shadow CI jobs and shadow admin access. And on observability, the shift is also real.

More companies want log retention. They want to correlate telemetry with business events. They want to ask what changed across logs, traces, config, and deploy history. Then they want AI to help explain it. That's powerful. But it also turns observability into governance. PII risk, access controls, data retention policy, cost controls.

Because if you keep all telemetry forever, you can easily build a compliance nightmare and a gigantic bill. So what do you do Monday? If MCP is showing up in your org, treat it like production integration, not a toy. You want ownership. You want access controls. You want an inventory. You want a default deny stance. And you want audit logs.

Because the first time an agent does something surprising, the only thing that will save you is knowing what it did and why. And for observability, if you are exploring AI there, don't start with let's let a model roam through prod logs. Start with specific measurable wins. Better incident timeline creation. Better alert deduping. Better correlation across services. Faster what changed answers.

Then measure if it actually reduces pager pain and toil. All right, time for the lightning round. All right, lightning round. GitHub Actions hosted runners had an incident this week where jobs queued, timed out, and people had a rough morning. It's a good reminder that hosted is not the same as immune. If CI is business critical, know what your fallback is. Docker patched an Ask Gordon issue.

And even if you don't care about the specifics, the theme matters. If you're feeding AI systems untrusted input, including metadata, you can accidentally build a prompt injection path into your workflow. Treat AI inputs like any other untrusted input. Terraform 1 .15 alpha dropped. Not a headline, but worth noting for platform folks.

Deprecation warnings for variables and outputs is the kind of feature that makes shared modules easier to evolve without surprise breaks. And Wiz wrote up the Malt Book situation. Misconfigured backend. Tons of exposed API keys and sensitive data. It's a perfect vibe coding meets production credentials cautionary tale. Also, a quick mention from the open source side.

Chainguard's Ameri -OSS effort is an interesting trend. Commercial orgs stepping in to maintain projects that everyone depends on, but nobody wants to own. You don't need to have a strong opinion on it yet, but it is worth watching. And a couple of quick follow -ups. Ingress Engine X had multiple issues disclosed and fixed versions called out. If you run Ingress Engine X, double check your versions and patch.

And N8n is still in the patch fast and treat your automation layer like a control plane category. If you are running workflow automation tools, assume they are part of your perimeter. And here's the human theme this week. We keep talking about agents like they remove humans from the loop. They don't. They move humans to a different part of the loop.

The hard part is shifting from can I run the command to can I decide what to trust? And then can I recover cleanly when the automation does something confidently wrong? Because with agents, the risk isn't the tool is dumb. The risk is that the tool is competent enough that people stop double checking. And the first time that that happens in pride, you get a weird kind of incident. Not a bug. Not a bad deploy.

More like automation drift. Everyone thought the automation was safe. Then one day it does the wrong thing and nobody notices until it's already rolled. So if you are adopting this stuff, the boring work is the real work. Guardrails. Approvals. Ownership. Audit trails. And clear stop the line rules. Because the future isn't no humans. It's humans overseeing more automation. And that's its own skill.

All right, time for a recap. We had five main stories for today. Azure reminded us the cloud control plane is part of the product, and it can absolutely be your incident even when your app is fine. GitHub is going multi -agent with Claude and Codex, and Actions got a little less painful with better workflow logic.

Opus 4 .6 pushed agent workflows further into the mainstream, and the real risk is trust and multi -step changes. Google is using Gemini CLI in real SRE outage work, which is basically the AI and ops playbook taking shape. And MCP plus AI -driven observability is turning into platform plumbing. And platform teams are going to own governance whether they want to or not.

And the lightning round was GitHub Actions hosted runners having an incident, Docker patching an Ask Gordon AI supply chain issue, Terraform 1 .15 alpha, the Malt Book credential exposure story, and the Emirate OSS maintenance trend. All links are in the show notes on shipitweekly .fm. If you made it this far, hit follow or subscribe wherever you are listening.

And if you can, drop a rating or review on Apple or Spotify. It helps more than you think. All right, I'll see you next week.