Ship It Weekly Transcripts

Deployment was supposed to get easier. We got cloud platforms, containers, Kubernetes, Terraform, GitOps, internal developer platforms, and now AI agents that can generate code faster than most teams can review it. And yet, for a lot of teams, shipping software still feels weirdly painful. Not because people do not know how to deploy. Because deployment is where all the hidden complexity shows up.

The app stack, the runtime, the registry, the secrets, the networking, the database, the cloud provider, the rollback path, the weird internal Jenkins job nobody wants to touch. And for smaller teams, that can be brutal. They do not always need every knob Kubernetes gives them. they do not always need a giant platform engineering program.

Sometimes they just need a clean way to build, deploy, get logs, roll back, and move on. So maybe the future of deployment is not more flexibility. Maybe it is fewer decisions, better defaults, and picking the right opinions to outsource. I'm Brian Teller from Teller's Tech, and this is Ship It Weekly.

Welcome back to Ship It Weekly, where I filter the noise and focus on what actually matters when you are the one running infrastructure and owning reliability. Most weeks, it's a quick news recap. In between those, I do conversation episodes with people who are building platforms, running infrastructure, organizing events, and thinking through where this industry is actually headed.

Today is one of those conversations. I'm joined by Evan Phoenix, CEO of Miren. Evan previously worked on Terraform Enterprise and Waypoint at HashiCorp. And he also built Puma and Rubinius. We talk about why deployment is still painful. What teams keep getting wrong when they try to simplify it. And why small teams may not need more knobs. They may need better opinions.

There is also a good thread in here around Terraform. OpenTofu, Terragrunt, Waypoint, platform abstractions, and what happens when tools make users do too much homework before they get value. We also get into AI because yes, AI can generate infrastructure code, but when that generated deployment setup breaks, who is on call for it? And does anyone actually understand why it was wired together that way?

So if you work in DevOps, platform engineering, infrastructure, developer experience, or you have ever stared at a deployment system and thought, why is this still so annoying? This one should be worth your time. All right, let's jump in. Today, I'm joined by Evan Phoenix. He is the CEO of Miren, previously worked at Terraform Enterprise and Waypoint at HashiCorp. and he built Puma and Rubinius.

We're talking about why deployment is so painful and what teams keep getting wrong when they try to simplify it. Evan, thank you for joining me. Oh, my pleasure. Glad to be here. So walk me through your history in tech. It's kind of interesting. I'm curious. How did you land where you are now? Well, I'm old enough that I graduated from college right after the bubble burst. The dot-com bubble burst.

And so a lot of my career is actually informed by lack of opportunity post bubble bursting. I did all kinds of crazy interviews trying to just get any job. I almost wrote web Perl for many interviews and stuff like that. I couldn't get anything. And so I end up working for like a tiny ISP in Seattle. And then from there, I kind of just.

was sort of scrapping i was like you know i met a guy in a hallway and he told me about this company he's working at and so i went to lunch and the lunch turned into a job interview and i worked at that company for a number of years and so like i didn't have any a moment where it was like oh there's this big thing and so i kind of was jumping around and the other out the other part of it was i started doing open source work in the late 90s at this point.

And I did it all the way through college and then into my first few jobs. My first few jobs were, I was working at a three -person ISP. So I wasn't programming. I was like, open source is what I had for programming. And so I did that for a long time. And then I actually, the company I was working for in Seattle at the time was spinning down. I had a Ruby project and I actually got.

hired by Engine Yard, for people who remember Engine Yard from the Ruby days, and to basically work on Rubinius, my Ruby implementation, and then also help out with sysadmin work. And I kept ignoring the sysadmin work and just working on Rubinius. And so they were like, I'll just do that. So I got to work on my open source project for like five years. They just basically paid me. It's like living the dream.

And so like that.

sort of balance like working on open source and then finding a job and finding interesting things and working on open source has been this sort of constant thing and then it it carried me through many jobs it carried me into uh a startup that i had before HashiCorp and then HashiCorp is a great sort of like job and open source thing that carried me all the way through that and then my current venture Miren is a similar sort of thing it's like job and open source and all kinds of things and so yeah it's been that's the that's the short long version Very cool.

So my first job was, to date myself to a degree, was doing dial -up tech support. Yeah. Me too. That was the ISP. Yeah. I did dial -up tech support. I did DSL tech support. And then we were putting wireless antennas on people's houses around Seattle. For long range. Yeah. It was sort of long range. But yeah, directional. Yes, yes. But big antennas.

That was actually how I got the, how that guy in the hallway talked to me was because I was carrying a giant antenna. And he was like, what is going on here? And I explained it to him. He's like, we should have lunch. And then, yeah, one thing or another. But yeah. That's awesome. So where does Miren fit in? What is Miren and how does it fit into the ecosystem? So, I mean, it's a software deployment system.

It comes out of me constantly feeling like we are always struggling with deployment systems, right? Like people, I have... People have lots of great ideas and especially now with coding agents, you know, getting an idea out there and getting it like, oh, my gosh, I got this idea and I want to put the code together. And then you have to figure out, like, how do I give this to somebody?

How do I let somebody else use this? It's always like it breaks you completely out of your rhythm. And so the like one of the original ideas was like, how do we integrate deployment?

actually into sort of the software development life cycle right so that it's like it's not this thing where like i spent all this time and now like oh god now i got to deal with deployment but it's like oh deployment is part of that i go through a deploy i can get the logs it feels like it's part of the process and so i i've worked with Kubernetes i worked with Nomad i worked with lot most of the AWS products to do deployment and i felt like They're always missing something.

And I wanted to feel like I had my own things to say about where that fit in. And the first cut of it is a self -hosted piece of software. So it's open source. You go and you run it wherever you want to run it. And so that makes it very accessible. That opens the doorway to lots of different people who have different needs. You want to data sovereignty. You want, oh, I'm really cost conscious.

I want to run it in one particular way. And one of the big things. to do it for us was like, hey, you've got this open source. You can run the deployment system on your laptop. You can just deploy to your laptop. And then when you're going and you're deploying to production, it's the same. It's exactly the same.

And so you've got all of these, all of your rhythm about how do I do it and how do I work is exactly the same. And so now you become that much more comfortable with the system and then you can sort of grow with it. That's awesome. That's great to hear. Okay, so. Why is deployment still such a mess after years of platforms and abstractions? Why? What's going on?

Yeah, I mean, it really is the final boss in some ways of software. I think that it has to do with a few different things. I think that if we all were working in sort of one stack. one system, I think it would be a lot simpler. I don't think we'd be as happy, but I think that deployment would be simpler. And because then there would be just fewer knobs.

There'd be, you know, like all the, you know, if Java, maybe, maybe not. I mean, it's not like Java beans really solved all of the problems of the, you know, like the late 90s, early 2000s. But there are so many different shapes that an application comes in. And some of that are just within the stack. One JavaScript app could be completely different and have a completely different profile than the next one.

But then you multiply that matrix by all the different stacks and all the different things. And now you've got sort of a giant 3D cube of different ways that an app can behave. And I think that deployment systems try to say, OK, I want to do I want to give the power of all of these things all at once?

And if so, I have to let you configure every possible thing on here so that I can have every possible stack here, right? And sometimes they're like, well, that's really too difficult. Most people are in this one quadrant. Let's actually focus on the kind of apps that look in this shape, your HTTP apps, request times under a second, whatever it might be.

And great, let's sort of focus on that and try to distill down the needs and the function and that kind of thing and make it better. I think that, I do actually think it's better now than it used to be. I actually think there was a time where it was.

better than it is now also but like it's just because of the number of variables that come into it and then what each platform each platform wants to have a unique say on it right so they want to say like oh well we've got this cool load balancer and so you should lean on the load balancer and now i'm like okay i gotta learn load balancer and i gotta learn all this kind of stuff right and so or uh you know AWS is like great for like oh we've got these AWS specific services to like make your app work in this one way of these different shapes and so it's the the amount of configuration the amount of of of functionality that you want to sort of eek from the platform too you think ai helps us bridge that gap better i mean in some ways yes in some ways because it's really good ai is really good at if you could if a platform can explain itself The AI is really good at being like, okay, great.

The platform works like this. I'm going to basically match the app to the platform, right? So the platform can have a bunch of different things it does well, things that it doesn't do well. And the AI can say like, okay, I'm going to make the app work. You wanted this platform. Great. I'm going to plug you into it. The inverse is also true, although I don't think we're seeing it that much yet.

The app exists as this thing, and the AI is like, great, let me find the deployment platform that fits this. I don't think we're seeing that quite yet, maybe eventually. I do think it's making it better. Okay, so you've lived through multiple waves of this will fix it. What did people misunderstand about Terraform Enterprise early on? You know, we were constantly trying to figure out the shape of it.

There's a whole Conway's Law situation with Terraform Enterprise, right? So we were constantly trying to help people figure out the shape of like, what should their repos look like? So in other words, like, should it be like there's a repo called Infra and it has like a thousand modules that are basically going and configuring all these different things.

And every time anybody changes anything, we have to run a plan that basically touches like, you know. 1 ,500 resources or something like that, right? Do you want that? Or do you want like, oh, well, I want my, like, should it be tiered? So it's like, okay, the load balancers are a module, a thing, there's like infra -loadlb, and that's just the load balancers.

And so it was, and I don't know that Terraform really ever figured this out. even to this day, if I'm honest. But people were constantly, that was the big question. We had white glove service at HashiCorp at one point that was like, how do I organize my Terraform modules, right? In a way that's like, that I can apply them quickly, that reduces the blast radius, all of those sorts of things.

And so I think that was a really interesting thing that, and that wasn't Terraform Enterprise specifically, that was Terraform writ large. I think Terraform Enterprise was a, The first version of Terraform Enterprise was the one that I worked on where it was for people who wanted the experience of having a platform that ran Terraform for them, but they needed to have it sort of inside their silo, whatever that meant.

And so we went through a few iterations where basically it was just like a – big machine like we scaled it vertically inside the actual instance and then you would just it would just run Terraform on on instance and it was a like so many products when you see how the sausage is made underneath you're like this is all we're doing you know like it was a little bit that in a good way because it basically made it made the product easy to manage and and simple and stuff like that like we were constantly like stripping things out and taking things out and like you know at one point the product was dependent upon RabbitMQ and we're like, no, we need to like tear, like slice this down.

Like that's too big of a complexity for a thing that's just running on a single machine. And so. Is that for like worker queues? Yeah, for worker queues.

Because that was, so at the time, this is Terraform Enterprise is a product before Terraform Cloud, which is a weird sequencing because we had Atlas at the time, which was the thing that predated HCP which was like our all-in-one platform and uh Terraform Enterprise existed as sort of a side thing to atlas and then atlas eventually gets broken apart part of it became Terraform Cloud we still have uh Terraform Enterprise and so that rabbit mq part came from the original vision of atlas yeah believe it or not i was an early uh customer to Terraform Enterprise yeah i we so probably much like a lot of the customers we we had a pci HIPAA requirement where we needed to have state files managed locally.

We needed to make sure that everything was self -contained and then eventually move to Terraform Cloud. Did you use it in the... You weren't at desk.com, were you? It was not, no. Was it a forms company at the time? There's three shapes of Terraform Enterprise. The first, the shape number one was only ever seen by like two customers. Oh, okay. One of which was desk.com when they got bought by Salesforce.

And then the second one was the AWS only Terraform Enterprise where we basically shared an AMI and a bunch of Terraform for you to run it. That was shape number two. Okay. And then shape number three was a rewrite that I did, which made it basically Docker -based. And then you can install it anywhere. I forget the name of the application that it was built on. Replicated. Replicated, yes.

I remember going through and dealing with weird Docker IP address allocation issues. All the time, yes. So yeah, I guess it was version three. But yeah, it was a good product. We eventually moved to Terraform Cloud. And then I think they've since, I don't know what the company does now. It was a few years ago.

But it got me excited about IaC because at the time, and this is going to cause a lot of people to cringe, I was at a company where we were using Ansible with no state. We were just using Boto3 to manage infrastructure, some CDK stuff. So going from that to Terraform was a huge jump.

More recently, the problem you were describing earlier, just about how do you manage state files, how do you manage environments, how do you manage modules, I feel like Terragrunt has really tried to answer that problem. I don't know that they've answered it well, to be honest with you, but I think they've tried at least to answer some of that dry issue. Yeah.

I mean, they were around right when I was at HashiCorp because the Terragrunt has been around for maybe not 10 years, but quite a while at this point. Yeah. And they had this idea of like, how do you slice your Terraform and we'll basically like put it in different shapes.

And they were very successful at basically saying, we're not going to let you like Terraform in some ways, let you do anything, let you put files wherever and let you organize it. However, in Terragrunt was like, what if I just put like a corral, like I put like a little, like a little.

guard around this thing you can't do all this stuff it's just like it's going to work like this and and that is very powerful from a workflows perspective because now people are they're having to comport themselves maybe to this smaller feature set but they're also not having to think as much because they're basically being like no it works like this like i'm supposed to supposed to put it in these boxes we think about that a lot about that at Miren because Part of it is what do you want someone to have an opinion about?

Like Terraform made you have opinions about like file structure and stuff like that. Right. Terragrunt is all about like, OK, well, let's not like people have an opinion about all of these things. Let's just make those opinions for them and we'll just tell them what they are. And then they're not having to go around, go about, figure out, like, should I have an opinion about this?

Like a lot of people, when they would start using Terraform, they're like, but just can you just tell me what the right pattern is? And they're like, well, like, did you do that?

or does it think this or what you know whatever and so uh they would we would sort of force them in some ways to have an opinion about stuff that they didn't really want to have an opinion about that's fair so where do you think OpenTofu sits in that opinionated stack against Terraform Terragrunt yeah at least whatever other alternatives there are well i think open tofu is the result of two specific things obviously number one was a license change without a question no no no question right but i think that the second one is part of it which is that the backlog of potential features for Terraform that we're sitting in PRs or sitting in issues that the Terraform team was like no we're not going to do that no we're not going to do that and so the the uh The license change catalyzed people to be like, okay, well, I want a different thing.

And then it got runway because it was like, well, look at all these PRs. Look at all these really cool features that Terraform team never wanted to do that will just do those things, right? And so now it's like it's growing all these different, growing in this sort of different direction because of that, right?

It might still exist even if there wasn't all those features, but I don't think it would have had the legs that it has had. if it didn't have that second part, right? And I think that what's interesting is that I was at HashiCorp during the creation of OpenTofu. So we were talking about it and we were trying to figure out what it meant.

And what was interesting is that OpenTofu was taking on these features that Terraform was rejecting.

for good reason right so there was a lot of interesting features there's encryption encrypted state management was one of the big things that OpenTofu took on that Terraform was like don't do this do not like the reason that they didn't do it for the longest time was because like we we run the math and if you do this what you end up doing is that you create like this other big hairy availability scenario that is now i want to do stuff and now i've lost the keys to do stuff and now i can't do anything i'm sort of like really dead in the water and so they were like an advocate for like do it a different way right they had all these different things and so and again like maybe that's an example of where Terraform was like actually don't let people do everything they were trying to sort of rein in the the sort of field of opinion and and in some ways OpenTofu is like even wider like now you can do all these things and maybe some of them are good ideas but at least people are trying them out so well in secrets probably shouldn't be in state right or anything too sensitive or if it is i mean lock it down otherwise but but maybe that's the answer like i'm sure they were pushing like Vault and like using yeah that was really that was that part of it was really interesting because that was the sort of the evolution of how of your state files as definitive reproducible data right so like one of the reasons early on in Terraform that Terraform would end up having secrets inside of it was because they were like look like the state file needs to be the thing the the the the jewel that you can use to recreate whatever you need right and so they they were like you should put secrets in your state file because you want to do this and the other thing and then what they sort of realized over time was that like that's a that's a hard position to to have defense for when Terraform depends on a jillion other APIs.

And so it's like, okay, well, if it already depends on all these other APIs to work anyway, can't you just make secrets an API that you have to depend on? And then that became a thing that they sort of moved towards. In my opinion, they didn't move towards it.

quick enough like in other words i i always felt like hey if you're gonna have secrets just make make a secrets part of Terraform Cloud and let secrets live there your state files can basically just like not have secrets in them no matter where you are you can just sort of outsource the secret management part to Terraform Cloud um that was kind of the idea with variable sets though right i mean yeah whether it's exactly right that's exactly right yes that's exactly right but it could have been at a deeper integration level no right so yeah for sure So what did you learn about what devs want versus what they think they want when it comes to tools like Waypoint?

Yeah, well, Mitchell and I started Waypoint in 2019, which was an interesting time to start a new product because right in late 2019. And I think that the thing that we were really focused on was a control plane. for backend deployment systems. So it would be like you would talk to Waypoint, and Waypoint would go off and do things like, oh, to do that, I need to build a container. Let me go build a container.

And then, OK, great, I will do this, and I will do this other thing, and I will do this third thing. And the idea was it was just a pure workflow engine for deployment. And it farmed out all of its backends to do different things. And one of the things that I learned working on that was You're asking a lot of someone who wants to use this thing to say it doesn't do anything until you do all of this homework first.

Right. It was a lot of homework to get to get Waypoint running. And that kind of sucked. Right. Like from just from a usability perspective. But it also it wasn't great from like we're working on a perspective. One of the things that we we found out that was like a like a really big hurdle was people.

people would get to the point where they're like, oh, like, oh, I just saw this on a system and it can run, it can build Docker containers. And that part was easy because we would just say, also run Docker on the same machine that you want to run Waypoint. Like that was sort of, we could easily pave over that. But then they'd be like, okay, now I want to deploy this to ECS.

And we'd be like, okay, well, you got to go. You have to go configure, like, container registry, and then you've got to push it to the configure push to push the container registry, and then you've got to tell the container registry over here to pull. And it was like just that, just getting over the, like, where does this image go was a leap. It was a big hurdle for people to get over. Sometimes it was too big.

Like, they wouldn't even get past that point. And so we worked on it to try to. to pave over it. One of the things that I did was I made it so that if you were using, let's say, ECS as an example, we would automatically configure a repo inside AWS's OCI registry to push those images to. So we would kind of just try to pave over that as much as we could.

But the learning experience I took away from it was like, a pure workflow engine, it's not enough product. It's not enough unless there's this really complicated thing that people are doing that you're like, I want to come in and I want to just revolutionize this one segment with a workflow engine. Great. But deployment wasn't that. And what they really needed was they needed answers.

It didn't need to be all the answers, but they needed a lot of answers that were already set in stone at the beginning. So when I was going to deploy stuff, I was like, it just deployed. It just did the thing. And part of that comes out of the idea of like. again, from HashiCorp that we learned was people wanted us to have opinions, right? They really wanted HashiCorp's opinions.

They want to say like, well, okay, well, like great, it's Terraform, it's Vault, it's Nomad, but how do I set it up? Like, what's the best way? All this kind of stuff. They'd constantly be asking for our opinions, even though we had these sort of generic tools. And so the takeaway that I took from it was like, well, let's just build a really opinionated tool, right? Let's go in and say, hey, we've got opinions.

We've been working in the field for all this time. We've got opinions about how this stuff should work and what it should do and how it should feel and what the primitive should be. And we think that if you agree with those things, you will really like our thing because it just does all those things, right?

It's sort of really defining that in -group of people who want something that looks and feels in a specific way. And so we... When we were working on Miren and from Waypoint, we took that opposite approach. We're like, let's just build all the pieces. Let's build the thing that builds stuff. That builds the thing that deploys it. Let's not tell people where stuff's being stored, where the images are.

Let's just make that not a problem at all that people have to think about and worry about. And that was a big takeaway from Waypoint, right? Waypoint, it never really got traction and it really got its legs underneath it. Partly, I think, because it didn't have enough opinions. It didn't have enough opinions about how someone should actually use this thing.

And then just because of things that were happening inside HashiCorp at the time, it ends up sort of migrating over to HCP and then it migrates away from deployment entirely. It migrates basically to a Terraform front end. And it wasn't doing deployment at all towards the end. It was sort of misnamed in my opinion. But yeah, that was the big takeaway from those Waypoint-as-deployment system days. Very cool.

So, OK, jumping back to Miren. And I don't want to turn this into a product pitch, but I am curious what what I do want to understand is what you're trying to fix and like what's broken for small teams trying to ship. What tradeoffs are you making to keep it simple? Yeah, I mean, I think that I'll start on the first one, right? Like what is the what makes Miren sort of interesting is that.

And where do people and small teams hit problems, right? I think that as I've been looking at deployment tools for years, you sort of have stuff that's really good for a single developer. Like, oh, I've got Docker on a machine somewhere, and I just SSH stuff to it or whatever, right? And then there's sort of a gulf.

And then you have like, OK, well, I want like a Kubernetes -based thing that's sort of eaten most of the market, right? There's not really much in that space for like, actually, I'm a team of five people that really just wants to like, that are application focused. They're just like, I just want to ship an application. I want to be able to deploy it 30 times a day if I need to, right?

And there's kind of not a lot there. There's a number of products that try to take what Kubernetes is and build facades and things inside Kubernetes in order to get. back to sort of this middle ground of like a thing that would feel good for a small team.

But what small teams always run into is now you have not this much stuff, like from Kubernetes to where they were, but like this much stuff, my arm is way over there because now it's like, it's not the, if something goes wrong, it's not necessarily in this top layer.

It's basically somewhere deep inside the Kubernetes stack of like your stateful pod had got evicted because the memory pressure was too high, whatever it is. Right. Yeah. And, we didn't want people to have to worry about that because that team of five people, they just want to deploy stuff. They want to get in the groove of deploying.

They need something that really is getting out of their way that just works the way they need it to work. The question was, how do you build something that isn't going to have a bunch of landmines, right? Like a lot of the deep Kubernetes stuff that's really important for big teams are landmines for small teams. And the answer was like, let's just strip it away.

Let's just keep stripping stuff away until we get to something that feels like it's the right size for a small team. And that meant like, yeah, it's not Kubernetes -based at all. We were like, we're stripping that away entirely. We're like, okay, what's the right?

what's the right primitive like containers are still the right primitive again you feel see where Docker is you see where people are writ large containers are still that thing but then it was like okay great let's say containers as a primitive and let's just start building on top of them and so there's things that miren does that Kubernetes does thousands of times better right that's no question right but i think that A lot of the things that Miren does, you could do with Kubernetes, but Miren does it out of the box.

And with Kubernetes, you have to become like a pretty decent Kubernetes expert to do. Like the simplest one that I can give you is we do a scale to zero application deployment by default. I have a really good reason for doing that is because you're going to run this on a machine, your own computer somewhere. And if it's an app that only is used once every month, don't run it. Don't run it.

Just let it run the one time a month that needs to run. Like don't take up resources running it, you know, for all 29 other days. And you can do that with Kubernetes. There's a bunch of Knative stuff. There's a bunch of pieces that you can sort of layer on top of Kubernetes in order to do that. But now you become an expert in a layer on top of Kubernetes instead of.

It's just a thing that does the thing by default, right? And so it becomes a question of those trade -offs, right? Like we don't do as much as Kubernetes, obviously, but the things that we do do are designed specifically to do in that way. And we think that those are the right trade -offs to make for your teams of like one to 20 is really sort of the sweet spot that we target.

So what kind of applications or workloads or problem statements do you see customers bringing to Miren? What do they come to Miren for that they wouldn't go to Kubernetes for? Yeah, I mean, they come to us a lot of times. The ambiguity around the Heroku situation is one reason people come to us. And they're like, I don't really want to take on, I really want a system that just does the system the right way.

I don't want to become an expert in how to layer a deployment system on top of Kubernetes. That is like all all the helm charts and all the things and keeping up on all the versions and all that kind of stuff. They're like, I don't want that. I don't have the time or the team size to do that. But I want a thing that is that just just works.

Like what we think about it, we think about ourselves as being application focused and workflow focused. So people come to us and they're like, hey, I just want to deploy my stuff. We're like, great. That's what we do. We deploy your stuff for you. You know, we we support. specifically because this is where 90 % of the workload is now.

It's like we're supporting HTTP applications, you know, out of the box as the first class thing, because that's where people are, right? We have stuff for other, we sort of started to layer on things for non -HTTP stuff, but like the bulk of our focus has been on how do we run HTTP based apps really well and quickly. And so. Where do you think deployment and IaC tooling goes next? Especially.

with AI speeding up code changes? Yeah, I mean, I think that people, one of the things people ask me is like, is Miren in danger of having an AI just code something to use AWS that works the same way? And I think the answer right now is no. But I think that even in the longer term, it's probably also no. And here's my take.

You can have an AI go through and code you up a deployment platform in the same that feels similar to the way that Miren feels today on top of AWS. It could, you know, those AWS graphs, it's like all the different services wired together. It could do that. It could wire all those services together. Is it on call? Is the AI on call when one of those things goes wrong? What happens when the AI...

is looking at it again after a month of it working and is now trying to figure out something is wrong and it's looking at it again. It's like, this doesn't make any sense. Who set this up? I'm actually going to delete all these things and rebuild it from scratch. You're like, oh my God, don't do that, right? And so there's not the discipline in order to do that.

Now, again, I think people will be like, well, maybe the AIs will get that discipline. Yeah, I mean, then you're effectively hiring an SRE to just build that up. And the question is, is that a good use of your time and or money?

right to to have an SRE spend all their time trying to balance AWS services you know on one hand uh for some people maybe it is uh i would i think our bet is that it's not you know like it's just not it it doesn't make sense in the the larger frame of things and so The other point about AI and IaC is that one of the things that we've seen is we've sort of pointed AI at our tooling and said, hey, go deploy this thing, is that our surface area is very small, but very obvious for deployment.

And so the AI gets it right every single time, right? Because it's kind of nothing to get wrong. It's like, hey, I see that I can set config variables. I see that I can do deployments. I see that I can do rollbacks. I see that I can get status. Great. So the surface area the AI has to interact with is small enough that it just doesn't get it wrong. Right.

Because it's like there's not a bunch of ambiguity about what is actually going on. And then, you know, the AI knows that like what doesn't know anything. But the API knows that behind all of that is the machinery that keeps all that stuff working in the shape that the API is made to work from. And so, I mean, I wonder like. what we're talking about here is abstraction levels.

I don't think abstraction levels are going to disappear as AI gets better. It's going to build new abstraction levels. And then you're basically like, okay, great. That abstraction level is done. We're good. We're solid. Now I can go to the next one. And the question is, do you, and this was the case for Waypoint way back in the day. People build deployment systems at their companies very commonly.

And the thing that we would always hear when we first was working at Waypoint was that, I got halfway done building this cool ass deployment system. And then my boss told me, hey, knock it off. Go back to the thing that actually we do as a company. And so then they'd have this half built deployment system that is just kind of a hunk of junk because it never got finished. And then they have to just suffer.

with it for years. And they're like, how come the deployment system is so weird that you have to like stand on one leg and shake a thing above your head in order to make it work? And it's like, well, because none of that was supposed to be there. That was all supposed to be paved over. But we never got enough time to finish it because we're a mortgage writing company. Right.

So like building a deployment system never made sense. And so I think that that's still true when people are. applying AI to AWS resources. It's just like, you're, you just needed a thing to work. You didn't need any of those other pieces. We want to be, we want to basically as Miren be a person who has opinions. You want to basically buy our opinions.

You want, we want, you want to outsource your opinions about how this is supposed to work to us. And then we'll just give you a thing that works. Oh, that makes sense. Yeah. I, I fell into the same trap. I think with Jenkins, like most people do. Oh yeah. Me too. You know, you didn't have time to. to fix it. And it was a house of cards. It just works. So you just left it alone. Right. Don't touch it. Yeah. Yeah.

No better to, to yeah. Go to a company that can give you the opinions around the stack. You pay for that, pay for that. You pay for that service that they're going to maintain. Yeah. I mean, I have, I know people who, uh, who pay for AWS consultants because, and then they're there, that's the same thing. Right. And, and in some ways, uh, you know, they're, you're paying for that person's opinions. So.

Okay, so wrapping up, what's one industry belief about DevOps or platform engineering that you think is just wrong? I think that people fall into the trap of, I guess it's not DevOps. I'll say what I was going to say, which is, yeah, I was going to say that I think that people fall into the trap of feeling like, oh, I can't.

i shouldn't rewrite that i shouldn't do how about my own version of this that and the other thing and i think that um it's a double -edged sword of basically being like oh don't ever always use this one version of this thing um even though it does no most of it doesn't do what we want but this one little slice does right i think that people should be more uh willing to say I'm going to do the thing that works best for me, and I'm going to do 5 % of what the other thing does, but it's going to be the 5 % that I want and now I can own.

It's sort of like the industry did a lot of hand -wringing around dependencies about 10 years ago, about how big your node_modules should be and all that kind of stuff, right? I think we're running into it again now with supply chain. And so I think that having your own versions of the things that fit just what you need, it's way underrated. So yeah, no, that's fair. I agree with that.

Evan, where can people find more about you? You have sOCIals? Yeah, yeah. You can find me on Bluesky. I'm evanphx.dev on Bluesky. We have a Miren Discord. If you want to hit that up, that's at miren.dev/discord. Yeah, that's probably the main two places he says. Awesome. Appreciate it. Thank you so much, Evan, for coming on. Really appreciate it. Absolutely. Thanks so much. All right.

That was my conversation with Evan Phoenix from Miren. My biggest takeaway from this one is that deployment is painful because it sits at the intersection of everything else. It is not just run the app. It is build the app, package it, push it somewhere, wire up the config, expose the service, watch the logs, handle rollbacks, and make sure the next person can understand what happened.

And every time we try to simplify that, we usually move the complexity somewhere else. Sometimes that is fine. Good abstraction hides the right things, but bad abstraction hides the thing until production is broken. And then suddenly, the team that didn't need to know Kubernetes has to understand pod eviction, node pressure, ingress behavior, registry auth, and whatever controller is angry three layers down.

I liked Evan's point that small teams often do not need the biggest, most flexible platform. They need something with a smaller surface area that handles the boring parts well. That does not mean Kubernetes is bad. It does not mean Terraform is bad. It just means flexibility has a cost. Every knob is a decision. Every decision becomes something to support. And eventually, the platform becomes its own pile of work.

The AI angle makes that even more interesting. AI can generate deployment code. It can wire together cloud services. It can probably build something that works for a while. But is AI on call? Does the team understand what it created? Does the next person know why those decisions were made? That is why good abstractions probably matter more, not less.

A small, clear deployment surface is easier for humans to use and easier for AI to interact with safely. So the practical question is not always how do we build the most powerful platform? Sometimes it is how little platform can we get away with while still shipping safely? I'll have links to Evan, Miren, the Discord, and anything else we mentioned in the show notes.

If you enjoyed this conversation, follow or subscribe to Ship It Weekly wherever you listen to podcasts. You can also find the show notes and links over on shipitweekly.fm. Thanks for listening, and I'll see you later this week.

This week, Amazon Q Developer and the AWS language servers had a pair of trust-boundary CVEs. JFrog found hijacked npm and Go packages using hidden VS Code tasks to run malware when a workspace opens. AWS WAF had HTTP/2 request body inspection issues. And AWS introduced Lambda MicroVMs for running user-generated and AI-generated code in isolated sandboxes.

Put those together, and the shape of the episode is pretty clear. Execution is the boundary now. The repo. The IDE. The AI assistant. The WAF. The sandbox. They all sit at the point where something gets to run, inspect, block, or decide. And that is where the risk lives. I'm Brian Teller from Teller's Tech. And this is Ship It Weekly.

Welcome back to Ship It Weekly, the show about the DevOps, SRE, cloud, platform, and security stories that actually matter when you're the person who has to keep the thing running at 3 a.m. If you're new here, follow or subscribe wherever you're watching or listening. And if you want the weekly story list and source links, check out OnCallBrief.com.

For past episodes, full show notes, and more from the show, head over to ShipItWeekly.fm. This week is about execution boundaries. Who gets to run code? Who gets to load config? Who gets to inspect the request? Who gets to decide whether something is safe? We open with Amazon Q Developer and AWS language server CVEs, then hijacked npm and Go packages using VS Code folder-open tasks.

After that, AWS WAF and HTTP/2 request inspection, then AWS Lambda MicroVMs, because sandboxing untrusted code is becoming a first-class platform problem in the lightning round we'll hit GitHub advisory volume git 2.55 valkey 9.1 and a quick fable 5 callback let's get into it first up AWS disclosed two issues in language servers for AWS and Amazon Q Developer IDE plugins. The first one is CVE-2026-12957.

AWS describes it as improper trust-boundary enforcement. The short version, if a user opened a malicious workspace, commands inside project configuration files could automatically execute. The user had to trust the workspace when prompted, but that is not exactly a giant comfort. Because developers trust workspaces all the time. They clone a repo. They open it in VS Code.

They click through whatever prompts get them back to work. And they assume the repo is code, not a local execution trap. The second issue is CVE-2026-12958. That one involved symlink validation.

A malicious workspace could use a crafted symlink that resolved outside the workspace trust-boundary both bugs point to the same bigger problem the IDE the language server the assistant and the project folder are all negotiating what is trusted and that boundary is messy it includes config symlinks workspace trust MCP server configuration environment variables, local files, cloud credentials, and whatever the assistant or plugin can reach.

That is why this matters. Amazon Q Developer is not just an editor feature. It sits near source code, local credentials, AWS profiles, repo context, terminals, and sometimes production access. So when a workspace can cross into command execution, it can become more than a local bug. It can become a cloud credential problem, a source control problem, a CI/CD problem. The takeaway?

Update Amazon Q Developer and the AWS IDE plugins, but also review the operating model. Do developers understand workspace trust? Are MCP configs reviewed?

can project config execute shell commands do developer machines have long-lived AWS credentials can a malicious repo jump from weird local folder to cloud account access AI coding assistants are becoming part of the local control plane local control planes need boundaries not vibes boundaries second story JFrog found hijacked npm packages using a clever execution path.

The malicious versions were not relying on the usual npm lifecycle scripts. No obvious preinstall, no obvious postinstall. Instead, the payload hid inside a VS Code task, specifically a hidden task configured with run on folder-open. So the trap is not just install the package and npm runs code. The trap is open the project folder in VS Code, trust the workspace, and the IDE runs the task.

That is a different threat model. And it matters because the ecosystem has been hardening the package manager path. npm version 12 is changing install script defaults. Teams are paying more attention.

to dependency execution ci pipelines are starting to care about lifecycle scripts that is all good but attackers adapt if npm install gets harder they look for the next execution surface and the IDE is a very good one developers open folders all day example projects reproduction repos proofs of concept vendor samples AI generated apps Random repos from Slack or GitHub issues. And the editor is not passive.

It runs extensions. It loads config. It starts language servers. It runs tasks. It reads environment. It can access terminals and local secrets. JFrog also found additional Go packages tied to the same malware pattern. So this is not just an npm story. It is a workspace execution story. The takeaway is to treat IDE workspace trust like a security control. Watch for hidden VS Code tasks.

Pay attention to run on folder-open. Be careful with VS Code tasks in random repos. Do not trust every folder just because it looks like code. And for teams using Cursor, VS Code forks, or AI-heavy editor workflows, this gets even more important. The package manager is not the only place code runs. Your editor is an execution environment now, so act like it.

Third story, AWS disclosed two AWS WAF issues involving HTTP/2 multi-frame request body inspection.

The CloudFront case was remediated server-side with no customer action required the application load balancer case is the one that operators should look at under certain conditions a crafted multi-frame HTTP/2 request could cause only part of the request body to be inspected that is the sentence that matters only part of the body gets inspected because when you put waf in front of an application the whole point is that the security layer sees the thing you are trying to block.

If a request is split across frames and the inspection layer does not accumulate enough of it before applying rules, then the security decision may be made on an incomplete view. That is not unique to AWS WAF. This is a classic security control problem. The proxy, load balancer, WAF, app server, and backend framework all have to agree on what the request means.

If one layer parses it one way and another layer parses it differently, attackers get room to move. That is where parser differentials and request smuggling-style problems come from. AWS released a configuration option on ALB so customers can control how WAF inspects HTTP/2 request bodies. So this is not only a provider fixed it story. If you use WAF with ALB and HTTP/2, there is something to review. The takeaway?

Do not just ask, do we have WAF? Ask what it sees. Where does TLS terminate? Is HTTP/2 involved? Are request bodies inspected the way you think they are? Do your ALB settings match the protection model you are depending on? Security controls are not magic shields. They are parsers, policies, and defaults. And this week, the parser mattered. Fourth story. AWS introduced Lambda MicroVMs.

This is the positive version of the same theme. If the first three stories are about unexpected execution paths, this one is about building a safer place for execution to happen. AWS describes Lambda MicroVMs as a new serverless compute primitive for running user-generated or AI-generated code in isolated, stateful execution environments.

You get VM-level isolation, fast launch and resume, lifecycle control, state control, and AWS manages the underlying infrastructure. The use cases are easy to imagine. AI coding assistants, interactive code environments, Plugin systems, data platforms, vulnerability scanners. Anything where your product needs to run code that your team did not write. That is becoming normal. And it is dangerous if handled casually.

Containers are useful, but shared kernel isolation is not always the boundary you want for untrusted code. Full VMs are stronger, but heavier. Traditional Lambda is great. but not built for every interactive or stateful sandbox use case. Lambda MicroVMs are AWS trying to fill that gap. Whether you use this specific product or not, the direction matters.

Running untrusted code is becoming a platform feature, not an edge case, not a just run it in Docker and hope situation, a platform feature. The takeaway. is to inventory your execution surfaces. Where do users, agents, plugins, tests, scripts, or generated workflows run code? Can that code reach internal services? Can it access credentials? Can it write to shared storage? Can it talk to the internet?

Can state leak between users? Can you kill it? Can you audit it? Can you prove it is gone? Execution is not going away. the platform has to make execution safer. Quick lightning round. First, GitHub says its advisory database hit a record volume. In May, GitHub published more than 1,500 reviewed advisories, more than five times the typical monthly output. And even that was not enough.

to keep up that is a real DevSecOps problem vulnerability management is becoming a queueing problem the hard part is not getting alerts the hard part is deciding what matters before the queue buries the team second git 2.55 is out the useful bit is incremental MIDX support through git repack That mostly matters for very large repositories and hosting platforms.

But it is a good reminder that developer platforms are infrastructure too. At enough scale, Git performance becomes engineering productivity. Third, Amazon ElastiCache now supports Valkey 9.1. The main angle is performance and efficiency improvements landing in the managed AWS service. Not every cache needs an urgent upgrade. But cache engines are production systems.

Memory efficiency, throughput, command behavior, and isolation matter when the cache is sitting directly in the request path. Fourth, quick Fable 5 callback. Anthropic's Fable 5 is back online after the export control issue that knocked it offline. And that does not weaken the earlier point. It strengthens it.

The model came back, but the dependency risk was still real AI reliability is not only uptime anymore it is access policy regional rules model behavior fallback behavior and vendor changes so yes fable 5 is back still put AI models in your dependency register the human closer this week is about execution a repo is not just source code if opening it can trigger the assistant language server or task runner an IDE is not just an editor if it can run commands load tools and inherit credentials a waf is not just a checkbox if it only inspects the part of the request it actually sees a sandbox is not just nice to have If your product runs code from users, agents, or plugins, and an advisory feed is not useful just because it is complete, someone still has to decide what matters.

That is the platform work hiding under this episode. Modern systems are full of places where something gets permission to run, inspect, transform, or decide.

Sometimes that thing is a package script sometimes it is an IDE task sometimes it is an MCP server sometimes it is a WAF parser sometimes it is a sandbox sometimes it is an AI model the better question is not only is it trusted the better question is what happens when it executes what can it reach what can it read what can it write what authority does it inherit what logs prove what happened And who owns that boundary?

Because execution is where trust becomes real. Before execution, trust is a policy. After execution, trust is a blast radius. So find the places where code runs. Find the places where tools make decisions. Find the places where security controls parse reality on your behalf. Then make those boundaries explicit. Because execution... is the boundary now. That's it for this week of Ship It Weekly.

We covered Amazon Q and AWS Language Server CVEs, hijacked npm and Go packages using VS Code tasks, AWS WAF HTTP/2 inspection issues, Lambda microVMs, and a lightning round on GitHub Advisory Volume, Git 2.55, Valkey 9.1, and Fable 5 coming back online. If this episode was useful, follow or subscribe wherever you're watching or listening. If you're on YouTube, hit subscribe.

If you're in a podcast app, follow the show there. And if you know someone dealing with AI coding assistants, IDE security, supply chain risk, WAF configuration, or sandboxing untrusted code, send them this one. It genuinely helps the show grow, and it helps me keep making this for people who actually live with these systems.

You can find the weekly brief at OnCallBrief.com and the full show notes, links, and past episodes at ShipItWeekly.fm. I'm Brian Teller from Teller's Tech. Thanks for listening. And remember, before execution, trust is a policy. After execution, trust is a blast radius.

AI security is in a weird place right now. On one side, there is real excitement. People are building faster. Researchers are moving faster. Bug hunters can narrow huge codebases down in ways that would have felt impossible a few years ago. The blank page is less scary. The first draft is easier. The first pass through a massive repo is not quite as painful. And honestly, that part is cool.

But on the other side, security teams are looking at the same acceleration and asking a much less fun question. What happens when attackers get faster too? Because this is not just about AI writing code. It is also about AI finding bugs, AI helping weaponize vulnerabilities, AI making it cheaper to search for weird code paths, injection bugs, RCEs.

And the kinds of flaws that used to take a lot more time and patience to find. And somewhere in the middle of that, we still have the same boring problems. Credentials, IAM, misconfigurations, unpatched systems, source code secrets, people getting tired, teams moving fast, systems nobody fully threat-modeled because the feature had to ship. That is really what this conversation is about. Not AI will destroy security.

Not AI will save security. More like what actually changes when AI speeds up both the people building software and the people trying to break it. I'm Brian Teller from Teller's Tech, and this is Ship It Weekly. Welcome back to Ship It Weekly, where I filter the noise and focus on what actually matters when you are the one running infrastructure and owning reliability. Most weeks, it's a quick news recap.

In between those, I do conversation episodes with people who are building platforms, running infrastructure, finding security issues, and thinking through where this industry is actually headed. Today is one of those conversations. I'm joined by Kat Traxler, a principal security researcher at Vectra AI.

Kat focuses on abuse techniques and vulnerabilities in the public cloud, especially around cloud, AppSec, IAM, and those uncomfortable places where everybody kind of assumed the layer underneath was safe. And I like this conversation because it lands in a more honest middle ground than a lot of AI security takes. There is definitely hype. There is definitely fear.

There was also a bunch of practical reality sitting underneath both of those things.

Kat had just come out of RSA and the Unprompted conference, so we talk about the current AI security mood, the excitement from startups, the nervousness from security teams, the San Francisco Consensus, the zero -day clock, and this growing belief that AI is changing how quickly vulnerabilities can be found, understood, and potentially weaponized. But we also talk about where teams actually get hurt first.

And the answer is not always the flashiest new AI attack. A lot of the time, it is still credentials, IAM, misconfiguration, known vulnerabilities, unpatched systems, the same boring fundamentals that have been ruining everyone's week for years. That tension is what makes this conversation useful. Because AI is absolutely changing security research.

Kat talks about using models to narrow the search space in bug hunting, why they are good at finding certain code-level issues, and why they are not as good at understanding context, privilege, deployment shape, threat models, or whether a flaw really matters in the system where it lives.

We also get into prompt injection, AI-assisted writing, using different models for different research tasks, and why being an expert still matters if you do not want the model to confidently lead you off a cliff. There is a really good thread in here around IAM too. Why do teams keep failing at it? Kat's answer is pretty simple and pretty true. IAM is where people and technology collide.

Credentials, authentication, access, human behavior, shortcuts. Fatigue and attackers taking the lowest friction path it is not just a cloud service problem it is a people problem with an API.

And toward the end we talk about overhyped AI security narratives what actually happens if the zero-day clock keeps shrinking whether we are headed towards more insecure by design flaws and why some of the hardest problems left maybe less about bad lines of code and more about bad assumptions.

So if you work around cloud, DevOps, SRE, AppSec, IAM, security operations, platform engineering, or you are trying to separate useful AI security thinking from panic and vendor fog machines, this one should be worth your time. All right, let's jump in. Today, I'm joined by Kat Traxler.

She is a principal security researcher at Vectra AI focused on abuse techniques and vulnerabilities in the public cloud, especially at the intersection of cloud, AppSec, IAM, and “we thought this layer was safe”. Kat, thank you for joining me. Thanks for having me, Brian. I'm excited to talk about the new accelerated world we find ourselves in. It is. It's definitely a new accelerated world for sure.

Tell me a little bit about what have you seen lately in the last couple of weeks or months as far as AI or security in general? I'm coming off the heels of RSA. So I spent a four-day pilgrimage out to the bay. The vibe was definitely rational exuberance. Exuberance? Yeah. That's not quite a word. But you know what I mean. Everybody was incredibly optimistic. A lot of two-pizza teams.

Somebody breaks off from a big company with... A set of like AI LLM skill sets and are like racing to the future. So in the startup space, people are jazzed. I haven't seen this level of excitement in a while. You contrast that with middle America, where I live and folks are much less enthusiastic about AI and its role in our life. Then you contrast that with, you know, the security sec op folks.

And people are very, very nervous about what's to come. So you just have so many different experiences depending on where you are at in this world. That's fair. I think I hold all of those in my head as well. I mean, just as a person, right? Like, I mean, there's excitement. We talked a little bit before we started. Like, you're able to iterate more.

You have all these projects that you wanted to start that you couldn't start. Now you are. But then there's also the security side where you see the RCEs and attack vectors that are more prevalent than they were maybe a year, two years ago. So, yeah, it's a different world for sure. The security reporters are busy, right? It seems like there's a new massive major story that they're covering weekly.

A new, you know, critical zero-day, a new supply chain attack. The frontier model companies, Anthropic and OpenAI, have showed us that these models are really, really good at finding these specific kinds of vulnerabilities. Not all vulnerabilities, but these specific kinds of code-level vulnerabilities. They can find them like no other tooling has found before.

So we're just having an onslaught of these vulnerability announcements now, and it can be really overwhelming. That's fair. So recently you wrote about, you had a right, you had an article, the San Francisco Consensus. Can you explain that in your words? Sure. That was coming off the heels of another pilgrimage to San Francisco, going to the Unprompted conference, which I was one of the organizers.

I was one part of the CFP process and had a blast communing with all of the fantastic AI researchers, but there was definitely a consensus amongst folks that coalesced and peaked with Nicholas Carlini's, I almost called it speech, but talk. He's with Anthropic's Frontier Red Team. If you haven't seen that talk, please go see it.

He really lays bare the issue of the zero-day clock, which is the idea, not the idea, I mean, the proven phenomenon that The time between a zero-day being announced and it being exploded in the wild had gone from two years. A couple of years ago, it was like two years it would take. You'd have this announcement, and then it would take attackers literally two years to weaponize it. Now it is just hours.

You have the announcement, they can weaponize it. Nicholas Carlini's talk at unprompted late is over there. And I'm telling my dog to not bark at everybody. Can you hear that? I'm sorry. Of course, right in the middle of what I'm talking to people.

This captures the zeitgeist and the, you know, kind of pure, the pure terror slash pure excitement in everybody's eyes that LLM can find code-level vulnerabilities like no tooling has found. Attackers are weaponizing it in minutes.

Therefore, and people don't really explain what the “and then therefore” is, but the and then therefore is kind of assumed that and then therefore systems will crumble, our global technology infrastructure will no longer work. We will have a complete collapse of critical infrastructure. We will go to the Stone Age, move to the woods.

So I'm kind of catastrophizing a little bit, but that's the natural conclusion of the zero-day clock. And that's the consensus, is that we are months away from this scenario occurring. So is it kind of a take on the doomsday clock, where we're counting down to... Yeah, yeah, yeah. Although I think the doomsday clock is sort of like... arbitrarily set by some people.

Yeah, this is, I think this is backed by some real data. And I've read a report from RAPId7 recently that was really good that has a lot of backing data. They never say zero-day apocalypse, but they have this like 26-page report that backs up all of the observed data points in the wild that just kind of come down to. In 2018, it was two years to exploitation. Now it's minutes. And the big question is, and then what?

And I think we should, as a community, spend a little bit more time filling out the “and then what”, as opposed to just kind of letting our minds go wild. And everybody has a different idea of what and then what means. So if you had to rank where teams actually get hurt first, what's at the top? Like, where should we start? Yeah. I mean, historically, it's been credentials.

Historically, it's been mishandling of credentials, reusing of credentials. It's been committing them to source code, having them be static, not having efficient rotation in them, and then misconfigurations. In the new RAPId7 report, I believe that they had said moving to the top of initial access was... just a small suite of known code-level vulnerabilities.

So I'll use Log4j, but just those kind of like reused over and over and over CVE exploits that are known and patchable and are just tried over and over and over again. So by the numbers, those are still the things that are biting people when it comes to initial access. Going forward, the zero-day clock and the San Francisco Consensus says They don't necessarily say that those things aren't going to matter.

What they say is that there's a whole new attack surface that's going to be open and that's all the zero-days that nobody knows about.

All of the code-level vulnerabilities within your exposed on-prem stack or any other piece of infrastructure that hasn't been exploited anywhere that because finding a zero-day is now just costs and tokens that's what you have to now protect for and i'm just saying like that's that's the san francisco consensus is like that that attackers will shift to that exploitation I doubt it because i think all the other mechanisms just work so well that I do not think that that will be, at least in the foreseeable future, the exploitation route that attackers will leverage, at least initially.

It might be a concern for some of the large institutions, Google, IBM, nation-states, but for your average mid-size technology companies, it's still potential reuse on patched servers. That kind of stuff. So what's your take on prompt injection as an ops problem? An operations problem? Yeah, over just like an app problem specifically. Tell me more. What are you thinking more about that?

Like the different layers it interacts with or like how it affects the infrastructure? Yeah. Where that may be. Or if you see it as an issue going forward, more so in the future, operationally. Yeah, I'm not exactly sure how to answer that. I mean, there are frameworks now to protect against a significant amount of prompt injection. They're notoriously difficult to implement and operationalize.

Ultimately, there are bypasses. So, I mean, it's always some level of a WAF. So it's always going to be some whack-a-mole that you're playing with the prompt injection defenses. But I don't see that folks are really even using the most basic level of prompt injection mitigations out there that are available to them. Here in 2026, there are... A significant amount of prompt injection frameworks that could help.

Again, you know, it might come out to be 96 % effective, 97 % effective, 98 % effective. There's always still going to be a bypass. Yeah. Does that answer your question? Yeah, it does. That's good. I probably didn't ask it well. I don't know if you wrote about this or I heard you talk about this.

We're talking about AI helping narrowed the search space in bug hunting oh yeah how are you using that in practice well i can't tell you I am using it to narrow my search space uh no i mean okay so for example, for code-level vulnerabilities you have these like huge massive gnarly codebases that you know, to some level or another, you're like just grepping your way through looking for bad patterns.

And then maybe you're using, this is the old way, maybe you're using a static code analysis tool to kind of help you with some of that work. But ultimately, you're just grepping your way through a problem. But the LLMs love this sort of problem, this sort of like injection issue, which... A large portion of the OWASP Top 10 just comes down to like injection issues.

And they love this sort of problem of taking this massive interwoven complex blob and being injected with the concept of like, you know, bad input in results in bad input out. And where are the potential mitigations? How can they potentially be mitigated along the way, doing all the heavy lifting for you and being able to get down from 500 lines of code to maybe 100 that it's going to flag for you as potentially bad.

I found that it's amazing for these known problems. These issues that you can come down to, this is this bad line of code, and you can then insert this particular type of sanitization or parameterization and fix it. It's something that it's really great at.

It's not so great at understanding context or the threat model around where it's deployed, who the users are, what data those users might be accessing, under what conditions, those subtleties that are going to change. That's a whole different problem set that they're not good at yet, and which makes me realize that I still have a job.

But those code-level things that are like needles in a haystack, they're good at finding needles. They're not good at finding, I would say that they're not good at finding all of the needles. If you want to find a needle, you'll find one, but you won't find all of them, if that makes sense. Is it primarily certain languages or certain query formats? Like, are we talking like SQL injection?

Are we talking like app function calls? Or specifically Python, maybe, because the LLM has that training data? You know, I really don't know what it comes down to. I've done a huge project on just trying to find remote code execution. And then ensuring that it's tracing back to some sort of untrusted input from a user. So being very clear about where... What inputs it is allowing, you know, from where.

SQL injection is great at finding. XSS is great at finding. Not as great as finding things like confused deputy, which are, again, contextual. Like, what's privilege? That's always, like, a squishy thing. You have to be really explicit with it. I mean, the people who are doing bug hunting.

On the client side client-side issues it's great finding IDOR um like it's really great at these very specific not things that don't require a ton of context issues and going from this massive code base to these handful of things and then proving exploitability yeah i i really couldn't tell you why it's good at that why that puzzle it it turns on really well as opposed to it doesn't understand the nuance of privilege.

Are you using public models or are you training like your own data? No, I'm using a lot of different public models, but I switch between a ton of different ones. Obviously the Claudes, I'm switching between the various versions of Gemini. I'm switching between also some local models like Gemma. So I try to be efficient with my token usage.

Folks that are wanting to get into bug hunting, I think there's a lot of advice that just says like run it all on Claude Opus 4.6 and just go to town. I'm much more, I'm just much more delicate around, which model I use when. Because I think that they can do... They play well off each other. You can use one to catch the bugs of another or to help catch the faulty process of another.

One might have a glaring gap that you just aren't aware of that another one can catch. Yeah, I know. ChatGPT is really good at spatial awareness, whereas Claude's very good at generally coding -type challenges and problems. Is there a specific model that maybe works better for prompt injection -type research, or is it more nuanced than that?

I mean, all of the bug bounty people will just say, turn Claude on to your target and go to town. I don't think you're going to go wrong with that. It just depends on how efficient you want to be with your token usage. I think it's overkill to use Claude on the vast majority of tasks.

Now, if that's your profession, if you are a full-time bug bounty hunter, you're going to scale up to as many Claude Max subscriptions as your budget will allow and just set them off. Me, not so much. I'm going to be a little bit more delicate with what I use when. And I think that's kind of fun because you get to learn all of their intricacies.

And I don't know if you heard, but just today there was the revelation from Anthropic on Project Mythos, or sorry, Mythos, their next iteration of models and Project Glasswing. which is their private preview of Mythos to about 40 different companies.

Particular researchers within AWS, Azure, Google, IBM, Cisco, Palo Alto, CrowdStrike, kind of like all the big ones, have private access to this incredibly powerful model, more powerful than the current iterations. And they're not planning on releasing it to the public. They are wanting it to only be in the hands of hand-picked organizations to allow them to shore up their infrastructure for potential onslaughts.

I wonder how the U.S. Government will react to that. They weren't on the list. I want to see that list of MAGIC 41, you know, Cisco, IBM, Red Hat, Kat Traxler.

That is what i'm looking for that'd be a cool list to be on for sure yeah that would be a great list yeah what parts are AI good at versus what do you still need a human for like is there any part where you think that like AI is not quite there yet i mean we talked a little bit about context and not having like contextual awareness around like large monorepos or large monolithic apps but are there other areas maybe where i mean for example For example, writing.

I do a ton of writing. And I have had LLMs just generate an article for me and it's crap. It sounds like garbage. I can't handle it. But I can go through and put together a decent rough draft. Write it as if I'm just writing for myself very casually. I do not concern myself too much with... Spelling mistakes and word order if I mess up here and there. But make sure I have like the needs of my argument.

Sometimes I'll even do it in bullet points and then send it to my LLM of choice. Have it with some very direct instructions on like the tone and how much to shift away from my tone. Have it be very constrained and then iterate on that from there. So I think it has a great place in writing. We just, I just personally, use it somewhat sparingly. So just getting your voice out there I think is great.

It's such a barrier to entry for some people to blog, to put articles out there. They see that blank page and they don't really know what to do. If they could say, you know what, just write a table of contents, a list of bullet points, sentence fragments, paragraphs here and there, and then start. I think that can maybe unlock a lot of knowledge that people have and they can publish their ideas.

But if you don't give it enough meat, it's really going to come out pretty poorly. So that's with writing. And then, you know, obviously you need that expert level knowledge because it's going to, it's not, the models aren't truth seeking. They're not going to write you research that's necessarily true. And when you challenge them on things, they'll be like, oh, I'm so sorry. That's not at all what I thought.

So it can supercharge an expert's workflow to really get your ideas out there. But if you're not an expert in the space, it can really lead you astray. Garbage in, garbage out to a degree. Garbage in, garbage out. Yeah, garbage in, garbage out. I was iterating on something now. And there was a ton of instances where there was an LLM just hallucinating on what you could really do with cloud flow logs, VPC flow logs.

And at some point I had an idea that it would have some really like packet-level revelations in it. And if I wasn't an expert in this space, I would have just assumed, okay, they're telling me I can do X, Y, and Z with these VPC flow logs. And as an expert in the space, I can challenge and say like, this is impossible. This is impossible. This is impossible because these are the, this is the.

These are the fields you have. And this is the kind of inferences you can make based on those fields. But it is not truth-seeking. It's going to try to make it fit. Yeah. And so if you don't challenge it. Okay. So let's stay in cloud a little bit. I'm curious your take on this. So if most breaches still start with boring fundamentals for the most part. Why do we keep failing at IAM? Why is that still such an issue?

It seems like. Hard. IAM is really, really hard. I mean, IAM has this thing where it's actually just about people and people's uses of things and people, process, technology. The people part is the hardest thing. So even though like IAM is another service, just like computer S3, it's actually just like the intersection of people.

And technology, which, you know, how people use their credentials, how people authenticate, all of that is about human behavior. That's just the worst. I mean, I don't know about you, but I'm the worst. So I don't know if it's a technology problem or it's just that within those three pillars, people, process, and technology, it's always the most difficult.

I feel like we all think we're not susceptible either to phishing attacks or whatever, like not having strong credentials or some sort of injection that happens that we think that we would catch. But everyone gets tired. Everyone gets complacent to a degree. We're all people just trying to do our best. And conversely, attackers, threat actors are all people just trying to do their best. And what are they going to do?

They're going to take the easiest low-friction route, which is going to be credentials. Yeah. Is there any overhyped AI security take that you disagree with? I think it's like the... The article that I published about the San Francisco Consensus is probably the biggest one that I take exception with.

Once we do get to the point where the zero-day clock gets to actually zero, that's not actually a catastrophe for the lives of average people. I think that if you're a technologist, in the space, deeply embedded in the space, and you see all of these crazy advancements within LLMs, and you see their capabilities for offensive research, it's very natural to be alarmed.

And then you're, you being alarmed, alarms other people. And then there's this like, there's a ferocity that goes around. And so I think the take is that we are on the precipice of a catastrophe. But if we step outside of that and we say, what actual catastrophe is going to happen?

I don't think that we will see large-scale, system-wide availability, confidentiality problems that will affect the vast majority of humans on Earth. I think it's going to lead to some no good rotten days in the SOC. But that's not the same. Where do we go from here then? What do we do? What do we do? Where do we go from here? Keep on, keep on, right? Like one foot in front of the other.

I think I take this maybe more middle of the road approach because I do live in middle America and I don't live on one of the coasts. And if you're in that bubble, it can be really easy to drink the Kool-Aid and live in the zeitgeist. But when you, Live elsewhere where the vast majority of people don't work in tech. The vast majority of people have other concerns within their life.

You can kind of see the full breadth and scope of humanity and you can kind of say like, yep, this problem is coming and we should absolutely be working on it. However, it actually won't lead to worldwide system collapses. So what do we do? Show up for work tomorrow. That's all we can do. That's all you can do. That's all you can do, Brian. Make sure you get your PTO in.

Don't fall into the fallacy that you and your skills are so important that you can't spend your hard-earned time resting and recharging. I think that's a common fallacy in IT and security folks that this is all going to come crumbling down if I step away for a minute, and it won't. So, okay, closing thought. Are we headed towards more insecure-by-design cloud flaws given that? Maybe. Maybe, Brian. Maybe.

You know, I am always struck when people talk about all of the vulnerabilities that LLMs can find. And I'm always asking people to define what they mean by vulnerability. And it always comes down to like a code-level vulnerability, not necessarily a design flaw.

So these insecure-by-design problems, these like “I did not threat model appropriately” ones, I'm not sure if they're going to increase as a result of LLMs, but I do think that they might be some of the last ones left on the table. The best way to think about insecure-by-design is to think about Apple AirTags. And, you know, when they were released, nobody thought about the threat model of somebody of stalking.

And yeah, right. So and it took somebody writing a series of articles about. Women being stalked by their partners with Apple AirTags before Apple took it seriously to then build in both Android apps and newer capabilities on your phones to then know if you're being followed by an AirTag. So that's kind of the classification of, that's like the best way to describe an insecure by design issue.

And often they are the most fun to find because they require some amount of convincing. Like convincing somebody that, no, you just didn't think about this correctly. So that's why I love looking at those specifically. If we're going to be relying more on LLMs to build our software and have less human oversight, maybe it does mean that we're going to have more of these insecure-by-design flaws. Hopefully not.

Hopefully it just means that I still have a job. Either way, I think they're not going to go anywhere anytime soon. I guess the problems just change over time, right? I mean, the problems don't go away. The shape of it might change. Yeah, and it's all about expectations. What does your consumer base expect from this product?

Maybe 50 years ago, if AirTags came out and they could be used for stalking abused women, maybe 50 years ago, it wouldn't have been considered a vulnerability. But in 2020, it certainly was. So it's also just about the expectations of your customers. If this was publicized, would there be an outrage? Yeah. Awesome. Well, where can people find more of your work and find you online?

I'm chronically online on Twitter, as NightmareJS. Also on Bluesky and then LinkedIn. Awesome. I will put links to all of those in the show notes. Thank you so much, Kat, for coming on. Really appreciate it. Thanks for having me. All right. That was my conversation with Kat Traxler from Vectra AI. My biggest takeaway from this one is that AI security is not one story. It is a speed story. It is a context story.

It is a fundamentals story. And it is very much still a people story. Because, yes. AI is changing vulnerability research. It can help narrow massive codebases. It can find certain code-level bugs faster than older tooling. It can help researchers move from where do I even start to here are the suspicious paths worth looking at. That matters. But it does not magically understand the whole system.

It does not automatically know the threat model. It does not always understand privilege. It does not know what your customers expect. It does not know which weird business rule turns a harmless-looking behavior into a real abuse path. That is where humans still matter. And honestly, that might be the more interesting version of this whole AI security conversation.

The model can help find needles, but the human still has to know which needles matter. That came up a few different ways in the conversation. Bug hunting. Writing, prompt injection, IAM, insecure-by-design flaws. The pattern is pretty consistent. AI can accelerate the work.

But if you do not have the judgment to challenge it, constrain it, validate it, and understand where it is guessing, it can absolutely make you faster in the wrong direction. And that is not just true for researchers. It is true for platform teams, DevOps teams, SRE teams. AppSec teams.

Anyone using AI to generate code, review code, write policies, inspect infrastructure, or explain some terrifying IAM graph that nobody wants to look at manually. The other thing that struck me is Kat's point about the boring stuff still being the place teams get hurt. Credentials still matter. IAM still matters. Misconfiguration still matters. Known vulnerabilities still matter. Attackers are practical.

They are going to take the lowest friction path that works. And a lot of the time, that path is not some cinematic AI-generated zero-day chain. It is a leaked key, an over-permissive role, a reused credential, a server that should have been patched, a service account nobody remembered existed. That does not mean the AI risk is fake. It means we need to hold both ideas at the same time.

AI may compress the time it takes to find and weaponize certain classes of vulnerabilities. That is real, and security teams should care. But if your IAM is a mess, your secrets are everywhere, your patching is inconsistent, and nobody owns the exposed attack surface, then the scary future problem is not an excuse to ignore the thing already sitting on fire. I also like the discussion around insecure-by-design flaws.

Those are not always easy for tools to find because they are not just bad code. They are bad assumptions. The AirTag example that Kat brought up is a good way to think about it. The issue was not just whether the device functioned as designed. It was whether the design accounted for how real people could abuse it. That is a different class of security work. It requires context.

It requires thinking about users, incentives, misuse.

Power dynamics expectations and what happens when a feature gets used by someone with bad intent and if AI removes some of the easier code-level bug hunting work or at least makes it faster i would not be surprised if more of the interesting work shifts towards those deeper design and abuse questions not is this line vulnerable more like what does this system allow someone to do that we did not intend that is harder and probably more important so my takeaway is not panic about AI security it is also not relax everything is fine it is more like keep your head use AI where it helps let it speed up the annoying parts let it help with first drafts first passes code search query generation pattern matching and research flows.

Do not outsource judgment. Do not let the AI output become the security conclusion. And definitely, do not let the excitement around future attacks distract from the fundamentals that are still hurting teams right now. Credentials. IAM. Patch the known stuff. Understand what your system actually promises. And when the model sounds confident, remember that confident and correct are still not the same thing.

I'll have links to Kat, her work, and anything else we mentioned in the show notes. If you enjoyed this conversation, follow or subscribe to Ship It Weekly wherever you listen to podcasts. It helps the show and it makes sure you get both these conversation episodes and the weekly DevOps, SRE, platform, cloud, and security news recaps. You can also find the show notes and links over at shipitweekly.fm.

Thanks for listening, and I'll see you later this week.