Ship It Weekly Transcripts

A lot of infrastructure stories sound small right up until they change how people build. S3 gets a file system interface. A fake plugin turns into a foothold. A security incident comes back for a second round and gets worse. An old Kubernetes path starts running out of runway. And the next generation of workloads is already pushing the platform to act differently. That's the story this week. Not just new features.

More like the layers around the system are becoming the system hey I'm brian teller I work in devops and sre and I run teller's tech this is ship it weekly where I filter the noise and focus on what actually changes how we run infrastructure and own reliability. Show notes and links are on shipitweekly .fm. If the show's been useful, follow it wherever you listen. Ratings help way more than they should.

And if you want more signal between episodes, check out oncallbrief .com. We have five main stories today, then the lightning round, and we'll wrap with the human closer.

We're starting with amazon s3 files because aws just changed the way that s3 can show up in applications and I think people are going to oversimplify that story really fast then we've got the malicious npm packages pretending to be Strapi plugins which is just a very 2026 supply chain story after that trivia again but this time as the uglier follow -on we talked about the earlier incident before and now the sequel is

Broader and hits the ci path even harder then we'll hit kubernetes ingress to gateway 1 .0 and the bigger reality that teams are going to have to migrate towards gateway api without pretending that this is a one -click swap and finally kubernetes agent sandbox because the platform is already starting to admit that agent -style workloads are different Story one.

Amazon S3 files is interesting, but I would not call it S3 is EFS now. Let's start there. AWS launched S3 files this week, and the headline version is simple. You can expose an S3 bucket as a file system and mount it from EC2, ECS, EKS, or Lambda. AWS says it supports NFS version 4 .1 operations, syncs changes back to S3, and uses a high -performance active data layer to make file access fast.

What I think is interesting here is how this compares to the old S3FS or generic FUSE -style pattern. Because people have been trying to make S3 feel like a file system forever. But that old approach usually came with the usual caveats. Weird semantics. Performance trade -offs, caching weirdness, consistency questions. General, yes, it mounts, but should it? Energy. This feels smarter than that.

Not because it magically turns object storage into a perfect shared file system, but because AWS is very clearly building and managing the translation layer for you instead of leaving you to duct tape it together yourself. So this is related to the old S3FS and FUSE idea in the sense that it solves the same human problem. Which is my app or tool wants files, not bucket APIs.

But it is not just a nicer wrapper around the same old hack. It is AWS putting a managed service boundary around the pattern. That said, I still would not read this as S3 is basically EFS now. I think the better read is that AWS put a managed file system layer in front of S3 so file -based tools and workloads can interact with bucket data more naturally.

Especially AI and ML -style workflows, without everybody inventing their own copy pipeline or mount trick. Where I still have questions is real -world behavior once this is under pressure, multi -writer contention. Sync behavior. What the latency story looks like once the happy path cache assumptions stop being happy. So for me, the takeaway is not S3 replaced file storage.

It is more like the object store just got a much friendlier interface for workloads that were never really written to think in buckets. And that matters because once the interface changes, people start building differently. Story 2. The fake Strapi plugin story is ugly, but very believable. Next up, the supply chain mess. Researchers found 36 malicious NPM packages disguised as Strapi plugins.

They followed the Strapi plugin naming pattern, used post -install scripts, and came with different payloads aimed at Redis and Postgres SQL. Credential harvesting, reverse shells, and persistence. That is the part that I think lands. Not just NPM had malware again, more like this is what the modern trust problem looks like.

The package ecosystem, the install workflow, the environment variables, the local services, the containers, the database paths. All of that is fair game once developers or pipelines run something they should not have trusted. And I think the Strapi angle is useful too. Because it plays on something real. Developers expect there to be plugins. They expect those plugins to be named in ways that feel obvious.

Attackers know that. So the trick is not brilliance. It is just getting close enough to expected behavior that nobody slows down. That's why these stories matter. Not because every team uses Strapi. Because every team uses something with a plugin model, a package manager, and a little too much trust in what shows up during install. Story 3. Trivia is back, and this one is the uglier sequel.

Next up, the trivia follow -up. We talked about the earlier trivia incident back in episode 24. This is not just the same story repeated. This is the second chapter, and Aqua's own write -up says that pretty clearly. On March 19th, Aqua said a threat actor used compromised credentials to publish malicious releases of trivia.

Trivia action and set up trivia and that this was a follow -up from the march 1st incident because the original containment was incomplete that matters because now we are not just talking about repo disruption or suspicious activity around the project now we are talking about official releases and ci tooling being turned into a distribution path for credential theft aqua said affected users should treat reachable

Secrets as compromised and rotate them Multiple security write -ups also called out that the compromised actions and releases were used to target CICD environments directly.

This is the nightmare version of security shifted left. The scanner is not outside the blast radius. The security tool is not outside the blast radius. The thing scanning your code may need the same level of paranoia as the thing deploying it. And I think that is why this one is worth covering even though we already talked about Trivi before. The first story was something is wrong.

The follow -on story is the earlier compromise turned into malicious releases and broader CI impact because the attacker still had a path back in. That is new enough and ugly enough to justify a follow -up. Kubernetes' ingress to Gateway is the kind of migration story platform teams actually live through. Now let's get back to platform work.

Kubernetes announced ingress to Gateway 1 .0, and the framing is pretty direct. With Ingress Engine X retired as of March 2026, the question is not really whether teams will move towards Gateway API, but how they do it safely. That is why I like this story. Because it is honest. A lot of migration stories get told like they are basically just YAML translation and maybe a few find and replaces.

But anybody who has lived through one of these knows the hard part is not converting manifests. The hard part is discovering all of the weird behavior your current setup has been carrying for years and deciding what actually needs to survive the move. And Gateway API is a real shift. More modular. More expressive. Better separation of concerns. Great.

But the migration still has to happen in environments where teams have old annotations, old controller -specific assumptions, and production traffic that does not care how elegant the destination architecture looks on a blog post. So I think this is one of those very real platform stories where the value is not the destination alone. It is having a safer path off the thing everybody relied on for years. Story five.

Agent Sandbox is Kubernetes admitting the workload model is changing. Last main story. Kubernetes also published a piece on Agent Sandbox. And I think it is worth paying attention to. Not because everybody needs it tomorrow, but because it signals where the platform sees things going. The argument is basically that long -running AI agents do not behave like normal short -lived stateless workloads.

They need stable identity, workspace persistence, isolation, scale to zero behavior, and warm pulls so they do not feel painfully cold every time they wake up. That feels right to me. Because even if you are not all in on agent hype, the platform problem is real.

Once you have workloads that hold context longer, need tool access, spin up and down differently, and may need tighter isolation than a normal web app, the old abstractions start feeling a little off. And that is part of why I like this story paired with S3 Files. S3 Files says here's a friendlier interface for data because these newer workflows want file semantics.

Agent Sandbox says here's a friendlier runtime shape because these newer workloads want different lifecycle semantics. Those are different stories, but they rhyme. The platform is getting nudged by new usage patterns, whether we want to call that AI, agents, or just another round of abstraction pressure. A few quick ones before we wrap. Fortinet customers have an actively exploited problem on their hands.

Fortinet released an emergency update for Fortinet EMS after CVE -2026 35616. A critical flaw was exploited in the wild, and CISA added it to the known exploited vulnerabilities list on April 6th. Karpathy had one of those posts making the rounds again that taps into the bigger conversation around how much software work is shifting from direct coding towards orchestration, review, and steering.

I do not think the useful takeaway there is blind hype. I think it is that a lot of engineers are already feeling the interface to software work changing, even if the results are still uneven. And then there is Proofshot, which I actually think is a pretty fun tool mention. It is an open -sourced CLI that tries to give AI coding agents eyes by recording browser sessions.

Capturing screenshots, collecting console and server errors, and bundling proof artifacts for humans to review or attach to pull requests. If agents are going to claim they built something, they should probably bring receipts. I think the cleanest closer for this one is this. The interface layer is becoming the story. Not just the database. Not just the bucket. Not just the cluster. And not just the scanner.

The layer in front of it. The workflow around it. The migration path off of it. The trust assumptions baked into it. This is where a lot of the real action is now. S3 is still S3. But the file system interface changes who can use it and how. Until the second incident shows it can turn into a broader distribution path for secret theft if the first breach was not fully closed. Ingress is still just ingress.

Until the controller everybody built around it starts aging out and you realize how much invisible behavior you were carrying the whole time. And Kubernetes is still just Kubernetes, but even it is starting to say out loud that newer workloads may need different shapes, different assumptions, and different guardrails. That's the human side of this stuff.

A lot of ops pain comes from realizing too late that the helper layer was never really a helper layer. It was part of the system, maybe the important part. And the teams that do well are usually the ones that figure that out before the incident does. All right, that's it for this week of Ship It Weekly. Quick recap.

Amazon S3 files and why I think it is more managed file layer over S3 than S3 is EFS now, especially compared with the old S3FS and Fuse -style approach. Malicious Strapi flavored NPM packages, the trivia follow -on incident, and why it was worth revisiting. Kubernetes ingress to gateway and the bigger gateway API migration pressure and agent sandbox as a look at where the platform may be headed next.

Links and show notes are on shipitweekly .fm. You can also find the video versions on YouTube. And if you want more signal before the episode, check out on callbrief .com. If this episode was useful, follow or subscribe wherever you listen, and send it to the person on your team who keeps having to explain that the thing everybody calls just glue is usually sitting directly on the blast radius. Thanks for listening, and I'll see you next week.

A lot of teams hear internal developer platform, and they immediately think new portal, new dashboard, new interface. But that is probably not where this is going, because the real problem was never just we need another place to click. It was always too much friction, too little context, too many handoffs, and too much time between idea and production.

And now with AI and coding agents starting to change how work gets done, that question gets even sharper. If developers are spending less time clicking through dashboards and more time working with agents, then the platform layer has to change too. It has to become less about UI and more about context, automation, and safer paths to action. Hey, I'm Brian. I work in DevOps and SRE, and I run Tellers Tech.

This is Ship It Weekly, where I filter the noise and focus on what actually matters when you are the one running infrastructure and owning reliability. Most weeks, it's a quick news recap. In between those, I do interview episodes with people building tools, running teams, and thinking hard about where platform engineering and DevX are actually headed. Today is one of those conversations.

I'm joined by David Tuite, founder and CEO of Roadie. A managed internal developer portal built on Backstage. And this one is really about what IDPs are for, how teams should think about them, and how that changes as agentic workflows become more normal.

We talk about the difference between a platform and a portal, the three big problems teams are usually trying to solve with an IDP, why adoption is often more of a human problem than a technical one, and why a lot of teams should start with automation first, not just a service catalog. We also get into self -hosted backstage versus managed backstage versus more opinionated SaaS options.

What trade -offs actually matter there? And why the future may be less about single pane of glass dashboards and more about packaging the right context for humans and agents at the moment they need it. And maybe my favorite part of the conversation is that David does not really answer this from a hype angle. He keeps bringing it back to something a lot more useful. Talk to your users. Define the real problem.

Solve what painful is, and do not build a platform strategy around a fantasy version of the future. If you like these kinds of conversations, follow the show wherever you listen, subscribe on YouTube, and check out shipitweekly .fm or tellerstech .com for more episodes, show notes, and everything else that I'm building. All right, let's jump in. Today, I'm joined by David Tuite. He's the founder and CEO of Roadie.

Roadie is a managed internal developer platform built on Backstage. And David's got a strong take on where IDPs go next as coding agents become normal. David, thank you for joining me. No problem. Thanks so much for having me. Yeah, I've been... Kind of around the IDP space since its genesis, I guess, about five or six years ago.

I was originally a software engineer and then an infrastructure product manager at Workday. We built a IDP internally inside Workday before IDPs were a thing, before Backstage was open source, before companies like mine existed, obviously. And so we kind of learned some of the lessons through that process. We experienced the pain points of why people want to have an IDP.

And when I say IDP, what I mean is internal developer portal. As opposed to platform. But yeah, we experienced the pain points of why you would want one in the first place. And that was what gave me the impetus to start Roadie originally. Give me the thought behind platform versus portal. Yeah, well, the way I differentiate them... Is kind of around the goal of what they're trying to provide.

So platform, I think about being, as being deeply integrated, like vertically integrated throughout the stack. So it's not just the visualization of what's running and where and so on, but it's also the orchestration. It's what is deployed where, how do I move it to something else? And it'll try and control that entire stack.

Whereas a portal, I think, and this is not necessarily true, but as a general rule is a thinner layer, which is trying to sit more broadly across more of your stack. So how this manifests itself really. With a platform, you often have like a pretty high initial setup cost where you literally have to move everything onto the other platform.

Whereas a portal should be more able to kind of work with whatever you've got, integrate with the tools that you already have, and just give you a unified interface across all of those tools. So if a team was deciding between self -hosted backstage versus a managed backstage or another IDP, what signals do you think matter most? So, I mean, well, let me maybe take it back a step, if you don't mind.

And talk about why you would want to have a developer portal in the first place. So, because if you can answer that question first, it's usually a good place to start when it comes to which solution should I have, right? But typically, and this is, I recommend that people do this all the time, but try to define the problem first.

And so what I see out there in the marketplace, and this has been broadly true, I think, for the past five years that Backstage and other technologies have existed, is... Three different problems or three different categories of problems, I would say. So the first one is that what I would call the discoverability problem. People feel like there's too much software.

There's too many humans producing all of this software. And it's just difficult to answer basic questions about, you know, where are the bills for the XYZ service? Does it even have bills? Who would I go to ask in the first place? And obviously that...

That extends to, or the complexity of those questions is multiplied by the number of different tools that you're using, the number of services that you've got, the number of humans, and so on. And so that becomes a problem in and of itself. Really, what you're trying to do there is get the right context to the right humans at the right time.

And we'll talk about this later on, but that problem is probably changing quite a bit as we become more agentic. The second category of problem is around speeding up the path to production. Really like speeding up the time it takes from an idea in a developer's head to a service which is actually running in production.

There's typically, I mean, back when I was at Workday, we had a 40 -page Confluence document full of checklists that you had to go through. Teams talk about ticket ops a lot. Most operations require opening a Jira ticket against the platform team and waiting for it to be done. You want to kind of take away all of that.

And essentially provide self -service automation to the developers who are running their software on the platform. And then the third category of problem is really about putting guardrails in place to ensure that software which is running on the platform meets a certain set of requirements, right?

So this could be simple things like, you know, if you want to have your software running on our platform, you should have a readme in the repository, right? And it can extend to far more complicated things around using the correct Docker base image, let's say, that the platform team has just released to run your software on the platform.

And really that's about improving homogeneity, trying to make it a more consistent environment in the platform. To improve reliability and stability and speed. So those are the three kind of ways that people typically explain their problem when they originally come to me. And that's the job of an IDP is to try and solve one or more of those problems for a team. Does that make sense? No, totally.

So with those three areas of focus in mind, is there a overwhelming majority like common? Problem statement that you see over and over again? Yeah. Well, they typically always have the discoverability problem. So they'll have that problem first and then they'll have one or more of the other problems. Now, it varies a bit in terms of size of company.

So firstly, for IDP purchasers, they're typically 100 engineers plus. That's the kind of minimum size of company. Now, we do have smaller customers and there are smaller teams out there who want IDPs, but to really be successful or really have the pain that an IDP can help you solve, you want to be 100 engineers or larger because you want to have a certain size.

Is a certain amount of complexity in order to be able to, in order to need to purchase a platform or a portal to solve the problem. But after that, it's really kind of maturity specific or it depends on the maturity of the team, right? A lot of the time, if they're a new platform just getting started.

Or if they're trying to migrate from a legacy platform to a new platform, then the top priority for them will be the automation pieces. It will be speeding up the path to production because that's pertinent for the job, the initiative that they're trying to complete at that time. We see a lot of teams also who are moving, let's say, from Bitbucket to GitHub or Azure DevOps to GitHub or something to GitLab, et cetera.

And so they want to provide automation to their teams in order to speed up that migration. And so they'll be less interested in the catalog and the discoverability pieces, but they'll be more interested in the automation pieces. So it depends a little bit on what the initiative is. That's kind of why one of the first things I'll often ask teams when they come to Roadie is, What is the initiative?

Why is now the right time to bring in an IDP? Why wasn't it six months ago? What's going on in the company? And really that informs the IDP that you choose, the features of the IDP that you want to roll out first and how you want to use it and how you want to communicate it to your teams. Cool. So I know that I want an IDP. I've figured out that problem statement.

Maybe I need a service catalog or I want that automation. How do I decide between port, roadie, backstage? You know, where do I start? Yeah, yeah, yeah. Well, I mean, regardless of which platform you're going to have to choose, and there is a way to think about that also, you want to think about how you're going to get it into the company or how you're going to drive adoption of it.

Uh throughout the company right so so I mean this is often much more of a human problem than a technical problem I think and people underestimate that so the if you take a simple question you're gonna know you're gonna want to have a catalog in your in your in your developer portal that's typically the piece where a lot of the functionality hangs off you know you go and look at a service and then you can see who owns

It or you can redeploy it or click a button to do something right So how are you going to populate your catalog is one of the first things that I would suggest that people think about.

The two paths that you have there are some sort of automated population where you're essentially going to connect it up to GitHub and pull in the repos and connect it up to Argo CD and pull in the applications from there. And it makes form some sort of relationship between those two things. That's going to be good for catalog completeness, but it's going to be poor for...

Giving the teams a sense of ownership over how their service is represented in the IDP because they'll feel like you just did that work and it's nothing to do with them really. The other end of the spectrum maybe then is backstage is kind of traditional YAML files where you're asking teams to put a YAML file in their repository and maintain that over time. Which is the other end of the spectrum.

It gives them a high sense of ownership, but it's going to be less up to date. There's going to be a lag between changes happening in the real world and changes being represented in that YAML file. And there's going to be a slower rollout there because you're asking teams to do something. But those are both ways that we've seen be successful.

But even beyond that, you have to think about what is our data model in this company? And that can often be a hard thing to pin down in the first place. Before you even try to define that in YAML files or in code or however you're going to do it.

You can get your architects in a room and talk to Bill and talk to Mary, and the two of them will disagree about what constitutes a service in the company, or is the payment service part of the payment system or the billing domain? What is the nomenclature that we use to describe the company, right? These are all things that need to be decided at a human level before you can dive into choosing your IDP, I think.

You would be surprised, or I think a lot of people who maybe work in smaller companies would be surprised at the level of chaos that exists out there around some of these operations in larger companies. We have plenty of customers who can't answer a simple question like, what's a team? If they go to Workday, they'll see one definition of who's on a team. If they go to GitHub Teams, they'll see another definition.

And these things mean... They're not incorrect. It's just that in different contexts, what you would call a team is slightly different. Maybe, you know, your manager in Workday is a certain person. And so you are in that team, but you're also on call for a service which is actually owned by a different team. And so you're kind of in that team too, right?

And so the messiness of human environments really takes a toll when it comes to organizing your data model and keeping everybody happy. And so I do think that teams should... Think deeply about how they're going to do that before they get an IDP at all. Other good things to think about are what is your first feature? What's the first problem you're going to try and solve with your IDP?

Which I think can have a large effect on whether or not the initiative is successful. Recommend that teams start with the automation use case in backstage -based deployments or in Ro. The reason being that catalogs have network effects effectively, right?

When you take some time to build out your catalog and in those initial stages, when people go to visit the catalog, they're not going to see the things that they need. And so they're going to be discouraged from adding their own software. It's kind of, it ends up being a bit like a social network, like Facebook or anything else. Whereas automation has immediate ROI.

If you can take a process that used to take a month and you can make it now 15 minutes, then you can very easily multiply the amount of times that that process is run by the time saved. And then you can easily explain that to your VP of engineering and claim a win and take that forward. There's no waiting period before you achieve that benefit. So we recommend people start there a lot of the time.

Of course, taking into consideration the actual needs of the organization. If you don't have a need for automation, don't start there, obviously. But then, okay, so after all of that happens, you've decided that you do have a real use case for an IDP. You've thought a bit about how you want to roll it out and so on.

The market, I like to think about, kind of divides itself on a spectrum of extensibility to ease of use or speed of deployment, let's say. So at one end of the market, you do have self -hosted backstage. It is, and let's be clear, self -hosted backstage is the market leader. Like probably 90 % of the market is using self -hosted backstage.

3 ,500 companies out there who have adopted Backstage from the largest companies in the world to small teams of 10, 20 engineers. The thing to keep in mind about Backstage, though, is that it's a framework for building a developer portal. It's not a developer portal itself.

It's kind of funny to think about, but back in the early days of the Backstage community, I noticed that there was a trend of people searching for Backstage Docker container or Docker image on Google because people were looking for not a... TypeScript library, a bunch of TypeScript libraries that they could build together into a developer portal, but they were looking for a Docker container that they could just run.

And so one of the first things that we did in the community was actually just build that Docker container and make it available. But that was a long time ago. Things have moved on quite a lot. From there. But the core nugget of truth is still true. Backstage is a framework for building a developer portal.

You mentioned just before we started the call that Spotify had said that teams should be six to eight engineers dedicated to Backstage to be successful for it. Our data shows the same thing, really. And we ran a survey last year called the State of Backstage. And when we analyzed the data from that, people who reported being very satisfied.

With their self -hosted backstage deployment had at least three engineers dedicated to it. That's kind of what we saw across the market. So that's all true. Time to value you're expecting to see is six to 12 months. This may have changed recently with the development of AI probably has come down in the past three months. I feel like Opus and improvements to Codex have really increased what AI can achieve.

So that might come down a little bit. But the core work that you have to do is still there. It's about going and talking to your internal developer community, distilling what they need. Turning that into features. A lot of that work you still have to do. Building in things like role -based access control, which doesn't come with Backstage out of the box.

You know, this is all kind of boring stuff that I think at least we would claim that your engineers are better off not doing and just using a kind of existing solution instead. But you do have Backstage, self -hosted Backstage. That's at one end of the spectrum. It's going to be the most extensible thing that you can get.

No matter what your internal tools are, no matter how homegrown or legacy they are, it's going to be able to integrate with them because you basically build your own developer portal.

Then at the other end of the market you have port cortex etc um really what you're trading off there is control for reduced time to value so you're going to get more features out of the box You're going to get things like scorecards, role -based access control, built -in AI, et cetera. All those things are going to come out of the box and you're going to be able to just use them on day one.

And then when you're trading off in terms of control is vendor lock -in, restrictions to the amount of integrations that they provide. You kind of are locked into their roadmap. Same as with any other SaaS solution, limited customizability, et cetera. But it's going to be more opinionated. You're going to get started more quickly. And so that's a path that you can go, of course.

Roadie is slightly different, and I don't want to turn this into a sales pitch for Roadie, but just to explain, because we are based on Backstage, we give every customer a full Backstage stack, but then we build in the kind of ease of use that you need on top of it. So, you know, adding integrations is just a matter of dragging and dropping things around the place. There's no re -employments.

We give people search, which is based on open search. Role -based access control, scorecards, et cetera, all those things are built in. So the idea really is just that you're getting the extensibility of backstage, but you're also getting the ease of use and the reduced time to succeed that you will get with port recording.

So speaking to the automation angle, how do you think agentic workflows change IDPs or how they may be implemented going forward? Yeah, I think, I mean, we're in the middle of a massive shift right now. I mean, I don't know how much you can, how much you're perceiving it. I still am hands -on keyboard writing code whenever I can, you know, which is typically just my spare time, but I'm making small changes.

I'm not allowed, the engineers don't allow me to make changes to the actual products anymore, but I hack on our marketing website whenever I can. And, you know, I haven't written a line of code in maybe three months or so, since Christmas, I would guess. Everything is produced by Claude. So there's a massive shift happening right now.

I think that IDPs existed to try and bring information from external sources, LaunchDarkly, CircleCI, whatever tools you're using. Into one UI so that you could have a kind of a single pane of glass experience. You know, that's what people would talk about a lot. I think that the need for that is shifting.

At least I feel now like I want to be in my terminal or in Visual Studio Code or whatever it is that you're using all of the time. And I feel like the information can come to me. So I think that's one huge shift which is happening. Why would I go to a catalog and click through an interface? To find a service, to be able to see a piece of information when I can just ask the question directly in my terminal.

So, I mean, that's how I think the shift is being pushed onto the market. And so the UI layers of IDPs, I think, are becoming less and less relevant over time. So that's one change. And then the other thing that we're seeing, and we're a little bit earlier in this, but the more and more work is being done by agents. And I don't just mean when you're in cloud code.

Plug code as an agent, but I mean an actually externally triggered, hey, something went down, an agent fires and makes decisions about what to do about that. So these are operations decisions. It could be, you know, add a second cluster or something, right?

We're earlier in that, but my expectation certainly is that that is going to become more and more prevalent over the next two years or so as we can trust OLMs more and more.

Obviously it's a lot riskier but the earlier stages of that I think are are humans or humans building working directly with llms quite closely to answer questions you know I mean take an example of something like figure out why the payment service went down last night between 3 and 4 a .m right like that's an actual use case from inside and roadie that we were discussing recently but if you think about the context that's required there for the agent to be effective it's What is the payment service?

Where is the logs for it? What log group? Where is the metrics? Which cloud? What region? Etc. There's so much context that an agent needs to be able to be successful. So I see the job of...

IDPs shifting a little bit from providing context to humans through UIs that they can click through to providing what I would call context bundles to agents so that they can have historical data if they need it, real -time data if they need it, concise, accurate, correct, to be able to make decisions about how certain changes in the environment should be reacted to. So building like a centralized MCP for all?

Information around the development cycle I think that can I think that can certainly be part of it yeah um that will give you the kind of real -time access to well let me let me let the agent ask a question to be able to get a result right I think that's certainly part of it I think that I'm not sure that that solves the entire conciseness part of the problem and it certainly doesn't solve the kind of historical data or what has changed over the past hour part of the problem But certainly that will be involved.

Yeah, for sure. So if coding agents make dev faster, why wouldn't DevX teams suddenly have time to build whatever, everything? Yeah, well, I mean, I guess like I would be, the engineer in me would be hopeful for a world where they would. But I guess the CEO in me is not necessarily sure that that's how it's going to pan out. You know, I mean, it certainly does make it easier to make changes, faster to make changes.

That's irrefutable at the moment. But I think that. That doesn't necessarily mean that an internal platform team is going to have all the time in the world to build whatever they want.

I worry that people in the C -suite are going to look down at that team and think, well, okay, now we can actually take two engineers off that team because everybody else is more productive and put them on the revenue -generating side of the business. I still think that that fundamental tension between the revenue generating side of the business and the platform team or the internal teams still exists.

And I still think that trade -off exists. And so it's not necessarily clear to me that you're going to have more time to work on internal projects. I still think that even with AI, I mean, the return on investment equation does change, but I still think it's difficult to prove the benefit of that or prove that return on investment. To the executive level.

So this is a challenge that we deal with a lot in the IDP space because IDPs are at their highest level supposed to improve productivity of developers. But we're not even all that good. And I mean, when I say we, I mean the industry at large is not all that good at measuring productivity of developers in the first place. So showing a delta between where it was before and where it is now is even more difficult.

And I still think that that's going to be true as well for internal projects, even with AI. You can obviously claim the cost is reduced, and that's true. But the cost reduction mostly... Shows up in the front of the application build -out lifecycle, right? So, you know, it's easier to get started. It still gets a bit slower to improve software over time, or it's still slower than it is to start a Greenfields project.

I mean, we see this with backstage upgrades internally, right? One of the benefits of Brody is just that you don't need to upgrade backstage anymore. But AI is not all that good at upgrading a sprawling backstage application, making broad changes across the code base. And so that's still going to be true if you build your own. And out of software.

Scope is going to increase over time and you're going to have to do wide changes internally. And that's still, I think, going to be difficult. And then the other large source of waste that I see inside software engineering organizations is not necessarily not building the thing fast enough, but it's building the wrong thing in the first place. And I still think that that's going to be a problem.

I don't think AI necessarily tells you what to build or how to solve the pain points of your users. And so one of the benefits of...

I think picking off the shelf software is that it's built out of the lessons which have been learned over many, many years from rolling out at lots of different companies and solving all sorts of problems that you don't necessarily want to slow yourself down by having to reinvent the wheel, solve those problems again. So, okay, given that it sounds like we're moving from dashboards to workflows, IDPs become...

The source of truth for context to some degree, which I think is an organic move. What should teams do now so they're not rebuilding everything again in 18 months? Like, how do we prepare? That's a million dollar question. I wish I had the answer to that myself. It seems like everything you put in your roadmap might get released by Anthropic six months from now and might disappear, right?

I think you have to stay close to your users. I have to, and I feel this too, right? Because as a startup founder, you somewhat... Have to live in the future, right? You're trying to keep track of what's happening out there in the world and what's coming in the next six months. But I have to go back and put every idea through the filter of, well, where are our customers now?

And I think that internally, if you have internal customers, which is what a typical platform team has, I think you need to keep that in mind too, right? Like, what are the needs of the business right now? Where are my users? What are they capable of? What does our security restrictions and our governance rules and everything else allow us to do? I kind of build for now with an eye on the future.

I think, you know, I guess some myself, I still think that developers are going to be around for quite a while. And this is, again, just pure speculation on my part. Nobody really knows, but I certainly feel like the human and AI loop.

Is is quite important and will be for for the next while I think we'll be augmented by ai I think it's a bit like having an exoskeleton you get stronger you get better um you get faster but I still think that the creativity part um exists and and will do for quite some time the state of dorametrics report end of year last year um I thought it was interesting they talk about how engineers in general say that they're

Getting more productivity out of ai or with ai but yet we're not actually seeing like more tangible deployments or like feature rich work being deployed where engineers think that they're getting more more out of ai but they're maybe not you know or at least not as quickly as we would think that they should maybe with the efficiency gains yeah Yeah, well, I think, I mean, this is, again, just personal thoughts.

So frame this however you want. I think that it's tempting to do more busy work now, I think is the first thing. Like things that where you wouldn't have necessarily made, you wouldn't necessarily made that feature before, but now because it's easier, now you will. But that doesn't necessarily add to like the bottom line, the overall productivity.

I think the developers who are able to make product decisions have gone up in value. A lot over the past three to six months as well. Because again, the AI won't tell you what to build. It will build it faster. So I think if I was going to give advice to someone who's starting out in software engineering now would be to try and be a well -rounded individual who knows some UX, who knows some...

UI design, who understands why people use the software, because then you can kind of build towards your customer and you can actually solve problems in a very real way for people versus just moving code around, which I think is a risk.

And I think there's also just a, there's a general economic diffusion problem that we have to solve at the moment as AI just percolates through the economy over the next couple of years. So, I mean, an experience I had recently was we had a, we had a full company offsite. And we're a remote company, so it's rare for people to all be in the same room together.

But we got the technical people in the same room as the non -technical people. And if you're not on camera, you can't see that I'm doing air quotes. But the air quotes are because I feel like everybody's a bit technical now. It was eye -opening for some of the non -technical people, people in customer success, people in sales, etc.

Eye -opening for them to see what the engineers were able to do in a very short amount of time with AI. And I'll give you a great example. I mean, one of our customer success people went, took the lessons from that offsite, you know, just kind of, it was, when I say lessons, it was more of an eye -opening experience. It was just, oh, wow, that's what you can do. And once he'd been shown what was possible.

He was able to go back and he's never written a line of code in his life, but actually make his own customer success dashboard, which pulls information from lots of different sources and shows the health of each of our customers, etc. So I think there's a certain amount of just realization of what's possible that has to happen throughout the economy. The models don't need to get much better, although they will.

The knowledge has to percolate through the economy. And I think that that's when you'll see the real productivity gains. Yeah. It's almost like we're all becoming managers now and we're all like the skill set is really knowing how to prompt an AI, but then also how to be critical of the output of the AI too. Like, because it's very, it's very good at being confidently incorrect.

So if you don't give it the context, like you said, it's going to make assumptions, you know, so you have to know how to. How to actually reason about what it's giving you and then be able to give it the right input so it gives you the output that you're expecting. Yeah. And just because you can build anything and everything doesn't mean you should. Yeah. We've all used bloated software.

I mean, it's one of the top complaints of people who actually produce software when software is bloated. But, you know, AI can help you make your software bloated much more quickly than you could before. Very true. Okay, so wrapping up, what's one piece of advice for platform or DevEx teams starting an IDP this year? Put your laptop away and go talk to your internal users. That's fair, yeah. Right?

You know, go and ask them, hey, what are you doing for 90 % of your day? What's the biggest frustration you've got? What takes the longest time? Because you might find out that it's actually, well, the bills are slow or whatever, right? And it's not that they're trying to search for who is. The owner of the payment service or who's the manager of that team or whatever else. I think go talk to your users.

That's the number one piece of advice. If you actually are in a situation where you want to have an IDP or you have a need for an IDP, I would suggest trying a little bit of self -hosted backstage, try a pure SaaS solution and try Roti just so that you get a good overview of the market. An IDP is something that you are going to roll out across your entire organization. You're going to live with it for quite a while.

And so you want to put some thought into that decision. It's not necessarily the most reversible thing in the world. So I think just take a look at a couple of different solutions and see how they each perform. Yeah, that's fair. Where should people follow you or read your stuff? I'm probably most active on LinkedIn, which almost makes me shudder to say. The inner developer in me hates that, but it's probably true.

So just search for David Tuite on LinkedIn. Yeah, that's probably the best place. Very cool. I'll leave all that information in the show notes as well. David, thank you for coming on. Really appreciate it. Super. Thank you very much. Thanks for having me. All right. That's my conversation with David Tuite. My biggest takeaway from this one is that the future of IDPs probably is not build a prettier dashboard.

It is building the right context layer. For developers, for platform teams, and increasingly for agents. That means discoverability still matters. Automation still matters. Guardrails still matter. But the shape of the experience is changing. Less clicking around for answers. More workflows. More context delivered where the work happens. More systems that can help humans and agents make better decisions faster.

I also liked that David kept this grounded. He was pretty clear that teams should not start by chasing hype. Or by building towards some imagined 18 -month future that may change again in six months. Start with the pain you actually have. Talk to your internal users. Figure out what slows them down. Figure out what causes friction. Figure out what should be self -service. Then build from there.

This is a much healthier way to think about platform work. And honestly, it is probably the right way to think about AI too. Because just because we can build faster now does not mean we automatically know what is worth building. If you enjoyed this episode, follow Ship It Weekly wherever you listen to podcasts.

If you want the full show notes, links to David, Roadie, and the resources we talked about, head over to shipitweekly .fm. Thanks for listening, and I'll see you later this week.

A lot of the work that keeps systems safe does not look important until the day it is. It's a pin commit hash, a safer config rollout, a cleaner restart, a daemon that starts before the app does, an IP range nobody can use your token from outside of. None of that is sexy. All of it matters. Hey, I'm Brian Teller. I work in DevOps and SRE, and I run Teller's Tech.

This is Ship It Weekly, where I filter the noise and focus on what actually changes how we run infrastructure and own reliability. Show notes and links are on shipitweekly .fm. If the show's been useful, follow it wherever you listen. Ratings help way more than they should. And if you want more signal between episodes, check out oncallbrief .com.

We have five main stories today, then the lightning round, and we'll wrap with the human closer. We're starting with GitHub Actions and Kubernetes, because the helper layer is officially part of the trust boundary now. Then Airbnb, with one of the better real -world platform stories I've seen in a while on shipping config changes safely.

After that, Cloudflare open sourcing the grateful restart plumbing behind zero downtime upgrades for Rust services. Then AWS ECS managed daemons, which is a nice clean platform engineering story. And finally, HCP Terraform getting more serious about narrowing access with IP allow lists and temporary AWS permission delegation. Story one, GitHub Actions is not just CI anymore.

Let's start there because I think the easy part of GitHub actions is over. On this week's on -call brief, we flagged that Kubernetes -related repositories are moving towards full 40 -character SHA pinning for actions, with non -compliant workflows set to fail after April 15th.

That sounds like a tiny implementation detail right up until you remember how many teams got burned by mutable tags and trusted automation over the last year. And the broader GitHub direction lines up with that. GitHub's 2026 action security roadmap is basically one big admission that CI and workflow automation are part of the software supply chain now.

GitHub says the roadmap includes dependency locking for workflows, centralized policy controls, and actions data stream for observability. And egress controls for hosted runners. GitHub's security team also said this week that recent attacks are increasingly about exfiltrating secrets and that many of them start by compromising a workflow on GitHub Actions. That's the story, not GitHub added a security feature.

More like the industry has finally stopped pretending build automation is some layer off to the side. If a workflow can publish artifacts, push code, assume cloud roles, or touch secrets, then it is part of the trust boundary, period. So the practical takeaway is pretty simple.

If you are still using broad version tags or workflows that nobody has really reviewed in a while, the platform is telling you where this is going. The old convenience model is getting squeezed out. And honestly, good. It probably should. Story 2. Airbnb built the kind of config platform people say they want. Next up, Airbnb. I really like this one because it is not hype. It is not we added AI to configs.

It is just good platform work. Airbnb wrote about its internal dynamic config platform, Sitarr. The basic idea is to make runtime config changes safer without making them painfully slow. Their setup uses a Git -based workflow by default, schema validation and review before rollout, staged rollouts, fast rollback, and a separation between the control plane and the data plane.

On the service side, there's an agent sidecar and a local cache. So services can keep running on the last known good config, even if the back end is degraded. That's the kind of story I want more of. Because config is one of those places where teams love the flexibility right up until it becomes the outage. And the answer is usually not ban dynamic config.

The answer is build the guardrails and the blast radius controls so the flexibility does not turn into roulette. The part I liked the most is the last known good behavior. This is such a real operator move. Not just our config platform is available, more like what happens to the service if the config backend is having a bad day? That's the right question.

A lot of systems look fine until their control plane gets weird. Airbnb is clearly thinking past that. So yeah, for me, the takeaway here is the safe speed is real engineering work. It's not just vibes. It is staged rollout, rollback, validation, and not making every config change a big bang. Story 3. Cloudflare opensourced the restart plumbing nobody notices when it works.

Now for one of my favorite kind of infra stories. Cloudflare opensourced a Rust library called Ecstasys that it says has been in production for five years and enables zero downtime upgrades across critical Rust services. Their write -up says the point is to restart network services without dropping live connections or refusing new ones, even at Cloudflare scale.

The way it works is a fork -and -exec handoff, where the child process inherits the listening socket, signals readiness, and only then does the parent stop accepting new work and drain existing connections. Cloudflare says that approach preserves millions of requests across its global network on every restart. This is exactly the kind of thing people outside ops do not think about much.

But it matters, because just restart the service sounds clean until the service is live. Handling real traffic and the wrong restart behavior turns into a customer -facing mess. The boring, invisible reliability work is usually where the real maturity is. Not in launch day demos, in the stuff that lets you patch, upgrade, and change things without making users feel it. And I also just like the honesty of the post.

It walked through why the naive restart is bad, why so -reused port is not enough for graceful restarts, and why the socket handoff matters. That's real engineering. You can hear the battle scars in it. So if you run long -lived services, proxies, gateways, or anything else where restart behavior is part of reliability, this is a good reminder that restart logic is not housekeeping. It is production behavior.

Story four, Amazon ECS Managed Daemons is a good platform team feature. The AWS story this week is Amazon ECS Managed Daemons. And honestly, I think this is a better story than yet another agent announcement. AWS says Managed Daemons for ECS Managed Instances lets platform teams independently deploy and manage logging, tracing, monitoring, and security agents without bundling all of that into application deployments.

AWS says ECS runs exactly one daemon task per managed instance, starts the daemon before application tasks are placed, drains it last, and supports rolling updates with rollback protection. There is no extra feature charge beyond the compute the daemon uses. That is clean. That is useful.

And that is one of those features where the value shows up immediately if you've ever had to coordinate host -level tooling through app teams that really do not want to think about your agent lifecycle. The part that I like is the separation of concerns. App teams own app rollout. Platform teams own platform tooling.

That should not be controversial, but a lot of environments still make those things trip over each other. So when AWS gives people a more explicit way to keep those concerns separate, that is worth paying attention to. And it fits the theme of the episode pretty well too. Another background layer becoming more explicit, more managed, and a little less improvised. Story 5.

HCP Terraform is narrowing access in the places people usually leave soft. The last main story today, HCP Terraform. HashiCorp announced two security features that I think fit together nicely, even though they came as separate updates. First, HTTP Terraform now supports IP allow lists at the organization and agent level.

HashiCorp says that means tokens are only accepted from trusted, predefined IP addresses, and that it closes a pretty obvious gap where valid credentials could previously be used from anywhere by default. HashiCorp also says agent pull allow lists can be scoped separately, which lets teams align access with real egress points like NAT gateways or trusted VPC egress.

Second, AWS permission delegation is now generally available in HTTP Terraform. HashiCorp says this uses AWS temporary permission delegation, so customers can grant trusted partners short -lived, customer -approved access for setup and onboarding tasks instead of handing out long -lived permissions. This is the kind of tightening I like.

Nothing magical, nothing flashy, just narrower access and less standing privilege. And that is usually where a lot of the real security value lives anyway, not in adding another dashboard, in making the default trust path shorter and more explicit. So the practical takeaway here is if your Terraform environment still feels a little too open by default, HashiCorp is giving teams some better primitives now.

Network -based restrictions and shorter -lived delegated access are both good moves. A few quick ones before we wrap. GitHub's secret scanning kept moving in March. GitHub says it added 28 new detectors from 15 providers, enabled push protection by default for 39 detectors, and added more validity checks. Then GitHub added secret scanning through the GitHub MCP server.

So AI coding agents can scan changes for exposed secrets before you commit or open a pull request. GitHub Codespaces is now generally available for GitHub Enterprise Cloud with data residency. GitHub says it supports Australia, the EU, Japan, and the US, but requires enterprise or organization -owned Codespaces. User -owned Codespaces are not supported in that setup.

And Kubernetes version 1 .36 is closing off a couple of old foot guns. The project says .spec .externalips is deprecated because of long -standing security risk. And the old git repo volume driver is now permanently disabled because it could allow code execution as root on the node. I think the cleanest closer for this one is pretty simple. Failure is loud. Prevention is quiet.

SRE Weekly put it that way this week, and I think it lands because it's true. Enterprises do not usually underinvest in reliability because they hate reliability. They underinvest because outages scream and prevention whispers. Budgeting systems respond to noise. Prevention work often looks boring right up until the day it would have saved everybody a lot of pain. And that is basically the whole episode.

Pinned actions are prevention. Safer config rollout is prevention. Graceful restarts are prevention. Managed daemons are prevention. And narrower terraform access is prevention. None of that usually gets celebrated the way incident response does. But if you've ever been the person on the hook at 2am, you already know which work matters more. And I think that is the human side of ops that sometimes gets lost.

A lot of the best work we do is the stuff that nobody notices because nothing happened. No page. No rollback scramble. No emergency patch. No weird config blast radius. No restart turning into an outage. No token working from somewhere it never should have. That work counts. Even when it is quiet. Especially when it is quiet. Alright, that's it for this week of Ship It Weekly. Quick recap.

We talked about GitHub Actions hardening and why CI is part of the trust boundary now. Airbnb showing what safe config rollout actually looks like. Cloudflare open sourcing the restart plumbing behind zero downtime rust upgrades. Amazon ECS managed daemons. And HCP Terraform tightening access with IP allow list and temporary AWS permission delegation. Links and show notes are on shipitweekly .fm.

You can also find the video versions on YouTube. And if you want more signal before the episode, check out oncallbrief .com. If this episode was useful, follow or subscribe wherever you listen. And send it to the person on your team who keeps getting asked to move faster while quietly doing all of the work that keeps the background layers from becoming the next incident. I'm Brian, and I'll see you next week.

Shortcuts keep turning into trust decisions. It's kind of the whole show today. A GitHub workflow gets treated like plumbing until it hands over a token. A version tag looks stable until somebody moves it. A cloned Windows image looks fine until the platform finally decides it cares. And a skills marketplace looks like discovery right up until you remember the code is coming from wherever that URL points.

That's where we're at. Hey, I'm Brian Teller. I work in DevOps and SRE, and I run Teller's Tech. This is Ship It Weekly. Where I filter the noise and focus on what actually changes how we run infrastructure and own reliability. Show notes and links are on shipitweekly.fm. If the show's been useful, follow it wherever you listen. Ratings help way more than they should.

And if you want more signal between episodes, check out oncallbrief.com. We have five main stories today then the lightning round and we'll wrap with the human closer we're starting by coming back to the trivy story but from a different angle we already covered trivy on the show on episode 24.

The reason to revisit it now is that it is clearly not just one ugly repo incident it was part of a bigger github actions campaign after that github has one of the better platform engineering stories I've seen lately They rebuilt search high availability in GitHub Enterprise Server because the old shape could literally trap people in a bad maintenance state.

Then a Windows Server 2025 identity mess that feels extremely real if you've ever lived through cloned image weirdness. And finally, agent skills, because apparently we are rebuilding package manager history all over again. Except this time, the packages can inherit shell access, file access, and credentials. Story 1. Trivy was part of a bigger GitHub Actions campaign. Let's start there.

I don't think the trivy angle is really the headline anymore. The bigger story is that OpenSSF warned on March 1st about an active campaign called HackerBot Claw. Their advisory said it was actively scanning public repositories for weak GitHub Actions workflows and exploiting them to execute code and steal credentials.

InfoQ's follow -up said the campaign hit projects tied to Microsoft, Datadog, CNCF, and aqua security and that it got remote code execution in five of seven targeted repositories that matters because it takes this out of the bucket of one repo got unlucky this is a real pattern now open ssfs write up called out the exact stuff most teams already know they should be nervous about privileged triggers like pull request

Target running untrusted code from forks inline shell unvalidated input week -auth checks before workflows run.

And they were pretty blunt about the response too. Review, harden, and monitor your workflows now. So for me, the real lesson is not wow, an AI bot did a thing. The real lesson is that repo automation has become part of the trust boundary. And a lot of teams still treat it like background glue. It isn't glue anymore.

If a workflow can write releases, publish artifacts, touch environments, or push back to the repo, that is real production adjacent power. Attackers know that. We should probably act like we know that too. One thing I'd actually do after this episode, pick a couple of workflows and ask a very boring question. Where exactly does untrusted input enter? And where does it land?

Branch names, PR titles, file names, checked out code. If you can't draw the trust boundary cleanly, you probably don't have one. Story two, Xygeni and the problem with mutable trust. The Xygeni one is a perfect follow -on because it is the same lesson from a different direction.

Step Security reported that the official Xygeni action was compromised on March 3rd when an attacker using stolen maintainer credentials injected a reverse shell, and moved the mutable v5 tag to a malicious commit. The nasty part is that repos using at v5 did not need to change their workflow file at all. The trust moved underneath them.

StepSecurity later said Xygeni removed the bad tag on March 10th, rotated contributor tokens, enabled release immutability, and added tag protection rules. That's what makes this one so useful. Nothing had to look different in your YAML. No scary PR. No obvious version bump. No, why did Dependabot open this? Just the same reference, now pointing somewhere else.

And that is why I keep coming back to this idea that mutable references are not a minor detail. They are a trust decision. Teams talk about major version tags like they are basically stable, but they are only stable if the permissions, protections, and release process behind them are stable. If the tag can move, the trust moved too. So I would not make the takeaway pin everything or you're reckless.

I'd make it a little more honest than that. Know what layer your trust lives on. If you are depending on a mutable tag, say that out loud. If you are okay with that trade -off, fine. But make it a real decision. Don't let it be an accidental one. And if you own internal reusable actions, this is a pretty good excuse to tighten those up right now.

Tag protection signed commits release immutability SHA pinning guidance the usual grown -up stuff story 3 github rebuilt high availability search in github enterprise server now for the one that is not mostly security doom GitHub published a nice write -up on rebuilding search high availability in GitHub Enterprise Server. The old model clustered Elasticsearch across the primary and replica nodes.

GitHub said that created ugly failure modes during upgrades and maintenance. A primary shard could move to a replica, then that replica could go down for maintenance, and the whole thing could get stuck waiting on itself. Their new design uses Elasticsearch cross -cluster replication and moves each Enterprise Server instance to its own single -node Elasticsearch cluster.

GitHub says support starts in G -H -E -S 3 .19 .1. I like this story because it feels like real platform work. Not we added an AI button. Not we made this graph go up. More like this architecture had a shape problem, and the shape problem kept showing up during normal operations. So we changed the shape. That's good engineering. And a lot of teams are sitting on something like that right now.

You know the kind of system I mean. It mostly works but everybody knows upgrades are touchy. Maintenance has to happen in the correct moon phase. Failover is theoretically fine but nobody wants to test it on purpose. That usually means the system does not need another warning label. It probably needs a different shape. That's the part worth stealing here.

If a core platform component keeps demanding perfect choreography just to survive normal admin work, stop asking how to babysit it better. Start asking whether the design is wrong for the job now. Story 4. Windows Server 2025 is surfacing old image sins. This one is a little less flashy, but man, it feels real.

Microsoft published exchange guidance in March for Windows Server 2025 systems built from non -generalized images. The root issue is duplicate computer SIDs. Microsoft says those can happen if you deploy an image without running sysprep, clone an existing Windows Server instance, or reuse a VM snapshot or image the wrong way.

And the reason this is showing up now is that KB5065426 introduced strict checks for duplicate SIDs. Microsoft also says PS GET SID is the way to verify it. I wanted this in the episode because this is such a classic ops problem. The bad habit can sit there for a long time. Maybe years. Everything mostly works. Then the platform hardens one layer of identity behavior.

And suddenly you're not dealing with security improvement in the abstract. You're dealing with a weird auth failure that sends people digging through the wrong part of the stack for hours. That's the pattern. Hardening doesn't just improve security. It also drags old shortcuts into the light.

So if you are in one of those environments with old templates, old images, or cloned Windows boxes nobody has wanted to touch, this is a nice reminder that it works and it was deployed correctly are not the same thing. One practical move here. If you've got any suspicion your Windows fleet has image history you don't fully trust, go check before the next update checks for you. Story 5.

Agent skills are replaying package ecosystem history. The last main story is the one that keeps getting more obvious. Socket wrote in February that skills .sh had already indexed more than 60 ,000 skills across tools like Cursor, Cloud Code, GitHub Copilot, and Windsor, and that anyone can publish a skill from any GitHub repository. Their point was simple.

Skills are decentralized by design, so they inherit the same supply chain problems we already know from places like NPM. Then Snyk took a harder look at the ecosystem, and their numbers were not exactly comforting.

Their February study scanned 3 ,984 skills from ClawHub and Skills .sh and said 13 .4 % had at least one critical issue, 36 .82 % had at least one security flaw of any severity, and 76 malicious payloads were confirmed through human review. They also called out the part that really matters operationally.

These things can inherit shell access, file access, credentials, messaging access, and persistence through memory depending on the agent they extend. That's why this story matters to me. Not because agent skills are risky is some hot take. Of course they are. It matters because this is the same old supply chain problem wearing a newer outfit. Lightweight distribution. Easy publishing. Borrowed trust.

Code coming from repos you don't control. And a whole lot of people treating discovery as if it implies review. It doesn't. Popular does not mean safe. Installed does not mean verified. And it's just a skill is probably going to age about as well as it's just a package.

So if your team is experimenting with skills, sub -agents, MCP servers, whatever flavor of extension system is hot this month, treat that install path like a real dependency path. Because that's what it is. A few quick ones before we wrap. GitHub paused enforcement of the minimum self -hosted runner version requirement. The version is still v2 .329 .0. Runners below that can still register for now.

But GitHub also said the long -term direction has not changed. So this is a pause, not a pardon. GitHub also rolled out March secret scanning updates. They added 28 new detectors from 15 providers, turned on push protection by default for 39 detectors, and added more validity checks. I like this one because it is the kind of boring improvement that actually saves people from themselves.

And one more little connective tissue point from the OpenSSF advisory. Pin third -party GitHub actions by commit SHAs. Keep workflow permissions minimal. And stop using privileged triggers unless you really need them. That advice is not new. But the campaign is a nice reminder that we know better and we configured it better are not the same thing.

I think the cleanest thread through all of this is that convenience keeps getting mistaken for safety. A workflow is convenient. A tag is convenient. A cloned image is convenient. A skill directory is convenient. And none of those things are neutral once they start making trust decisions for you. That's the real pattern here. The Trivy follow -up story is really about CI being part of the trust boundary now.

Xygeni is about mutable trust hiding under a stable looking reference. The GitHub search story is what it looks like when a platform team admits the system shape is the problem and they fix it. The Windows story is what happens when the platform finally starts enforcing a thing you were sloppy about years ago.

And the skills ecosystem is just the latest reminder that the minute installation gets easy, trust gets fuzzy, unless somebody does the work to pin it down. So the operator version of the episode is pretty simple. Stop calling these things helpers if they can change outcome. If it can execute, publish, authenticate, install, or inherit credentials, it is not just automation. It is part of the control plane now.

And that means you owe it the same questions you always owed production. What is mutable? Who can change it? What is isolated? What is logged? What fails open? And what are you trusting by default that you probably shouldn't be? All right, that's it for this episode of Ship It Weekly. Quick recap.

We came back to Trivy by zooming out to the bigger HackerClaw campaign, then Xygeni and tag provisioning, GitHub rebuilding high availability search in GitHub Enterprise Server, Windows Server 2025 surfacing old duplicate SID problems, and agent skills replaying package ecosystem history, except with even more access and worse defaults. Links and show notes are on shipitweekly .fm.

You can also find the video versions on YouTube. And if you want the signal before the episode, check out OnCallBrief .com. If this was useful, follow or subscribe wherever you listen. And send it to the person on your team who still thinks CI, tags, images, and just helper tools live outside the real trust boundary. I'm Brian, and I'll see you next week.

Everybody wants AI to help run infrastructure. A lot fewer people are asking where that AI is allowed to fail. Because the hard part is not getting an agent to suggest a change. The hard part is making sure that change can be tested, challenged, and debugged before anything touches real cloud infrastructure. And that is what made this conversation interesting to me. Not just the AI angle.

The idea that before we trust agents with real systems, they may need a real training ground first. Thank you. Hey, I'm Brian Teller. I work in DevOps and SRE, and I run Teller's Tech. Ship It Weekly is where I filter the noise and focus on what actually matters when you are the one running infrastructure and owning reliability. Most weeks, it's a quick news recap.

In between those, I do interview episodes with people building tools, systems, and ideas that could actually change how this work gets done. Today is one of those conversations. I'm joined by Ang Chen. Associate professor at the University of Michigan.

He's working on Project Vera, which is now being positioned as a high -fidelity multi -cloud emulator you can run locally on your laptop with support for AWS EC2 and GCP compute. At a practical level, the pitch is pretty straightforward. Test cloud infrastructure locally, use standard tooling, avoid real accounts, real spend, and real blast radius while you are iterating.

But the bigger idea behind Vera is what really got my attention. Ang frames this as part of a longer -term vision for giving AI agents a safe learning environment. Or what he calls a kind of world model for digital systems, where they can build operational intelligence before ever touching real infrastructure.

So in this conversation, we get into what high fidelity really means, how Vera works at the API layer, how it can sit under workflows that already use CLI tools, SDKs, or Terraform, and why that matters if you want faster feedback without pointing tests at the real cloud. We also get into the skeptical operator questions. How close does something like this actually need to be before you trust it?

Where is it strong today and where is it still early? And if AI is going to play a bigger role in infrastructure, what kind of safety layers should exist first? That's the real conversation here. Not whether AI can generate infra work. Whether it can be forced to prove itself somewhere safe before it earns access to the real thing. If you like these kinds of conversations, follow the show wherever you listen.

Subscribe on YouTube and check out ShipItWeekly .fm or TellersTech .com for more episodes, show notes, and everything else that I'm building. All right, let's jump in. Today, I'm joined by Ang Chen, an associate professor at the University of Michigan. He's working on Project Vera, which is basically trying to build a high -fidelity cloud emulator using AI agents, starting with EC2. Ang, thank you for joining me.

Thank you, Brian. I'm excited to be here. So tell me about Project Vera. What is it? It's an effort that automatically generates a digital twin of your cloud deployment. DevOps can be very tricky to get right. And we don't want any downtime or security issues when we actually push the program to the cloud. So you can think of it as a sandbox. That's a digital copy of your actual infrastructure.

Within this sandbox, the DevOps programs can be tested. They can be debugged. You can even deploy an AI agent to play with the sandbox and get to know more about your deployment without actually reaching into the actual deployment itself. And what's interesting about the sandbox is that it's actually generated by AI agent itself. Have an AI agent that reads.

The cloud documentation, and it could also observe traces and logs about the deployment. And it uses a very specialized program synthesis pipeline to generate an emulator framework. The emulator framework will mimic the behavior of EC2, for instance, in terms of how to respond to a certain call, what should be the responses and formats in a very high fidelity manner.

And the same idea generalizes to other services in AWS and it generalizes to other clouds as well. Actually, we're building it for Azure and GCP and other clouds as well. So it's an agent building a simulator of the cloud. And on that cloud, on that simulator, the DevOps engineers can do a lot of their works much easier. What's the target audience for a tool like this? Right.

It would be primarily for DevOps engineers who... Want to test their programs in the sandbox. The DevOps engineer can deploy their programs in the sandbox and observe the behavior and debug their programs before they push it to the actual cloud. And that sandbox can also be used to support DevOps like AI assistance.

AI is getting very powerful every day, but we often don't want the AI to directly work on the infrastructure. Because it could hallucinate. So having an AI agent testing its proposed actions in the sandbox before putting it to the cloud would be another use case of the sandbox. Is it interfacing with like IAC, like a Terraform or CloudFormation? Or how does that integrate? Right.

The simulator emulates the cloud at the API level. Basically, every API that creates virtual machines and subnets is captured here. So basically, it can support SDK scripts, but it can also support CloudFormation and Terraform because eventually they all call into the APIs. And in the release that we have, we have like CRI test cases that mimic Amazon, but also Terraform programs that can be booted on this emulator.

When you say high fidelity, what does that mean in practice? Right. It means that there are two key properties of this emulator because this is generated by an AI co -developer.

So to speak that reads the cloud documentation and test against the cloud we want to make sure that this is not vulnerable to hallucination ais are getting very good but they still have hallucination and we have two ways to prevent this from happening and the first is the ai agent that we have built that's behind vera is using formal abstractions is using formal methods and verification to make sure that the code eliminates classes of hallucination problems.

So it's built to be correct by construction without suffering from arbitrary errors that an AI model would otherwise introduce. And the second is that the AI agent also takes this simulator and strategically tests this against Amazon.

Because this emulator is generated by the agent, the agent understands the inner workings of the emulator and it can understand what might be some edge cases and what might be some places where strategic testing would be helpful. So the agent also takes this emulator and produce traces and send it to the cloud, observe whether the behaviors are the same or not.

And if they're same, that's what we mean by high fidelity. And if there are discrepancies, the AI agent will then consume these two traces and automatically patch the emulator so that in the next test case, they will be aligned with each other. Interesting. I guess I'm curious, how does it get around the non -deterministic behavior of an AI or an LLM specifically? Right. And that's a very good question.

That's where the formal abstractions come in. Instead of having the AI write code in a freeform style, We actually have a lot of scaffolding. That's the key part. The structure of the emulator is a deterministic framework. And what we ask the AI to do is essentially fill in the blanks that we have left out instead of being creative about writing everything about the emulator.

So it's a combination of neural and symbolic methods where the neural framework constrains the behavior. And it's fully deterministic. And there are strategic parts where the AI needs to read the documentation and understand what it's supposed to do. And it's only filling in these blanks in a way that's constrained by the scaffolding. So it's like a spec then that you're built around, okay. Exactly.

Or in Cursor or Kiro, it's like a plan file that it's reading. Is it like a pre -prompt or is it more specific than that? It's more specific than that. So we use a special kind of phone methods that builds classes and abstractions, almost like a template. And the template has a very well -defined structure. And we know that the structure cannot go wrong because it's deterministic.

But the structure also has certain stops. And the stops are where the AI agents will generate code and insert them into. So it's more specific than a pre -prompt, almost like... A class that can be inherited and can be turned into a compute instance, can be turned into a subnet, a firewall, and so forth. So can you walk me through the shape of the system?

Like if I'm calling an EC2 API, what's happening behind the scenes? Right. So if it's calling into the EC2 API, so that API will be captured by the emulator framework and it will create a class, so to speak. That captures the behaviors as specified in the EC2 virtual machine documentation. For instance, there is a run instance which creates the virtual machine.

And you could destroy it, you could attach disks to it, and so forth. And that will trigger some state modification, almost like IAC, where Terraform contains the state. So here, the emulator mimics that workflow, and it also contains state. But now we have a virtual machine. And the virtual machine could have a specific name.

And if there's another API that attaches a disk to the virtual machine, the emulator will also capture that by modifying the internal state. So it is a hierarchy of these services where you could instantiate a virtual machine and the virtual machine could be contained in a VPC. So when you're updating the virtual machine, for instance, the emulator knows that it also must. Updates the VPC.

So talking about state, how do you deal with weird edge cases like eventual consistency, retries, throttling, quota errors, that sort of thing? Right. So the emulator framework itself is generated by an AI agent that reads the cloud documentation. So the cloud documentation describes the key behaviors of the cloud, but it doesn't describe everything. So the question that you ask is a very important class of problems.

Which are not fully documented in the documentation. As an example, eventual consistency and consistent guarantees are often not described in detail. But the API behaviors, how it should perform, is documented very extensively.

So what we are doing here is that we are taking the emulator and bootstrap it to a fully functional emulator, but doesn't capture some of the nuances regarding throttling, rate limiting, consistency. But we have another simulator. In the backend that can supply some of these semantics.

So this functional emulation can be, if there's a call into an API, that API, we can also emulate latency for that API throughout behaviors and quotas. So there are an orthogonal subsystem that supplies that kind of intricate detail to the emulator. That's a great question. What's the success bar? Is it like same response, same timing, same failure mode? Yeah. So there are two milestones.

The first milestone is that it should enable the same inputs and outputs for the APIs so that DevOps engineers doesn't have to actually go to the cloud to understand whether their program is working. So then we just tested it and... Observe the actions in this emulator. And the second milestone is that actually this emulator can help with DevOps perform better debugging than the cloud can.

And the reason is that when the cloud has an error, it gives you some trace, but that trace is often verbose. It doesn't really help with pinpointing which line of code is problematic in your Terraform file or in your SDK file. Because there's an AI agent living in the sandbox. The agent can analyze the traces and produce better debugging information and even pinpoint the problems in Terraform.

So the second milestone is actually to do better debugging than what the cloud can do. Interesting. So how do you prove that it's not lying to me? That is a heart of the question. How do we make sure that this emulator is actually producing the same responses in the first release?

In the github we have more than 200 test cases and these are cri command lines that you would type into aws and we run a test between vera and an existing emulator so what we've shown is that vera is already doing much better than existing emulators but the same set of test cases I've also shown that Vera sometimes fails to produce the same behavior because this is a agent that continuously improves itself.

And the first release gets a 70 % based on our measurement. And by this agent, we have another version that's continuous running and improving itself until it hits all the test cases. So one way that people test it is to use a leading emulator called local stack. Local stack is this really nice tool. That emulates AWS APIs. It's not one -to -one. I've found it's good in some ways, but yeah, it's...

It's not one -to -one. It's close enough to enable classes of DevOps testing. In the open source release, actually, we did a comparison between Vera and local stack. So what we found is that Vera covers more than 70 % of the cases, whereas local stack covers 40%. So the first version of Vera is already performing. Quite well in that regard. And we also have another version that's continuously improving itself.

And the goal is to simulate the behavior of the cloud to Terraform and DevOps programs, almost like the Turing test. The ultimate goal is when we run a DevOps program against the simulator versus against the cloud, the DevOps program doesn't feel any difference. It doesn't necessarily mean that has to be line by line.

Character by character the same regarding the logs and the outputs, but we want it to be high fidelity enough that DevOps engineers can test it thoroughly in this simulator. So if I'm a platform team, where would I actually plug this in? Local dev versus CI versus prepod, validation, like what would be a good first step? Right. One way of using this is to integrate it to the CI -CD pipeline.

When there are code changes, there's a new Terraform file. The agent can take the changes and validate it in the sandbox first and suggest changes to the program if there are errors and fix these errors and generate corrections for the DevOps engineers so that this would be integrated to the CICD before it's actually pushed to the cloud. Is there any use cases where maybe it's not well suited for yet?

Maybe it doesn't have enough testing around it or validation. So there are two things that we know about the limitations of Vera. The first limitation is that it doesn't yet cover all resources in AWS. It does cover EC2, which is a key service. There's also a lot more beyond EC2. That's the first limitation that we know.

And the second limitation is that the current version, the current version doesn't do some of the things that we... Are thinking about. For instance, I've talked about AI -based debugging suggestions to DevOps engineers. So that tooling is not fully ready yet. So currently, if there's a bug, error doesn't automatically diagnose the bug for you, which is part of our ongoing plan.

For the first limitation, which is that it doesn't support all APIs and it doesn't support customization. For instance, it doesn't automatically understand how a specific deployment is like. That deployment for an enterprise may not all use all the APIs. They may use the APIs in a very specialized way. So these kind of customizations are also not there yet, but they are on our agenda.

So it sounds like EC2 is its area where you've had a lot of focus and it seems like you have trusted output there. What's the nastiest EC2 edge case that you've had to emulate?

There's a very interesting edge case that we have found in this exercise, which is that sometimes our AI co -developer that writes the emulator uses one types of string formats, like a camel cases, where it's easy to make format the same string differently. And that's very interesting because a Terraform program expects a certain type of format.

And if it's formatted slightly differently, then Terraform program won't run. So there we had to create specialized directions for the agent so that you would only produce camel cases when it's supposed to be camel cases. In other cases, produce snake cases and so forth.

I thought that was a very interesting and unexpected edge case where the initial version of Vera didn't produce the exact response and we had to do extra engineering to make that align. Interesting. Do you think clouds will ever make official emulators good enough or is learned emulation the path? Right.

One thing that's very special about cloud emulation compared to other types of emulators is that the cloud is a moving target. There are new services every week and there are API changes. Many of these changes will introduce different behaviors. So it's building an emulator for the cloud. Our experience is that there are two key challenges.

One is that the size of the cloud is so big and there are so many different clouds with different behaviors. Beyond AWS, we've also investigated Azure and GCP, which is on our agenda as well. They all have different APIs. They have different behaviors. The emulation for one cloud doesn't really generalize to the other. The second is that the APIs go through constant evolution.

Because the clouds need, they want to stay competitive. They're introducing new services, new ways of using these services. We really believe that learned and AI agent built emulator is the path. Because the AI agent doesn't have to spend a lot more extra effort once the emulator framework is there. It still has to align the emulator periodically.

Whenever there's an API change, the agent has to understand what has changed. It has to generate strategic test cases. To realign that API, but it doesn't have to do everything from scratch. So this agent can keep up with the changes that happen in the cloud and it can gradually expand to different clouds. So this is an ever -expanding emulator that can catch up with the speed of the cloud.

And that's something that we are very excited about regarding this learned emulation. Have you done much as far as GCP training yet? I'm just curious because... The IAM approach in GCP is completely different than the IAM approach in AWS. Right. Or even like Cloud Run versus Lambda is also completely different. Fundamentally different services, although the same general idea or same general focus. Yeah.

And that is a very good question. The clouds call the same service differently. So they're almost aliases. So here is where AI will shine. Because as long as we can... Make the AI understand. There are certain concepts across clouds that are similar. Instances are called virtual machines in a different cloud. Then there are certain knowledge base in the AI that can transfer from one cloud to another.

So there's a core of the learned knowledge that can transfer, but not everything. The APIs are still different and the services do not always have a counterpart across clouds.

What I'm excited about this approach is that some core knowledge of the cloud can be transferred so that when we're building the second emulator for GCP, it will be much faster than the first one for AWS, where it has to learn everything, all the concepts from scratch. Where can people find more information about Vera? We have an open source release called Project Vera.

And it's project -vera .github .io, where we have an open source release, as well as the publication that we had over the years that eventually led to this paper, to this simulator. And that could be a good source of information that not only is about the release.

But also the rationale behind the release and the specific approach that we take in designing Vera and other tools that we have built in the past couple of years surrounding AIOps and DevOps. And we are looking for contributors to help us improve Vera.

And if you're interested in contributing to the open source release or contributing new ideas, or if you have a service that you would like to see emulated, this is something that we are very excited to help you with. What's the license model for Vera? Currently? It's under MIT license in open source. That's good to hear.

Too many new open source projects like to limit their open source initiatives, which is a little frustrating. Right. It's fully open source under MIT license. Thanks. Well, thank you, Ang, for coming on. Really appreciate it. Thank you very much, Brian. It's great to be here. All right. That's my conversation with Ang Chen.

My biggest takeaway for this one is that AI for infrastructure gets a lot more believable when it has to survive a sandbox first. That's what makes Vera interesting to me. Not just that it emulates cloud behavior, but that it is trying to create a local, high -fidelity environment where cloud workflows can be tested without real credentials, real billing, or real production risk.

And since we recorded this, the project has clearly kept moving. It's now being presented publicly as a multi -cloud emulator, not just an EC2 -focused idea, with AWS and GCP support and a stronger public story around local testing and safer iteration. I also liked that the bigger vision was not just AI does ops. It was more grounded than that. Give agents a training ground.

Let them learn inside something rule -based. See how they behave, see where they fail, then decide what, if anything, Thanks for listening, and I'll see you later this week.