Ship It Weekly Transcripts

Webhooks might be the simplest distributed system most teams underestimate. You give a service a URL. It sends an HTTP POST when something happens. Done. Until your endpoint is down during a deploy. Or the sender retries and you process the same event twice. Or events arrive out of order. Or a valid request triggers something it should not. Or you discover the provider stopped retrying hours ago and nobody noticed.

Suddenly, that simple HTTP POST needs queues, idempotency, signature verification, replay, observability, retention, and someone willing to own it when it breaks. That is the strange thing about webhooks. The interface is simple. The operational contract is not. I'm Brian Teller from Teller's Tech. And this is Ship It Weekly.

Welcome back to Ship It Weekly, where I filter the noise and focus on what matters when you are the one running infrastructure and owning reliability. Most weeks, it's a quick DevOps, SRE, platform, cloud, and security news recap. In between those, I do conversation episodes with people building and operating the systems we all depend on.

Today I'm joined by Jay Lark, Principal DevOps Engineer and the founder of Hookbridge. Jay has spent years dealing with the operational problems hiding behind webhooks. Failed deliveries, duplicate events, retry behavior, signature verification, replay, payload visibility, and the awkward process of testing webhook integrations locally.

We talk about what at-least-once delivery actually means, why the receiver usually carries most of the reliability burden, what teams need to log without accidentally storing sensitive data, and why replay feels optional right up until the first incident.

Jay also explains why polling may be more useful than engineers give it credit for, and why every team probably does not need to build its own webhook delivery platform. All right, let's jump in. Today I'm joined by Jay Lark, principal DevOps engineer and founder of Hookbridge.

We're talking about webhooks, why they became the default way services talk to each other, why they're harder than they look, and what teams need to understand before relying on them in production. Jay, thank you for joining me. Thanks for having me. Glad to be here. So I guess let's start at the top. What are webhooks? Yeah, we should define it so we all know what we're talking about here.

They're deceptively simple. At least they seem like they're simple until they're not. But basically, it's an HTTP POST request from one website to another website, one app to another. Just a real simple example. If you want to, say, get a notification when something happens on GitHub, like your PR merges in or something, you can give GitHub a URL.

And when your PR merges in, GitHub will send an HTTP POST to the URL you gave it with a little JSON body saying PR merge or whatever. And that's it. It's real simple. HTTP POST. Why are they harder than they look? There's a whole bunch of things that can happen. If the receiver is down, what happens then? Many webhooks don't retry. They just kind of sit there.

And some webhook providers give you an option to go manually hit the retry button, but many don't. A lot of the big webhook providers will send webhooks kind of out of order. So you might get an order updated before order created, and your system receiving it might not be able to update an order that it doesn't have yet. There are latency issues.

If your servers are slow to say, yes, I received this webhook, the sending party may decide, hey, they didn't get that. There's an error. Let me send it again. And now all of a sudden you have duplicates. There's a whole host of things that can go wrong in the whole transaction. So on paper, a webhook is just an HTTP POST, as you said. Why is that mental model the source of so much pain?

Probably because it seems so simple. So people write kind of the happy path in their code, right? It's one or two lines of code to send an HTTP POST. But all of the things that you would have to do when it fails, and, you know, you're wishing you did a little bit more than the one or two lines of code. All of the things that you end up having to do to kind of catch all those edge cases, it's pretty significant.

Separating events by different IDs, deduplicating events if you get the same one, and verifying that the webhook came in from whoever you expect it to. It's a whole set of things that you really need to account for and to build for. A lot of people don't. And then there's at-least-once delivery. What does that actually mean for the person receiving the webhook? Yeah, that's good wording, isn't it?

The big webhook senders, like Stripe or PayPal, say, at-least-once delivery: We will send you the event at least once. We might send it 10 times. We might send it more, yeah. So, you know, if the webhook is charging your credit card for an order, you certainly don't want to process that 10 times. So you're going to want to make sure you only process it that once. Okay, so there's an asymmetry with webhooks.

The sender fires an event, but the receiver has to expose an endpoint, keep it available, verify it, process it, and not lose anything. Is that a fair burden to put on every consumer? Fair? That's a tough call. That's the way it is now. You know, before webhooks became the main thing, we used to poll for events.

So if you go back to our example of a GitHub PR merge, you could change your app to check every five minutes or every 10 minutes, or maybe every minute: Has that PR merged in? Has that PR merged in? And most of the time you're going to get nothing back. The thing didn't happen, so it's pretty wasteful. And also then, on GitHub's end, they're getting hammered by you every minute and then like a million other people.

And I'm not blaming GitHub; literally everybody does this. But they've switched the contract now, right? We'll just send it to you as it happens. So it's good for you. You get the event as soon as it happens. But yes, there's a huge burden on you now. You have to be up all the time. You can't go down, or like we said, they may not retry. You might never get it. You have to do all these deduplications.

The events may not come in in the order you expect, so you have to maybe queue them somewhere and wait for event number one in a chain to arrive before you process events two, three, and four. Then you have to do all the verification and all those types of things. So is it fair? Maybe. I mean, it's your data you're looking for, really. Maybe. But it is just a fact. That's how the modern web works now.

If you're optimizing for like six nines with like 31 seconds per year or whatever, if it happens to go down during that 31 seconds, you lose out on that webhook, right? I mean... For a lot of providers, yeah. Somebody like Stripe has a web console you can go in and you can hit retry and it'll resend it to you. But you have to know that you missed it.

You don't know that somebody submitted an order and you didn't get it. So you're building it. Then you're building some checker to go on to Stripe's API and check for a missed order. Like, wow, you could have built the redundancy into your system in the first place. So I guess that leads to debugging the failed webhooks. That's been miserable for so long. Is there a way to combat that? How do we improve it?

Yeah, it's very, very difficult. A lot of the time, if you've ever debugged a failed webhook that GitHub sends you, it's this big red failed message in the GitHub console, but there's not a lot of detail about what happened. You might get the HTTP error code back that your system sent, but there's really not that much there.

So really, if you're building this yourself, if you're receiving webhooks, you have to build in a whole bunch of logging to find out what happened. Did the signature verification go wrong? Is there just too much of a latency? A lot of webhook senders will just fail pretty fast if you don't respond and acknowledge and process the data quickly enough. A whole host of things can go wrong.

So you need that detailed logging and information to be able to debug it. And most senders don't give you what you need. How do we get around the latency requirements or the availability requirements without just throwing a ton of infrastructure at it, right? Yeah, it's tough. It's a tough call. We have a lot of customers that use n8n to receive webhooks. It's a great product, really easy to use.

And a lot of the workflows get kicked off by webhooks. I think a lot of people use n8n, and it's hard for me to say, because it's cheap, and they end up putting it on pretty under-specced servers, which just take a little bit longer to respond. And a lot of providers time out really fast on that. So the answer to your question is that you have to give it enough horsepower. That's just the fact of the matter.

Going down for deployments and things like that, you have to build in that redundancy. So however many listeners you have up, they have to stay up while the new ones come take their place. It's not all that different from kind of rotating on a web server underneath a load balancer, but it's just something you do have to account for when you're receiving webhooks because you have no idea when they might come in.

Whose problem is it when there is a failure? Is it on the sender? Is it on the receiver? Is it, I guess, case by case? Yeah, I mean, it can be both, right? Depending on why. Ultimately, it's kind of the receiver's problem to deal with, right? They want the data.

They have to kind of figure it out, and whether that's creating a whole bunch of support tickets with the upstream provider and, you know, maybe someday getting a response, or beefing up your infrastructure and, you know, solving whatever problem is causing the delivery problem. Ultimately, I think it comes down on the person receiving the webhook. That's fair.

And I guess before we even know, right, we don't know if there was an issue if the webhook didn't go through if it was a latency drop-off, if our servers just didn't receive the webhook for whatever reason during that time that it was sent. How much visibility should a provider have or service have to look into webhook calls? Like what's a standard as far as visibility? Yeah, I think you need a lot there.

You need good logging, like I said, between the whole TCP handshake to see whether things are getting cut off because of latency. You need visibility into where that might be dying. You need the request and the response headers. Maybe the signature verification is just not working. A lot of people, at first, if they're actually trying to do signature verification, actually verify on the wrong part of the message.

And that's going to be pretty clear if you're looking at the headers of the response, but sometimes you just don't get those. So you're going to need the full request and response headers. You really need everything except for probably the body of the message, right? There's some privacy and security issues with seeing whatever might be sent in that body.

But other than that, you really need a pretty comprehensive observability view into that transaction. So that was actually going to be my next question on the privacy compliance angle. Is it just the body? Is that the only thing that we're really concerned with, I guess, with PII? I mean, well, that's a huge one. People use webhooks for anything. So there's no telling what could be in the body.

Obviously, listeners out there, they'll know what their app does. Maybe there's nothing super sensitive in a PR merged message, but typically there's going to be something more sensitive in there. But it's not only the body. Tokens and signature verification data can be in the headers. Those can also be intercepted and used for nefarious purposes if somebody has access to things they shouldn't.

What about replayability? Is that a core requirement, or does that feel optional until the first incident? It probably feels optional to a lot of people at first. Because going back to the idea of sending an HTTP POST, it's usually one or two lines of code, right? You just do it. The first time you have an incident, you're going to want to be able to replay things.

And the fact is a lot of providers just don't allow that. But yeah, you can, especially if you are receiving a webhook and your system is sending a 200 response back, yeah, I got it. But something internal to your system broke and you didn't process the message correctly. It might be gone now from the sender.

So, you know, maybe you'll fix your back end and say, oh, things work now, but you need to be able to go and replay those messages. Okay, so we have observability. We have webhooks being received. We maybe have the ability to replay them.

Now from the security side of things, the trust boundary: Once you expose that public endpoint for incoming events, what are the security basics that people should get right from the get-go? Yeah, this isn't everywhere, but most providers now are going to give you a cryptographic signature that you can verify that the sender is who you expect. It's them sending it.

So if the provider you're receiving webhooks from offers that, please do that. Verify it. But there's a second part that I definitely see people ignoring is just because the webhook came from a trusted source doesn't mean whatever the payload is asking you to do in your system is okay. Like you need to verify the business logic and say, okay, yeah, this order can go to this or this thing is owned by this project ID.

So it's not just blind trust. It's a trusted message, but still do all the things you would normally do on an HTML form or something, you know, in your app. What does HMAC signing actually prove? Like more than just like tokens or? Yeah, so they'll give you a key and then they'll sign their message and your system can compare, make sure that the message is signed by them with the key that they give you.

So if you're doing this signature verification with Stripe, the messages that are coming in, you can verify those are coming from Stripe. Okay. Like I said, you still want to make sure that, you know, the orders, or probably more importantly, refunds, are going to the right accounts and you know you're somebody's not spamming you with refunds to one credit card. But that's all it does.

It verifies that the message is coming from the person you expect it to come from. And in regard to security models, how do replay attacks work in this context? Yeah, so this is a big problem. I don't know if it's a big problem. I don't know if people are intercepting a lot of webhook messages, but it's a security precaution that you need to take.

With signature verification, you need to be able to kind of dedupe messages. So if a refund is coming in, hopefully your provider has an event ID attached to it. Some providers don't do event IDs or anything like that. So, you use an event ID so you only process that message once, and probably attach timestamps to it.

So a timestamp will say, okay, this, you know, maybe you have some rules that say only process this message within a certain window of the timestamp. So you can do these things to protect yourself from somebody who has access to those keys or whatever, so they can't not spamming you with webhooks you shouldn't follow. Okay. And so moving from that, I guess, into like local development. So we have security in place.

I want to do local testing of my webhooks. Why is that so painful? It is a very painful thing. Because in our earlier example, if you want to monitor, if you want a message from GitHub when a PR merges, you have to go into GitHub and you have to put in, like, a URL of where you want GitHub to send that message. GitHub's not going to be able to send a message directly to your laptop, your localhost environment.

Maybe you have some sort of tunneling program. There's all sorts of them out there. Set up some secure connection that pops your laptop out to the web with a unique URL. But 99% of those programs, the URL changes like literally every time. So every time you boot up your laptop, you've got to go into GitHub and change something.

I mean, the other option is deploying to your staging environment or your testing environment somewhere up in the cloud where you can receive webhooks, but every single time you change, even if it's a simple spelling mistake, you've got to do this whole CI deployment pipeline, and it just takes time. So the time spent deploying up to the cloud or every single time going up and changing this URL, it's annoying.

It's a lot of friction. I'm assuming you thought about this too while building Hookbridge. So how did you build that out for local development? Yeah, so even developing a product like Hookbridge, because we ran into this firsthand. So we built a little program, a little binary you can run on your computer. You go into our Hookbridge app and you create a one-time URL to hand to GitHub or whoever you want.

And you put that in the app and that URL never changes. And then you run this little binary on your computer, and it'll create a secure tunnel from your laptop up to our system. And our system will just kind of forward those webhooks through to your laptop. So GitHub can send it to us, and then we'll send it down to your laptop. You set it up once, just like any other webhook, and then that URL is live forever.

Interesting. So when you started building webhook infrastructure. What kind of things surprised you the most? Just how many ways it can break and how, you know, not our system, just webhooks in general. Just our whole conversation, how many things you have to account for on something that seems so simple.

And it really turns into this huge distributed thing that you have to build to account for all the ways things can go wrong. And that was kind of the inspiration for building Hookbridge. I've been in the industry for quite a while now, and I've seen this play out over and over everywhere I've been that has accepted webhooks. You have to do it right.

You do it right at the beginning or eventually you learn, you have to do it, but you have to eventually build this huge thing to accept something that seems so simple. Yeah, it's just a cURL command, right? It's simple. Yeah, it's deceptively simple for sure.

Okay, so if a platform team is building or buying a webhook infrastructure, what would you say are the most non-negotiables that they should look for or should be table stakes? Yeah, I mean, table stakes, obviously, a lot of the security stuff that we talked about, I mean, I'm obsessed with security. So you have to be able to do the signature verification and things like that.

But you need the tools to be able to troubleshoot when there are problems. All the observability and logging that we talked about, and inspecting all the request and response headers. That's what I'm trying to say. You just need to be able to see the whole chain to see where it's going wrong because it doesn't matter if you're sending webhooks or if you're receiving them.

The other party is usually not very helpful in troubleshooting the problem. And then obviously all the other problems that we talked about, like you got to be up all the time, that kind of thing. So what prompted you to build Hookbridge to solve this? Yeah, because literally every place that I've been that uses webhooks runs into these exact same problems. And finally, I'm like, just going to build it.

Build it once and we can use it forever. And what kind of users are you seeing use Hookbridge and what kind of problems are they solving? Yeah, a lot of it is the n8n users, like I talked about. We'll just act as an intermediary in front of their n8n box.

We'll keep retrying for as long as they want, and we'll try a little bit longer, and we won't time out as fast as maybe some providers that are having issues with n8n. But an interesting newish thing is, I mean, I'm sure we've all heard about OpenClaw and all these other claws, all the home AI automation things that are out there.

A lot of people are setting up systems, tools at home that are great, do a lot of different things. A lot of them are requiring to receive data through webhooks. Don't let OpenClaw open holes in your router to send them to your laptop.

So one thing that one thing we've added to Hookbridge that's really caught on is we call it a pull endpoint, where you go into Hookbridge, you add an endpoint, a webhook-receiving endpoint, just like you would normally. You hand that to Stripe or whoever is sending you webhooks and they send webhooks normally. But we don't forward them onto you.

We just hold them and then OpenClaw, whatever system you have that wants to receive webhooks, they can just hit our API and they can pull down the events from us when you want them. So you don't have to open up any firewall holes or anything like that. It also solves some issues with ordering and things. So you can pull down by event ID if you want all the orders first and then all the updates and things like that.

So the out-of-order processing and process-once issues kind of go away when you're pulling things with an API instead of just receiving them whenever they might happen. Yeah, that's cool. Okay, so if I wasn't ready to start with a platform yet, but I want to shore up or fix my webhook issues so I don't run into these problems, what are the top three things that we should do right off the bat?

We kind of talked a little bit about this, but is it HMAC? What is it? My number one thing is always going to be the HMAC signature. Like everyone should do that. Just don't accept random webhooks from whoever's pretending to be somebody else. Beef up your observability platform.

Whatever you have now, just make sure that you can diagnose any problems, whether they're on your end receiving, or as much as you can kind of gain insight from whoever's sending you, or if you're sending webhooks, you know, the responses you're getting back from the person you're trying to send to. Just make sure that your observability platform is as good as it could possibly be.

Have you noticed, we talked a little bit, you talked a little bit about OpenClaw. Fable is now back online after being taken away for a couple weeks. Have you noticed any trends on the security side? There's been a lot of RCEs, I know, outside of just webhooks, but there's been a lot of CVEs out there lately.

Have you noticed anything with the platform, like from your customers or from other friends in the industry around webhooks, trends related to AI or related to just infiltration in general, like trends that are happening? Maybe AI is making that easier, but maybe it's not even AI related. Yeah, a couple. Like I said, the people running OpenClaw at home that need to receive webhooks.

That was kind of an unexpected surprise. It's nice. It's more people than you might think, too, that are all of a sudden like, I need to receive webhooks to my iMac. That's weird. Okay, I get it. I get a lot of, and this has always happened, even when I didn't run the business.

Just being on the DevOps team, you do occasionally get these emails, but there's a lot of people now that have access to OpenClaw or AnyClaw, any AI, whatever, who are security researchers. And they send you all sorts of emails: "I found whatever vulnerability. I'll tell you about it for whatever fee," you know, so you get a lot of that. And I think that has probably increased since the dawn of AI.

I don't think they're finding legitimate things. The couple that I've asked, oh, hey, can you show me some proof? And maybe I'll give you the bounty. They're not finding real things, at least from what I've seen. Locally, personally, I'm using it. Yeah, it gives me good confidence when I can run the code through multiple AI models and say, give me a thorough security review.

And we can definitely find things that way. Yeah, out on the open web, I haven't seen a whole lot coming our way yet. Fingers crossed, because I know some communities are really under fire from it. And what kind of customers primarily do you have or users do you have in the platform? And what are they using Hookbridge for? Like, what problems are they solving? Yeah, big one is the n8n users.

Someday I'll be able to say that well. But a lot of them are the opposite of what we've been talking about. We've mostly been talking about how to build a system if you are going to receive webhooks. And it's a big thing, like we discussed. You've got to have a queue to hold all these messages and make sure you process them in the right way and do all this signature stuff. A lot of people don't do it.

So if you are sending webhooks and the person you are sending to doesn't have a system that is reliable and can receive webhooks and things, you need to kind of build the reverse of it, be able to retry and stuff. So instead of having to do that, people just use us because we will queue the message, keep retrying it, sign it for you, and do all that kind of stuff.

So yeah, everything we talked about receiving, you kind of have to do it on the reverse too. It's interesting. So okay, wrapping up, is there any webhook opinion that you have that maybe other engineers might disagree with? Well, as the owner of Hookbridge, I'll probably say, you know, most teams shouldn't build their own webhook infrastructure. I would say a controversial webhook take is to not use webhooks.

I would say polling is an underrated thing. Polling still has a place. Like, obviously, be a good citizen. Don't hammer someone. Or, you know, this wasn't my intention, but again, use the Hookbridge pull endpoint. Poll us. We don't care. Let's say someone was looking into growing their webhook journey and they don't want to do all of this themselves.

What can Hookbridge offer them and what does it cost to get started? What's the barrier to entry? Yeah, I don't blame you if you don't want to build this huge thing because it's a lot. We have a free plan. You can sign up. All the features are there. There's no big thing that you can't do with the free plan. I think you can send like a thousand webhooks over a month.

That should hopefully plenty to test something out. And for a lot of our users, that's enough for their home OpenClaw installs, like we talked about. A thousand a month is fine, and they can just send and receive that many for free. But then obviously we have paid plans if you need to send more messages per month or you need a whole bunch of endpoints and things like that. The paid plans have long retention periods.

So if you're receiving webhooks and you're down or actually you're probably going to want to get your own site up really fast too. But you have no control over your customers, right? If the people you're sending to are down for a lengthy amount of time, we'll hold the message and we'll keep retrying it for a long time. And our paid plans have kind of lengthy retry periods, retention periods there. Very cool.

So where can people find out more about you and Hookbridge? Hookbridge is at www.hookbridge.io. Send me an email, jay@hookbridge.io, J-A-Y. Happy to talk about webhooks. Happy to talk about anything. Yeah, let us know what you think.

Let us know your pain points with webhooks, because I know there are tons of them out there, and we are happy to discuss new use cases and how you might use webhooks or how Hookbridge could work for you. Very cool. Thanks so much, Jay, for coming on. Really appreciate your time. Thank you. That was my conversation with Jay Lark, Principal DevOps Engineer and founder of Hookbridge.

The biggest takeaway for me is that webhooks are not unreliable because HTTP is unreliable or because the basic pattern is bad. They become unreliable when teams build only the happy path and treat delivery like a single request instead of a distributed workflow. Once a webhook matters to your business, you need to assume duplicates will happen. Events may arrive out of order.

Endpoints will occasionally be unavailable. And something inside your own system may fail after you already acknowledged the request. That changes what success means. A 200 response does not necessarily mean the event was processed correctly. A valid signature does not mean the requested business action is safe. And a retry does not help much if nobody can see what failed or replay the event later.

I also liked Jay's point about the burden placed on the receiver. The sender gets to fire the request and move on, while the receiving team has to expose a public endpoint, keep it available, validate the sender, protect the business logic, deduplicate events, and build enough observability to figure out what happened when something goes wrong.

And sometimes the answer really might be polling, not hammering someone's API every second, but using polling intentionally when it gives you better control over timing, ordering, or infrastructure exposure. So if you are building a webhook integration, start with the failure path. Verify signatures. Use event IDs for idempotency. Acknowledge requests quickly. Retain enough information to debug failures.

And make sure you have a replay story before you need one. You can learn more about Jay and Hookbridge at hookbridge.io. I'll have that and anything else we mentioned linked in the show notes. If you enjoyed this conversation, follow or subscribe to Ship It Weekly wherever you listen to podcasts. You can find the show, previous episodes, and the weekly DevOps and SRE news recaps at shipitweekly.fm. Thanks for listening, and I'll see you later this week.

AWS CloudFormation Express mode can report a deployment complete up to four times faster, but the resources may still be stabilizing in the background. So the green check arrives faster, but it means something different. Apache Spark 4.2 adds native vector operations, allowing some AI data workloads to stay inside the platform teams already operate.

GitLab says that AI is making developers faster at producing code, while review, testing, governance, and deployment struggle to keep up. And a platform team at Sevdesk has a question for anyone rolling out new controls. Will your internal users hate you by the time you are finished? Put those together, and the theme this week is local optimization.

Making one part of the system faster does not guarantee the whole system gets better. Sometimes it just moves the waiting, risk, or ownership problem somewhere else. I'm Brian Teller from Teller's Tech, and this is Ship It Weekly. Welcome back to Ship It Weekly, the show about the DevOps, SRE, cloud, platform, and security stories that matter when you are the person keeping the thing running at 3 a.m.

For the weekly story list and source links, check out OnCallBrief.com. For full show notes and past episodes, head over to ShipItWeekly.fm. This week: CloudFormation Express mode, vector functionality in Spark 4.2, GitLab's warning that AI is accelerating code faster than software delivery, and minimum viable governance for platform teams.

In the lightning round, OpenShift security updates, a GitHub Actions outage, Cloudflare dropping some POST requests, and GitHub Code Quality becoming generally available with billing attached. And we close with Reid Savage on the difference between what a capable team could do and what it should do. Let's get into it. First up, AWS introduced CloudFormation Express mode.

AWS says it can reduce deployment completion time by up to four times in some workflows, but it does not necessarily make the underlying resources provision four times faster. It changes when CloudFormation declares the operation complete. In standard mode, CloudFormation applies the configuration and waits for resources to pass stabilization checks.

Express mode reports completion once the configuration has been applied, while the resource may continue becoming operational in the background. AWS showed an SQS deployment dropping from 64 seconds to around 10 seconds. That is meaningful when you are repeatedly changing and testing infrastructure, especially for AI agents that are generating, deploying, and correcting configurations.

But deployment status is a contract. Pipelines, dashboards, agents, and engineers all make decisions based on what "complete" means. What happens immediately afterwards? Do you run integration tests? Deploy an application? Shift traffic? Start another stack? With Express mode, those steps may need their own readiness checks. CloudFormation can be finished while the resource is not ready.

Express mode also disables rollback by default, although teams can turn it back on. That makes sense for development where you may not want every failed experiment followed by a long rollback. In production though, faster confirmation without readiness validation, monitoring, or rollback could let the pipeline move ahead too early. The takeaway is not that Express mode is unsafe. It looks useful.

The takeaway is that teams need to separate three states they often treat as one. The configuration was accepted. The resource finished provisioning. And the application is ready for users. For development, testing, and agent-driven workflows, the faster loop could be a major improvement. For production, the pipeline needs to understand exactly what that green check now means. Faster confirmation is useful.

Confirmation is not readiness. Second story: Apache Spark 4.2 adds native support for more vector-oriented operations. Spark SQL now includes functions for similarity, distance calculations, normalization, and vector aggregation. It also adds NEAREST BY, a top-K ranking join for matching rows based on similarity or distance.

That supports workloads like recommendations, entity matching, candidate generation, and offline retrieval. This does not mean Spark replaces every vector database. A specialized system serving low-latency application requests is solving a different problem. But not every vector workload is an interactive search request. Sometimes the data is already in Spark.

Sometimes you are evaluating embeddings, enriching large datasets, or generating candidate lists in batches. Moving that data into another platform means another credential flow, network path, scaling model, upgrade process, and bill. So the first question should not automatically be, which vector database should we add? It should be, what does this workload require? Is it online or offline?

What latency and freshness do we need? Do we need a serving API? Or can the work stay inside a platform we already operate? Large similarity operations can still be expensive. Teams still need to think about scale, partitioning, and compute costs. But Spark 4.2 reflects a broader shift. Established platforms are absorbing more AI-oriented capabilities. Sometimes a specialized system is justified.

Other times, the best new platform is the one that you do not have to introduce. Third story: GitLab released its 2026 AI Accountability Report, based on a survey of more than 1,500 developers and technology buyers. 78% said developers are writing and committing code faster with AI. But 85% agreed that the bottleneck is shifting toward review and validation. That makes sense. Software delivery is a chain of queues.

Implementation. Review. Testing. Security. Deployment. Operations. AI may accelerate one part, but if the next stage has the same capacity, work piles up there. If developers create twice as many pull requests, but the organization has the same number of reviewers, review gets slower.

If an agent generates Terraform in five minutes, but every plan still needs a senior engineer to interpret the blast radius, that engineer becomes the queue. And if more code reaches production without better validation, on-call becomes the final quality gate. Producing the implementation faster is a real gain. But what happened to lead time, rework, review quality, and change-failure rate?

GitLab also found a gap between confidence and traceability. Most respondents believed they could quickly determine whether AI-generated code contributed to an incident. But among organizations that had experienced an incident, roughly a third could not actually determine it. That leads to three useful questions for any AI-generated change. Where did it come from? What was it intended to do?

And who owns the result in production? Those are not just compliance questions. They are incident-response questions. When something breaks, teams need to know what generated the change, what context it had, what validation ran, and who accepted responsibility. The exact percentages come from vendor-sponsored research. But the broader point is sound. Generating the artifact is getting cheaper.

Understanding, validating, and operating it is not getting cheaper at the same speed. The goal is not to maximize code entering the pipeline. The goal is to safely deliver useful changes through the entire system. A faster coding agent does not automatically create a faster engineering organization. Fourth story.

There was a talk called "Road to Compliance: Will Your Internal Users Hate Your Platform Team?" The story comes from Sevdesk, where the platform team needed to improve cloud security, spending, and resource governance. The bad version of that rollout is familiar. The platform team creates a long list of requirements, gives developers a deadline, and threatens to block deployments.

The controls may be technically correct, but the rollout treats governance only as a configuration problem when it is also a change management problem. One issue at Sevdesk was inconsistent resource tagging. Rather than immediately blocking deployments, the platform team first built visibility.

They used AWS Config and Security Hub to find resources, connected findings to repository and CI information, grouped notifications to avoid flooding teams, and sent feedback through Slack. The rollout followed three stages: inform, soft-enforce, then enforce. They call this minimum viable governance. The idea is not to weaken standards.

It is to begin with the controls that address the company's most important risks and introduce them in a way that teams can adopt. If Security Hub contains thousands of findings, treating all of them as equally urgent does not create security. It creates noise. Without prioritization, the platform team becomes the enforcement layer for everybody else's queue. That is how a paved road starts feeling like a toll booth.

The deeper lesson is that the platform team's product is not the policy. The desired product is changed behavior. A failed deployment does not teach someone how to fix an issue. An automated message is not useful unless it reaches the correct team, provides context, and explains the remediation. Important controls should eventually become real controls. But the sequence matters. Build visibility. Find ownership.

Prioritize by risk. Give teams useful feedback. Then, enforce what has already been explained and supported. Internal users are still users. You can win the policy argument and still lose the platform. Quick lightning round. First, OpenShift Container Platform 4.22.5. Red Hat rated this an important security advisory and recommends OpenShift 4.22 users update when the release reaches their channel.

The reminder is simple. Supported does not mean current. And a functioning cluster is not necessarily a fully patched cluster. Second, GitHub Actions had a disruption on July 13. At the peak, 30% of Actions jobs failed to start. Pages builds and Copilot cloud-agent sessions were also affected because both depend on Actions. GitHub traced the issue to an autoscaling change using outdated capacity thresholds.

The configuration was valid. Its relationship to the live system was wrong. Validation needs current operational context, not just correct syntax. Third, Cloudflare had an incident where some POST requests did not reach customer origins correctly. The takeaway is to test more than GET. A homepage can load while logins, purchases, API writes, and webhook deliveries are failing.

Monitor the actions that users actually depend on. Fourth, GitHub Code Quality is now generally available. It includes CodeQL analysis, AI-assisted detection, quality gates, and Copilot Autofix. It is also now a paid product at $10 per active committer per month, plus some usage and Actions costs. Billing began automatically for organizations that had it enabled during preview.

Preview features can become production dependencies and production bills. Know what has been enabled before the invoice explains it for you. The human closer this week comes from Reid Savage at Honeycomb. Their article is called "Could vs. Should: The First Year Managing an SRE Team." Reid says SRE teams and engineering managers have something in common.

There are many things they could do, but only some are things they should do. Separating the two requires context and comfort with ambiguity. During their first year managing Honeycomb's SRE team, Reid sometimes stepped too far away from technical decisions because they wanted the team to maintain ownership. But staying completely silent was not always useful either.

Their experience could identify roadblocks or add context. The lesson was not that managers should take over. It was that they need judgment about how to participate. Sometimes you make the call. Sometimes you recommend a direction. Sometimes you offer context. And sometimes you stay quiet. The same applies to SRE ownership. A capable SRE team can solve many problems.

But capability alone does not determine responsibility. Will owning this work create reusable leverage? Will it improve reliability across the organization? Or will it make one strong team permanently responsible for everyone else's ambiguity? That connects back to every story this week. CloudFormation can report completion faster. But should this production workflow use it? Spark can perform more vector work.

But should this workload live there? AI can generate more code. But should all of it immediately enter the review queue? The platform team can block every noncompliant deployment. But should blocking be the first interaction developers have with the policy? Capability expands the decision space. It does not make the decision. Good engineering increases what is possible. Good leadership decides what is useful.

And good teams keep asking whether the thing they could do is actually the thing they should do. That's it for this week's Ship It Weekly. We covered CloudFormation Express mode, vector operations in Spark 4.2, AI accelerating code faster than delivery, and minimum viable governance for platform teams.

Plus, OpenShift updates, a GitHub Actions autoscaling failure, Cloudflare dropping POST requests, and GitHub Code Quality becoming generally billable. Follow or subscribe wherever you are watching or listening. And send this episode to someone trying to fix an organizational bottleneck by making one tool run faster.

You can find the weekly story list at OnCallBrief.com, and the show notes and past episodes at ShipItWeekly.fm. I'm Brian Teller from Teller's Tech. Thanks for listening. And remember, faster is not the same as finished. And could is not the same as should.

AI demos are easy. That is kind of the problem. You can wire an LLM into a tool, give it a few prompts, let it call an API, and suddenly it looks like magic. It writes queries, it explains dashboards, it investigates issues. It might even suggest a fix. And in a demo, that feels great. But production does not care about demos. Production cares if the answer was right. If the agent used the right tool.

If it leaked data. If it got confused by logs. If it made things worse after a deploy. If the cost graph quietly turned into a crime scene. And that is where this gets interesting because the hard part is not proving AI can do something impressive in a controlled demo. The hard part is figuring out what happens after people start depending on it.

When it is connected to real tools, real telemetry, real production systems, real costs, real failure modes, at that point, AI is not just a feature anymore. It is another thing you have to operate. I'm Brian Teller from Teller's Tech, and this is Ship It Weekly.

Welcome back to Ship It Weekly, where I filter the noise and focus on what actually matters when you are the one running infrastructure and owning reliability. Most weeks, it's a quick DevOps, SRE, platform, cloud, and security news recap. In between those, I do conversation episodes with people building and operating the systems we all end up depending on. Today, I'm joined by Mat Ryer from Grafana Labs.

Mat is Senior Director of AI at Grafana, a longtime Go developer, author, open source contributor, and podcast host. And in this episode, we talk about what happens when AI moves out of the demo booth and into production.

We get into Grafana Assistant, AI observability, evals, LLM-as-judge patterns, telemetry cost, OpenTelemetry, agent guardrails and why a chat interface is not enough if the system underneath is making decisions people need to trust one of the threads I liked most in this conversation is Mat's point that AI gives us a new primitive that sounds big and abstract but his example is pretty simple a conversation plus a loop suddenly becomes an agent Or as Mat put it, for loops are back.

But once people rely on that agent, the problem changes. Now you need to know whether it helped, whether it answered the question, whether it picked the right tools, whether a prompt change improved one workflow while breaking another, whether your AI feature is actually observable enough to operate like the rest of your production stack.

We also talk about why AI observability is not just latency, logs, and HTTP 200s. Those still matter, obviously. But now you also care about behavior, cost, tool selection, user feedback, eval results, model versions, prompt changes, and whether the agent is producing something useful.

Or just confidently wandering around your telemetry matt also gets into UX which I think is more important here than people give it credit for because if the AI gives you a wall of text you still have to trust that text but if it can show you the graph deep link you into the right Grafana view apply the filters and let you inspect the source data yourself that is a very different experience and near the end We talk

About where AI can actually help operations teams today, not some giant six -month AI transformation project.

More like pick something small. Use it to enhance the workflows you already have. Let it write the query. Let it help with the first pass. Let it automate the boring parts, but build the guardrails and feedback loops around it, which honestly fits the name of the show pretty well. Ship something small. Learn from it, then ship again. All right, let's jump in. Today, I'm joined by Mat Ryer from Grafana Labs.

Mat is a senior director of AI at Grafana, where he's focused on how AI fits into observability and production systems. He's also a longtime Go developer, author, open source contributor, and podcast host.

And we're talking about what happens when AI moves from demos into production, why observability for AI systems is not the same as basic service monitoring, and what teams should be thinking about as telemetry volume, cost, and operational complexity keep climbing. Mat, thank you for joining me. Thank you, Brian. Pleasure to be here.

So starting out, I'm curious, as AI systems move from experiments into production, What are teams underestimating? Yes. Well, first of all, I think it's very exciting, this whole space. And I think that's where I always start with this. We have a new primitive now. We have a new capability, something that we couldn't do before. And that, I think, is just a very exciting thing. There are concerns with AI and valid concerns.

Alongside them are I think is the fact that we are now in a new world and this is a very exciting thing you see people making little agents to solve all kinds of little problems we see people building big agents and big complex ones as we are doing as well at Grafana Labs to solve more complex and deeper problems and just now like having a a new primitive like a conversation Being able to now, even just putting that in a loop, by the way, which was when things became agentic.

That's just a for loop. That's just like, oh, for loops are back. They're so back. They're cool again. And I think agents are just amazing. Just a simple primitive put together with something that we're already very familiar with can produce something new. So we're still in that phase. And I love to see all the little things that people are building.

At some point you do end up with, like we have a product in production, Grafana Assistant, and people are relying on it. They're using it as part of their day-to -day. Some people have leaned very much so into it and it's really unlocked them for all the observability stuff they're trying to do. So now suddenly we can't just vibe it. We can't just feel it out.

We've got to grow up and make sure that this is going to work for them. And I think people underestimate how hard that is. It is quite easy to wire up an LLM to anything, do a few simple tools, and suddenly you unlock this new capability. But how do you really know that's good? How do you know when you change it, you aren't making it worse in some other place?

You know, we used to have unit tests, test suites protecting us from all this. We didn't really have that now. So how do we make sure?

When we make changes to it it is getting better and how do we make sure it's even like correct in the first place you know it demos very well like you say we've kind of I think checked off that from the mission like has to demo well for people to care to it demos pretty well uh but beyond the demo yeah how do you how do you make sure it does that and we we have we had to solve this problem in assistant which we built our Grafana Assistant we launched that I think it was last year now.

And yeah, we've had to solve that problem. So at a high level, how do you start with that problem statement? How do you, do you build special testing around that? How do you actually solve that? Yeah, so I'd say, right in the beginning, I think people still vibe testing, manual testing, I still think is the thing to do in the beginning. Because it's the really when you're just feeling it out, you need to know.

Is this even possible? Is this feasible? Does this make sense? And there's so much innovation and there's so much opportunity to learn that actually you still have to have that same attitude of build things and ship it. That's why I came on the Ship It Weekly podcast because it's all about shipping it. So I would say don't wait for there to be good evals.

Don't wait for you to have this problem solved before you progress. But it is something to think about in the back.

In the background and yeah when when we did assistant we essentially had to build we had lots of different tools of course there's like the normal telemetry that we're very good at Grafana Labs metrics for measuring latencies and things like this and we even have metrics for costs measuring the the cost of different operations and things uh logs of course kind of makes sense I think tracing becomes more interesting

Now because you could think of a conversation really as a as a sort of trace you could imagine it in that you know in that model um and and I think so so you get a lot already just of the basics but there are new things you care about suddenly uh you know it's not just getting a 200 back from an LLM but now how do we how do we make sure that the content the the way it behaved the tools it chose to use how do we make

Sure that stuff is right and we did it um partly By vibing tools internally, literally we'd have like, we've got a handful of them and various dashboards and things and different databases and different ways of storing things, you know, just trying to sort of cobble together the solution of this.

And again, I think that was the right approach in the beginning because it's a playground where we can experiment and we can try things. We can solve problems without having to think about solving the entire problem.

Of this we can just pick one piece of it and just solve it with a standalone thing so there's a lot of freedom in that and I think the culture that we had in the engineering team when we're building this played a lot into that because yeah we had to make sure that people could try things just solve that particular problem not go for months and have big design sessions and just do this enormous project to try and

Solve the whole thing After we'd done all that, of course, you then end up in a situation where you do have quite a complex little set of internal tools that are all slightly overlapping and doing things in different ways.

They don't all agree with each other. Some of them are agentic themselves. So that's even more getting like Inception, where the agents are monitoring the agents to make sure that they're behaving properly. And so then we were able to take all that learning and put that together into our AI observability product, which is a new feature that we have in Grafana Cloud. And that's our answer to this.

And we think we've picked the most important things that people should care about and built that into the product. And you can see we're different to some kind of competitive products. We're different in some interesting ways because I think some of them were imagined. Whereas we built ours from the real day-to-day of actually operating the assistant at scale. Yeah, very cool.

What does production level observability for AI actually look like? Well, you do need evals. You need to be able to test whether what's actually being asked for is being answered. And we have a... We have a technique inside where we actually have an LLM as a judge looking at a conversation. And you can just do this. You can try this. It works. Just ask it, did you answer the user's request?

Because you can see their first message. You can see the path you went on and where you ended up. Did you answer it? Did the user have to intervene in any way? Were you ambiguous? You can kind of ask it these things in, and you're dealing with natural language now. So it's like you write these tests also in natural language. And then we can measure that.

We can start to measure and start to put some data and some science behind the behavior of agents. So evals, I think, are going to be increasingly more and more and more important. We want these AIs, we want these technologies to be trusted and to be used. Some people write them off just because they know that they can be wrong. So they can't trust it, and so therefore they're not going to use it at all.

I don't think that's the right attitude, personally. You know, the internet, the network connections aren't reliable. They error all the time. And we kind of paper over those errors so you don't really notice it. Same kind of thing with AI. We have these non-deterministic systems.

In some ways, when it comes to doing investigations and things, for like digging around your telemetry, that can actually be turned into an asset where... The fact that multiple agents take random routes through things can actually be good. That's what humans might do if they all take a different approach.

But then you've got an even bigger observability problem because now this one thing, suddenly it's multiple conversations with models, not even the same model, multiple models often, certainly different system prompts, different sets of tools. All this complexity now is kind of a big...

Mess that you have to deal with and we think yeah simple observability simple uh and new signals as well as part of this is the solution to so that you can just answer those questions and make sure that your agents are on the right track so kind of following up with that a lot of teams instrumented everything then they realized they were drowning in telemetry costs uh what changed yes I think we're going to go through that as well for AI.

So I think you're right. People would just monitor everything. It feels really cheap. Like when you're writing something, you just add a counter in code. It feels like the cheapest thing you can do. As soon as you then hit that scale, suddenly you're counting. You're just counting at scale is hard. So, yeah, you kind of create these new kind of problems when you hit scale. The same thing is definitely true for AI.

I think a lot of people aren't really monitoring much at the moment. But when they do, and as this increases, we'll have the same kind of problem. And, you know, Grafana Labs has this technology called adaptive telemetry, which is basically what they do is they look at all the telemetry that you've used. They look at all the queries you're making. And then they identify bits where you've never queried this metric.

So we're going to automatically sample it down. So we'll just take a sample of it. We won't capture all of it. And, of course, you can tune it and tweak it and things. And this was really an answer to that problem, the fact that people's telemetry was just growing and growing and growing, and not just in the history of it, but even just the now, because people's systems and complexity is going up.

So I think that we'll end up with another kind of adaptive telemetry. We have already in our solution sampling, so you can choose how often the evaluators will run on conversations. So we're kind of thinking about that already. And I think this is the advantage of building this solution at Grafana Labs, because we had to solve a lot of the observability problems already.

And a lot of the thinking applies directly, which is very cool. We have these drill down apps, which... Basically present the data and you can just click and drill into it. That turns out to be good for humans, but actually quite good for agents as well, because it's the same problem. There's just too much stuff there and to look through it all and find it is hard.

So how do we, yeah, the fact that we solve that for humans turns out it's quite good to solve for agents. Where does OpenTelemetry fit into this AI observability story? I think it's going to have a...

Big part of it we actually are talking already with um ted young who was one of the co-founders of OpenTelemetry yeah and we're kind of um I've seen some proposals and I've seen some things where the conversation data gets put into trace data so that they they are stitched together in that way we didn't take that approach because of scale issues um that we saw so we haven't got that approach in our solution but It could well end up being some new standard like that.

But I just think with OpenTelemetry, it's going to just take some time. You know, we like to move very fast in the AI department in Grafana Labs and in Grafana Labs in general. In some places, you do have to go slowly. Where you're dealing with standards or foundational things that you really need to get them right, you're going to go a bit slower. So I think it'll take some time. But I do think OpenTelemetry should...

Embrace this new world and and see which new signals are common enough that we are going to then capture them we got to make those decisions ourselves when we built our solution so of course we capture you know the the basic things you expect metrics for latency and things we add cost into there using metrics and stuff that that's also turns out to be quite complicated to do but very powerful if you have accurate

Cost data um And with the way that we have labels and things, kind of like the Prometheus model, you can use labels for the different versions of your app or different agent names or something.

So now you can compare performance, not just cost and metrics, but the other dimensions as well. And if you've got evals running, you can compare behavior between different models as well. So you can say, version one of our model. Took a long time to answer this kind of question. And it's because we hadn't prompted it much or, you know, we just gave it one tool that it could use. We did some work.

We added a couple of tools. We tweaked some prompts and our new version is now much faster. It solves this problem. It's much more accurate. The user's happy. We capture ratings as well of users, you know, thumbs up, thumbs down, as well as comments. All the sort of practical things you need to actually have some kind of feedback loop where you can improve your agents.

Is that what you have found to be, I guess, the best feedback loop is like providing like right in the chat, the ability to give feedback immediately like, hey, this was not the right response or hey, this was not detailed enough or it went off on a tangent that was not really related to the question. Or are you feeding that in with other information as well? And then how does that work?

Yeah, I think user feedback is the most valuable. And it is nice that you can, even just a thumbs up, thumbs down is enough to give you something to go on. I think actually typing in a comment is even better, right? Because we can genuinely have LLMs consider that commentary and even suggest PRs.

Into our repo to improve prompts and things like this um so mostly though we don't get uh we don't get as much of that as we would like so we can't rely on that and the other thing is we're very sensitive about learning from people's data without them realizing that's what we're doing so yeah we are we are very um yeah we want to make sure that we are uh not yeah just being very trustworthy basically if we're going

To build AI that we expect people to use I think the standards have to be very, very high for things like that.

Make sure that your data is protected. It's easier than ever to leak data. We had a case where, luckily it was test data, but we set up a GitHub permission for an MCP server. We didn't realize that you don't need write permission to open an issue in a public repo. So we didn't give it write permission, but still it started to open an issue. It was a test box, so it was no problem.

It was putting data that it saw in there into the issues very cool feature of course and extremely useful for like IRM and incident management and things especially useful when it opens a PR for you very nice but we've got to do that carefully and safely and I think knowing that your systems are safe and aren't behaving and that they will react uh properly and appropriately to prompts that are kind of probing and trying to do malicious things.

I think that all is a very important part of this that we're still scratching the surface on. Yeah. I think it also kind of highlights the need for guardrails because LLMs are over-eager, right, to solve the problem, get to the root cause and provide a solution. And if it has the means to do that, it will. Yeah. It's true for tools as well.

It's true for like, if you give it lots of tools, it's just going to be like a kid in a candy shop. It's going to be so excited to just go and use all the tools. And given that it's non-deterministic, you can easily end up in a situation where it's using quite random tools to try and solve problems.

And it's trying to use a hammer to open a door, for example, which I'm not a DIY expert, but I don't think that's how you do that. But yeah, so. So how do you sort of tackle that? I think that's kind of another interesting angle on this. Yeah, so from a guardrails perspective, I'm sure at the IAM level, like you're providing read-write permissions or scoped permissions based on what's needed.

But are you doing like pre-prompting as well? Or are you not relying on pre-prompting at all? Because that's just too close to the LLM and, you know, it could be influenced.

Uh well yeah so we have how do you build that system yeah yeah we've got a few different things one of the so our AI observability solution lets us uh write kind of security type tests and assert them basically on the model on the model so that kind of gets us we can do some kind of testing there just to set test the end point to see what how are you going to behave if a user asks for this um you know it's also

Dealing with customer data so it's also dealing with data in logs which can just be any text so you have to be careful to make sure that it doesn't just load some logs and think oh I'm just going to execute something in in here um and the real answer in in assistant is we just have lots of different checks and different gates and things happening because there's so much it's a very complex um solution I think it

Looks simple when you use it and the UX there's another thing I will get to talk about hopefully is the the user experience of this is paramount here more than ever and it's always been important I think in tech but the um you know the fact that you you the fact that you need to uh just just have well the fact that we have this complex system we use multiple models we've got like specialist tools inside assistant all

Running in Grafana Cloud there's preemptive work that happens around discovering the infrastructure And so we put that into a semantic vector database.

So it's the agents have access to kind of learning that it's had before. When incidents and investigations are happening, that also is a great source of information for agents. But again, how do you make sure that's all safe? So I think at any point when you're integrating and bringing some data in, you have that security question of in what ways could this be malicious?

What protections can we do here to help and make sure it's not going to be a risky system for people to use? And how have you thought about the UX side of the user experience side of the AI agents? How is that architected or how do you think about providing that experience in a quality environment that is actually useful? Yeah, this is somewhere where I think the team have excelled. And we have heard this feedback.

One of the big labs told us that at the time, this was the best looking version of something that they'd seen using their models, which was a big, big compliment, of course. I think the key is we on the team just got really annoyed by walls of text.

It was a kind of a byproduct of this is that, you know, the fact that you are dealing in conversation means it's generating text as what kind of how the whole thing works. And it's just so much text. Now, when it comes to trusting LLMs, you know, in that text generation, that's where they can hallucinate things. So we, instead of using text wherever we can in the assistant. We'll use a Grafana visualization.

So, you know, luckily we have this library of these beautiful tools. Grafana has been around a while. So anything we needed to represent from logs or from traces or metrics or profiles or other ways with graph views and Mermaid diagrams, things like this, we would ask the assistant to do that and to prefer that. And so it's...

It's really about, I think, having people with excellent taste, kind of designer engineers in one, really, having them obsessed with that mission of we're going to fight this wall of text. Because if, you know, a picture tells a thousand words.

So if we can show you, here's what the data is doing, not only can you see it as a user, but that is still better as an experience for users than reading what someone tells you about that graph. Someone might say, oh, it's a flat graph. Is it flat but high? Is it flat but low? You can describe it. It's actually easier. Just show me the graph.

This is why I don't think dashboards are going to go away in the AI world because we're always going to care about what's true. And if the agents are correct, we're going to want to see that data. So the Grafana Assistant is very visually beautiful for that reason because it piggybacks a lot on Grafana. Yeah, it's important to, it's going to make a big difference.

And I think you can separate yourself if you put a bit more effort into making sure that that experience is excellent. I think you can separate yourselves from similar capabilities. It turns out for some reason, and we have a few theories on this, but it turns out that LLMs are quite good at Grafana by default. And we think it's because of so much of its open source. Grafana.

The whole code base is open source, but also all the community stuff, all the discussions around there, like community dashboards out there. And a lot of the connected ecosystem, Prometheus and OpenTelemetry, all these other things are also kind of very open source. And I think the LLMs have hoovered all of that up, so they naturally have quite good context. So when we first stitched together the LLM...

With Grafana and gave it a few tools and had the UI in the Grafana front end. It was surprisingly already quite good. And this is what people are finding when they wire up Claude Code with our gcx, which is a CLI tool that we released. That thing, you know, for certain tasks, it's very good. And if you've got your code right there as well, like it's very handy to be working alongside with production code.

With telemetry you know and having the having them considered at the same time but all the work we then did in assistant with the prompting the fine tuning of certain little pieces of it the specialist tools and lots of other kind of secret sauce that's gone in created this uh complexity that we have to deliver in a simple way so I think if you use assistant it'll look hopefully just looks and feels very intuitive

You know it does cool things like it can build deep links so it knows how to jump you to the right page in Grafana but not just the right page because all the your because all the filters are in the URL parameters it will apply those filters as well so it takes you straight to the right view um and so that was another good thing I think we did was We aren't going to reinvent everything and give you an AI-only experience here.

We are going to use all the capabilities that you're already familiar with and some of the amazing tools that already exist in Grafana as part of this too. And the fact that the assistant knows what page you're looking at. Little things like that. If you go to a page and ask and say, tell me about this, it knows what you're talking about. So that really matters.

Those kinds of little details, that experience that the user's having, I think is what makes it feel like excellent, helps you trust it. You see the data itself as well. So you can see if it's made a mistake. And I think UX is more important than ever. Very true. Okay, so for a team that is starting out wanting to introduce AI, where... Where do you think AI can actually help operations teams today?

Like where should they start? What's the most important telemetry or piece of data to start using to analyze? Well, it will depend probably on each of their cases. The cool thing is they will know, they'll have an instinct already for things that they already care about. So the thing is, the nice thing is probably people already are using some kind of telemetry. They're using some kind of observability.

And really, AI is there to enhance that experience. It's there to make that easier to do. Since it writes the queries for you, you don't have to learn PromQL and LogQL. And I've used those languages before, of course, but I would always have to go and look up again how to do something or prefer the query builder, honestly.

Now you don't even need to do it at all because genuinely, through natural language, it can generate those things. And it generates very complex ones as well. It's far out, you know, surpassed me, my abilities. But I would say like, use it initially to enhance what you're already doing and see ways that you can automate. In some ways, that's really what we're doing.

It's just automating it, but automating it in a trustworthy way. And I would say from those few little early things and do stuff, ship it, Ship It Weekly. I like that actually as a motto, Ship It Weekly. Not just the fact that this podcast runs every week. But I actually ship things every week. I like that because you're kind of forced to focus on scope. Well, the scope that's most important.

You know, if you have a short window for shipping things, then you've got to really focus on what's important. And I think that applies to people picking up AI things. Don't do some enormous AI project that's going to take months and months. Because honestly, everything's different by then anyway. Pick something small, do it, action it, and talk about it and share it.

And see what see if there's excitement you can drum up it's what happened at Grafana Labs and then now you see people all over the company building all sorts of things and doing amazing things with agents some of it will eventually be a product I'm sure but it's it comes from play and that's something that we do all right at Grafana Labs we we have a lot of space for people to play and get creative and try things and

Fail for things to fail um there's a great analogy that I love where they they did a study where they got these two groups and they told they were making pottery you know on the pottery wheel but they were all amateurs they told one group you've got to make the best pot you can and they told the other group just just make as many pots as you can doesn't matter what they're like just make lots of them and then by the

End of it the the group that were making as many pots as they could were making better pots than the group that was trying to make the best pot and I think That tells you a lot, I think.

It's the trial and error that you learn. You know, there was a lot more waste on the floor of this other team where pots hadn't worked and they had to throw it away or smash it up and tear it down and turn it into something else. So it feels wasteful, I think, to some. And they might be tempted to kind of optimize that away. But honestly, that's often where the learning is. So I think people should pick up something, tackle it.

Something small have a play see what you can do and from there yeah you create new problems for yourself but they're good problems to have where where do you think AI is going with observability this year I mean obviously it's it's the talk at every conference but is there overarching areas in observability that you think that AI um that we're just at the forefront like we're able to do now with larger token uh more

Tokens more uh better models um faster models is there anything that like lends itself now that maybe we couldn't have done six months ago or we could do in six months um based on these these rapid changes to AI I think seeing seeing the models get better and get smarter is nice we are finding that we have to tell it less which is interesting we're also finding that we We are basically tuning our prompts to a specific model.

We noticed with Claude 3.5 to 3.7 even, 3.7 didn't perform as well as 3.5 by default. We had to make it work by changing it and tweaking it because it was a kind of different thing. And so I think we'll get to the point where agents are, yeah, you have to tell them a lot less. I think their instincts would be good. We already have plenty of tools to help with skills and things.

In Assistant, you can write skills, which lets you basically control how the agents behave. This is nice if you've got expert SREs where they've spent years building these patterns and practices and they've got these high standards. You can then give those same high standards to the agents and they'll follow your behaviors. So you can already customize and tune it like that. But I think...

That's going to just improve. And I sort of expect that to happen. Bigger context windows, not necessarily seeing that improve things too much. We already have kind of too much context. A big challenge that we're doing at Grafana Labs is, and with Assistant when it does an investigation, it's about going and finding the right context and keeping the right context and discarding the things that's not right.

So big context windows, kind of they're tempting. You just think, oh, we could just fill it all up with everything. I don't think that's going to help it at all. It's just more for it to think about. But I do think we are going to see more things automated. And one example I think is an easy one is rolling back a deploy.

Let's say you kick off a deploy through GitHub, whatever your process is, at the end, once it's out, Maybe, you know, there's some, you do this in staging and things, whatever you already have, all the great stuff. In production, have agents actually go and do a series of kind of critical tests, check things, check all the dashboards, you know, they know how to use Grafana, so they know how to use all your telemetry.

So how can we get all of that and make good decisions from it? And then a decision might be that.

That deploys not I'm not happy with it for whatever reason and I'm I as the agent I'm just going to decide to roll it roll it back so I think that's a safe quite a safe operation because it's you should always be able to roll back a release that should be a safe operation um it's not always so it's you know it's not without risk but um yeah that feels like something like that I think we're going to get we're going to

See more and more of that and then I want to go further what else are we happy to let it do can it can it deal with scale issues based on unexpected events can it um what else can it do where else can it take action and I think uh we need to get a bit braver sometimes uh on some of these things it's on us to build the tools and make sure that they are behaving well and you can see what they're doing and you know

People will some people will always have the the human in the loop to give the approval and hit the button and I actually think that's also fine but but all the work that leads up to that point Because there's still a big space there to explore.

And we are exploring it. It is very exciting, some of the things that can happen. But I do think we're going to have agents doing more of this operating things for us. At least the simple stuff. And that frees us up to then focus on the cases that aren't simple and that are more complicated. Or frees us up to build more things and do more other things. Very true.

Okay, so wrapping up, is there any belief about AI or observability that you think is just wrong? I think one thing that's wrong is because it's not 100%, you therefore can't trust it. As we mentioned earlier, yes, it's not perfect. It gets things wrong. So do people, but you still hire people, hopefully, for now. So I think that, I think, but that's not just AI. I think people often struggle with that.

I remember, I remember like in COVID times, I was wearing a mask and a guy in the elevators like thought I was silly. Like I thought I just believed some hype or whatever. So he's like, why are you wearing that? They're not even effective. They only like, only prevents 20% of spread or something. So it's like, oh, is it? So it's only 20% fewer people going to be. But because it wasn't 100%, he dismissed it.

I see people doing the same thing with AI. Or they think with coding, it's not as good as me because I prompted it, asked it to do something, it didn't do it as well as I would have done it. And so I did that last year and I'm not going to touch it. AI is not for me. That's also a mistake. Keep trying them. It's usually you haven't prompted it well enough if it's not working.

You can prompt it differently and there's different things you can try. And you can also be much more specific. In fact, the more context you give it, the better. So, yeah, it's not 100%. It's not perfect, but it's good. It's changing the world. Yeah, for sure. Mat, any closing thoughts for our audience?

No, but I just want to say keep shipping it weekly because, you know, that's the only real way to know you're doing anything useful. Iterate, yeah. And keep listening to this great podcast as well with Brian. Appreciate it. Thanks, Mat. Thank you so much for your time. Thank you for coming on. Really, really appreciate it. Pleasure. All right. That was my conversation with Mat Ryer from Grafana Labs.

The thing that stuck with me the most is that AI in production has to grow up pretty fast. It is fun when it is a prototype. It is fun when someone wires up a model.

Gives it a few tools and suddenly it can answer questions that used to require bouncing between dashboards logs traces docs and half a dozen slack threads that part is genuinely cool but once people start depending on it the bar changes now it needs observability it needs evals it needs feedback loops it needs cost awareness It needs guardrails.

It needs some way to tell whether the agent actually helped or just produced a convincing answer. And that is where I think this conversation gets useful for platform and SRE teams. Because the trap is thinking AI observability just means watching the AI service like any other service. Did it return a 200? How long did it take? How much did it cost? Those are good signals. But they are not enough.

With agents, you care about the path it took, the tools it used, the context it kept, the context it threw away, whether it answered the actual user request, whether the user had to correct it, whether the model changed behavior after a prompt update or model upgrade. That is a much messier kind of production system. I also liked Mat's take on trust.

A lot of people treat AI like it has to be perfect or it is useless. And I get the instinct, especially when we are talking about production systems. But we already operate plenty of imperfect systems. Networks fail. APIs time out. Humans miss things. Dashboards lie by omission. Runbooks rot. The answer is not blind trust, but it is also not refusing to use the thing because it is not 100%.

The answer is instrumentation, feedback, constraints, and judgment. It's the same as most other production problems, honestly. The other big takeaway for me was the UX side. AI should not just bury operators in a wall of generated text.

If the system can show the graph, link to the right view, apply the right filters, and expose the evidence behind the answer, That is way more useful than a paragraph that sounds right. Because at the end of the day, operators still want to see what is true, not just what the model said. So my takeaway is pretty simple. Use AI to make the work easier. Let it help search through telemetry. Let it summarize patterns.

Let it assist with the first pass of an investigation. But do not skip the production discipline. Measure it. Test it. Watch the cost. Build evals. Keep humans in the loop where the blast radius is real. And when you give an agent tools, remember that it may actually use them. That sounds obvious, but it is going to matter a lot. I'll have links to Mat, Grafana Labs, and anything else we mentioned in the show notes.

If you enjoyed this conversation, Follow or subscribe to Ship It Weekly wherever you listen to podcasts. It helps the show and it makes sure you get both these conversation episodes and the weekly DevOps, SRE, platform, cloud, and security news recaps. You can also find everything over at shipitweekly.fm. Thanks for listening and I'll see you later this week.

This week, Datadog is tracking coordinated GitHub API enumeration using ghost accounts, leaked tokens, and normal-looking API traffic, which is a reminder that public metadata can still become operational intelligence. XAI's Grok Build CLI reportedly transmitted repository data without redaction. So the AI coding tool question is not just, does it help me build faster? It is, what did it read?

And where did that data go? AWS Security Hub added Network Scanning and exposure impact analysis, which gets cloud security closer to the attacker's view. What is actually reachable? And what can that reachable thing access next? And Microsoft says AI-powered vulnerability discovery is changing patch management.

More bugs found faster means more pressure on the teams that have to test, deploy, and recover from patches. Put those together, and the theme this week is exposure. Who can map your systems? Who can read your code? Who can reach your infrastructure? And who owns the response when the map gets too accurate? I'm Brian Teller from Teller's Tech, and this is Ship It Weekly.

Welcome back to Ship It Weekly, the show about the DevOps, SRE, cloud, platform, and security stories that actually matter when you are the person who has to keep the thing running at 3am. If you are new here, follow or subscribe wherever you are watching or listening. And if you want the weekly story list and source links, check out OnCallBrief .com.

For past episodes, full show notes, and more from the show, head over to ShipItWeekly .fm. This week, we start with GitHub API enumeration because attackers do not always need an exploit to build a useful map of your organization. Then we get into Grok Build CLI and the data boundary around AI coding tools. After that, AWS Security Hub Network Scanning and impact analysis.

Then Microsoft's warning that AI-powered vulnerability discovery may mean more patches, faster timelines, and less room for slow change management. In the lightning round, HalluSquatting, Cloudflare control plane incidents, node-tar CVEs, ServiceNow authentication risk, quantum procurement pressure, and Lambda Managed Instances for Java cold starts.

And we close with something from SRE Weekly about post-incident reviews, authority gaps, and the difference between documenting an incident and actually learning from it. Let's get into it. First up, Datadog Security Research published a write-up on coordinated GitHub API enumeration and access token abuse. This one matters because it is not a flashy exploit story.

No zero day, no dramatic breach chain, just API traffic. Datadog is tracking campaigns that use GitHub's API to enumerate organizations, repositories, users, members, followers, public repos, private repo paths. And related metadata. Some of that data is public by design. That is the uncomfortable part.

Public repos, org details, user profiles, REST endpoints, GraphQL queries, relationships between people and projects. Individually, those answers may look harmless. At scale, they become a map. Who works at the company? What naming patterns exist? Which internal project names leak into public references? Which private repo paths might exist? Which users are worth targeting?

Recon is not harmless just because the data is public. Datadog also called out ghost accounts. Older GitHub accounts created years ago, left dormant, and later used for API traffic across many organizations. An old account blends better than one created yesterday. Then add compromised OAuth tokens or personal access tokens, and enumeration can turn into access.

Now, the actor may be probing private repo paths, listing repos, checking commit history, or cloning code. The takeaway is simple. Treat GitHub like a production surface, not just source control. Watch Token Scopes. OAuth apps, old PATs, dormant users, outside collaborators, audit logs, GraphQL usage, private repo path probes, and sudden bursts of API activity.

GitHub holds the instructions for how your company builds and ships software. Attackers know that. Your GitHub org is part of your attack surface. Second story. On Call Brief flagged reports that xAI's Grok Build CLI transmitted repository data and file contents without redaction. The details around one tool may change. The bigger lesson does not. If an AI coding tool reads your repo, that is access.

If it reads your dot env file, Terraform, Helm charts, Kubernetes manifests, runbooks, incident notes, or internal docs, that is access. And if it sends that context to a cloud service, that is data movement. That means security, legal, compliance, and platform teams need to know about it.

We spent years building controls around source code, private repos, SSO, branch protections, secret scanning, vendor reviews, access reviews, least privilege. Then an AI coding tool shows up and says, I can help you build this app. Cool. But what can it read? Where does the data go? Is it retained? Is it used for training? Does it respect ignored files? Does it read Git history?

Does it upload the whole repo or selected context? Can the organization audit usage? Those are not boring procurement questions. Those are production security questions. Repos are not just code. They contain architecture, service names, build pipelines, IAM assumptions, deploy rules, vulnerability hints, and sometimes secrets. The lesson is not never use AI coding tools. That ship has sailed.

The lesson is to stop treating them like harmless editor plugins. They are repo readers. Sometimes they are command runners. Sometimes... They are package installers. Sometimes they are cloud -connected agents sitting next to your source code and local environments. So they need policy, review, and boundaries around where they can be used. Because it was just on my laptop is not a data boundary.

Quick break from the news. The Ship It Weekly shop is officially open at shop.tellerstech.com. I've got a few Ship It Weekly t-shirt designs up there now. And I spent way too much time sourcing the actual shirts because I wanted them to feel like good trade show shirts people actually keep. Think of those soft Datadog style shirts that you get at conferences and somehow still wear two years later.

That was the target. I'm wearing one of those now so you can get a feel for it on video. If you want to support the show and rep the merch a little bit, head over to shop.tellerstech.com. For the next few weeks, coupon code SHIPTHESTORE gets you 20% off your order. That is SHIPTHESTORE. Again, shop.tellerstech.com. All right, back to it. Third story.

AWS Security Hub added network scanning and impact analysis for exposure findings. The short version. Cloud security is moving from this might be exposed based on configuration to we checked from the outside, this is actually reachable, and here is what that exposure could lead to. That is useful.

Network scanning probes resources from the internet to identify actual reachability, not just what the route tables, security groups, and diagrams suggest. Actual reachability. Public IPs. Virtual machines. Load balancers. Reachable ports. Services behind them. That matters because cloud reality drifts. A public IP gets attached during troubleshooting. A port gets opened and forgotten.

A test load balancer lives forever. A Kubernetes service becomes public because someone copied a helm value from dev. Config analysis is helpful, but active scanning gets closer to the attacker's view. Can I reach it? What port is open? What service is there? What evidence proves it? Then, impact analysis adds the blast radius side.

AWS says Security Hub can map downstream resources that could be compromised beyond the initial exposed resource. That matters because exposure alone is not the whole story. An exposed test box with no useful permissions is one kind of risk. An exposed instance with a role that can read production secrets, write to S3, or reach a database is a very different risk. So do not only prioritize by severity label.

Prioritize by reachability and blast radius. Public and reachable matters. Public and reachable with dangerous IAM matters more. And when something is exposed, do not stop at closing the port. Ask who owns it, how long it was exposed, what it could access, what logs prove activity, and what prevents it from coming back. The tool finding the exposure is only step one. Someone still has to own the cleanup.

Fourth story. Microsoft published a post about evolving Windows vulnerability management. For AI-powered discovery. And even if you are not managing Windows endpoints, the larger trend matters. AI is not just helping people write code faster. It is helping defenders find bugs faster. It is helping researchers analyze code faster. And eventually, attackers get better at this too.

Microsoft talked about using an agentic scanning harness called MDASH. To analyze Windows at scale with multiple models, validation, human review, and then high confidence findings going to engineers. The operator takeaway is that faster discovery means more update pressure. A lot of companies still treat patching like a monthly chore. Patch Tuesday happens. Vulnerability teams triage. Endpoint teams test.

App owners delay. Exceptions pile up. Someone asks if it can wait until the next window. That rhythm already struggles. AI-accelerated discovery makes the tempo worse. More findings. Faster analysis. Shorter response windows. More pressure to know what is actually affected. This does not mean patch everything instantly and hope. Patching is still production change. Patches break things. Agents break things.

Middleware breaks things. Security fixes can have side effects, but patching needs to look more like a real operational system. Known inventory, staged rollout, rollback, ownership, emergency paths, and coverage reporting, not spreadsheet theater. If your patch process depends on 10 people manually asking app owners how they feel, AI is going to make that pain worse.

Security updates are becoming production operations, so they need production-grade process around them. Quick lightning round. First, HalluSquatting. AI coding agents can hallucinate package or repo names, and attackers can squat those names ahead of time. If the agent tries to install the fake dependency, it can pull malicious code. Do not let agents install packages just because the name sounds plausible.

Second, Cloudflare had dashboard and API incidents. The edge may keep serving traffic, but if the dashboard or API is down, incident response can still be impaired. Data plane and control plane are different things. Both matter. Third, node-tar had a cluster of denial of service issues. Archive parsing sits in package managers, build systems, CI jobs, artifact handling, upload pipelines, and scanners.

If you accept archives from untrusted sources, tar parsing bugs matter. Fourth, ServiceNow had a requires authentication equals false issue. One unauthenticated REST endpoint in a platform that holds tickets. Assets, workflows, and incident context can become an enterprise data exposure problem fast. Fifth, France is pushing quantum risk into procurement language. That is how post -quantum migration becomes real.

Procurement, compliance, vendor questionnaires, and data retention expectation. And last, AWS wrote about Lambda -managed instances for Java cold starts. Serverless keeps getting features that look more like managed capacity under the hood. That is not bad, but the abstraction is not magic. Cold starts, warm capacity, runtime behavior, and cost still matter. The human closer this week comes from SRE Weekly.

Issue 525 had a few pieces around post-incident reviews, human-in-the-loop authority gaps, and whether teams confuse collecting artifacts with actually learning. That fits this episode, because this whole week is about maps and exposure. Datadog is talking about attackers mapping GitHub organizations. Grok Build raises the question of what AI coding tools read.

Security Hub is mapping what is reachable from the internet. Microsoft is talking about AI finding vulnerabilities faster. Maps are useful, but maps do not fix anything by themselves. An exposure finding does not close the port. An API log does not revoke the token. A patch bulletin does not patch the fleet. A postmortem timeline does not create learning.

A human approval step does not guarantee judgment if the human has no context. That last one matters. Human in the loop sounds safe. But if the human is pulled in at the worst moment with the least context under the most pressure, the approval step can become theater. Same with post-incident reviews. A meeting is not learning. A timeline is not learning.

A document in the wiki is not learning those things can help but the learning happens when the system changes when ownership becomes clear when the runbook improves when alert noise goes down when the risky default gets removed when the next operator is less dependent on luck so the question is not just did we review the incident the question is what changed because of it visibility is not ownership.

A map is not a fix. That's it for this week of Ship It Weekly. We covered GitHub API enumeration, Grok Build CLI, and AI coding tool data boundaries, AWS Security Hub network scanning, and impact analysis.

Microsoft's AI -powered vulnerability discovery and patch cadence, plus HalluSquatting, Cloudflare incidents, node-tar CVEs, ServiceNow authentication risk, Quantum Procurement Pressure, and Lambda Managed Instances. If this episode was useful, follow or subscribe wherever you are watching or listening.

If you know someone dealing with GitHub org security, AI coding tools, cloud exposure, or patch pressure, send them this one. Also, the Ship It Weekly shop is open at shop.tellerstech.com. Use coupon code SHIPTHESTORE for 20% off your order for the next few weeks. Again, SHIPTHESTORE at shop.tellerstech.com.

You can find the weekly brief at OnCallBrief.com and the full show notes, links, and past episodes at ShipItWeekly.fm. I'm Brian Teller from Teller's Tech. Thanks for listening. And remember, visibility is not ownership. A map... Is not a fix.

This week, Amazon EKS gets Kubernetes version rollbacks, which sounds like an undo button, but rollback is not the same thing as readiness. A new supply chain report called Cordyceps found exploitable GitHub Actions patterns across major open source projects, which is another reminder that CI/CD YAML is code with credentials.

Tenet Security showed how a fake Sentry error could hijack AI coding agents into running attacker-controlled code. So logs, errors, and telemetry are not just evidence anymore. They may also be instructions. And CloudWatch can now create alarms directly from log queries. Useful, yes, but also a very fast way to turn noisy logs into noisy pages. Put those together, and the theme this week is pretty clear.

The safety nets are getting better, but the blast radius is getting wider. I'm Brian Teller from Teller's Tech, and this is Ship It Weekly. Welcome back to Ship It Weekly, the show about the DevOps, SRE, cloud, platform, and security stories that actually matter when you are the person who has to keep the thing running at 3 a.m.. If you're new here, follow or subscribe wherever you are watching or listening.

And if you want the weekly story list and source links, check out OnCallBrief .com. For past episodes, full show notes, and more from the show, head over to ShipItWeekly .fm. This week, we start with EKS version rollbacks because Kubernetes upgrades finally get a safety net in a place where a lot of teams have wanted one. Then, Cordyceps and why GitHub Actions workflows need to be treated like production code.

After that, agentjacking, where fake telemetry can become a way to steer AI coding agents.

Then, CloudWatch log query alarms and the line between better signals and easier noise in the lightning round ECS Service Connect zone-aware routing etcd 3.7, GitHub innersource advisories, secret scanning metadata and CloudWatch application signals service events and we close with a simple reminder safety nets are good but they do not replace ownership let's get into it First up, Amazon EKS now supports Kubernetes version rollbacks.

And this is a big one. For a long time, Kubernetes control plane upgrades have felt like a one-way door. You do the checks, you read the release notes, you look at your add-ons, your nodes, your admission controllers, your autoscaling, your old helm charts, and whatever legacy thing nobody wants to touch. Then eventually, you upgrade. And hope the cluster is still boring. EKS is changing that with version rollback.

If you upgrade an EKS cluster and hit a problem, AWS now lets you roll the Kubernetes version back to the previous version within a seven-day window. That gives teams a real recovery path when an upgrade exposes a compatibility issue. That is useful, especially if you run a lot of clusters.

Have strict change windows, or operate in environments where Kubernetes upgrades are already surrounded by approvals and nervous Slack threads. But this does not make upgrades free. It does not make your workloads compatible. It does not fix every add-on problem. It does not replace staging. And it does not mean your whole cluster magically goes back in time like nothing happened.

AWS also added rollback readiness checks, which is important because rollback has its own requirements. Your nodes matter. Your add-ons matter. Your disruption budgets matter. And if you use EKS Auto Mode, node rollback becomes part of the story too. That means rollback can take time. And in an incident, people hear rollback and assume fast. But safe rollback is not always instant rollback.

The takeaway is pretty straightforward. Use this feature. Put it in your upgrade runbook. Test the path. Know what it does and does not cover. But do not treat the rollback button like permission to skip upgrade discipline. A rollback window is a safety net. It is not a test plan. EKS got an undo button. That is great. Just make sure that someone still owns the upgrade.

Second story, Novee Security published research called Cordyceps, and this one lands directly in the your CI/CD is production code bucket. The report looks at exploitable GitHub Actions workflow patterns across high impact open source repositories. Not one weird repo, not one random bad workflow. A repeated class of CI/CD mistakes.

Novee says that they scanned around 30,000 high-impact repositories and confirmed more than 300 fully exploitable attack chains. The patterns include command injection, artifact poisoning, broken trust boundaries between workflows, and untrusted pull request or comment data making its way into privileged automation. That last part matters because most teams still treat workflow YAML like config.

It gets copied from another repo, tweaked until the build is green. Reviewed quickly. Then forgotten. But workflow YAML can run commands, read secrets, publish packages, sign artifacts, assume cloud roles, push containers, cut releases. That is not just automation. That is a privileged execution path. The nasty part is that these problems are often not obvious from one line. A low-trust workflow produces an artifact.

A high-trust workflow consumes it later. A branch name lands in shell. A pull request title becomes command input. A cache crosses a privilege boundary. Each piece looks normal enough. Together, you accidentally gave untrusted input a path into trusted automation. And now, add AI coding agents to the picture.

If agents are generating CI/CD configuration, they can reproduce insecure workflow patterns very quickly across a lot of repos. The fix is not to stop using GitHub Actions.

The fix is to review workflow logic like code with production authority, threat model your triggers, separate trusted and untrusted workflows, be careful with pull_request_target, watch shell interpolation, limit token scopes, treat artifacts and caches as trust boundaries, and stop assuming YAML is harmless because it looks boring. Cordyceps is a good reminder that your pipeline is part of your supply chain.

And attackers already know that. Third story. Tenet Security published research on agentjacking. The short version: They showed how a fake Sentry error report could influence AI coding agents into running attacker-controlled code. That should make platform and SRE teams pause. Because this is not only about one agent or one vendor. The bigger issue is that operational context is changing.

Logs, errors, stack traces, issue descriptions, pull request comments, incident notes, runbooks. That stuff used to be mostly passive. A human read it, decided what mattered, ignored the nonsense, and took action. Now, agents are reading that same material, and sometimes they are acting on it. In Tenet's example, the attacker injects a crafted error event. The event looks like normal diagnostic context.

An AI coding agent is asked to investigate or fix the issue, and the agent treats malicious content like instructions. That is the trust boundary. Not did the attacker compromise the server, but can the attacker put text somewhere the agent will trust? That is a much lower bar and a pretty uncomfortable one. A lot of our systems were built assuming that logs and error text were just evidence, not instructions. Maybe we sanitize them for display. Maybe we redact secrets.

But we were not always asking whether a string in telemetry could become part of an agent's decision loop that has to change if an agent can read from observability tools open repos run commands install packages access cloud credentials or modify code then untrusted context becomes a security problem the control is not the agent is smart the control is boundaries can it read without executing Can it propose without running?

Can it separate trusted remediation guidance from user-controlled error content? Can you audit what it saw and what it did? And are there live credentials sitting nearby when it makes a bad decision? The takeaway is simple. Do not wire agents into developer and incident workflows and only ask about productivity. Ask about authority. What can it touch? What can it run? What does it trust?

And what happens when that context is hostile? Agentjacking is not just an AI story. It is a trust boundary story. The text around your systems can now cause action. So your trust model has to include the text. Fourth story, Amazon CloudWatch now supports creating alarms directly from log queries. This is one of those features that sounds small, but it changes how teams build alerts.

Before this, alerting on logs usually meant extra glue, metric filters, custom metrics, another system. Some pipeline where a log pattern turns into a metric and then the metric turns into an alarm. Now CloudWatch can create alarms from log queries directly. Those alarms can use normal CloudWatch alarm actions, including SNS and EventBridge. That is useful. A lot of incidents show up first in logs.

A specific exception. A repeated auth failure. A weird timeout. A dependency error. A line that says the thing is broken before the normal metric catches up. So yes, I like this feature. But I also fear it a little. Because easier alert creation can become easier pager noise. Someone sees an error. Someone says we should alert on that. Nobody checks how often it happens. Nobody asks whether it is user impacting.

Nobody asks who owns it. Nobody checks whether the log message changes every other deploy. And suddenly your on-call rotation is getting paged by a regular expression with abandonment issues. The right use case is targeted signal. Known failure mode. Clear owner. Clear threshold. Clear action. Clear reason why a human needs to be interrupted.

If the alarm fires and the on-call has no idea what to do, you probably made a search shortcut, not an alert. So this is a good CloudWatch feature. It removes glue. It shortens the path from log signal to operational response. Just be careful. The fastest way to create an alarm is not always the fastest way to create reliability. Sometimes it is just the fastest way to create noise. Quick lightning round.

First, ECS Service Connect now supports zone-aware routing. AWS says It is enabled by default for new and existing services. But existing services need a one-time redeployment to pick up the behavior. The idea is to prefer same-AZ traffic when possible. That can help with latency, cross-AZ cost, and failure domain behavior. Service-to -service networking is never just connectivity.

It is cost, latency, and resilience. All tangled together. Second, etcd 3.7 is out. Not the flashiest headline, but etcd is still the backing store under Kubernetes. This release includes RangeStream for streaming large result sets in chunks, plus performance work, cleanup of old v2 store remnants, Protobuf updates, and dependency updates.

For most managed Kubernetes users, This is not an immediate action item, but it is a good reminder that Kubernetes reliability still depends on the boring distributed database underneath it. Third, GitHub innersource security advisories are now generally available. This gives GitHub Advanced Security customers advisory workflows for internal packages and internal repos.

That matters because not every vulnerable dependency is public. Internal libraries, shared actions, platform modules, and private packages need disclosure and remediation workflows too. Fourth, GitHub secret scanning added extended metadata and multipart validation. For some secrets, the token itself is not enough to validate the risk. You may need a workspace URL, host, endpoint, project, or other related field.

That sounds small. But during incident response, it matters. Finding a secret is step one. Figuring out whether it is real, live, owned, expired, scoped, and safe to revoke is where the pain starts. Fifth, CloudWatch Application Signals now captures service events automatically. AWS says that it can capture exception snapshots, latency event snapshots, and deployment events.

For instrumented applications that is useful because during an incident teams do not only need to know the service is slow they need to know what changed more context near the signal is good as long as it does not turn into another wall of events that nobody owns the human closer this week is about safety nets because that is the shape of the episode. EKS gets rollback. CloudWatch gets easier log-based alarms.

ECS routing gets more zone-aware. GitHub gets better advisory and secret scanning workflows. Etcd gets reliability and performance improvements. Even the AI agent security story is really about figuring out where the new guardrails need to go. And I like all of that. I like rollback buttons. I like better alerts. I like scoped workflows.

I like dependency and secret cleanup that does not require five spreadsheets and a prayer. But safety nets can trick teams into thinking the hard part is solved. It usually is not. A rollback button does not tell you whether the rollback is safe. An alarm does not tell you whether the signal is worth waking someone up. A scanner finding a secret does not tell you how to rotate it without breaking production.

An AI agent finding a fix does not mean it should run commands on your laptop. And zone-aware routing does not mean you understand your failure domains. Tools give you better options. They do not own the system for you. That part is still on us. So take the safety nets. Use them. Be glad they exist. But do not let them replace ownership.

Because in production, the worst failures are usually not the ones where you had no tools. They are the ones where everyone assumed the tool meant the thinking was done. That's it for this week’s Ship It Weekly. We covered EKS version rollbacks, Cordyceps, and CI/CD supply chain risk, agentjacking through fake telemetry.

CloudWatch log query alarms, ECS zone-aware routing, etcd 3 .7, GitHub advisories, secret scanning, and CloudWatch Service Events. If this episode was useful, follow or subscribe wherever you are watching or listening. If you are on YouTube, hit subscribe. If you are in a podcast app, follow the show there.

And if you know someone dealing with Kubernetes upgrades, GitHub Actions security, AI Coding Agents, or Alert Noise, send them this one. It genuinely helps the show grow, and it helps me keep making this for the people who actually live with these systems. You can find the weekly brief at OnCallBrief.com and the full show notes, links, and past episodes at ShipItWeekly.fm. I'm Brian Teller from Teller's Tech.

Thanks for listening. And remember, safety nets are good. But they do not replace ownership.