Ship It Weekly Transcripts

This week is a reminder that the stuff we treat like glue is now a primary failure domain. CI trigger roles, cert renewals, helm charts, automation tools. One tiny assumption goes sideways and suddenly you're dealing with a supply chain risk, a global build break, or an RCE in the thing that holds all your credentials. Hey, I'm Brian. I work in DevOps and SRE and I run Tellers Tech.

This is Ship It Weekly, where I filter the noise and pull out what actually matters when you're the one running infrastructure and owning reliability. If something's hype, I'll call it hype. If it changes how you operate, we'll talk about it. Quick bit of housekeeping, the show notes and links are on shipitweekly .fm. If the show's been useful, follow it wherever you listen.

Also, a rating helps way more than it should. Four main stories for today. First, code breach. A CI trigger and filtering issue in a small set of AWS managed repos that's a perfect reminder that CI glue is part of your security boundary now. Second, the Bazel TLS cert expiry incident. The kind of failure that is boring, binary, and absolutely capable of blocking your entire engineering org. Third, Helm chart reliability.

Prequel reviewed over a hundred popular charts and the results are basically a post -mortem template for why it installed fine is not a reliability guarantee fourth n8n two new high severity flaws disclosed by jfrog that can lead to code execution we're going to treat this one like a mini story because workflow automation tools are basically a control plane holding your secrets then a quick lightning round with a few

Operator friendly tools and takeaways and a human closer tying the theme together alright let's get into it So code breach.

AWS published a security bulletin describing a misconfiguration involving unanchored account underscore ID webhook filters for code build. Used by a small set of AWS -managed open -source repos. AWS says they mitigated it quickly, rotated credentials, reviewed logs, and added additional mitigations and protections around build processes with credentials in memory. Wiz's research frames the risk clearly.

If you can trigger a privileged build in a repo that's part of a supply chain, you potentially get access to tokens and credentials that can be used to push changes or create malicious artifacts. That's why it's not just CI, it becomes supply chain. Now, the operator lesson is not code build is bad. The operator lesson is stop treating pipeline trigger logic like it's harmless.

If an untrusted event can cause a trusted pipeline to run, you do not have CI. You have an execution environment exposed to the internet. And almost every org drifts towards this risk without meaning to. Here's the drift pattern. You start with PR checks, lint, test, build, great. Then someone adds integration tests.

And those tests need credentials to hit an environment, or to pull from a private registry, or to call a third -party API. Then someone adds preview deployments. Those need cloud creds, or at least some deploy token. Then someone adds artifact publishing because it's easier if the PR build produces the image. Now your PR pipeline can build and push images. Then someone uses those images in staging.

And now your PR pipeline is part of your release path. And at that point, you've accidentally created a supply chain path where a PR can influence something that runs in your environment. This is why the most dangerous sentence in CI is, it only runs in CI. Because CI is usually the thing that holds credentials that can touch everything else. So the practical question to ask is simple.

Can untrusted events cause trusted actions? Can a forked PR run a job that has secrets available? Can a PR comment trigger a workflow that can deploy? Can a PR workflow push artifacts that are later deployed? Can a PR workflow assume cloud roles? If the answer is yes, or even I'm not sure, you've got a boundary problem. Now, I want to give you a do this Monday playbook that doesn't require a big platform rewrite.

Step one, label pipelines mentally as untrusted and trusted. Untrusted pipelines is PRs and forks. No secrets, no publish, no deploy. It answers, does it compile? Do tests pass? Trusted pipelines run on merges to main, tags, or explicit manual approvals. That's the one allowed to publish and deploy. Step two. Pick one repo, your most sensitive one, and map the event chain in plain English.

What events trigger workflows? Which workflows can access secrets? Which workflows can write artifacts? Which workflows can deploy or change infra? Step 3. Check for the classic foot guns. Are secrets injected into your PR workflows at all? Are you using conditions like actor checks or branch checks as auth? Are you relying on the code won't print secrets as a control?

Do you allow PR builds from forks to run with privilege tokens? If you find these, the fix is not perfect security. The fix is strong separation. PR builds can still run. Just keep them in the untrusted sandbox. If you need integration tests, run them in a separate environment with a separate low privilege credential set. If you need to build images, build them but don't push them to a production registry.

Or push to an isolated registry that cannot be used for deploy. If you need a preview environment, require approval before anything privileged runs. And the last point, even if you fix the triggers, scope matters. Least privilege for CI tokens is not optional. Your pipeline credentials should not be able to do everything. That's story one. Story two is Bazel.

On December 26th, 2025, the TLS certificates for many star .bazel .build domains expired and it caused widespread build breakage. The Bazel team's postmortem says the outage lasted around 13 hours before it was resolved. This is one of those incidents that's both boring and terrifying. Boring because it's just a cert. Terrifying because cert failures are a binary cliff. Everything works, then it doesn't.

And the blast radius is immediate because every client that depends on that endpoint fails at the same time. Also, auto -renew does not prevent this class of incident. Auto -renew is one link in a chain. The full chain is issuance, renewal, deployment, Reload and verification. A lot of real cert outages are renewal succeeded but deployment didn't reload. Or new hostname wasn't included.

Or monitoring checked the wrong endpoint. Or DNS validation broke and nobody noticed. So here's the practical operator version of this story. You'd need external monitoring for cert expiry from the outside. Against the actual endpoint users hit. Not an internal health check. Not a dashboard in the cert system. The real endpoint. You need ownership. A named owner. A team. A channel.

Someone that at 2am on call can tag and get traction. And you need runway. Alerts well before expiry. If your alert fires 24 hours before expiry, you are still basically doing incident response. If it fires 30 days before expiry, you can fix weird edge cases like DNS changes, migrations, or validation issues calmly. Now, the do this Monday pass. Pick your top three engineering org blockers.

These are not always customer facing. Often they're internal systems that block shipping. Artifact Registry, Git Host, CI Endpoint, SSO Login, Webhook Receiver, Package Download Host, any of those. For each one, answer. Do we have external monitoring for cert expiry and chain validity? Does it alert at least 14 days out, ideally 30? Is there an owner written down? Do we know where to fix it if renewal breaks?

If the answer is no, that's a cheap reliability win you can fix without rewriting anything. And if you want a meta lesson, certs are one of those dependencies where the failure is so preventable that it's painful. Which is why teams tend to overcut corners until they get burned. Don't wait for the burn.

Story 3 helm chart reliability prequel reviewed 105 popular open source helm charts and they found the average reliability score was roughly 3 .98 out of 10 with a median around 4 out of 10. Their point isn't helm is bad it's that many charts ship demo friendly defaults not reliability -friendly defaults. This matters because charts aren't just packaging.

They encode operational behavior, readiness and liveness, resource requests, update strategies, disruption behavior, security contexts, sometimes topology and scheduling assumptions. So when you install a chart, you are adopting a set of operational opinions whether you realize it or not. Here's how this bites teams. A chart has no resource requests. In dev, it looks fine.

In prod, under pressure, it becomes unpredictable. Pods are throttled, get evicted, or get starved. Probes are missing or sloppy. Traffic gets routed to pods that aren't ready. Or probes are too aggressive and under load, they trigger restarts. No pod disruption budget, no topology spread. And then routing node maintenance becomes a cascading outage.

Everything was highly available until you drained a node and lost a majority of replicas in one place. Unsafe update strategy and rollouts turn into brownouts. And the worst version is when Kubernetes says everything is green while your app is melting. That's where chart defaults turn into long incident timelines. So what do you do without forking every chart? You create a baseline overlay and a checklist.

Baseline overlay is a thin layer in your GitOps repo or Terraform or Helm release config where you enforce defaults. Resources required. Probes required. Explicit update strategy. PDB when appropriate. Spread constraints if the service needs real availability. Security context that matches your cluster policy.

And the checklist is just does this chart behave predictably under rollout, under node drain, under load spike. Now, the do this Monday pass. Pick one chart you run in production that matters. The one that would page if it went sideways. Open its values and answer. Are resource requests set? Are probes set and meaningful? Do we have safe rollout behavior? Do we have disruption behavior planned?

Will replicas spread across failure domains, or can they all land on one node? What happens if a node gets drained? If you can't answer quickly, that's your signal. Add the defaults, document them, and move on. You don't need perfect charts. You need boring, predictable behavior. Now, N8n. And this is new. Hacker News covered two high severity vulnerabilities in N8n discovered by JFrog.

The short version is these flaws can let authenticated users escape the sandbox and execute code. One is in the expression sandbox and the other involves Python code execution in internal mode. Here's the important part. People hear authenticated and they relax. In workflow automation platforms, the permission model is often broader than you think.

Authenticated can include a lot of people who can build or edit workflows. And in tools like N8N, workflow editing is basically code execution. Because workflows can evaluate expressions and interact with credentials. So this isn't, oh no, an attacker needs an account. The real question is, who in your org has an account? And what can their workflows touch?

And what makes this class of bug extra painful is these tools often sit in the middle of your environment holding keys. Slack, GitHub, Jira, AWS keys, database credentials, webhooks, secret managers, all of it. So a sandbox escape is not just someone ran code. It's someone ran code where the keys live. That's why we keep coming back to N8N on this show. It's not because N8N is uniquely bad.

It's because the category is high leverage. Okay, practical actions. First, patch. Don't debate it. If you self -host N8n, patch quickly when sandbox escapes drop. The blast radius is too high to slow walk it. Second, reduce who can author workflows. Don't treat workflow editor as a casual permission. Treat it like can run code in a privileged environment because effectively that's what it is. Third, reduce exposure.

If your N8n UI is public on the internet, you are playing on hard mode. Put it behind SSO, VPN, IP allow lists, whatever fits your org. You want fewer people able to even reach the attack surface. Fourth, isolate it. If it's holding keys to a bunch of systems, at least give it a narrow runtime permission set. Least privilege on the credentials it uses. Separate credentials per workflow if you can.

Don't run it with god mode access to AWS just because it was convenient once. And now the tie back to our last episodes where we talked about N8N CVEs. The theme has been consistent. Workflow automation tools are basically control planes. They need the same operational rigor you'd give to an internal platform. Patch fast, lock down authorship, reduce exposure, least privilege the credentials.

If you treat it like just a tool, it will eventually treat you like just a breach. All right, time for the lightning round. First. Use Tusk Fence. It's a lightweight sandbox for running commands that block network by default. If you are experimenting with agents, runbooks that execute, or any workflow where code runs on behalf of a user, this is the kind of primitive you want. Safe by default beats clever by default.

Next, HashiCorp agent skills. This is part of a trend I actually like. Vendors shipping structured reusable skills and guardrails instead of just telling you to prompt better and hope. Next, Marimo. It's a reactive Python notebook that stores as normal Python.

That sounds small, but it matters because Git -friendly notebooks are actually useful for incident analysis, runbooks, or one -off ops experiments you want to keep without the notebook JSON misery. And a quick one from the register, the Ralph Wiggum clawed loop story. It's funny, but the real point is dead serious. People are already building loops that keep agents running until they produce output.

Without constraints and verification, that becomes confident nonsense at scale. The ops lesson is the same as alert fatigue. If your system floods you with low -quality output, humans stop trusting it. Okay, time for the human closer. Every story today is a glue failure. CI trigger logic becomes security boundary and someone implemented it like it was just config.

Cert renewals were treated like solved and then the cliff happened. Charts were treated like installers, not operational dependencies. And workflow automation tools are treated like a convenience layer even though they hold the keys. So the takeaway isn't stop using tools. It's treat guardrails like product work. Make untrusted pipelines truly untrusted. Make cert monitoring external and owned.

Make helm baselines explicit. Make workflow authoring privileged. Make credentials least privileged. Because if you only build accelerators, you are not building a better platform. You are just building a faster incident. All right, time for a quick recap. We talked about code breach and why CI triggers and filters are real security boundaries. We talked about Bazel and the CertCliff problem.

And we talked about Helm chart reliability and why defaults matter more than installs. And we talked about the new N8n sandbox escape flaws and why workflow automation needs to be treated like a control plane. The lightning round was Fence, HashiCorp Agent Skills, Marimo, and The Agent Loop Cautionary Tale. Links and show notes are on shipitweekly .fm.

If you got something out of this, follow the show wherever you are listening. And if you can, leave a quick rating or review. It helps a ton. I'm Brian, and I'll see you next week.

Bye. Bye. Hey, I'm Brian Teller. I work in DevOps and SRE, and I run Teller's Tech. Ship It Weekly is where I filter the noise and focus on what actually matters when you're the one running infrastructure and owning reliability. Most weeks, it's a quick news recap. In between those, I drop interview episodes with folks who are actually building in the space. Today is one of those interviews.

I'm joined by Austin Reed from Horizon .dev. He helps small and mid -sized businesses save serious time with AI and automation. And this isn't theoretical. We talk about what actually works when you're automating real workflows with real revenue behind them. Not just demos.

We get into the biggest misconceptions, why projects fail more from communication than code, where human in the loop makes sense, and what tools his team is using behind the scenes. From GPT and Claude to Cursor, CICD, and code review bots. Today, I'm joined by Austin Reed. Austin Reed, your automation expert with Horizon .dev. He helps SMBs save serious time with AI and automation.

And we're going to talk about what actually works in the real world, not just demos. Austin, nice to meet you. Thanks for coming on. Thank you, Brian. I appreciate it. So I'm curious, what do you do? Day -to -day in your job? Man, a little bit of everything. I do a lot less programming nowadays than I did when we started. When I started, it was just me and my ex -business partner, Mattel.

We were just keyboard warriors, always programming, me and him. And I would do sales and accounting, whatever. But nowadays, I mostly manage the team, lead the team. I hire people. I help the team with project structure and architecture. I help debug. More advanced problems. Sometimes I jump into code, but it's not that often. But I'd say the majority of my time is with client relations and on sales calls.

What is the real unlock or benefit for AI and automation for small to medium businesses right now, in your opinion? Well, I guess it would depend on what area of the business you're talking about unlocking, right? Every business is unique with its own problems. Some businesses, you know, they would really benefit by having some sort of sales automation, sales AI enablement, right?

Helping them generate more leads or helping them to where they don't drop leads or helping them with statistics on that side of things. Other businesses, they have that unlock and they would benefit more from fulfillment side of things, making sure everything is tight, everything's efficient, working properly. There's not any errors going on. AI can be used. Across, I'd say, big portions of a lot of businesses.

Most businesses, I mean, they essentially function the same. They take in some sort of customer, they service the customer in some way, or they give them some sort of product, right? And then there's the, after you turn in the project, like transition period, right? Where you get like referrals on the back end and stuff. And so adding AI in any of those core areas, I think really, really helps. 100%.

Yeah, that makes sense. So what would you say is like the biggest misconception that companies have when they say, we want automation, but they don't necessarily know what that means, right? They want automation. They want to move quicker, maybe, but they don't necessarily know what automation is or what that actually entails. Yeah, I mean, I think there's two.

One, on one side of things, you have people who think that AI is going to cause more problems than it's going to solve, or they don't trust the quality of AI. They don't have...

A firm belief in the powers of what it can do and on the other side of things they're definitely too gung -ho to get things done I I find the biggest issue is usually around some sort of content generation so they want to automate their instagram they want to automate their video creation things like that and so they think that ai can just do everything and another thing is they also think that because there's ai and everything's easy we have things like cursor and stuff now that It's really easy to build these things.

And so like they think it's going to cost like a couple hundred bucks or a thousand bucks to get a system that's robust enough to like manage the business. It's like, hey, you know, your business is working like. $100 ,000, $200 ,000 worth of sales a month. And you honestly think that like a $300 product is going to be able to sustain that. That doesn't really make a whole lot of sense, right?

So you have a lot of these problems. It's just underestimating and overestimating their capabilities, depending on what side you're on. And then definitely overestimating how cheap it is. Yeah, no, that's fair. Okay, so I know, like we talked about this a little bit, depending on the company, there's a lot of different... Automation wins.

But what, based on your experience for companies, what is the most popular or common win for automation? Okay. Yeah. So it's going to be one of two things. Usually it's either on the customer service side of things, or it's going to be on the sales side of things. So specifically for B2B businesses.

It'd probably be more on the sales side of things like I said automating lead flow increasing the amount of leads you're reaching out to enabling your bdrs sdrs aes to be able to do their jobs even better that drives a massive amount of impact in sales organizations whereas just two years ago like sdrs bdrs they were sending 50 100 emails a day they were handwriting the emails right now Then they moved on to doing sequencers.

And so then all of a sudden they were sending thousands of emails, but they were all templated with spin tax, whatever. And now it's all AI generated. And I know businesses that are sending millions of emails a month, which is absolutely insane, right? On the other side of things, you have the customer service. And it really kind of depends, but like you could imagine your mom and pop dental shop, like.

They're not always there to answer the phone. It would be nice to be able to call on like Saturday at six when people are busy or maybe people aren't working and be able to schedule an appointment without having somebody there needing to take the call, right? And so that also translates to a chatbot or an email responder. It's essentially the same thing. It's just the medium of which you communicate changes.

But the AI is still doing the same thing. It's taking schedules, doing follow -ups, taking information, you know, filling out forms and stuff so that the customer service side of things is on point. Is there any part of a business right now that you specifically try to not automate or you stay away from automating? Yeah, I'd say one, finances. And I don't think it's because it's bad to automate finances.

I think it's just because I'm not the best finance person. So because I don't understand it at a very intimate level, I tend to stay away from it. And I don't mean like automating like your accounts receivables and whatever. No, I mean like actually automating like statistics and things like that with accounting. Like I don't, I'm not an expert at that, right? The other thing I stay away from.

Firstly, is anything medical or HIPAA related? I don't want to touch, right? Because yeah, same thing with government. I don't want to touch it. But in terms of business, I'd say mostly just finances and the things that really require a personal touch or somebody overlooking it. So we'll do something called men in the loop, which is like, maybe we'll do an automation, but then a human has to approve it.

So the human edits it, approves it, says it's good, right? So for very bespoke services, you know, it's always good to have that little bit of human touch, right? So you can have some automations, but the automations, they don't do the job. They enable the person to do the job better. So it sounds like you're using AI as a tool. It's not replacing someone too. I mean, that's kind of an important point.

Like if you're an expert in something, you can use it to help further along. Development or use as a tool, but you're not replacing the individual. It's not because you don't understand finance. It could potentially hallucinate, right? So you don't want to just recklessly use a tool without knowing what the output could be. Right, exactly.

And so I guess the reason why I'm really good at automating sales and customer service and different fulfillment things is because I intimately understand a lot of those problems because, you know, I have those problems within my own business. I'm not saying I don't have finance problems in my business. I totally do. But I'm like not looking at that as well. I'm more focused on the day -to -day operations, right?

So yeah, no, I totally agree. Not replacing the individual is key. I mean, even like with my assistant, like every once in a while, I'll be like, hey, what do you spend most of your time on every single day? And she'll tell me, I'll be like, hey, well, let's take this off your plate, but let's add this new thing. You know, now that you have more time, now that you have more leverage, let's make the...

Return on time even greater for what you're working on. Now that makes sense. Is there anything that you've gone to automate with AI that's backfired that you're willing to talk about? I mean, I'm going to say not really in the sense of it backfired because it wasn't a good automation. And the reason why is because before we dive deep in automation, we had already been a programming agency for a while.

So we already knew the capabilities of what we we'd already been through the crap right like we've already made a lot of mistakes and learn from them and whatever but I would say the ones that have failed most of the time they don't fail because of the job they fail because of communication or improper vetting of the client or just clashing personalities is it ever where they don't maybe give the right spec on what

They actually need or they're not even sure of what the output is that they need I mean that happens a lot we've gotten better at identifying that problem so by first of all don't automate something that hasn't happened yet you know When we first started out with automations, there was a lot of people that were like, oh, we're going to have a lot of demand in February, so we need to build this system so that when the demand comes, we'll be ready to go.

And then February comes and there's no demand. It's like, okay, so you just spent $10 ,000 on a system. That doesn't help you. It doesn't make sense, right? And so I always try to make sure that there's actually a need there. And I think also the way we build kind of weeds some of that out. So we always like a lot of business owners.

Common they're like oh we want this and this and this and this and that it's like a huge menu order of like a restaurant menu of what they want right and that's cool but we're like okay that's cool but how about we what's the most impactful quick win we can do for you right and so we'll start with that that one impactful thing and then we give it to them as quickly as possible so we work a lot with cicd framework I

Don't know if you know what that is but basically we'll develop a small prototype as quickly as we can as best as we can and we'll give it to them as quickly as they as we can in order to get feedback on on how that is because maybe like you know we build an automation but then it doesn't quite work the way that they wanted or maybe the owner has this great idea but then when the team members get their hands on it

Like their reality is a little bit different because the owner isn't doesn't have his head in the trenches, so he doesn't know exactly what their day -to -day looks like.

So we always try to get... MVPs and prototypes in the business owner's hands as quickly as possible so that they're able to play around with it and test. And then as we stack on more features, you know, we get feedback like, hey, we like this. We don't like that. We'd like to change that. And it would be cool if we could add this. We're like, OK.

And so we can start adding those little tweaks early on instead of having a project where it's like, OK, cool, we'll get it done. And then two months later, we turn something in. That's not what they wanted. Right. What's the tech that you're generally using behind the scenes to build these automations? You're talking about like the code itself or like the tools we give our developers? The models, the code, APIs.

Everything? AWS, yeah. Is it more like AWS? Are you using like Gemini models? Are you using Claude models? Yeah. Yeah. We use a lot of GPT and Cloud. So I find Cloud is a lot better when it comes to sales and anything writing related, you know, like blog posts or customer service. Like Cloud is just sounds more human for some reason. I don't know why. GPT is a lot better at analytical things.

GPT Codex 5 .2 that just came out is really good. We've been using that with Cursor to build. Apps. So that's been helping out our development team a lot. We've also been experimenting with having AI review pull requests that programmers are implementing into code bases to see if it's good or not. And it's been working quite well. That's been quite interesting.

We've been thinking about implementing even a bot that whenever somebody adds an issue to GitHub, it'll maybe do a code to see what it looks like. And then the developer just reviews the code and edits it. We know that AWS has a a service that does that. We haven't fully implemented that part yet, but we're definitely experimenting with it now. So, I mean, there's definitely a lot of changes in the industry.

And then in terms of like servers, we don't use AWS or Google as much unless the client already has it. We actually prefer to use Volter, which is digital, same thing, right? I mean, same server stack, same technology, just a different name. And a lot cheaper, I would say. We use a lot of Kubernetes, a lot of Docker, Docker Swarms, Django, Node .js, NNN, yeah.

Has there been any technical challenges that you've run into with building out these automations? One thing is getting credentials from the customer. I know that sounds simple. Yeah, I've run into the same issue. Yeah, it sounds like a simple thing. It's more of a customer service problem than it is like a programming problem. But I mean, a lot of these people, they don't even know what their passwords are.

So it's like trying to get, or like you made a custom app and you need to get into their Google console to add whatever. And it's like, that's... That's difficult to do with some of these people who aren't as tech savvy, right? Anything else that I find makes it particularly difficult? Scope creep is totally a thing. Definitely have to push back on that. People get overly zealous about things.

That's why we always try to, we record all our meetings. We always try to put project plans right. So this is what you agree that we build, you know, so make sure everything's very transparent. But even with that, scope creep comes in sometimes. And then other than that.

I mean, at first we didn't, there were some clients where we had trouble fully understanding what their vision was, but I think that was more an experience than it was actual, actually a problem because now we don't see that as much because we have a lot more things in place, like making sure like there's already something to automate, giving them the prototype first so that we can, right.

So we flushed out a lot of those issues. How do you handle? The this is scary or I don't trust AI reactions pushback that you may get. Well, I mean, if they came in from a cold email, I'll be like, well, who do you think wrote the email? No, that's a hard objection. And for a lot of people, that's pretty hard to know because they just have some sort of mental block there.

And, you know, I'm not here to convince people. If they don't want the automation or if it doesn't serve them, I mean, it will serve them.

If they truly don't want it like I'm not gonna sit there and try to convince somebody otherwise like I'd much rather go to the guy who's like super pumped and excited about getting some new things done in the industry that hasn't been done before you know like that's much more exciting but I guess where we see some of that pushback is like in a business where it's like an old father and he's passing it on to the son

And so the son wants to implement these new things but the father's like no like the old way works why would we change it You know, we see some of that and they're usually a quick win.

Something super small and kind of insignificant is the best way because you can talk to them all you want. But when you actually show them something, it really works. So even just showing them a case study, but like really running through and be like, hey, you know, this. Similar business, different industry and problem or whatever. Like, hey, look what we did there. This is what it looks like for them.

We can do this for you. And they're like, oh, really? Is that possible? I'm like, how about we start small? We'll do something really small and quick, you know, quick win. It won't cost you very much. It won't take a lot of time. And you can just get your feet wet and see if you like it. Where do you think AI is going? If you had your crystal ball, where do you think like three, five years?

Obviously, it's a changing market. It's crazy. The last couple of years have been crazy. Yeah. If you were to hypothesize that, where do you think it's going? Well, AI influencers is already a thing, even though people don't fully believe it yet. I think that software agencies are going to, a lot of them are going to go out of business.

Because business owners are going to be able to make software themselves so you're going to see a big flood of apps and new stuff coming out because there's going to be a bunch of normal people launching their own apps and so maybe the new game is more sass related and less programming related I still think there will be programmers but I feel like it'll only be like very advanced niche stuff or old stuff like like

Like cobalt right for banks or something but so yeah I think At least my industry is going to, like, we need to pivot hard because, like, engineers now are not, I mean, they know code now, but maybe in five years they'll still know code.

But in 10 years, they might not know how to program, you know, and that's kind of weird. Outside of that, man, I think it's very good and very bad in a lot of ways. And it's really hard to say. Yeah, that's fair. Yeah, just like the car did, the horses that will probably open up new jobs and, you know, maybe business use cases that.

We're not even sure of yet you know but yeah it's definitely interesting is there any hype takes about ai that that you disagree with or you want to kill I don't think ai is quite to the point yet where you can fully automate social media I mean you can for like those info talking head videos not talking head but like the ones where it's like they say a story like you know oh this person went missing in 1994 because

Whatever like yeah those can be automated but I mean like truly valuable content I don't think can be automated to a high enough quality extent yet right I think that's coming I think it's close but I still think that there needs to be that human element the other thing is is because we're posting so much ai content now the ai is just rewriting ai content yeah so it's just learning from itself at this point So as time goes on, I believe the information gets more and more diluted.

And that's a problem in and of itself. I think that kind of covers it, at least my view. I know just being on LinkedIn even, but more so Instagram, Facebook, TikTok, it's all AI -generated Sora content now. A lot of it, yeah. It's hard. It's hard to just have real, genuine content that's not AI -derivative. Yeah. To be real. It cuts through the noise pretty well. So we talked about this a little bit earlier.

How do you think AI is going to influence development in the future? I mean, now we're using Cursor, right? We're using MCP servers to get information on Kubernetes clusters or whatever. But we don't have new junior engineers that are getting hired at the same rate they were just a couple of years ago. What happens to engineers and developers, you know, in the next five, 10 years?

They become prompt warriors or they switch industries, right? I mean, even our guys, like I handed them Cursor like a couple months ago. And before that, it was like Microsoft Copilot, right? Like two years ago, something like that. And even Microsoft Copilot. Like, dude, the difference in productivity between my guys was insane when I gave them Copilot. It was like, it's not like they don't know how to write code.

It's just that they can write it so much faster. They can be like. I want this, this, and that. And there's a huge difference. It's like I see a lot of business owners who don't know programming, who try to program with Cursor, and then they jump on Cursor, and it doesn't work. Cursor sucks, blah, blah, blah, and it's horrible. And I'm like, you don't know how to talk to it. It's really what it is.

I mean, there are some things like, yeah, it probably isn't the best app, but understanding how to architect. How to describe how it works with data, what order to build things in. That's huge, right? Because, you know, you get a certain database model set up, certain APIs set up, and all of a sudden you add something crazy in the mix and you have to redo everything, right?

And so it's like a lot of people just jumping in like they don't know that. So to answer your question, man, I mean, like I said, we're implementing AI more and more. All of our dev workflows, we're implementing it in code review and debugging and our CICV type of stuff to make sure that nothing broken gets pushed. We're implementing it and helping them even.

On issues you know they get an issue they talk to cursor cursor spits out some code they review some of the code they write some code themselves they tell it to write a certain function a certain way they sip their coffee they wait they test it you know and for testing too it's huge for like unit testing and integration testing like you write out a whole api in django and then you're like hey unit test this you know

Integration test this like that's huge the speed of which it can write out some of that code that, you know, before it would take, you know, weeks is insane.

Yeah, I agree. I know for me, being able to vet the output that Cursor gives you and not just trusting it, like you mentioned business leaders that may go use Cursor for the first time, but they don't really understand what they're, even understanding what to ask, but then understanding what the output is that they're given.

Like you need to have some awareness around that, at least currently, maybe in the future you won't to the same degree, but right now you, you know. It still can hallucinate. It could still mess up. So you need to know how to frame the question in the right way so it gives the output that you want, but then also vet the output that it's giving you.

And if you don't know anything about the subject that you're asking it to help with, you don't know if it's a good output or not necessarily. Right, exactly. Any closing thoughts? Not really. I think the world's going to change a lot really quickly. And we're along for the ride. Everybody needs to pivot. Every business, every industry needs to pivot, not just ours. Right.

So the faster you can kind of get on board and start testing new stuff, I think the better off you're probably going to be. Yeah. Be ready for it because it's coming regardless. Like you'd even mentioned, you know, the father and the son, the father may not want AI. He probably is talking to an AI chatbot on a website every day. He's probably getting phone calls from AI chatbots. He just doesn't know it.

He's already interacting. He's already in that world. He just doesn't know. Yeah. Yeah. All right. Well, appreciate it. Thank you, Austin, for coming on. Really appreciate it. I appreciate it, Brian. Definitely. All right. That's the conversation with Austin Reed from Horizon .dev. My biggest takeaway is his framing that automation isn't magic.

The win is picking a real pain point that already exists and getting a quick win live and then iterating. Also, the AI is cheap now mindset is a trap. If your business is doing real volume, you need systems that can actually hold up with guardrails, approvals where it matters, and someone who can sanity check the output.

If you want to connect with Austin or check out what they're building, I've got links in the show notes to his LinkedIn and to horizon .dev. If this episode was useful, share it with a DevOps, ops, or business friend who's trying to automate workflows without creating a bigger mess. Hit follow wherever you listen so you don't miss the weekly news recaps plus these guest interviews.

We'll be back with a regular Ship It Weekly News episode later this week. See you then. Thanks.

We keep trying to automate our way out of pain. Faster deploys, faster builds, faster ops. But the funny part is, the faster the system gets, the more expensive the mistakes get too. This week is basically three versions of that lesson. Hey, I'm Brian Teller. I work in DevOps and SRE, and I run Teller's Tech.

This is Ship It Weekly, where I filter the noise and pull out what actually matters when you're the one running infrastructure and owning reliability. If something's hype, I'll call it hype. If it changes how you operate, we'll talk about it. Quick bit of housekeeping. The site is shipitweekly .fm. That's where the show notes and links live.

And if this show has been useful, hit follow or subscribe wherever you are listening. If you've got 10 seconds, a quick rating or review also really helps way more than it should. All right, three main stories for today. First, Curl basically said we're shutting down our bug bounty program because we're getting flooded with AI garbage. That sounds petty until you realize what it's actually about. Signal to noise.

Burnout and how incentives can accidentally wreck your security pipeline. Second, AWS just made RDS blue -green deployments faster, with writer switchover downtime typically five seconds or less in single region setups. If you run production databases, this is a legitimately big lever, but it also comes with the usual, yeah, but what about caveats? Third, ECR finally supports cross -repository layer sharing.

Which sounds like a tiny registry feature until you're the person paying for storage and watching CI push the same base layers across 200 microservice repos like it's a hobby. Then the lightning round. A really good Kubernetes post on building kubectl -like CLIs without re -implementing the whole universe. A Helm charts write -up worth bookmarking, plus a couple of tools that are worth looking at.

And we'll close with the human side, the irony that automation doesn't remove responsibility, it moves it. And it usually moves it into the most overloaded humans at the worst possible time. So, Curl. If you haven't seen this one, the Curl project posted changes to stop their bug bounty program at the end of January 2026.

And the reason is basically the volume of low -quality AI -generated Vauln reports has become a tax they can't afford anymore. And I want to be careful here because the headline can turn into dumb discourse like AI bad or security people are cranky. That's not the point though. The point is incentives. Bug bounties are already a weird incentive machine.

They work best when the incoming reports are rare -ish, thoughtful, reproducible, and written by someone who actually validated something. But if you can create an environment where someone can crank out 50 convincing -looking reports in an afternoon, and they only need one to hit to maybe get paid, you've just created an attacker. Except the target is your maintainer's attention span.

This hits platform people in a few ways. One, it's the same pattern as alert fatigue. The first 10 useless alerts are annoying. The 500th useless alert changes how you behave. You stop trusting the channel. You start ignoring it. And then eventually, the real one shows up and everybody gets hurt. Two, it's a reminder that intake is part of your system design.

We talk about security tooling like it's just scanners and policies. But the workflow is the product. If your workflow can be spammed, it will be spammed. Three, this is also a very real maintainer sustainability problem. Curl is foundational internet plumbing. If the people maintaining that plumbing have to spend their evenings debunking hallucinated CVEs, it's not free.

It's a direct drain on the project's ability to ship real fixes. So what do you do with this besides nod sadly? If you run internal security intake, or even just your normal ops intake, steal the lesson. Add friction where spam happens.

That can be as simple as requiring real repro steps, requiring environment details, requiring a minimal proof, or forcing the submitter to do one more round trip before you spend time. Not because you hate people, but because you're protecting the team's attention. And honestly, I think we're going to see more of this. More projects putting guardrails around community intake.

Not because they don't want reports, but because they want fewer, better reports. Alright, let's switch from human bandwidth is the bottleneck to database downtime is the bottleneck. AWS posted an update that RDS blue -green deployments can now reduce writer upgrade downtime to typically five seconds or lower for single region configs.

If you're not deep in this world, blue -green is basically you stand up a green environment that's kept in sync, you test your changes there, and then you switch traffic so green becomes the new production. The reason this matters is the painful part of database upgrades is not you. Usually the upgrade itself. It's the how long is my primary not writable window.

For a lot of teams, that window is where you end up negotiating with product, doing off hours work, or just punting upgrades longer than you should. So AWS tightening that switchover window is real value. But here's the part I'd say out loud on an on -call team. Don't confuse fast switchover with safe upgrade. You still need to think about the stuff that bites you in real life.

If your application can't tolerate a brief blip, you still need retry behavior that doesn't melt your connection pools. If you've got long running transactions, you need to understand what the switchover does to them. If you are doing anything spicy with replication slots, parameter groups, extensions, or oddball maintenance jobs, you still need to test the green environment like it's going to betray you.

Because it might. Also, and this is the platform angle, reduced downtime is not the same as no coordination. A lot of orgs treat DB upgrades like a once -a -quarter fire drill because the process is fragile. If blue -green gets you to a world where upgrades are boring, that's the win. Boring is the goal. So my takeaway is if you are on RDS MySQL, Postgres, or MariaDB, this is worth revisiting.

Even if you tried blue -green before and shrugged, the improved switchover timing might be enough to change how often you are willing to use it. Now, let's talk about a feature that is way less glamorous than database downtime, but might save your CI bill and your sanity. AWS ECR now supports cross -repository layer sharing.

Meaning repositories can share common image layers when blob mounting is enabled at the registry level. This is one of those features where the AWS headline is polite. But the real world version is stop storing the same base image layers 300 times because you split the world into 300 repos.

If you've got a lot of services that all build off of the same base layers, like the same distro base, the same language runtime, the same internal golden image. ECR can now deduplicate that across repos. So what changes? Push performance can improve because you are not re -uploading layers ECR already has.

Storage can be less stupid because the registry can reuse layers instead of treating every repo like a separate universe. And in CI, especially if you do a lot of rebuilds, this can reduce the pain of why is this build pushing 800 megabytes again when nothing changed? Now the platform and governance angle. This is also a boundary question. A lot of people use separate repos as a mental model for isolation.

Not security isolation exactly, but organizational isolation. That team owns that image. Layer sharing doesn't mean the images are merged, but it does mean the backend storage model is more global. If you've got strict compliance requirements, you might want to understand how that interacts with retention, scanning, and who can access what. But overall, this is a very rare thing in Cloudland.

A simple feature that makes life better with basically no downside for most teams. If you run a big ECR footprint, this is worth enabling and measuring. You'll feel it fastest in why are our pushes so slow? And why is our registry bill weird? All right. Lightning round. First up, the Kubernetes blog post. Uniform API server access using client command. This is for anyone who has ever written a KubeCTL plugin.

Or wanted to build an internal cli that feels like cube ctl and then realized cube ctl has a million flags and conventions the post is basically a guide for how to use the same client cmd pattern so your tool behaves like users expect next building robust helm charts I like this one because it's not here's helm basics it's more the stuff you learn after you've maintained charts that that break in production, values

Structure, templates that don't explode, and patterns that keep your chart from turning into a pile of conditionals nobody wants to touch.

Next, a tool called TTL. This is basically traceroute grew up. With a TUI and a bunch of diagnostics, you usually end up doing manually. MTU discovery, NAT detection, route flap alerts, that kind of stuff. If you do network debugging and you're tired of bouncing between MTR, traceroute, and vibes, it's worth a look. And last quick one. Docker Canvas. InfoQ did a write -up on it.

And the short version is Docker is trying to bridge Docker Compose workflows into cloud deployment workflows. And visually, it's aiming at that make deployments less mysterious audience. I don't think it replaces Helm or Customize Tomorrow. But it is worth watching because it's another signal that developer -friendly workflows are creeping closer to production deployment patterns. Thank you. Alright, human closer.

For the human closer this week, SRE Weekly linked Honeycomb's interim report for their extended EU incident. And the reason I want to mention it is not because the vendor had an outage. It's the operational reality of multi -day incidents. And specifically, the part that a lot of orgs pretend doesn't exist. Managing human energy. In the report, the theme that stuck out was fix wasn't just restart thing.

It involved real engineering work over multiple days, new functionality and procedures, and coordination that stretched people thin. And if you've been through one of those, you know the hardest part is not always the technical root cause. It's keeping the team effective on day two. Because day one is adrenaline. Day one is everyone showing up, focused, and ready to grind. Day two is where mistakes happen.

Because people are tired. Context gets dropped. Communication gets sloppy. You start repeating the same investigations. You start making changes just to feel progress. And if you don't manage that, you can accidentally extend the incident by exhausting the team trying to fix it. So here's the point I'd steal for your own org. You need an incident process that assumes humans are a limited resource. Rotations matter.

Sleep matters. Handoffs matter. Having someone whose only job is to keep the timeline and reduce chaos? Matters. Also, you need leadership behavior that doesn't make it worse. If the team is deep in a multi -day incident and leadership is demanding constant updates, you are literally stealing cycles from the people doing the work. The ideal incident machine is one where the humans are protected enough to stay sharp.

And this loops back to the theme of the episode. Automation speeds up systems. But if you don't build the oversight and resilience around it, You just accelerate your way into bigger failure. You want faster deploys, safer upgrades, more automation. Sure, but you also want breaks, blast radius controls, and an incident process that doesn't burn your people down when the systems get weird.

Because the real bottleneck in reliability work is almost always humans. Not because humans are bad, but because humans are finite. Alright, quick recap and links and then I'm out. That's it for this episode of Ship It Weekly. We covered Curl ending their bug bounty program after getting flooded with low -quality AI -generated reports.

And why that's really a story about incentives, noise, and protecting maintainer time. We covered AWS improving RDS blue -green deployments with writer switchover downtime, typically 5 seconds or less. And why the real win is making upgrades boring. As long as your app behavior is mature enough to handle the blip. And we covered ECR cross -repo layer sharing.

The lightning round was Kubernetes client CMD patterns for better CLIs, robust Helm chart lessons, Docker Canvas, and the TTL networking tool. Links and show notes are on shipitweekly .fm. If you got something out of this, follow the show wherever you are listening. And if you can, leave a quick rating or review. It helps a ton. I'm Brian, and I'll see you next week.

All right, real talk for 2026. If a tool can run code, touch credentials, or trigger a deployment, it's not just a tool anymore. It's production adjacent. It's a control plane. And this week is basically three different angles of that same reality. One, N8n is back in the headlines with another max severity CVE.

It's a different bug than the one we covered in episode 12, and it changes what you should care about inside the platform. Two, GitHub introduced a new fine -grained permission for artifact metadata. It sounds boring, but it's supply chain and lease privilege getting sharper edges, and there's a deadline involved. And three, AWS wrote up what it actually took to take a DevOps AI agent from prototype to product.

The best parts are not AI is magic, it's the unsexy stuff. Evals, guardrails, and making the agent safe when it's 2am and everything's on fire. Alright, let's get into it. Hey, I'm Brian. I work in DevOps and SRE, and I run Tellers Tech. This is Ship It Weekly, where I filter the noise and pull out what actually matters when you're the one running infrastructure and owning reliability.

Quick bit of housekeeping, shipitweekly .fm has the episode links and show notes, and you can also find all of the past episodes there as well. All right. Three main stories today, then a lightning round, and then we'll wrap. Story 1. N8N CVE -2026 -21877. Authenticated RCE. That means your risk isn't only are we exposed to the internet. It's who can log in and who can edit workflows. Story 2.

GitHub Actions and the new artifact underscore metadata permission. Small change, big blast radius reduction. Story three, AWS DevOps Agent. A rare look at what it takes to ship an agent like a real product, not a demo. Then the lightning round. A few smaller things worth knowing this week. All right, N8N. Quick callback to episode 12 because it matters that these are two different issues.

Episode 12 was CVE -2026 -21858. The unauth nightmare one. That was the scary public exposure angle. This week's story is CVE -2026 -21877. It's a different vulnerability, still max severity, but it's authenticated RCE. And authenticated RCE is the your internal trust model is wrong reminder. Because internal only doesn't protect you from compromised credentials, session hijacks, leaked tokens, or lateral movement.

The gist here is if someone can authenticate and they can create or modify workflows, they may be able to abuse the Git node to get to arbitrary file write and then chain that into remote code execution on the host. So the real question isn't just did we patch? It's who is allowed to do what inside N8n. And in a lot of orgs, workflow automation tools start with a do -it -yourself vibe. Anyone can build.

Self -service. Super convenient, but convenience plus credentials plus code execution equals control plane. So treat it like one. What do you do? First, patch. It was fixed in 1 .121 .3. Second, remove optional sharp edges. If you don't need the Git node, disable it. Don't keep it enabled just in case. Third, tighten who can create and edit workflows. Treat workflow creation as privileged.

If the tool lets you do role separation, do it. If it doesn't, lock down who can access the UI. Fourth, inventory what creds live in there. Because the nightmare scenario with automation tools isn't just the server got popped. It's the attacker used the tool's existing access to pivot. The Do This Monday mini playbook is check your version, upgrade, and disable the Git node unless you truly need it.

Restrict workflow creation and editing. Get the UI off the public internet or at least behind strict access controls. List the integrations and credentials it holds and decide what you'd rotate if you had to assume compromise. All right, story two. GitHub added a fine -grain permission called artifact underscore metadata, and it's GA now.

There is also a deadline for moving away from using broad contents permissions for artifact metadata APIs. This is one of those changes that's not flashy, but it's exactly what mature platforms do. They reduce the default blast radius of common workflows. Because historically, a lot of folks granted contents permissions just to make artifact related calls work. And contents permissions tend to be broader than needed.

So what's the operational takeaway? This is your forcing function to clean up actions permissions hygiene. If your workflows don't have explicit permissions blocks, that's your first fix. If they do, audit where you've got contents read or contents write out of habit. Then, for workflows that touch artifact metadata APIs, migrate them to artifact underscore metadata.

And then bake it into your baseline templates and reusable workflows. Because the real win is not we fixed one workflow. It's future workflows don't start with broad perms by default. Do this Monday mini playbook. Identify workflows and reusable workflows using Artifact Metadata APIs. Update permissions to use artifact underscore metadata. Set a minimal default permissions block in templates.

Document the approved exceptions so future teams don't just slap contents right on everything. All right, story three. AWS DevOps Agent. This post is refreshing because it's not agents will change everything. It's here's what it took to ship an agent as a product. And the punchline is prototype is easy. Reliability is hard. Three lessons I'd steal. One, evals are not optional.

If the agent is part of your ops workflow, you need repeatable evaluation scenarios like it's a test suite. Two, you need observability into the agent's decisions, not just the final output.

Tool calls context inputs what it did what it tried what failed three you need an explicit control loop if the agent can take actions it needs breaks confidence thresholds approval gates scope rate limits a big red stop button because an agent that can open prs trigger CI or touch infrastructure is basically a junior engineer with a caffeine problem.

Fast and helpful, but absolutely capable of doing something dumb at speed if you give it too much freedom. Do this Monday mini playbook. Define what the agent is allowed to do and scope it hard. Put approvals on anything that changes state. Log tool calls and outcomes. Create a small set of eval scenarios based on your real workflows. And have a fallback plan when the agent toolchain is down.

All right, time for the lightning round. Some smaller stuff that's worth knowing this week. First quick hit, GitHub secret scanning is going to auto -enable extended metadata checks for certain repos starting February 18th, 2026. This is separate from the artifact metadata permission story. The practical implication is governance.

If you run GHAS or secret protection at scale, Expect new signal to show up without someone explicitly flipping a switch in every repo. Good for security coverage, but you'll want to sanity check how it impacts alert volume, ownership, and what actionable means in your org. Next, Claude Code and Guardrails.

There is a good real -world reminder in a thread where somebody's using Claude Code through Bedrock and their configured guardrails work via other paths, not through Claude Code. The punchline is simple. Don't assume your guardrails apply just because you turn them on somewhere. Validate the exact client and execution path you are using, especially if the agent can run tools or touch repos. Next, blocks goose.

This is another example of agent workflows getting operationalized instead of being a toy. They've got a repeatable research to plan to implementation workflow and a whole section on managing sessions and context so the agent doesn't go off the rails as conversations get long. The takeaway? If you're going to use agents for real work, you need structure, not vibes.

Treat it like an operational workflow with checkpoints, resumable sessions, and explicit context management. And last, open code as an agent runner platform. What's useful here is less the tool and more the pattern. Running agents locally with plugins that handle stuff like safe shell execution and background PTYs so you don't hang your terminal or let an agent freestyle destructive commands.

An actionable tip if you're experimenting, run it in a sandbox environment. Use scoped API keys and make the default posture read -only plus proposals, not write access everywhere. That's the lightning round. Here's the human part. A lot of teams are pushing hard on automation right now because it feels like the fastest path to leverage. And it is. Automation is leverage.

But leverage without controls is just a faster way to hurt yourself. N8n is powerful because it's connected. That's also why it's dangerous. GitHub Actions is powerful because it can do everything. That's also why permissions matter so much. Agents are powerful because they compress time. That's also why oversight becomes the job. So if you're building platforms in 2026, the job isn't add more automation.

The job is add automation, but with breaks. Scope permissions. Limit blast radius. Make actions auditable. Make rollback easy. And design your system so you can survive the weeks where something goes sideways. Alright, that's it for this episode of Ship It Weekly. We covered N8N CVE -2026 -21877, the authenticated RCE angle, and why workflow creation rights matter as much as patching.

We covered GitHub's new artifact metadata permission and how to treat it as a chance to standardize actions permissions instead of letting every repo reinvent bad defaults. And we covered AWS DevOps agent lessons from prototype to product. And the real takeaway.

Evals guardrails observability and control loops are what make agents usable in real operations if you got something out of this hit follow or subscribe wherever you are listening and if you've got 10 seconds a rating or review helps way more than it should links show notes and past episodes are on shipitweekly .fm I'm brian and I'll see you next week Thank you.

Hey, I'm Brian Teller. I work in DevOps and SRE, and I run Teller's Tech. Ship It Weekly is where I filter the noise and focus on what actually matters when you're the one running infrastructure and owning reliability. Most weeks, it's a quick news recap. In between those, I drop interview episodes with folks who are actually building in the space. Today is one of those interviews.

We're talking with Gracious James about TARS, his human -in -the -loop fixer bot wired into CICD. We get into how he segments incident response into sub workflows, how he keeps agents from turning into SSH with a chat bot, and the guardrails he thinks are non -negotiable if you're going to let AI anywhere near production. Today, I'm joined by Gracious James.

He's been building TARS, a human in the loop fixer bot wired into CICD. And we're going to talk about what it takes to make AI automation safe enough for real teams. Gracious, thank you for joining me. Thanks, Brian. Great to be here. Like you do it. I'm doing really well. Thanks for asking. So I'm really interested in this TARS bot that you built.

Can you give me a thesis of what you were actually building with TARS? And why is human in the loop the core idea? All right. So I am a software engineer. And when I coded and pushed my code into, you know, CI CD pipelines, I saw that it Took a lot of time to get through a lot of teams, especially the DevOps ones.

So I started looking at the processes of how DevOps go through, the incident response and everything like that. There's a lot of steps, manual steps that each and every person has to cover through. Going through it, I realized that all of these things are very manual, very repetitive. And don't get me wrong, everything is important and can break the system at any point.

So having a human in the loop is very important. But looking at the processes, I also see that all of these things can be automated. So having an AI agent where human in the loop approach increasingly helps us to fasten the processes is what got me going into all of this. Interesting. So what problem were you trying to solve that like existing tooling didn't handle? Right.

So all of this, all of the tools that I used into this workflow. Are existing tools, only that these are not custom made for, you know, my requirement, my company or whatever it is. So I had to custom make the workflow using multiple tools from different, different domains.

So I custom made, you know, the SS access in which there are, you know, validated SSH commands that my AI agent can utilize to find out what is going on into my Docker containers. To get a proper system status report.

And in order to do a comprehensive check, I compare those logs or the errors or whatever it is that the Docker is pointing out to my previous GitHub comments, be it the previous ones, the latest ones, or whatever it is, I compare them and get a comprehensive check. And based on those checks, it helps me to figure out where I should narrow my focus to.

And once I do that, I kind of understand what are the fixes that I should be figuring out in a very short incident response time that I get. That makes sense. Okay, so this was originally Friday, and then you morphed into TARS, right? Yes. Can you walk me through this evolution from Friday? Like, what was Friday and what is TARS? Well, Friday in itself is an Aniton workflow. Now, Aniton is a...

High -level abstract API, sorry, multiple API application system where we can connect different programs, subsections, and sub -workflows in which we take the data from one workflow, one action, and use it for the next process and step -by -step so that based on our custom input, we get a custom output which we utilize for something.

But Friday in itself used a lot of tools, like some of the tools I myself created and named, like visit development server, visit production server, visit this API container, that container. And based on all of this, my LLM creates a system status report in which I get to know whether some container is down or up or what is going on, if everything is working fine. And if anything is down.

Then it sends me a report via Telegram. And based on that report, it's kind of an alarm going off like in a plane's cockpit. And I am the pilot. All I see that, okay, Friday is telling me something is wrong over there. So it's my duty to go and evaluate, check what's going on over there. So yeah, it's kind of an alarm alert system. Interesting.

How do you keep this from turning into like a chatbot just with like shell access? Okay, so yeah. Everything is revolving around guardrails. So when it comes to prompt, the AI agent for doing system checks has its own prompt, and the prompt involves not just doing any actions.

So it's very limited to commands like Docker PS, Docker logs, the errors, and all of this information that it can gather to make a comprehensive update as to what the system is going through. It actually cannot do commands that are harmful for the system. And that's given in the prompt. Now I get your question as to like, the LLM can go ahead and do anything if it has the SSH access.

For that, what I have included is that in the tools like visit development server or production server, those are sub workflows. And those sub workflows. Don't just contain the SSH access. It has some validation parameters. It won't just let the LLM run any command it wants. It has validation checks based on what I have programmed it for.

For logs, it has only Docker PS, Docker logs, and it can never actually go and maybe Docker stop or Docker restart because all of those commands I have deliberately blocked in that sub workflow. So it is kind of a manual override over there. But yeah, that is a limitation that I have translated as a guardrail. That makes sense.

So, OK, speaking on guardrails a little bit more online, you had mentioned explicit rollback phases. What do those look like? OK, rollback phases. So, well, as a software engineer, when I do push some version into my development branch or from development to production, there's a lot of chances like. After the CI, CD testing and integration, something does break when it actually goes out into the real world.

So what we do know always as a software engineer is that the previous version, well, right, the previous version was pretty stable. So our go -to response is, you know, of course, fixing everything and putting it into the right place. But if it's going to take time, if we know based on the system status check.

That, okay, this error is a big one and it's going to take time rather than, you know, just sitting on it for a couple of days and, you know, just letting the clients know that, okay, I'm going to take my own time and do this. I'm just going to roll back to the previous stable version so that their work don't get, you know, hindered or whatever. You get it, right? Oh, that makes sense. Okay, so given the guardrails that you have in place, where do you require...

Human approval or is there human approval like in the process yeah there is there is the human approval uh does not come during the system status check so I don't really tell the ai agent you know where to go to read the data where to go to debug the code or whatever it is I have given it a set of instructions like you can check the docker logs you can check the terminal logs you can check the logging system that I

Have implemented per code base you can check the previous github commits to correlate as to what's going wrong but I don't really say it to you know go and check over here I think it might be wrong over there it doesn't do that what the human in the loop approach where it comes in is to do some kind of action now the action involves you know pressing some kind of button or running a pipeline or a workflow that actually changes some things in the system.

All of these things, the system checks and everything is just reading the data and creating a status report. But where it actually comes to change something, change in the system, that's where the human in the loop approach comes in. And without my, you know, verbal API key, me actually telling the agent to run this pipeline, run this workflow. It won't really do that.

Okay, so you had mentioned too that you had prompts with like guardrails. What are the hard never run this rules that you have set up? So it is kind of a corollary. What I'm trying to say is without a specific phrase, like run the CI CD pipeline to deploy this, this version. This is the exact phrase it requires to run the CI CD pipeline. So in my prompt, I have written it so that it.

Looks for the specific phrase to actually activate that function. So without it, without an actual phrase, without those actual wordings, it can't run it. And that's in the prompt, I get that. But as I told you before, as a guardrail, I have put it also into the sub workflow so that it doesn't just run anything. Once Friday gives me the comprehensive system check, I have two options usually.

One is to ask it again for a comprehensive system check where it has access to GitHub repositories and the GitHub actions. And that's when TARS activates because Friday doesn't have access to the GitHub, TARS does. So TARS goes through the normal Docker logs and compares it with the GitHub commits and what might have gone wrong. And it gives me a proper report as to this is the area that I need to focus in.

Now, it does ask me for a response as to what to do next, because after all, something is wrong in the system. So it's waiting for my approval or some kind of message that I need it to get. So I can again ask it to do some kind of comprehensive check.

But if I'm good with whatever information it has given me, what I'll do is I can tell it to, you know, run the CI CD pipeline to redeploy the previous stable version, the previous GitHub commit. So what it does is that based on this specific phrase, it activates the CI CD pipeline because in the sub workflow of the CI CD.

Pipeline, I have actually written that in order to run this, you need the specific phrase, run the CSED pipeline to redeploy the previous stable version. So without this phrase, so if this phrase involves like, okay, run the pipeline without the CSED or run the pipeline that I last deployed in the GitHub actions, it won't ever run because it does not match that specific phrase.

And that's kind of rudimentary, but it gives me more, you know. Believable focus as to, you know, it wouldn't just run anything just because I wrote it. Like, it is not up to its comprehension. That makes sense. Okay, so let's say that a team wants to adopt human -in -the -loop AI SRE automation. Where do you think they should start?

Okay, they should start, honestly, where they feel that a lot of steps that they do are not just rudimentary, but repetitive in tasks. Because that's where I started. I saw that the incident response procedure in itself has a lot of manual and repetitive steps. And not that I wanted to automate all of it, because after all, it requires access to SSH and a lot of system. And I can't just let an LLM run wild into it.

So kind of segmenting. Yeah, that's the right word. Segmenting the procedure into different steps.

Converting those steps into sub workflows and integrating all of those sub workflows into a pipeline one by one where I have the control between the input of one sub workflow to the output of other one so that I get in I keep in control as to what the whole picture is without actually letting one llm know what the entire big picture is so that it can manipulate is the go -to for integrating these type of tools.

How would you pitch human -in -the -loop AI SRE automation to someone that's maybe more skeptical? All right. So if a person has their own skepticism about AI actions, LLMs, thinking on their own, having access to all of these tools, I get that. I completely get that because that's where I started as well. But when you delve deep into how an LLM works, it works on an input and an output.

So as long as you can use that particular stage of giving an input to an LLM and phrasing that output in a way that I want to see and using that output into another LLM so that that LLM only gets the input that I am giving it, keeps the big picture to myself. And never the AI letting it know what the big picture is. So you obviously introduced this because you're trying to, you know, reduce toil.

You're trying to iterate quicker. What was the first thing that you used this AI tool to help solve? Okay, so... Where'd you start? Yeah, it never started as a fixer bot. It actually started just on the basis of, you know, go into the system and tell me what went wrong and why did the Docker container stop? That's all.

What I used to do is creating a simple LLM workflow in which I copied the output of the Docker logs, pasted it into the LLM, letting it know whatever the data I have, I'm giving it to you, tell me what went wrong. So then it explains me what went wrong one by one. And that's when I thought that, okay, this is one portion of it.

Why not I use the same portion for every single container that I have, but I want to automate it so that all I tell it is that, okay, go to this container and check it out rather than me copy pasting the logs and putting it and whatever it is. So for that, I needed to give it access to the terminal, right? So that's where the SSH access came in.

And that became a sub workflow because I still needed to keep it safe so that the LLM just can't, you know, write anything. I put the guardrail over there so that it can only write stuff like Docker PS or Docker logs. And it did the same thing for me, told me what is going wrong or how the system is working. And that's where it started.

And that's where it saved tons of time, me just copy pasting stuff, me just looking in the wrong places, because I have to look to all of those containers to see what is going wrong. So instead of that, I know exactly where I need to narrow my focus. That's where it all started. And then you just iterated on that and started adding more features. Yeah, yeah, absolutely.

What does future development look like for TARS? Like, let's say the next three, six months, assuming you have the runway and you have the time, what features would you like to add? That is a really good question. Well, TARS in itself is, I don't want to say limited. Limited is not the right word because two, three months ago, I was pretty astonished as to what TARS could do with the tools that it had.

And that itself was a huge accomplishment over there. But now after a couple of months when I use it regularly, I feel that I could go more. So I could give it access to more tools. I can give it access to the infrastructure, maybe Terraform and go from understanding what resources my system needs.

And if it's lagging somewhere, what is the focus that I should maybe increase some resources somewhere or maybe give it access to. Change stuff in Terraform or something like that. That's Terraform side of things, infrastructure. And that's one more tool to think about other than just, you know, GitHub actions and SSH access.

Let's say you have teams that are interested in experimenting with agents in CICD, but they're apprehensive. Do you have any advice for how to get started or where to start? Yep, yep. So it never starts with an LLM, I'll tell you that. Yes, we are trying to embed AI into it, but it never starts with an LLM. It always starts with...

Actually getting your thoughts and putting it into a paper, putting it into steps, logical steps that I can program first and then take that program and understand how an LLM will be effective in what positions. Because if you don't really have a work plan or a game plan to begin with, then the AI will also be as confused as you were. Yeah. That's where I'll start.

So what's your take on fully autonomous AI ops agents? It's scary. Because I myself am not a very big fan of, you know, I don't want to say it out loud, but I don't really believe a lot in experience counts. Because I believe if you're smart enough to do the smart thing, then sometimes even you can, you know, outgrow experienced people.

If you know where to look and how things work very logically instead of, you know, just gut feeling. Because experienced people tend to do that more. And that's where the AI and LLM lacks. It's kind of a hypocritical thing to say that thinking about the AI and LLM, always taking the smart choice instead of, you know, just having lots of experience, just a human like a human has.

But what I'm trying to say is that I still believe in the gut feeling that I truly hate in a lot of people. And that gut feeling is never going to be at least.

With the information that I know about AI and LLMs, it has never proved to have a conscience or a gut feeling that even if it sees all the data pointing to one thing, maybe a human knows that where exactly the error is, but the LLM will always go towards the probable cause instead of the gut feeling. So that's what the... That's where LLMs lack.

And that's why they can never replace humans because we have that creative aspect to us. And it's just a machine at the end of the day. Yeah, that's fair. Okay, so going back to guardrails, what do you think? We've talked about guardrails a lot. We've talked about AI and... Autonomy a lot. What would you say is the single most important guardrail for an AI fixer bot?

In terms of maybe single most, I would say validating every output that an LLM gives before you use that output for anything else. Yeah, that's fair. Okay, so closing thoughts, just curious, what would be like one of the main reasons to start using AI and CICD? What can that buy you? Well, AI is like an assistant. And it works to automate a lot of repetitive tasks that humans don't want to do.

And that primarily saves time. And time saved is money saved. But whenever we do tend to automate these things, we tend to get carried away in a way that, okay, I can automate this. I can automate that. I don't need you. I don't need that. I can put all of this into code. Well and good, but... Anything can go wrong when an AI hallucinates. Yeah, for sure.

All right, so closing out, what advice do you have for people using AI in their pipelines? Well, I understand there are a lot of disadvantages for a full -scale AI taking over a traditional system, but don't let that hamper you. Don't let that stop you from understanding your own process, understanding your own thoughts, writing it down.

Writing a program and, you know, running along with it because everyone gets their own time and you know how to utilize that time.

So you can always stop at a point, introspect as to what you have done till now, how it's going to affect the future, how it has changed your past, how it's going to help other people, how it's going to hamper other people and take all of these things into confidence and building the next step. Makes sense. Appreciate it. Thank you, Gracious, for coming on. Thanks a lot, Brian.

It was great being on this podcast with you. All right. That's the conversation with Gracious. I really liked his angle of designing the process first and only then dropping LLMs in as helpers, not magic. If this episode was useful, share it with a platform, SRE, or DevOps friend who's been playing with AI agents and trying not to blow up prod.

Hit follow wherever you listen so you don't miss the weekly news recaps plus these guest interviews. We'll be back with a regular Ship It Weekly news episode later this week. See you then. Thank you.