Ship It Weekly Transcripts

This week is basically a tour of the safety system became the incident. A protection rule that made sense once, then quietly started blocking real users. A Kubernetes permission that everyone treats like read -only, but it absolutely is not. A platform example that actually got the control plane versus data plane separation right.

And compliant scope expanding in a way that is easy to underestimate until you are drowning in evidence requests. So yeah, not a week of flashy tech. More like a week of go -check -your -assumptions. Thank you. Hey, I'm Brian, and this is Ship It Weekly. If you like the show, follow or subscribe wherever you are listening. It helps a ton. Everything lives at shipitweekly .fm.

Also, I'm starting another round of interviews. If you'd like to come on and talk about real world ops, hit me up at shipitweekly .fm.

All right let's get into it four main stories for today github reworked layered abuse defenses after legacy rules blocked legitimate traffic kubernetes node proxy git the telemetry permission that can turn into cluster wide rce hcp vault and what actually stayed up during a real aws regional disruption AWS PCI DSS scope expansion and the operational reality of compliance scope changes. Then a quick lightning round.

Then a human closer on reasonable assurance turning into busy work and what to do about it. Some of GitHub's legacy defenses are blocking legitimate traffic. So what happened? GitHub had users hitting unexpected too -many -request errors. And it's not because everyone suddenly got evil, but because old abuse mitigation rules were still active long after the original incidents that created them. So they went back.

Traced it, and reworked how layered defenses are managed, including how those rules get maintained and retried. This is a February 2026 story, and it's the kind of thing every platform team recognizes instantly. So why does it matter? If you ship defensive controls and you don't give them a lifecycle, they become permanent production dependencies. And that's the sneaky part. A security layer is not just security.

It sits on the request path. It can block revenue. It can break signups. It can break API clients. And it can create phantom outages that look like the app is slow when the app is fine. The failure mode is also brutal for teams because when the source of the problem is an old mitigation rule, the people on call might not even know it exists.

There's no muscle memory, there's no runbook, there's no customer complaints, and a bunch of dashboards that lie by omission. Old mitigations are like dead code until they suddenly run your business. So what do you need to do Monday? Do a quick inventory of your traffic safety layers. That could be CDN rules, WAF rules, bot protections, rate limits, edge middleware.

App throttles, whatever sits between the user and your service. Now you need to ask yourself two annoying questions. Who owns each layer? Not the team and actual owner. And how do you disable it safely if it starts blocking legitimate traffic? If you don't have a kill switch plan, you need to make one. Even if it's ugly, even if it's toggle this feature flag and accept higher risk for an hour.

Then add one reliability metric you probably don't track today. False positives. Not in a philosophical way, in a how many legitimate requests did we reject way. Because once your defensive layer starts rejecting good traffic, it's not a security success. It's an availability incident with a security label on it. Okay, that's story one.

For our second main story today, it's Kubernetes node proxy get and the read -only permission that isn't. So what happened? There is a Kubernetes RBAC permission that a lot of orgs grant to monitoring and observability tooling. It looks harmless. It's get on nodes slash proxy. The intent is basically let this thing scrape node metrics, stats, logs, healthy endpoints, that kind of stuff.

But research and write -ups in late January 2026 show how that permission can be abused to reach kubelet permissions and turn into arbitrary command execution in pods. The punchline is simple. A permission that teams treat like read -only telemetry can collapse trust boundaries if you hand it out broadly. So why does this matter? This is exactly how clusters get popped in real life. Not always by some fancy zero day.

Sometimes by a permission that was granted for convenience. Because observability stacks are the classic just give it cluster admin snowball. You start with it just needs to scrape metrics. Then it just needs to list pods. Then it needs node stats. Then you're granting node proxy access because charts and docs tell you to. And the scary part is psychological. Permissions like this don't trigger your gut.

They don't look like exec. They don't look like secrets. They look like plumbing. So nobody thinks to threat model it. Kubernetes RBAC is a minefield because the dangerous stuff looks boring. So what do you need to do Monday? First, go find it. Search your cluster roles for nodes slash proxy. Then list every subject that binds to those roles. Don't stop at Prometheus.

Look for logging agents, APM agents, cluster UIs, anything installed by Helm charts with default RBAC. Second, force the uncomfortable conversation. Which tool truly needs this? Is it using it or is it just granted because it was in the chart? If you truly need it, scope it tighter. Separate service accounts per tool. List namespace access where possible.

And document the justification so six months from now, somebody knows why it exists. Third, add detection where it matters. A lot of teams only monitor Kubernetes API server audit logs and call it a day. But if the abuse path is Kubelet -level behavior reached through proxying, you want visibility into that access pattern too. At minimum, treat node proxy access granted as a review trigger.

That permission should not be invisible. That's story two. Story three is HCP vault resilience during a real AWS regional disruption. Okay, so what happened? HashiCorp published a write -up on how HCP Vault behaved during a real AWS US East 1 disruption. Their control plane experienced elevated HTTP 500s and intermittent panics around 7 a .m. UTC.

But they say customer HCP Vault dedicated clusters maintained 100 % uptime and kept serving workloads. So the management plane had issues while the data plane stayed up. That's the key. So why does this matter? This is the kind of architecture decision that sounds like overkill until it saves you.

Control plane outages are common, not because everyone is bad at engineering, but because control planes are complex and they're multi -tenant and often exposed to weird edge cases. But customers don't care if your admin dashboard is having a bad day. They care if secret resolution fails and their apps stop booting. So that separation matters.

If your management layer being flaky can break your production usage path, you built one shared blast radius. And you see this pattern everywhere. Terraform is down so we can't deploy is annoying. Terraform is down so production can't read config is unacceptable. If admin plane downtime breaks prod reads, you don't have separation. Okay, so what are the action items? Do this for your top three critical systems.

Write down what is control plane and what is data plane. Then answer one brutally honest question. If the control plane disappears for two hours, what still works? Can apps still authenticate? Can apps still read what they need to run? Can you still scale? Can you still recover? Also, check your runbooks. A lot of runbooks quietly assume the control plane is healthy.

They tell you to click here, or run this automation, or use this UI. If the whole point is resilience, right, the control plane is down path too. Even if it's ugly CLI, because the day you need it, you will not be in a calm and well -rested mental state. Okay, that's story three. Story four is AWS PCI DSS compliance package expansion and what it really means for teams.

So AWS announced updates to their fall 2025 PCI DSS compliance package. They added two services to the scope of their PCI DSS certification. AWS Security Incident Response, and AWS Transform. They also added the Asia Pacific Taipei region into the PSI DSS scope. On paper, that sounds like a nice checkbook update. In practice, scope changes have consequences. So why does it matter?

Compliance scope changes do not usually break production, but they absolutely create work. And they create it in the most dangerous way. Slow, distributed, easy to underestimate.

And easy to turn into chaos when audit season hits so here's the trap a scope change lands security is happy leadership is happy then six months later some poor team is asked to reprove a pile of controls except now the region lists changed the service list changed and nobody knows what in scope even means in your org that's how reasonable assurance turns into hours of evidence churn And it's why compliance can feel like it fights delivery, even when it's trying to protect the business.

Audits aren't the enemy. Recreating proof from scratch is. So if this compliance change affects you and you're in a PCI environment, do a quick boundary check. What accounts are in scope? What regions are allowed? What services are approved. Then decide how you will generate evidence, not we'll do it when asked. Pick the repeatable pieces now. A reusable evidence package is the goal.

One place that says what the control is, how it's implemented, and how you prove it. You can point to it, update it, and stop rewriting narratives every time a new spreadsheet shows up. If you want compliance to be sustainable, you have to treat evidence like a product, not like a one -off homework assignment. Okay, that's story four. Now it's time for the lightning round. Here's some quick hits.

GitHub Actions extended the timeline for self -hosted runner minimum version enforcement. Starting March 16th, 2026, older self -hosted runners get blocked if they're below the minimum version. With a brownout period between February 16th and March 16th to help you find the stragglers. This is one of those, it won't be urgent until it's urgent things.

If you run self -hosted runners, go check what versions are actually deployed. Next, Headlamp, the Kubernetes UI project, is now officially part of the Kubernetes SIG UI, and they posted a 2025 highlights recap. The reason I care about this is simple. Kubernetes UIs are finally moving beyond toy dashboard into useful day -to -day tooling.

And teams need sane, supported ways to visualize cluster state without giving everyone kubectl to prod. And last, AWS Network Firewall Active Threat Defense. AWS wrote up how it draws near real -time intelligence from MadPot, their honeypot sensor network, and uses that to detect and block threats faster. Even if you never buy the feature, the idea is worth stealing. Speed matters.

Threat intel that shows up three weeks later is mostly trivial. Okay, that's the lightning round. Time for the human closer. And this week we're going to be talking about reasonable assurance turning into busywork. There's a thread in r slash sre that's basically the most relatable sentence ever. At what point does reasonable assurance turn into busywork? And the vibe is not we hate audits.

It's more like why are we spending engineer time formatting proof instead of reducing risk? That thread pairs perfectly with the GitHub story this week. GitHub had legacy protections that stuck around. They outlived the incident that created them and started blocking real users. That's what happens when a control doesn't have an owner and a lifecycle. Compliance busywork is the same failure mode, just slower.

A control exists. Maybe it's good. Maybe it's necessary. But over time, the evidence requests multiply. The templates change. The risk language changes. And suddenly, the work has not proved this system is safe. It's translate what we already do into 12 different formats. One comment in this thread nails it. If it's consistent and repetitive, automate the documentation. Audits matter. Formatting shouldn't.

That's the move. You don't win by arguing with auditors. You win by making your proof boring. One control. One source of truth. One evidence artifact that stays alive. And then you update it when reality changes, not when somebody sends you a spreadsheet. So my Monday challenge is simple. Pick one control you get asked about constantly.

Access reviews backup validation change management whatever now measure how many engineer hours go into evidence churn for that one control in one cycle if it's painful good you just found your next high leverage reliability project because busy work is not just annoying It steals time from actually making the system safer. Okay, that's the human story. Time for a quick recap.

We talked about how GitHub reworked layered defenses after legacy mitigations started blocking legit traffic. We talked about Kubernetes node slash proxy Git and how it's a permission you should treat like a real threat boundary. We talked about HCP Vault and how it's a clean example of control plane pain not taking down data plane availability. And we talked about AWS expanding their PCI DSS scope.

And that's your reminder to productize compliance evidence before it becomes chaos. And the human story was reasonable assurance turns into busy work the moment you are just formatting proof. If you like the show, follow or subscribe wherever you're listening. And everything, including the show notes and links, will be on shipitweekly .fm.

And if you want to come on for an interview round, reach out at shipitweekly .fm. I'll catch you next week. Thanks.

This week felt like AI went from cute helper to, okay, this is now part of the platform, not just in your IDE. It's showing up in CI, incident response, and in the places where mistakes become expensive. Also, Azure had one of those outages that doesn't just break a service. It breaks your ability to operate the service. And that one always hits different. Welcome back to Ship It Weekly.

I'm Brian from Tellers Tech, and this is Ship It Weekly, the DevOps and SRE news show for people who actually get paged. All links are in the show notes on shipitweekly .fm. And a quick ask up front. If you're getting value out of the show, hit follow or subscribe wherever you listen. And if you're on Apple Podcasts or Spotify, a rating helps a lot. Even a quick review is huge for discoverability.

All right, five main stories for today. One, the Azure control plane incident and why it creates the most annoying type of failure. Two, GitHub leaning harder into multi -agent dev with Claude and Codex in the loop, plus actions getting a little less painful. Three, Claude Opus 4 .6 and the real difference between better model and new failure mode. Four.

Google talking openly about SREs using Gemini CLI during outages. Five, the connective tissue story, MCP as plumbing, and observability getting pulled into the AI platform gravity well. Then we'll do a lightning round. Then a quick follow -up on Ingress Engine X and N8n since we covered those in previous episodes.

And we'll close with a human closer around trust, guardrails, and why - why agentic work is not a free lunch. Story one, Azure incident. VM management ops gets weird and that ripples everywhere. So what happened? Azure had a platform incident that impacted VM management operations across multiple regions. So not a VM went down. More like the thing that manages VMs is degraded. Create, update, scale, start, stop.

The day two operations that every platform team builds automation around. And when that gets weird, you feel it everywhere because it's not just VEBS, it's auto -scaling, it's instance group updates, it's node pools trying to heal, it's pipeline steps that assume I can always start a node or I always roll a deployment. So why does it matter?

This is the outage category I hate the most because it creates misleading symptoms. Your app might look up. Your load balancers are still answering. Your metrics are fine until they aren't. But deploys start hanging. Scale -outs don't happen. Rollbacks don't apply cleanly. And you end up in this gray zone where everyone is arguing about whether it's our code or the cloud. And here's the key point.

Most teams build reliability thinking about the data plane, requests, latency, error rates, but control plane failures break your ability to respond. If you can't change the system, you can't mitigate. And it turns a 15 -minute incident into a two -hour incident purely because you can't operate your own platform.

And if you've ever been stuck in a loop where you're trying to fix something and all your fixing tools are failing too, that's what this is. So what do you do Monday? Go look at your automation paths and ask a boring question. What happens when the control plane is slow or degraded? Do you have retries with exponential backoff? Or do you have hammer it until it works? Do you have timeouts that fail safely?

Or do you have jobs that hang forever? Because a lot of people accidentally build infinite retry storms into their pipelines. And the control plane is exactly where that becomes self -inflicted pain. Also, do you have a decision point in your incident process that says, stop trying to change the system, stabilize, communicate, wait for the provider? That's hard to do because it feels like giving up.

But sometimes it's the correct move. And one more thing. If your response plan assumes you can always add capacity or always recreate nodes or always scale out, make sure you have a mitigation that doesn't require that. Sometimes the right move is degrade features instead of add servers. Sometimes it's shift traffic instead of rebuild. And you only learn that when you talk through it before the outage.

All right, that's story one. Story two is GitHub leaning into multi -agent dev. And there's a bunch of practical implications for platform teams hiding under the surface. So story two is GitHub multi -agent in the workflow, plus actions getting less painful. What happened? GitHub rolled out AgentHQ support for using Anthropic Cloud and OpenAI Codex in GitHub and VS Code alongside Copilot.

So instead of co -pilot only, it's more like pick the agent that's best for the job. And GitHub Actions shipped workflow updates too, including a case function in Expressions. If you've ever had a workflow where the logic is basically a pile of nested ands and pipes, you probably just felt your shoulders drop a little. So why does it matter? Two reasons. First, multi -agent is an emission of reality.

People are already switching between tools. Cursor. Copilot, Claude Code, whatever. Different models are better at different things. Some are better at refactors. Some are better at reasoning through messy requirements. Some are better at quick code completion. GitHub putting multiple models in the same surface is them trying to become the control plane for AI -assisted development.

And if you are a platform team, that matters. Because the moment these tools can open PRs and modify workflows, they are part of your software supply chain. Second, actions changes matter because CI is still where a ton of production risk lives. Workflow changes are not just developer productivity. They are permissions. They are secrets. They are build artifacts. They are deployment paths.

So if you are enabling AI agents in that world, you should treat it the way that you treat any automation. Not cool, ship it. More like cool, what's the blast radius? So what do you do Monday? If you want to test agent PRs, do it like onboarding a new engineer. Start with read -only or low -risk tasks. Have it summarize failing CI. Have it draft release notes.

Have it propose a change and explain why, but don't let it merge. And keep it out of workflow files and infra repos until you've decided what your governance is. Because workflow files are basically production code. Now, about that case function. That sounds tiny, but here's what it really buys you. It lets you reduce clever logic and replace it with readable logic. And readable CI is safer CI.

Because the number of incidents that start with why did this deploy to prod is not zero. If your CI logic is hard to reason about, you are one sleepy reviewer away from a bad day. Also, a quick mid -episode reminder. If you are enjoying the show, hit follow or subscribe. It's a small thing, but it helps a ton. That's Story 2. Story 3 is Opus 4 .6.

Story 3 is Opus 4 .6, and I want to talk about this in a way that's actually useful, not just the new model dropped. Claude Opus 4 .6, the real change is trust and longer running work. Anthropic released Claude Opus 4 .6. They are positioning it as better at agentic tasks, longer horizons, more reliable multi -step work, more give me a goal, I'll do a thing behavior. So why does it matter? Here's the trap.

When people say better model, a lot of folks hear cool, the answers will be a little smarter. But the bigger shift is behavioral. When the tool gets good enough, people start delegating bigger chunks of work. Not rewrite this function, more like implement this feature. Or refactor this module.

Or update these workflows or go fix the failing tests and that's where the failure mode changes because now it's not the assistant gave me a wrong answer it's the assistant completed a plan that was subtly wrong And it did it confidently. And it did it across 15 files. So now you've got to do a different kind of review. Not just does this compile, more like does this match intent?

And also, did it change anything surprising? Because agents are really good at making changes that look plausible. So what do you do Monday? If you are using these tools seriously, I think teams need a new habit. Spec first, even lightweight. Before you let the agent touch the repo, you want a short plan. What are you changing? Why? What does done look like? What are the risks? What test will it prove?

Because without that, the agent will happily fill in the blanks with something that feels right, but isn't. Also, decide what you consider sacred ground.

For a lot of teams that's ci workflows iam network policy and anything that changes prod deployment behavior if an agent is going to touch those that should be behind stricter review two humans minimum no auto merge and probably a show me exactly what changed and why requirement and if you are a security minded org this is the week to remind people AI capacity upgrades help defenders and attackers.

It's easier to find bugs. It's easier to exploit them, too. So the defense is still the same boring stuff. Least privilege, secrets hygiene, fast patching, and not letting random automation have right access to the keys to the kingdom. That's story three. Story four is Google talking about SREs using Gemini CLI in outages. And this is where the AI and the ops conversation gets real.

Google SREs Gemini CLI as an outage sidekick. What happened? Google published a write -up on SREs using Gemini CLI during real outages. The framing is basically AI in the terminal as an incident companion. They're using it to classify symptoms, pull context, suggest mitigations, and reduce the early incident chaos. So why does it matter?

This is the direction a lot of teams want to go, even if they're not saying it out loud. Because the hardest part of incident response is not typing commands. It's context. What changed? What's impacted? What are the known safe mitigations? What's the fastest way to reduce customer pain? And under stress, humans drop context. We forget what we already tried. We lose track of timelines.

We waste cycles writing status updates that are vague because nobody has the full picture. So the AI as scribe idea is actually super compelling. You let the tool do the relentless bookkeeping. Summarize logs, pull the last deploy, list recent config changes, draft a timeline, draft a status update. Draft a mitigation checklist based on the runbook.

If you can reliably do those things faster, you've bought the human's time to think. And time to think is basically MTTR reduction. But there's an obvious danger. The moment you let AI run commands, you have to treat it like any other automation with prod access. You don't give it a shell and vibes. You give it narrow tools. And you build guardrails. So what do you do Monday?

If you want to copy this pattern without being Google, start with the safe version. Make an incident helper that is read -only. It can query logs. It can query metrics. It can pull deployment history. It can draft comms. And it can suggest mitigations from the runbook. But a human has to execute. That alone is valuable. And it's a good stepping stone.

Because the minute you jump straight to AI executes mitigation, you are going to have trust problems. And trust problems become adoption problems. And then this whole thing turns into a failed experiment. So start small, prove it helps, and build from there.

That's story four story five is the this will matter more over time story mcp and the platform layer that's forming around agents ai is becoming platform plumbing mcp governance and observability consolidation We are seeing vendors rally around MCP, the Model Context Protocol, as a way to connect agents to tools and data sources.

Miro launched an MCP server story to connect collaboration artifacts into AI coding tools. Kong announced an MCP registry angle for governance and tool discovery. So we are moving towards a world where agents can be given tools in a standardized way.

And in parallel observability is getting pulled into the ai orbit moves like snowflake planning to acquire observe are basically saying telemetry is data and we want ai sitting on top of it so why does this matter let me translate mcp into normal person ops language it's basically a standard way to say here are the tools an agent can use here's how it calls them and here's the context it can see And if that becomes common, platform teams are going to have a new inventory problem.

What agents exist? What tools do they have access to? What credentials are behind those tools? What data sources are exposed? What's logged? What's audited? And who owns the thing when it breaks? Because if you don't handle this like a real integration layer, you will end up with shadow agents the way we ended up with shadow CI jobs and shadow admin access. And on observability, the shift is also real.

More companies want log retention. They want to correlate telemetry with business events. They want to ask what changed across logs, traces, config, and deploy history. Then they want AI to help explain it. That's powerful. But it also turns observability into governance. PII risk, access controls, data retention policy, cost controls.

Because if you keep all telemetry forever, you can easily build a compliance nightmare and a gigantic bill. So what do you do Monday? If MCP is showing up in your org, treat it like production integration, not a toy. You want ownership. You want access controls. You want an inventory. You want a default deny stance. And you want audit logs.

Because the first time an agent does something surprising, the only thing that will save you is knowing what it did and why. And for observability, if you are exploring AI there, don't start with let's let a model roam through prod logs. Start with specific measurable wins. Better incident timeline creation. Better alert deduping. Better correlation across services. Faster what changed answers.

Then measure if it actually reduces pager pain and toil. All right, time for the lightning round. All right, lightning round. GitHub Actions hosted runners had an incident this week where jobs queued, timed out, and people had a rough morning. It's a good reminder that hosted is not the same as immune. If CI is business critical, know what your fallback is. Docker patched an Ask Gordon issue.

And even if you don't care about the specifics, the theme matters. If you're feeding AI systems untrusted input, including metadata, you can accidentally build a prompt injection path into your workflow. Treat AI inputs like any other untrusted input. Terraform 1 .15 alpha dropped. Not a headline, but worth noting for platform folks.

Deprecation warnings for variables and outputs is the kind of feature that makes shared modules easier to evolve without surprise breaks. And Wiz wrote up the Malt Book situation. Misconfigured backend. Tons of exposed API keys and sensitive data. It's a perfect vibe coding meets production credentials cautionary tale. Also, a quick mention from the open source side.

Chainguard's Ameri -OSS effort is an interesting trend. Commercial orgs stepping in to maintain projects that everyone depends on, but nobody wants to own. You don't need to have a strong opinion on it yet, but it is worth watching. And a couple of quick follow -ups. Ingress Engine X had multiple issues disclosed and fixed versions called out. If you run Ingress Engine X, double check your versions and patch.

And N8n is still in the patch fast and treat your automation layer like a control plane category. If you are running workflow automation tools, assume they are part of your perimeter. And here's the human theme this week. We keep talking about agents like they remove humans from the loop. They don't. They move humans to a different part of the loop.

The hard part is shifting from can I run the command to can I decide what to trust? And then can I recover cleanly when the automation does something confidently wrong? Because with agents, the risk isn't the tool is dumb. The risk is that the tool is competent enough that people stop double checking. And the first time that that happens in pride, you get a weird kind of incident. Not a bug. Not a bad deploy.

More like automation drift. Everyone thought the automation was safe. Then one day it does the wrong thing and nobody notices until it's already rolled. So if you are adopting this stuff, the boring work is the real work. Guardrails. Approvals. Ownership. Audit trails. And clear stop the line rules. Because the future isn't no humans. It's humans overseeing more automation. And that's its own skill.

All right, time for a recap. We had five main stories for today. Azure reminded us the cloud control plane is part of the product, and it can absolutely be your incident even when your app is fine. GitHub is going multi -agent with Claude and Codex, and Actions got a little less painful with better workflow logic.

Opus 4 .6 pushed agent workflows further into the mainstream, and the real risk is trust and multi -step changes. Google is using Gemini CLI in real SRE outage work, which is basically the AI and ops playbook taking shape. And MCP plus AI -driven observability is turning into platform plumbing. And platform teams are going to own governance whether they want to or not.

And the lightning round was GitHub Actions hosted runners having an incident, Docker patching an Ask Gordon AI supply chain issue, Terraform 1 .15 alpha, the Malt Book credential exposure story, and the Emirate OSS maintenance trend. All links are in the show notes on shipitweekly .fm. If you made it this far, hit follow or subscribe wherever you are listening.

And if you can, drop a rating or review on Apple or Spotify. It helps more than you think. All right, I'll see you next week.

This week is a reminder that the stuff we treat like glue is now a primary failure domain. CI trigger roles, cert renewals, helm charts, automation tools. One tiny assumption goes sideways and suddenly you're dealing with a supply chain risk, a global build break, or an RCE in the thing that holds all your credentials. Hey, I'm Brian. I work in DevOps and SRE and I run Tellers Tech.

This is Ship It Weekly, where I filter the noise and pull out what actually matters when you're the one running infrastructure and owning reliability. If something's hype, I'll call it hype. If it changes how you operate, we'll talk about it. Quick bit of housekeeping, the show notes and links are on shipitweekly .fm. If the show's been useful, follow it wherever you listen.

Also, a rating helps way more than it should. Four main stories for today. First, code breach. A CI trigger and filtering issue in a small set of AWS managed repos that's a perfect reminder that CI glue is part of your security boundary now. Second, the Bazel TLS cert expiry incident. The kind of failure that is boring, binary, and absolutely capable of blocking your entire engineering org. Third, Helm chart reliability.

Prequel reviewed over a hundred popular charts and the results are basically a post -mortem template for why it installed fine is not a reliability guarantee fourth n8n two new high severity flaws disclosed by jfrog that can lead to code execution we're going to treat this one like a mini story because workflow automation tools are basically a control plane holding your secrets then a quick lightning round with a few

Operator friendly tools and takeaways and a human closer tying the theme together alright let's get into it So code breach.

AWS published a security bulletin describing a misconfiguration involving unanchored account underscore ID webhook filters for code build. Used by a small set of AWS -managed open -source repos. AWS says they mitigated it quickly, rotated credentials, reviewed logs, and added additional mitigations and protections around build processes with credentials in memory. Wiz's research frames the risk clearly.

If you can trigger a privileged build in a repo that's part of a supply chain, you potentially get access to tokens and credentials that can be used to push changes or create malicious artifacts. That's why it's not just CI, it becomes supply chain. Now, the operator lesson is not code build is bad. The operator lesson is stop treating pipeline trigger logic like it's harmless.

If an untrusted event can cause a trusted pipeline to run, you do not have CI. You have an execution environment exposed to the internet. And almost every org drifts towards this risk without meaning to. Here's the drift pattern. You start with PR checks, lint, test, build, great. Then someone adds integration tests.

And those tests need credentials to hit an environment, or to pull from a private registry, or to call a third -party API. Then someone adds preview deployments. Those need cloud creds, or at least some deploy token. Then someone adds artifact publishing because it's easier if the PR build produces the image. Now your PR pipeline can build and push images. Then someone uses those images in staging.

And now your PR pipeline is part of your release path. And at that point, you've accidentally created a supply chain path where a PR can influence something that runs in your environment. This is why the most dangerous sentence in CI is, it only runs in CI. Because CI is usually the thing that holds credentials that can touch everything else. So the practical question to ask is simple.

Can untrusted events cause trusted actions? Can a forked PR run a job that has secrets available? Can a PR comment trigger a workflow that can deploy? Can a PR workflow push artifacts that are later deployed? Can a PR workflow assume cloud roles? If the answer is yes, or even I'm not sure, you've got a boundary problem. Now, I want to give you a do this Monday playbook that doesn't require a big platform rewrite.

Step one, label pipelines mentally as untrusted and trusted. Untrusted pipelines is PRs and forks. No secrets, no publish, no deploy. It answers, does it compile? Do tests pass? Trusted pipelines run on merges to main, tags, or explicit manual approvals. That's the one allowed to publish and deploy. Step two. Pick one repo, your most sensitive one, and map the event chain in plain English.

What events trigger workflows? Which workflows can access secrets? Which workflows can write artifacts? Which workflows can deploy or change infra? Step 3. Check for the classic foot guns. Are secrets injected into your PR workflows at all? Are you using conditions like actor checks or branch checks as auth? Are you relying on the code won't print secrets as a control?

Do you allow PR builds from forks to run with privilege tokens? If you find these, the fix is not perfect security. The fix is strong separation. PR builds can still run. Just keep them in the untrusted sandbox. If you need integration tests, run them in a separate environment with a separate low privilege credential set. If you need to build images, build them but don't push them to a production registry.

Or push to an isolated registry that cannot be used for deploy. If you need a preview environment, require approval before anything privileged runs. And the last point, even if you fix the triggers, scope matters. Least privilege for CI tokens is not optional. Your pipeline credentials should not be able to do everything. That's story one. Story two is Bazel.

On December 26th, 2025, the TLS certificates for many star .bazel .build domains expired and it caused widespread build breakage. The Bazel team's postmortem says the outage lasted around 13 hours before it was resolved. This is one of those incidents that's both boring and terrifying. Boring because it's just a cert. Terrifying because cert failures are a binary cliff. Everything works, then it doesn't.

And the blast radius is immediate because every client that depends on that endpoint fails at the same time. Also, auto -renew does not prevent this class of incident. Auto -renew is one link in a chain. The full chain is issuance, renewal, deployment, Reload and verification. A lot of real cert outages are renewal succeeded but deployment didn't reload. Or new hostname wasn't included.

Or monitoring checked the wrong endpoint. Or DNS validation broke and nobody noticed. So here's the practical operator version of this story. You'd need external monitoring for cert expiry from the outside. Against the actual endpoint users hit. Not an internal health check. Not a dashboard in the cert system. The real endpoint. You need ownership. A named owner. A team. A channel.

Someone that at 2am on call can tag and get traction. And you need runway. Alerts well before expiry. If your alert fires 24 hours before expiry, you are still basically doing incident response. If it fires 30 days before expiry, you can fix weird edge cases like DNS changes, migrations, or validation issues calmly. Now, the do this Monday pass. Pick your top three engineering org blockers.

These are not always customer facing. Often they're internal systems that block shipping. Artifact Registry, Git Host, CI Endpoint, SSO Login, Webhook Receiver, Package Download Host, any of those. For each one, answer. Do we have external monitoring for cert expiry and chain validity? Does it alert at least 14 days out, ideally 30? Is there an owner written down? Do we know where to fix it if renewal breaks?

If the answer is no, that's a cheap reliability win you can fix without rewriting anything. And if you want a meta lesson, certs are one of those dependencies where the failure is so preventable that it's painful. Which is why teams tend to overcut corners until they get burned. Don't wait for the burn.

Story 3 helm chart reliability prequel reviewed 105 popular open source helm charts and they found the average reliability score was roughly 3 .98 out of 10 with a median around 4 out of 10. Their point isn't helm is bad it's that many charts ship demo friendly defaults not reliability -friendly defaults. This matters because charts aren't just packaging.

They encode operational behavior, readiness and liveness, resource requests, update strategies, disruption behavior, security contexts, sometimes topology and scheduling assumptions. So when you install a chart, you are adopting a set of operational opinions whether you realize it or not. Here's how this bites teams. A chart has no resource requests. In dev, it looks fine.

In prod, under pressure, it becomes unpredictable. Pods are throttled, get evicted, or get starved. Probes are missing or sloppy. Traffic gets routed to pods that aren't ready. Or probes are too aggressive and under load, they trigger restarts. No pod disruption budget, no topology spread. And then routing node maintenance becomes a cascading outage.

Everything was highly available until you drained a node and lost a majority of replicas in one place. Unsafe update strategy and rollouts turn into brownouts. And the worst version is when Kubernetes says everything is green while your app is melting. That's where chart defaults turn into long incident timelines. So what do you do without forking every chart? You create a baseline overlay and a checklist.

Baseline overlay is a thin layer in your GitOps repo or Terraform or Helm release config where you enforce defaults. Resources required. Probes required. Explicit update strategy. PDB when appropriate. Spread constraints if the service needs real availability. Security context that matches your cluster policy.

And the checklist is just does this chart behave predictably under rollout, under node drain, under load spike. Now, the do this Monday pass. Pick one chart you run in production that matters. The one that would page if it went sideways. Open its values and answer. Are resource requests set? Are probes set and meaningful? Do we have safe rollout behavior? Do we have disruption behavior planned?

Will replicas spread across failure domains, or can they all land on one node? What happens if a node gets drained? If you can't answer quickly, that's your signal. Add the defaults, document them, and move on. You don't need perfect charts. You need boring, predictable behavior. Now, N8n. And this is new. Hacker News covered two high severity vulnerabilities in N8n discovered by JFrog.

The short version is these flaws can let authenticated users escape the sandbox and execute code. One is in the expression sandbox and the other involves Python code execution in internal mode. Here's the important part. People hear authenticated and they relax. In workflow automation platforms, the permission model is often broader than you think.

Authenticated can include a lot of people who can build or edit workflows. And in tools like N8N, workflow editing is basically code execution. Because workflows can evaluate expressions and interact with credentials. So this isn't, oh no, an attacker needs an account. The real question is, who in your org has an account? And what can their workflows touch?

And what makes this class of bug extra painful is these tools often sit in the middle of your environment holding keys. Slack, GitHub, Jira, AWS keys, database credentials, webhooks, secret managers, all of it. So a sandbox escape is not just someone ran code. It's someone ran code where the keys live. That's why we keep coming back to N8N on this show. It's not because N8N is uniquely bad.

It's because the category is high leverage. Okay, practical actions. First, patch. Don't debate it. If you self -host N8n, patch quickly when sandbox escapes drop. The blast radius is too high to slow walk it. Second, reduce who can author workflows. Don't treat workflow editor as a casual permission. Treat it like can run code in a privileged environment because effectively that's what it is. Third, reduce exposure.

If your N8n UI is public on the internet, you are playing on hard mode. Put it behind SSO, VPN, IP allow lists, whatever fits your org. You want fewer people able to even reach the attack surface. Fourth, isolate it. If it's holding keys to a bunch of systems, at least give it a narrow runtime permission set. Least privilege on the credentials it uses. Separate credentials per workflow if you can.

Don't run it with god mode access to AWS just because it was convenient once. And now the tie back to our last episodes where we talked about N8N CVEs. The theme has been consistent. Workflow automation tools are basically control planes. They need the same operational rigor you'd give to an internal platform. Patch fast, lock down authorship, reduce exposure, least privilege the credentials.

If you treat it like just a tool, it will eventually treat you like just a breach. All right, time for the lightning round. First. Use Tusk Fence. It's a lightweight sandbox for running commands that block network by default. If you are experimenting with agents, runbooks that execute, or any workflow where code runs on behalf of a user, this is the kind of primitive you want. Safe by default beats clever by default.

Next, HashiCorp agent skills. This is part of a trend I actually like. Vendors shipping structured reusable skills and guardrails instead of just telling you to prompt better and hope. Next, Marimo. It's a reactive Python notebook that stores as normal Python.

That sounds small, but it matters because Git -friendly notebooks are actually useful for incident analysis, runbooks, or one -off ops experiments you want to keep without the notebook JSON misery. And a quick one from the register, the Ralph Wiggum clawed loop story. It's funny, but the real point is dead serious. People are already building loops that keep agents running until they produce output.

Without constraints and verification, that becomes confident nonsense at scale. The ops lesson is the same as alert fatigue. If your system floods you with low -quality output, humans stop trusting it. Okay, time for the human closer. Every story today is a glue failure. CI trigger logic becomes security boundary and someone implemented it like it was just config.

Cert renewals were treated like solved and then the cliff happened. Charts were treated like installers, not operational dependencies. And workflow automation tools are treated like a convenience layer even though they hold the keys. So the takeaway isn't stop using tools. It's treat guardrails like product work. Make untrusted pipelines truly untrusted. Make cert monitoring external and owned.

Make helm baselines explicit. Make workflow authoring privileged. Make credentials least privileged. Because if you only build accelerators, you are not building a better platform. You are just building a faster incident. All right, time for a quick recap. We talked about code breach and why CI triggers and filters are real security boundaries. We talked about Bazel and the CertCliff problem.

And we talked about Helm chart reliability and why defaults matter more than installs. And we talked about the new N8n sandbox escape flaws and why workflow automation needs to be treated like a control plane. The lightning round was Fence, HashiCorp Agent Skills, Marimo, and The Agent Loop Cautionary Tale. Links and show notes are on shipitweekly .fm.

If you got something out of this, follow the show wherever you are listening. And if you can, leave a quick rating or review. It helps a ton. I'm Brian, and I'll see you next week.

Bye. Bye. Hey, I'm Brian Teller. I work in DevOps and SRE, and I run Teller's Tech. Ship It Weekly is where I filter the noise and focus on what actually matters when you're the one running infrastructure and owning reliability. Most weeks, it's a quick news recap. In between those, I drop interview episodes with folks who are actually building in the space. Today is one of those interviews.

I'm joined by Austin Reed from Horizon .dev. He helps small and mid -sized businesses save serious time with AI and automation. And this isn't theoretical. We talk about what actually works when you're automating real workflows with real revenue behind them. Not just demos.

We get into the biggest misconceptions, why projects fail more from communication than code, where human in the loop makes sense, and what tools his team is using behind the scenes. From GPT and Claude to Cursor, CICD, and code review bots. Today, I'm joined by Austin Reed. Austin Reed, your automation expert with Horizon .dev. He helps SMBs save serious time with AI and automation.

And we're going to talk about what actually works in the real world, not just demos. Austin, nice to meet you. Thanks for coming on. Thank you, Brian. I appreciate it. So I'm curious, what do you do? Day -to -day in your job? Man, a little bit of everything. I do a lot less programming nowadays than I did when we started. When I started, it was just me and my ex -business partner, Mattel.

We were just keyboard warriors, always programming, me and him. And I would do sales and accounting, whatever. But nowadays, I mostly manage the team, lead the team. I hire people. I help the team with project structure and architecture. I help debug. More advanced problems. Sometimes I jump into code, but it's not that often. But I'd say the majority of my time is with client relations and on sales calls.

What is the real unlock or benefit for AI and automation for small to medium businesses right now, in your opinion? Well, I guess it would depend on what area of the business you're talking about unlocking, right? Every business is unique with its own problems. Some businesses, you know, they would really benefit by having some sort of sales automation, sales AI enablement, right?

Helping them generate more leads or helping them to where they don't drop leads or helping them with statistics on that side of things. Other businesses, they have that unlock and they would benefit more from fulfillment side of things, making sure everything is tight, everything's efficient, working properly. There's not any errors going on. AI can be used. Across, I'd say, big portions of a lot of businesses.

Most businesses, I mean, they essentially function the same. They take in some sort of customer, they service the customer in some way, or they give them some sort of product, right? And then there's the, after you turn in the project, like transition period, right? Where you get like referrals on the back end and stuff. And so adding AI in any of those core areas, I think really, really helps. 100%.

Yeah, that makes sense. So what would you say is like the biggest misconception that companies have when they say, we want automation, but they don't necessarily know what that means, right? They want automation. They want to move quicker, maybe, but they don't necessarily know what automation is or what that actually entails. Yeah, I mean, I think there's two.

One, on one side of things, you have people who think that AI is going to cause more problems than it's going to solve, or they don't trust the quality of AI. They don't have...

A firm belief in the powers of what it can do and on the other side of things they're definitely too gung -ho to get things done I I find the biggest issue is usually around some sort of content generation so they want to automate their instagram they want to automate their video creation things like that and so they think that ai can just do everything and another thing is they also think that because there's ai and everything's easy we have things like cursor and stuff now that It's really easy to build these things.

And so like they think it's going to cost like a couple hundred bucks or a thousand bucks to get a system that's robust enough to like manage the business. It's like, hey, you know, your business is working like. $100 ,000, $200 ,000 worth of sales a month. And you honestly think that like a $300 product is going to be able to sustain that. That doesn't really make a whole lot of sense, right?

So you have a lot of these problems. It's just underestimating and overestimating their capabilities, depending on what side you're on. And then definitely overestimating how cheap it is. Yeah, no, that's fair. Okay, so I know, like we talked about this a little bit, depending on the company, there's a lot of different... Automation wins.

But what, based on your experience for companies, what is the most popular or common win for automation? Okay. Yeah. So it's going to be one of two things. Usually it's either on the customer service side of things, or it's going to be on the sales side of things. So specifically for B2B businesses.

It'd probably be more on the sales side of things like I said automating lead flow increasing the amount of leads you're reaching out to enabling your bdrs sdrs aes to be able to do their jobs even better that drives a massive amount of impact in sales organizations whereas just two years ago like sdrs bdrs they were sending 50 100 emails a day they were handwriting the emails right now Then they moved on to doing sequencers.

And so then all of a sudden they were sending thousands of emails, but they were all templated with spin tax, whatever. And now it's all AI generated. And I know businesses that are sending millions of emails a month, which is absolutely insane, right? On the other side of things, you have the customer service. And it really kind of depends, but like you could imagine your mom and pop dental shop, like.

They're not always there to answer the phone. It would be nice to be able to call on like Saturday at six when people are busy or maybe people aren't working and be able to schedule an appointment without having somebody there needing to take the call, right? And so that also translates to a chatbot or an email responder. It's essentially the same thing. It's just the medium of which you communicate changes.

But the AI is still doing the same thing. It's taking schedules, doing follow -ups, taking information, you know, filling out forms and stuff so that the customer service side of things is on point. Is there any part of a business right now that you specifically try to not automate or you stay away from automating? Yeah, I'd say one, finances. And I don't think it's because it's bad to automate finances.

I think it's just because I'm not the best finance person. So because I don't understand it at a very intimate level, I tend to stay away from it. And I don't mean like automating like your accounts receivables and whatever. No, I mean like actually automating like statistics and things like that with accounting. Like I don't, I'm not an expert at that, right? The other thing I stay away from.

Firstly, is anything medical or HIPAA related? I don't want to touch, right? Because yeah, same thing with government. I don't want to touch it. But in terms of business, I'd say mostly just finances and the things that really require a personal touch or somebody overlooking it. So we'll do something called men in the loop, which is like, maybe we'll do an automation, but then a human has to approve it.

So the human edits it, approves it, says it's good, right? So for very bespoke services, you know, it's always good to have that little bit of human touch, right? So you can have some automations, but the automations, they don't do the job. They enable the person to do the job better. So it sounds like you're using AI as a tool. It's not replacing someone too. I mean, that's kind of an important point.

Like if you're an expert in something, you can use it to help further along. Development or use as a tool, but you're not replacing the individual. It's not because you don't understand finance. It could potentially hallucinate, right? So you don't want to just recklessly use a tool without knowing what the output could be. Right, exactly.

And so I guess the reason why I'm really good at automating sales and customer service and different fulfillment things is because I intimately understand a lot of those problems because, you know, I have those problems within my own business. I'm not saying I don't have finance problems in my business. I totally do. But I'm like not looking at that as well. I'm more focused on the day -to -day operations, right?

So yeah, no, I totally agree. Not replacing the individual is key. I mean, even like with my assistant, like every once in a while, I'll be like, hey, what do you spend most of your time on every single day? And she'll tell me, I'll be like, hey, well, let's take this off your plate, but let's add this new thing. You know, now that you have more time, now that you have more leverage, let's make the...

Return on time even greater for what you're working on. Now that makes sense. Is there anything that you've gone to automate with AI that's backfired that you're willing to talk about? I mean, I'm going to say not really in the sense of it backfired because it wasn't a good automation. And the reason why is because before we dive deep in automation, we had already been a programming agency for a while.

So we already knew the capabilities of what we we'd already been through the crap right like we've already made a lot of mistakes and learn from them and whatever but I would say the ones that have failed most of the time they don't fail because of the job they fail because of communication or improper vetting of the client or just clashing personalities is it ever where they don't maybe give the right spec on what

They actually need or they're not even sure of what the output is that they need I mean that happens a lot we've gotten better at identifying that problem so by first of all don't automate something that hasn't happened yet you know When we first started out with automations, there was a lot of people that were like, oh, we're going to have a lot of demand in February, so we need to build this system so that when the demand comes, we'll be ready to go.

And then February comes and there's no demand. It's like, okay, so you just spent $10 ,000 on a system. That doesn't help you. It doesn't make sense, right? And so I always try to make sure that there's actually a need there. And I think also the way we build kind of weeds some of that out. So we always like a lot of business owners.

Common they're like oh we want this and this and this and this and that it's like a huge menu order of like a restaurant menu of what they want right and that's cool but we're like okay that's cool but how about we what's the most impactful quick win we can do for you right and so we'll start with that that one impactful thing and then we give it to them as quickly as possible so we work a lot with cicd framework I

Don't know if you know what that is but basically we'll develop a small prototype as quickly as we can as best as we can and we'll give it to them as quickly as they as we can in order to get feedback on on how that is because maybe like you know we build an automation but then it doesn't quite work the way that they wanted or maybe the owner has this great idea but then when the team members get their hands on it

Like their reality is a little bit different because the owner isn't doesn't have his head in the trenches, so he doesn't know exactly what their day -to -day looks like.

So we always try to get... MVPs and prototypes in the business owner's hands as quickly as possible so that they're able to play around with it and test. And then as we stack on more features, you know, we get feedback like, hey, we like this. We don't like that. We'd like to change that. And it would be cool if we could add this. We're like, OK.

And so we can start adding those little tweaks early on instead of having a project where it's like, OK, cool, we'll get it done. And then two months later, we turn something in. That's not what they wanted. Right. What's the tech that you're generally using behind the scenes to build these automations? You're talking about like the code itself or like the tools we give our developers? The models, the code, APIs.

Everything? AWS, yeah. Is it more like AWS? Are you using like Gemini models? Are you using Claude models? Yeah. Yeah. We use a lot of GPT and Cloud. So I find Cloud is a lot better when it comes to sales and anything writing related, you know, like blog posts or customer service. Like Cloud is just sounds more human for some reason. I don't know why. GPT is a lot better at analytical things.

GPT Codex 5 .2 that just came out is really good. We've been using that with Cursor to build. Apps. So that's been helping out our development team a lot. We've also been experimenting with having AI review pull requests that programmers are implementing into code bases to see if it's good or not. And it's been working quite well. That's been quite interesting.

We've been thinking about implementing even a bot that whenever somebody adds an issue to GitHub, it'll maybe do a code to see what it looks like. And then the developer just reviews the code and edits it. We know that AWS has a a service that does that. We haven't fully implemented that part yet, but we're definitely experimenting with it now. So, I mean, there's definitely a lot of changes in the industry.

And then in terms of like servers, we don't use AWS or Google as much unless the client already has it. We actually prefer to use Volter, which is digital, same thing, right? I mean, same server stack, same technology, just a different name. And a lot cheaper, I would say. We use a lot of Kubernetes, a lot of Docker, Docker Swarms, Django, Node .js, NNN, yeah.

Has there been any technical challenges that you've run into with building out these automations? One thing is getting credentials from the customer. I know that sounds simple. Yeah, I've run into the same issue. Yeah, it sounds like a simple thing. It's more of a customer service problem than it is like a programming problem. But I mean, a lot of these people, they don't even know what their passwords are.

So it's like trying to get, or like you made a custom app and you need to get into their Google console to add whatever. And it's like, that's... That's difficult to do with some of these people who aren't as tech savvy, right? Anything else that I find makes it particularly difficult? Scope creep is totally a thing. Definitely have to push back on that. People get overly zealous about things.

That's why we always try to, we record all our meetings. We always try to put project plans right. So this is what you agree that we build, you know, so make sure everything's very transparent. But even with that, scope creep comes in sometimes. And then other than that.

I mean, at first we didn't, there were some clients where we had trouble fully understanding what their vision was, but I think that was more an experience than it was actual, actually a problem because now we don't see that as much because we have a lot more things in place, like making sure like there's already something to automate, giving them the prototype first so that we can, right.

So we flushed out a lot of those issues. How do you handle? The this is scary or I don't trust AI reactions pushback that you may get. Well, I mean, if they came in from a cold email, I'll be like, well, who do you think wrote the email? No, that's a hard objection. And for a lot of people, that's pretty hard to know because they just have some sort of mental block there.

And, you know, I'm not here to convince people. If they don't want the automation or if it doesn't serve them, I mean, it will serve them.

If they truly don't want it like I'm not gonna sit there and try to convince somebody otherwise like I'd much rather go to the guy who's like super pumped and excited about getting some new things done in the industry that hasn't been done before you know like that's much more exciting but I guess where we see some of that pushback is like in a business where it's like an old father and he's passing it on to the son

And so the son wants to implement these new things but the father's like no like the old way works why would we change it You know, we see some of that and they're usually a quick win.

Something super small and kind of insignificant is the best way because you can talk to them all you want. But when you actually show them something, it really works. So even just showing them a case study, but like really running through and be like, hey, you know, this. Similar business, different industry and problem or whatever. Like, hey, look what we did there. This is what it looks like for them.

We can do this for you. And they're like, oh, really? Is that possible? I'm like, how about we start small? We'll do something really small and quick, you know, quick win. It won't cost you very much. It won't take a lot of time. And you can just get your feet wet and see if you like it. Where do you think AI is going? If you had your crystal ball, where do you think like three, five years?

Obviously, it's a changing market. It's crazy. The last couple of years have been crazy. Yeah. If you were to hypothesize that, where do you think it's going? Well, AI influencers is already a thing, even though people don't fully believe it yet. I think that software agencies are going to, a lot of them are going to go out of business.

Because business owners are going to be able to make software themselves so you're going to see a big flood of apps and new stuff coming out because there's going to be a bunch of normal people launching their own apps and so maybe the new game is more sass related and less programming related I still think there will be programmers but I feel like it'll only be like very advanced niche stuff or old stuff like like

Like cobalt right for banks or something but so yeah I think At least my industry is going to, like, we need to pivot hard because, like, engineers now are not, I mean, they know code now, but maybe in five years they'll still know code.

But in 10 years, they might not know how to program, you know, and that's kind of weird. Outside of that, man, I think it's very good and very bad in a lot of ways. And it's really hard to say. Yeah, that's fair. Yeah, just like the car did, the horses that will probably open up new jobs and, you know, maybe business use cases that.

We're not even sure of yet you know but yeah it's definitely interesting is there any hype takes about ai that that you disagree with or you want to kill I don't think ai is quite to the point yet where you can fully automate social media I mean you can for like those info talking head videos not talking head but like the ones where it's like they say a story like you know oh this person went missing in 1994 because

Whatever like yeah those can be automated but I mean like truly valuable content I don't think can be automated to a high enough quality extent yet right I think that's coming I think it's close but I still think that there needs to be that human element the other thing is is because we're posting so much ai content now the ai is just rewriting ai content yeah so it's just learning from itself at this point So as time goes on, I believe the information gets more and more diluted.

And that's a problem in and of itself. I think that kind of covers it, at least my view. I know just being on LinkedIn even, but more so Instagram, Facebook, TikTok, it's all AI -generated Sora content now. A lot of it, yeah. It's hard. It's hard to just have real, genuine content that's not AI -derivative. Yeah. To be real. It cuts through the noise pretty well. So we talked about this a little bit earlier.

How do you think AI is going to influence development in the future? I mean, now we're using Cursor, right? We're using MCP servers to get information on Kubernetes clusters or whatever. But we don't have new junior engineers that are getting hired at the same rate they were just a couple of years ago. What happens to engineers and developers, you know, in the next five, 10 years?

They become prompt warriors or they switch industries, right? I mean, even our guys, like I handed them Cursor like a couple months ago. And before that, it was like Microsoft Copilot, right? Like two years ago, something like that. And even Microsoft Copilot. Like, dude, the difference in productivity between my guys was insane when I gave them Copilot. It was like, it's not like they don't know how to write code.

It's just that they can write it so much faster. They can be like. I want this, this, and that. And there's a huge difference. It's like I see a lot of business owners who don't know programming, who try to program with Cursor, and then they jump on Cursor, and it doesn't work. Cursor sucks, blah, blah, blah, and it's horrible. And I'm like, you don't know how to talk to it. It's really what it is.

I mean, there are some things like, yeah, it probably isn't the best app, but understanding how to architect. How to describe how it works with data, what order to build things in. That's huge, right? Because, you know, you get a certain database model set up, certain APIs set up, and all of a sudden you add something crazy in the mix and you have to redo everything, right?

And so it's like a lot of people just jumping in like they don't know that. So to answer your question, man, I mean, like I said, we're implementing AI more and more. All of our dev workflows, we're implementing it in code review and debugging and our CICV type of stuff to make sure that nothing broken gets pushed. We're implementing it and helping them even.

On issues you know they get an issue they talk to cursor cursor spits out some code they review some of the code they write some code themselves they tell it to write a certain function a certain way they sip their coffee they wait they test it you know and for testing too it's huge for like unit testing and integration testing like you write out a whole api in django and then you're like hey unit test this you know

Integration test this like that's huge the speed of which it can write out some of that code that, you know, before it would take, you know, weeks is insane.

Yeah, I agree. I know for me, being able to vet the output that Cursor gives you and not just trusting it, like you mentioned business leaders that may go use Cursor for the first time, but they don't really understand what they're, even understanding what to ask, but then understanding what the output is that they're given.

Like you need to have some awareness around that, at least currently, maybe in the future you won't to the same degree, but right now you, you know. It still can hallucinate. It could still mess up. So you need to know how to frame the question in the right way so it gives the output that you want, but then also vet the output that it's giving you.

And if you don't know anything about the subject that you're asking it to help with, you don't know if it's a good output or not necessarily. Right, exactly. Any closing thoughts? Not really. I think the world's going to change a lot really quickly. And we're along for the ride. Everybody needs to pivot. Every business, every industry needs to pivot, not just ours. Right.

So the faster you can kind of get on board and start testing new stuff, I think the better off you're probably going to be. Yeah. Be ready for it because it's coming regardless. Like you'd even mentioned, you know, the father and the son, the father may not want AI. He probably is talking to an AI chatbot on a website every day. He's probably getting phone calls from AI chatbots. He just doesn't know it.

He's already interacting. He's already in that world. He just doesn't know. Yeah. Yeah. All right. Well, appreciate it. Thank you, Austin, for coming on. Really appreciate it. I appreciate it, Brian. Definitely. All right. That's the conversation with Austin Reed from Horizon .dev. My biggest takeaway is his framing that automation isn't magic.

The win is picking a real pain point that already exists and getting a quick win live and then iterating. Also, the AI is cheap now mindset is a trap. If your business is doing real volume, you need systems that can actually hold up with guardrails, approvals where it matters, and someone who can sanity check the output.

If you want to connect with Austin or check out what they're building, I've got links in the show notes to his LinkedIn and to horizon .dev. If this episode was useful, share it with a DevOps, ops, or business friend who's trying to automate workflows without creating a bigger mess. Hit follow wherever you listen so you don't miss the weekly news recaps plus these guest interviews.

We'll be back with a regular Ship It Weekly News episode later this week. See you then. Thanks.

We keep trying to automate our way out of pain. Faster deploys, faster builds, faster ops. But the funny part is, the faster the system gets, the more expensive the mistakes get too. This week is basically three versions of that lesson. Hey, I'm Brian Teller. I work in DevOps and SRE, and I run Teller's Tech.

This is Ship It Weekly, where I filter the noise and pull out what actually matters when you're the one running infrastructure and owning reliability. If something's hype, I'll call it hype. If it changes how you operate, we'll talk about it. Quick bit of housekeeping. The site is shipitweekly .fm. That's where the show notes and links live.

And if this show has been useful, hit follow or subscribe wherever you are listening. If you've got 10 seconds, a quick rating or review also really helps way more than it should. All right, three main stories for today. First, Curl basically said we're shutting down our bug bounty program because we're getting flooded with AI garbage. That sounds petty until you realize what it's actually about. Signal to noise.

Burnout and how incentives can accidentally wreck your security pipeline. Second, AWS just made RDS blue -green deployments faster, with writer switchover downtime typically five seconds or less in single region setups. If you run production databases, this is a legitimately big lever, but it also comes with the usual, yeah, but what about caveats? Third, ECR finally supports cross -repository layer sharing.

Which sounds like a tiny registry feature until you're the person paying for storage and watching CI push the same base layers across 200 microservice repos like it's a hobby. Then the lightning round. A really good Kubernetes post on building kubectl -like CLIs without re -implementing the whole universe. A Helm charts write -up worth bookmarking, plus a couple of tools that are worth looking at.

And we'll close with the human side, the irony that automation doesn't remove responsibility, it moves it. And it usually moves it into the most overloaded humans at the worst possible time. So, Curl. If you haven't seen this one, the Curl project posted changes to stop their bug bounty program at the end of January 2026.

And the reason is basically the volume of low -quality AI -generated Vauln reports has become a tax they can't afford anymore. And I want to be careful here because the headline can turn into dumb discourse like AI bad or security people are cranky. That's not the point though. The point is incentives. Bug bounties are already a weird incentive machine.

They work best when the incoming reports are rare -ish, thoughtful, reproducible, and written by someone who actually validated something. But if you can create an environment where someone can crank out 50 convincing -looking reports in an afternoon, and they only need one to hit to maybe get paid, you've just created an attacker. Except the target is your maintainer's attention span.

This hits platform people in a few ways. One, it's the same pattern as alert fatigue. The first 10 useless alerts are annoying. The 500th useless alert changes how you behave. You stop trusting the channel. You start ignoring it. And then eventually, the real one shows up and everybody gets hurt. Two, it's a reminder that intake is part of your system design.

We talk about security tooling like it's just scanners and policies. But the workflow is the product. If your workflow can be spammed, it will be spammed. Three, this is also a very real maintainer sustainability problem. Curl is foundational internet plumbing. If the people maintaining that plumbing have to spend their evenings debunking hallucinated CVEs, it's not free.

It's a direct drain on the project's ability to ship real fixes. So what do you do with this besides nod sadly? If you run internal security intake, or even just your normal ops intake, steal the lesson. Add friction where spam happens.

That can be as simple as requiring real repro steps, requiring environment details, requiring a minimal proof, or forcing the submitter to do one more round trip before you spend time. Not because you hate people, but because you're protecting the team's attention. And honestly, I think we're going to see more of this. More projects putting guardrails around community intake.

Not because they don't want reports, but because they want fewer, better reports. Alright, let's switch from human bandwidth is the bottleneck to database downtime is the bottleneck. AWS posted an update that RDS blue -green deployments can now reduce writer upgrade downtime to typically five seconds or lower for single region configs.

If you're not deep in this world, blue -green is basically you stand up a green environment that's kept in sync, you test your changes there, and then you switch traffic so green becomes the new production. The reason this matters is the painful part of database upgrades is not you. Usually the upgrade itself. It's the how long is my primary not writable window.

For a lot of teams, that window is where you end up negotiating with product, doing off hours work, or just punting upgrades longer than you should. So AWS tightening that switchover window is real value. But here's the part I'd say out loud on an on -call team. Don't confuse fast switchover with safe upgrade. You still need to think about the stuff that bites you in real life.

If your application can't tolerate a brief blip, you still need retry behavior that doesn't melt your connection pools. If you've got long running transactions, you need to understand what the switchover does to them. If you are doing anything spicy with replication slots, parameter groups, extensions, or oddball maintenance jobs, you still need to test the green environment like it's going to betray you.

Because it might. Also, and this is the platform angle, reduced downtime is not the same as no coordination. A lot of orgs treat DB upgrades like a once -a -quarter fire drill because the process is fragile. If blue -green gets you to a world where upgrades are boring, that's the win. Boring is the goal. So my takeaway is if you are on RDS MySQL, Postgres, or MariaDB, this is worth revisiting.

Even if you tried blue -green before and shrugged, the improved switchover timing might be enough to change how often you are willing to use it. Now, let's talk about a feature that is way less glamorous than database downtime, but might save your CI bill and your sanity. AWS ECR now supports cross -repository layer sharing.

Meaning repositories can share common image layers when blob mounting is enabled at the registry level. This is one of those features where the AWS headline is polite. But the real world version is stop storing the same base image layers 300 times because you split the world into 300 repos.

If you've got a lot of services that all build off of the same base layers, like the same distro base, the same language runtime, the same internal golden image. ECR can now deduplicate that across repos. So what changes? Push performance can improve because you are not re -uploading layers ECR already has.

Storage can be less stupid because the registry can reuse layers instead of treating every repo like a separate universe. And in CI, especially if you do a lot of rebuilds, this can reduce the pain of why is this build pushing 800 megabytes again when nothing changed? Now the platform and governance angle. This is also a boundary question. A lot of people use separate repos as a mental model for isolation.

Not security isolation exactly, but organizational isolation. That team owns that image. Layer sharing doesn't mean the images are merged, but it does mean the backend storage model is more global. If you've got strict compliance requirements, you might want to understand how that interacts with retention, scanning, and who can access what. But overall, this is a very rare thing in Cloudland.

A simple feature that makes life better with basically no downside for most teams. If you run a big ECR footprint, this is worth enabling and measuring. You'll feel it fastest in why are our pushes so slow? And why is our registry bill weird? All right. Lightning round. First up, the Kubernetes blog post. Uniform API server access using client command. This is for anyone who has ever written a KubeCTL plugin.

Or wanted to build an internal cli that feels like cube ctl and then realized cube ctl has a million flags and conventions the post is basically a guide for how to use the same client cmd pattern so your tool behaves like users expect next building robust helm charts I like this one because it's not here's helm basics it's more the stuff you learn after you've maintained charts that that break in production, values

Structure, templates that don't explode, and patterns that keep your chart from turning into a pile of conditionals nobody wants to touch.

Next, a tool called TTL. This is basically traceroute grew up. With a TUI and a bunch of diagnostics, you usually end up doing manually. MTU discovery, NAT detection, route flap alerts, that kind of stuff. If you do network debugging and you're tired of bouncing between MTR, traceroute, and vibes, it's worth a look. And last quick one. Docker Canvas. InfoQ did a write -up on it.

And the short version is Docker is trying to bridge Docker Compose workflows into cloud deployment workflows. And visually, it's aiming at that make deployments less mysterious audience. I don't think it replaces Helm or Customize Tomorrow. But it is worth watching because it's another signal that developer -friendly workflows are creeping closer to production deployment patterns. Thank you. Alright, human closer.

For the human closer this week, SRE Weekly linked Honeycomb's interim report for their extended EU incident. And the reason I want to mention it is not because the vendor had an outage. It's the operational reality of multi -day incidents. And specifically, the part that a lot of orgs pretend doesn't exist. Managing human energy. In the report, the theme that stuck out was fix wasn't just restart thing.

It involved real engineering work over multiple days, new functionality and procedures, and coordination that stretched people thin. And if you've been through one of those, you know the hardest part is not always the technical root cause. It's keeping the team effective on day two. Because day one is adrenaline. Day one is everyone showing up, focused, and ready to grind. Day two is where mistakes happen.

Because people are tired. Context gets dropped. Communication gets sloppy. You start repeating the same investigations. You start making changes just to feel progress. And if you don't manage that, you can accidentally extend the incident by exhausting the team trying to fix it. So here's the point I'd steal for your own org. You need an incident process that assumes humans are a limited resource. Rotations matter.

Sleep matters. Handoffs matter. Having someone whose only job is to keep the timeline and reduce chaos? Matters. Also, you need leadership behavior that doesn't make it worse. If the team is deep in a multi -day incident and leadership is demanding constant updates, you are literally stealing cycles from the people doing the work. The ideal incident machine is one where the humans are protected enough to stay sharp.

And this loops back to the theme of the episode. Automation speeds up systems. But if you don't build the oversight and resilience around it, You just accelerate your way into bigger failure. You want faster deploys, safer upgrades, more automation. Sure, but you also want breaks, blast radius controls, and an incident process that doesn't burn your people down when the systems get weird.

Because the real bottleneck in reliability work is almost always humans. Not because humans are bad, but because humans are finite. Alright, quick recap and links and then I'm out. That's it for this episode of Ship It Weekly. We covered Curl ending their bug bounty program after getting flooded with low -quality AI -generated reports.

And why that's really a story about incentives, noise, and protecting maintainer time. We covered AWS improving RDS blue -green deployments with writer switchover downtime, typically 5 seconds or less. And why the real win is making upgrades boring. As long as your app behavior is mature enough to handle the blip. And we covered ECR cross -repo layer sharing.

The lightning round was Kubernetes client CMD patterns for better CLIs, robust Helm chart lessons, Docker Canvas, and the TTL networking tool. Links and show notes are on shipitweekly .fm. If you got something out of this, follow the show wherever you are listening. And if you can, leave a quick rating or review. It helps a ton. I'm Brian, and I'll see you next week.