Ship It Weekly Transcripts

Developer tooling used to feel like the safe part of the system. Production was the scary place. The database, the cluster, the load balancer, the pager. And then everything around that was just tooling. GitHub, CICD, merge queues, package publishing, code review bots, AI agents. But that line is getting harder to defend.

Because when the tooling can merge code, publish artifacts, leak secrets, spend money, or decide what gets shipped, it is not just sitting beside production anymore. It is part of production. And this week, that idea showed up everywhere. Hey, I'm Brian Teller. I work in DevOps and SRE, and I run Teller's Tech.

This is Ship It Weekly, where I filter the noise and focus on what actually changes how we run infrastructure and own reliability. Show notes and links are on shipitweekly .fm. If this show's been useful, follow it wherever you listen. Ratings help way more than they should. And if you are watching on YouTube, subscribe there too.

We have six main stories today, then the lightning round, and we'll wrap with the human closer. We're starting with GitHub's critical Git Push remote code execution vulnerability, including the AI reverse engineering angle from Wiz. Then we'll talk about prompt injection against AI agents tied into GitHub workflows.

After that, Elementary's CLI supply chain incident, where a GitHub actions workflow became the path to a malicious release. Then GitHub's April 23rd Merge Queue regression. Then cal .com going closed source, which is probably the most argument -starting story in this set.

And finally, GitHub Copilot moving to usage -based billing because AI tooling is becoming a governance and cost management problem, not just a productivity experiment. Then in the lightning, we'll hit MinIO, Ghostty, Docker -hardened images, and Azure DevOps security updates. Story one. GitHub's GitPush RCE is a toolchain wake -up call. Let's start with GitHub.

GitHub disclosed a remote code execution vulnerability in the GitPush pipeline. The issue affected GitHub .com, GitHub Enterprise Cloud, GitHub Enterprise Cloud with Data Residency, Enterprise Managed Users, and GitHub Enterprise Server. GitHub says Wiz reported the bug through the bug bounty program on March 4th.

GitHub reproduced it internally within about 40 minutes, fixed GitHub .com in under two hours, and says it found no evidence of exploitation. That is the reassuring part. The uncomfortable part is the shape of the bug. The vulnerability involved user -supplied Git push options. Those push options ended up influencing internal metadata passed between GitHub services.

Because of how that metadata was formatted, an attacker could inject extra fields that a downstream service treated as trusted internal data. And once user input can shape trusted internal metadata, things get ugly fast.

GitHub says the researchers showed they could override the environment where the push was processed, bypassing sandboxing around hook execution, and ultimately execute arbitrary commands on the server handling the push. That is a lot hiding behind the phrase Git push. And that is why this belongs in a DevOps and SRE conversation, not just a security newsletter. Every team treats Git as normal plumbing.

Humans push code. Bots push code. Automation responds to pushes. CI kicks off. Release workflows start. Security scanners run. Deployment paths wake up. But behind that one command is a hosting platform. Internal service boundaries. Hook execution. Sandboxing. Audit logs. And identity. That is infrastructure.

For GitHub .com and enterprise cloud users, GitHub says the platform was patched and no customer action is needed. For GitHub enterprise server customers, this is a patch now situation. GitHub specifically recommends upgrading supported G -H -E -S releases and reviewing audit logs for suspicious push operations. But the bigger story may be how Wiz found it.

Dark reading covered the AI -assisted reverse engineering side. Wiz used AI tooling to help analyze GitHub Enterprise Server, understand compiled binaries and internal service behavior, and find a bug that historically may have taken weeks or months to uncover. They reportedly went from the idea to working exploit in under 48 hours. That is the part that should stick with us.

Not because AI found a bug, like the robot woke up and became a hacker. The real story is that AI lowered the cost of understanding complex systems. A lot of software has been protected informally by being annoying. Hard to reverse engineer. Hard to understand. Too much glue. Too many binaries. Too many internal assumptions. That was never real security, but it was friction. And friction matters.

If AI removes enough of that friction, then systems that were probably fine because nobody will spend the time become more interesting targets. So the takeaway is not panic. It is that boring security engineering matters more when discovery gets faster. Input validation. Trust boundaries. Least privilege. Patch velocity. Audibility. Because when the cost of finding bugs drop, the cost of sloppy assumptions go up.

Story 2. Prompt injection is now in the CI -CD blast radius. Next up, AI agents and GitHub actions. The register covered research showing prompt injection attacks against AI agents from Anthropic, Google, and Microsoft tied to GitHub workflows. The basic shape is pretty simple.

AI agents read GitHub data, pull request titles, issue bodies, issue comments, review comments, markdown, the stuff developers and contributors write all day. So if an attacker can place malicious instructions into that data and an agent later processes it with access to tools or secrets, the attacker may be able to steer the agent.

In the Claude case, researchers showed that a malicious PR title could influence the bot and leak credentials into a review comment. With Gemini, issue comments and prompt injection were used to expose an API key. With GitHub Copilot Agent, malicious instructions could be hidden in an HTML comment that humans would not see in rendered markdown.

The victim assigns the issue to Copilot, but the payload is already sitting there in the context. The researchers demonstrated theft of Anthropic and Gemini API keys, multiple GitHub tokens, and potentially any secret exposed to the GitHub Actions Runner environment. That last phrase is the one I care about. The GitHub Actions Runner environment. Because this is not just an AI story. This is a CICD story.

If your agent runs in GitHub Actions, and GitHub Actions has access to tokens, deployment credentials, package publishing rights, cloud permissions, Slack webhooks, or JIRA credentials, then prompt injection becomes part of that blast radius. The researcher called this comment and control, which is a play on command and control. And it fits. The attacker may not need malware. They may not need an external callback.

They may not need to trick a human into clicking anything. They can write data into GitHub, let automation read it, and let the automation leak the secret back into GitHub. That is a nasty little loop. And this is where I think teams are a little too cautious about agents. A code review bot should not have more access than it needs. An issue summarizer should not have repository write access.

A documentation bot does not need cloud keys. And if you are passing untrusted GitHub content into an agent that can use tools, run commands, or read secrets, that is not a harmless productivity flow. That is untrusted input hitting an execution environment. We have had that category of problem for decades. AI did not invent it. AI just made the parser weird. Story 3.

Elementary's CLI incident is the supply chain story in miniature. Now let's talk about elementary. Elementary Data published a security incident report for a malicious release of the elementary open source Python CLI version 0 .23 .3. On April 24th, a malicious CLI package was published to PyPI, and a malicious Docker image was published to their registry. Those artifacts were not produced by the elementary team.

According to elementary, the attacker opened a PR with malicious code and exploited a script injection vulnerability in one of their GitHub actions workflows to publish the release. That is the modern supply chain anxiety loop in one story. A pull request. A workflow vulnerability.

A release -looking artifact, PyPI, Docker, and downstream users who now have to assume any credentials accessible to the environment where that CLI ran may have been exposed. Elementary removed the malicious release, published a safe 0 .23 .4, removed the vulnerable workflow, rotated affected credentials, and moved the OIDC authentication where possible.

That response sounds pretty solid, but the attack path is the real lesson. A lot of teams still think of GitHub Actions as a convenience layer. But for many projects, GitHub Actions is the release authority. It builds containers. It publishes packages. It signs things. It pushes to registries. It decides what gets shipped.

So if that workflow has script injection, broad permissions, long -lived tokens, or too much trust in PR -controlled input, then your release system can become the attack path. The question is not only do you trust your source code? It is also do you trust the automation that turns source code into artifacts? Those are not the same question anymore. You can have a clean main branch and a vulnerable release workflow.

You can have signed releases and a compromised signing path. You can rotate app credentials and still forget that the package publisher token has been sitting in CI forever. So the practical playbook is boring but important. Check workflow permissions. Avoid long -lived publishing tokens where OIDC is available.

Be careful with shell interpolation from PR titles, branch names, issue comments, tags, and commit messages. Split, build, and publish. Protect release environments. And threat model GitHub Actions like production. Because if GitHub Actions can publish your package, it is production. It might not serve customer traffic, but it controls customer trust.

Story four, GitHub's merge queue regression is a control plane reliability story. Next one is also GitHub, but from a reliability angle. On April 23rd, GitHub had a merge queue regression. GitHub says whole requests merged through merge queue using squash merge produced incorrect merge commits when a merge group contained more than one pull request.

In some affected cases, changes from previously merged PRs and prior commits were accidentally reverted by later merges. GitHub says that 658 repositories and 2 ,092 pull requests were affected. No data was lost because the commits still existed in Git. But affected default branches ended up in an incorrect state. And GitHub could not safely repair every repository automatically. That is a painful incident.

And it is painful because merge queues exist to reduce risk. That is the whole point. They are supposed to make high -change environments safer. Instead of everyone racing to merge against a moving base branch, the queue group changes, validates them, and lands them in a more orderly way. So when that control plane gets a bad behavior, it creates a weird kind of incident. Not the site is down.

Not the database is corrupted. But the mechanism we trust to preserve correctness just produced a branched state that is not what we think it is. That is subtle. And subtle is dangerous. Because if a deploy fails loudly, you know. If an API goes red, you know. If a merge queue quietly reverts earlier changes, you might not notice until the wrong code is already in production.

The practical takeaway is not don't use merge queues. I still like merge queues. The takeaway is that any environment with merge authority needs SRE thinking. Good audit trails. Easy ways to identify what changed. Detection when a branch state does something weird. Clear ownership when the thing that protects main becomes the thing that damages main.

Merge queues, feature flag systems, CI schedulers, artifact registries, deployment orchestrators, GitOps controllers. These are control planes. They may not be in the request path, but they shape what production becomes. Story five. Cal .com going closed source is the argument everyone will want to have. Now for the spicy one. Cal .com announced that it is going closed source.

They said that after five years as open source champions, they are moving the production code base closed because AI has changed the security landscape. Their argument is basically this. In the past, exploiting an application required skilled humans. Time, and patience. Now AI can be pointed at an open -source codebase and used to systematically scan for vulnerabilities.

Cal .com says that being open -source is increasingly like giving attackers the blueprints to the vault. They are keeping an open version available as cal .diy under the MIT license. But they say that the production Cal .com codebase has diverged, including rewrites in areas like authentication and data handling. I have mixed feelings on this one. On one hand, I understand the concern.

This week's GitHub and Wiz story gives some support to the idea that AI can make vulnerability discovery faster and cheaper. If you run a SaaS product handling sensitive calendar data, authentication flows, integrations, and customer information, it is not crazy to worry about AI -assisted vulnerability mining. On the other hand, closed source equals safer is not a clean argument.

Closed source software still gets reversed. Closed source software still has bugs. Closed source software can lose the benefit of external review. And hiding code is not the same as securing it. So I would not frame this as cal .com is right or cal .com is wrong. The better framing is that AI is putting real pressure on the business and security model of commercial open source.

For maintainers, open source can mean transparency, community, contributions, and trust. For customers, it can mean self -hosting, auditability, and less vendor lock -in. For attackers, it can mean a searchable map of the systems. And for a commercial company, it may mean all of those things at once, plus support burden, vulnerability reports, hosted customer risk, and investor pressure. That tension is not new.

AI just makes it louder. For platform teams, the takeaway is not stop using open source. That would be ridiculous. The takeaway is stop treating project governance and licensing as background noise. When a project changes its source model, repo model, release model, or commercial posture, that is supply chain signal. You do not have to panic, but you should notice.

Story six, co -pilot usage -based billing makes AI a governance problem. Last main story. GitHub Copilot is moving to usage -based billing on June 1st. GitHub says all Copilot plans will transition to GitHub AI credits. Instead of counting premium requests, plans will include a monthly credit allotment, and usage will be calculated based on token consumption, including input, output, and cached tokens.

Base subscription prices are not changing, according to GitHub. Code completions and next edit suggestions stay included. But agentic usage, longer sessions, heavier models, and multi -step workflows move more directly into the cost model. And one detail jumped out at me. Copilot code review will also consume GitHub Actions minutes in addition to GitHub AI credits.

That is what makes this a platform engineering story. Because now AI usage is not just a seat cost. It is tokens, model choice, cached context, review automation. Actions minutes, budget controls, and admin visibility. And this is the normal cloud story happening again. First, it feels magical. Then everyone adopts it. Then the bill shows up. Then leadership asks, why is it growing?

Then teams discover they do not have tagging, budgets, ownership, quotas, or usage visibility. Then platform teams get asked to govern it after it is already everywhere. We saw this with cloud. We saw it with observability. We saw it with CI minutes. Now we are going to see it with AI -assisted development. I am not anti -copilot. I use AI tools constantly. But the economics are changing.

A quick autocomplete suggestion and a multi -hour agentic coding session are not the same cost profile. A lightweight code explanation and a repo -wide refactor are not the same cost profile. A human asking a few questions and an agent iterating across a codebase are not the same cost profile. GitHub is saying that more directly now. And teams should probably take the hint.

If AI behaves like cloud spend, you need cloud spend habits. Visibility. Ownership. Guardrails. Budgets. And somebody who notices before the bill becomes the incident. A few quick ones before we wrap. MinIO's main GitHub repo was archived by the owner on April 25th and is now read -only. The readme says the repository is no longer maintained and points users towards other options.

It also says that the community edition is now source -only and that legacy pre -compiled binaries will not receive updates. That is a big dependency signal if you have MinIO in your stack. Ghostty is leaving GitHub. Mitchell Hashimoto wrote that Ghostty plans to move off of GitHub while keeping a read -only mirror at the current URL. The interesting part is his framing. He says the problem is not Git itself.

The problem is the surrounding infrastructure that people rely on. Issues, pull requests, actions, and platform workflows. Git may be distributed, but your actual engineering workflow usually is not. Docker published a one -year reflection on Docker -hardened images. They say Docker -hardened images crossed 500 ,000 daily pulls with more than 2 ,000 hardened images, MCP servers, Helm charts, and ELS images.

That is a nice counterweight to the rest of the episode. Not every supply chain story is a compromise. Some of the work is making safer defaults easier to adopt. Azure DevOps added one -click CodeQL setup and org -wide alert triage to advance security. Microsoft says teams can enable code scanning at the repo, project, or organization level without manually configuring a pipeline.

And security admins get a combined alerts view across repositories. Not super flashy, but useful when security has to scale beyond fixing repo number 417 by hand. I think the human thread this week is that engineers are still emotionally treating developer tooling like it lives outside the blast radius. And it does not. The build system is not outside the blast radius. The merge queue is not outside the blast radius.

The package publisher is not outside the blast radius. The AI review bot is not outside the blast radius. GitHub Actions is not outside the blast radius. Copilot billing is not outside the governance conversation. The open source project that you depend on is not outside your supply chain just because you starred the repo five years ago. All of this is part of production now.

Maybe not production in the narrow sense, maybe not serving customer traffic at this exact second, but production in the operational sense. If this breaks, we cannot ship. If this is compromised, customers may be affected. If this gets expensive, leadership will care. If this silently does the wrong thing, production may become wrong later.

If this changes its licensing or maintenance model, our roadmap may get weird. That is production. And I think that that shift is hard because developer tooling used to feel like the comfortable part of engineering. It was where we moved fast, where we automated things, where we glued things together with tokens and YAML and a little bit of we'll clean this up later.

But the shortcut became the path, and the path became the control plane. And the control plane now has secrets, merge authority, release authority, billing impact, AI behavior, and platform dependency risk. That does not mean every team needs to panic. It means we need clearer labels. A GitHub Actions workflow that publishes to PyPI is not just CI. It is a release system.

A merge queue that lands code on main is not just a productivity feature. It is a source control control plane. An AI agent with repo access and secrets is not just a bot. It is a principal with tools. An open source dependency whose repo gets archived is not something to look at later. It is a dependency management event. And a co -pilot billing change is not just pricing. It is a governance signal.

Once you label the system correctly, you can manage it correctly. You can put the right permissions around it. You can monitor it. You can budget for it. You can patch it. You can separate trusted and untrusted input. You can stop pretending that the thing with keys to production is somehow not production. Reliability is not just keeping servers up. It is keeping the paths to production trustworthy.

And this week, those paths looked a little more serious than usual. Alright, that's it for this week of Ship It Weekly. Quick recap. GitHub patched a critical Git push remote code execution vulnerability. And the AI -assisted reverse engineering angle may be just as important as the bug itself.

Researchers showed prompt injection attacks against AI agents tied into GitHub workflows, which is a pretty clear reminder that agents are now in this CI -CD blast radius. Elementary Data dealt with a malicious CLI release published through a vulnerable GitHub Actions workflow. GitHub had a merge queue regression that affected more than 600 repositories and more than 2 ,000 pull requests.

Cal .com is going closed source and putting a spotlight on the tension between AI, security, open source, and business models. And GitHub Copilot is moving towards usage -based billing, which means AI tooling is becoming a cost and governance conversation, not just a productivity conversation.

Then in the lightning round, we hit MinIO archiving its main repository, Ghostty leaving GitHub, Docker hardening images crossing 500 ,000 daily pulls, and Azure DevOps making CodeQL setup and org -wide alert triage easier. Links and show notes are on shipitweekly .fm. You can also find the video version on YouTube. If this episode was useful, send it to the person on your team who still says it's just CI. I'm Brian, and I'll see you next week.

You know that moment when a platform stops sounding helpful and starts sounding serious? That's this week. Kubernetes is cleaning house. Gateway API keeps getting more official. AWS is moving on from Copilot. Airbnb is showing why production should not be your first alert test. And Cloudflare is reminding everybody that a bot with a token is still a principal with blast radius. Hey, I'm Brian Teller.

I work in DevOps and SRE and I run Teller's Tech. This is Ship It Weekly, where I filter the noise and focus on what actually changes how we run infrastructure and own reliability. Show notes and links are on shipitweekly .fm. If the show's been useful, follow it wherever you listen. Ratings help way more than they should.

We have five main stories today, then the lightning round, and we'll wrap with the human closer. We're starting with Kubernetes 1 .36 because this feels like one of those releases where the project keeps sanding off older sharp edges while pushing more production grade features into stable territory. Then Gateway API version 1 .5, which is basically SIG network saying the future is not just coming.

It is getting promoted into the stable path now. After that, AWS Copilot CLI end of support because this is a very real platform story. One easy path is aging out and AWS is pretty clearly nudging people towards the next preferred path. Then we've got Airbnb on alert development, which is probably my favorite SRE story in the set because it is really about how better feedback loops beat blaming culture.

And finally, Cloudflare on non -human identities. Because this is one of the clearest examples lately of security vendors saying out loud that scripts, agents, and third -party tools need to be treated like first -class identities, not side characters. Story 1. Kubernetes 1 .36 feels like a maturity release. Let's start there. Kubernetes 1 .36 shipped on April 22nd.

The release includes 70 enhancements with 18 graduating to stable, 25 entering beta, and 25 moving to alpha. That alone gives you the usual big release energy, but the more interesting part is what it says about where the project is spending its maturity budget. One of the clearest examples is service .spec. External IPs.

Kubernetes says that field is now deprecated, calls it as a known security headache, and points back to the long -running man -in -the -middle risk around CVE -2020 -8554. You'll see warnings now, and full removal is planned for version 1 .43.

Kubernetes is also permanently disabling the old Git repo volume plugin in 1 .36, saying that path is closed for good and existing workloads need to move to alternates like init containers or external Git sync style tools. That's why I like this story, because this is not just look at all of the new stuff.

It is Kubernetes continuing to act like a platform that wants fewer half -legacy foot guns hanging around forever. And honestly, that is what maturity often looks like. Not more knobs, fewer weird things everybody knows are sketchy, but keeps tolerating anyway. And there's another angle here that makes this release feel more grown up than flashy.

Kubernetes 1 .36 is also pushing more of the supply chain and packaging story towards cleaner primitives. The release highlights support for packaging read -only application data. Models, and static assets as OCI artifacts and delivering them to pods through the same registries and versioning workflows teams already use for container images.

It also calls out staleness migration work for controllers, which matters because a lot of weird controller behavior does not show up as a dramatic crash. It shows up as a controller acting on stale assumptions at the wrong time and doing the wrong thing very confidently. So the practical read for me is this.

If you run Kubernetes at any real scale, 1 .36 is the kind of release where you should not just skim the release nodes for what's new. You should skim them for what old thing are they finally done tolerating? And what newer path is now stable enough that we should stop calling it experimental in our own heads? Because that's usually where the real platform signal is.

Story 2, Gateway API version 1 .5, is the networking future getting more real. Next up, Gateway API. SIG Network announced Gateway API version 1 .5 on April 21st, called it the biggest release yet, and said the focus was moving existing experimental features into the standard, meaning stable, channel. The release promotes six features that people actually care about.

Listener set, TLS route, HTTP route, CORS Filter, Client Certificate Validation, Certificate Selection for Gateway TLS Origination, and Reference Grant. They also moved to a release train model, which basically means features ship when they are ready at freeze time instead of waiting for some perfect bundle.

That matters because Gateway API is no longer just a nice future -facing idea for people who like cleaner abstractions. It keeps becoming the actual road forward. Especially now that Kubernetes itself is pointing people towards gateway API in places where older patterns are being deprecated. And especially after all of the broader ingress and controller retirement pressure we've already been seeing this year.

So to me, the real read here is simple. The networking control plane in Kubernetes keeps getting more explicit, more standardized, and less willing to leave everything. In the controller specific magic plus annotations bucket forever. And some of the specific gateway API promotions are worth slowing down on for a second.

Listener set is interesting because it gives teams a cleaner way to contribute listeners to a gateway without forcing everything into one giant resource owned by one team. TLS route going stable matters because it makes the TLS pass through and terminate use cases feel more first class.

The CORS filter moving into the standard channel is also one of those small looking things that matters a lot in real life because it is exactly the kind of behavior people used to bury in controller specific config or app side workarounds. And the move to a release train model is its own kind of maturity signal too. It means the project is optimizing less for perfect bundling and more for predictable forward motion.

So if I'm a platform team, the takeaway is not just cool, more gateway API stuff. It is probably how much of our ingress and edge behavior is still trapped in annotations. How much of it is controller specific and how much of it now has a cleaner upstream shaped home we should actually plan forward. Story three, AWS Copilot is reaching end of support. And that tells you a lot. Now to AWS.

AWS says Copilot CLI reaches end of support on June 12th, 2026. It will remain available as an open source project on GitHub. But AWS says it will no longer receive new features or security updates from AWS. In the same announcement, AWS points users towards Amazon ECS Express Mode and AWS CDK Layer 3 constructs as the migration paths they want people evaluating.

This is one of those stories I like because it is not really about the tool alone. It is about the platform preference. Copilot was AWS's opinionated developer -friendly path for developing containerized apps on ECS and AppRunner. Now AWS is pretty clearly saying the newer opinionated paths are somewhere else. That does not mean Copilot users did anything wrong. It just means the center of gravity moved.

And that is a very real cloud platform lesson. Sometimes the easy path you picked was genuinely the right call at the time. Then the provider evolves. The preferred abstractions change. And now your job is not debating whether the shift is fair. Your job is planning the migration before the old path becomes operational debt with a calendar attached to it.

And there is a migration planning lesson here that I think gets missed when people hear end of support. Copilot did a lot more than just offer a nicer CLI. AWS says it used CloudFormation stacks for the app and service layers, which means a lot of teams probably have more Copilot -shaped infrastructure under the hood than they remember. So the right response here is not panic. It is inventory.

What workloads are still on Copilot? Which ones are app runner versus ECS? Which parts of the deployment flow are tied to Copilot convention? And whether the real destination should be express mode for simplicity or CDK L3 constructs for teams that want stronger IAC controls and customization. AWS is pretty explicit.

That Express Mode is meant to preserve a lot of the simplicity that made Copilot attractive, while CDK L3 is the more customizable path. That is the kind of thing I would actually say out loud to a team. Do not turn this into a philosophical debate about whether Copilot was good. Assume it was good for when you picked it. Now ask, what will be annoying to unwind later if you ignore the date now? Story 4.

Airbnb says alert pain was a workflow problem, not a culture problem. This one is probably my favorite in the whole episode. Airbnb says the issue with alert development was not that engineers did not care. It was that their observability as code workflow had a blind spot. Code review could validate syntax and logic, but not actual alert behavior against real -world data.

So production kept becoming the proving ground. Airbnb says they built fast feedback loops to preview, validate, and surface alert behavior before PR submission, cut development cycles from weeks to minutes, and used the workflow to help migrate 300 ,000 alerts from a vendor to Prometheus. They also say what used to take a month of iteration can now take an afternoon. This is such a good SRE story.

Because it is very easy to look at noisy alerts, weak alerts, or slow alert iteration and say, this is just a culture problem, or people just need to care more. Sometimes, sure. But a lot of the time, the workflow is just bad. If the only way to see whether an alert behaves correctly is to merge it and wait, then you built a system where production is the testing harness and on -call is the feedback loop.

That is not a motivation issue. That is a tooling issue. And Airbnb's fix is basically the kind of thing platform teams should love. Earlier feedback, more confidence inside the PR, less wasted iteration after the fact. And Airbnb made another design choice here that I really liked. They explicitly chose compatibility over novelty.

Instead of inventing some exotic, proprietary, alert analyst model, they took Prometheus role groups as the input. Used Prometheus's own rule evaluation engine, and wrote the results back out as Prometheus time series blocks, exposed through the standard query API. That is a really smart platform move because it means the preview and analysis system fits into the workflows engineers already understand.

Instead of becoming one more internal snowflake, everybody has to relearn. That matters because a lot of internal platform tooling dies, not because the idea was bad. But because the workflow becomes learn our special thing first. Airbnb's version sounds more like use the standards, use the real engine, show people the delta before they merge, and make the right thing easier than the lazy thing.

That is honestly a great pattern well beyond alerting. Story five. Cloudflare is saying non -human identity is now the real identity story. Last main story. Last time in episode 34, Cloudflare was talking about the network fabric for agents. This time, they're talking about the identity, token, and permission model around them. Cloudflare's framing here is very direct. Identities are not just people anymore.

They are agents, scripts, and third -party tools acting on your behalf. Their update packages that into three practical areas. Scannable API tokens, better OAuth visibility and revocation, and more granular resource -scoped RBAC. Cloudflare says new tokens are easier for scanners to recognize. Customers now get a central connected applications experience for OAuth access and revocation.

And resource -scoped permissions are available for more resources so both users and agents can be right -sized more tightly. They also say these scopes can be assigned through the dashboard, the API, or Terraform. And I think that this story matters because it is one of the clearer examples of the industry dropping the pretense. An agent with a token is not some cute helper. It is an identity with power.

A script with standing access is not background noise. It is an identity with power. A third -party OAuth app is not just a convenience. It is an identity with power. And once you accept that, the rest of the story gets more normal. Token scanning, connected app visibility, permission scoping, leased privilege. This is just IAM growing up around modern workloads and agent -heavy environments.

And I also like that Cloudflare is not treating this as some abstract future of security thing. The token changes are practical. They added a recognizable prefix and checksum so scanners can identify Cloudflare tokens with much higher confidence. The OAuth work is practical too. You can review the app name.

Publisher, requested scopes, and which accounts the app is asking to access before you approve it, and then see those connected applications later in one place to revoke them if needed. And their resource scoped permissions framing is probably the most useful mental model in the whole post. The token gets you in the building. But scopes should decide which rooms you can enter.

That is the part I think teams should take seriously. If you are letting agents, scripts, and third -party automations pile up with broadstanding access, you do not really have an AI governance problem. You have an IAM hygiene problem that got new branding. Okay, a few quick ones before we wrap.

Microsoft shipped April patches for Azure DevOps Server and says they strongly recommend staying on the latest secure version. The patch fixes a null reference issue that could break pull request completion during work item auto -completion, improves sign -out validation to prevent potential malicious redirects, and fixes PAT connection creation for GitHub Enterprise Server. That is a very practical PatchNow item.

Google also pushed more on OTLP metrics for cloud monitoring. Google says you can send metrics to cloud monitoring through a provider -agnostic OpenTelemetry pipeline, store that data in the same format as managed service for Prometheus, and query it through the same cloud monitoring interfaces. That is a nice observability standard story because it is not just support the protocol.

It is make the protocol path actually first class. I think the human thread underneath this week's episode is that a lot of engineering pain comes from waiting too long to make responsibilities explicit. Kubernetes is making some of that explicit by deprecating or removing paths it clearly does not want to keep carrying forever. Gateway API is making networking intent more explicit.

AWS is making platform preference more explicit by telling people where Copilot stops and where the next preferred paths begin. Airbnb is making alert quality less dependent on vague craftsmanship and more dependent on visible feedback before merge. And Cloudflare is making it harder to pretend that agents and scripts are somehow outside the normal identity and access conversation.

And this is where I think platform work gets misunderstood sometimes. People talk about maturity like it means more automation, more abstraction, more paved roads. Sometimes it does. But just as often, maturity is really about being honest sooner. Honest about which patterns are legacy. Honest about which workflows are broken. Honest about which tools are losing support.

Honest about whether production is secretly your only validation environment. Honest about who or what actually has access in your environment. That is not as fun as a big shiny launch. But it is usually where a lot of the real risk reduction comes from. Because the longer a team stays fuzzy on ownership, the more that fuzziness turns into toil. And the more that toil turns into staffing pain.

And the more that that staffing pain turns into reliability pain. Not because anybody is lazy. Not because people do not care. Usually just because too many systems stayed ambiguous for too long. So yeah, that's probably my biggest takeaway from this week. Better platforms do not just make things easier. They make certain kinds of vagueness harder to sustain. And honestly, that is usually a good thing. All right.

That's it for this episode of Ship It Weekly. Quick recap. Kubernetes 1 .36 and why it feels like a maturity release. Gateway API version 1 .5 moving more core networking features into stable territory. AWS Copilot reaching end of support and what that says about shifting preferred paths. Airbnb proving alert pain was a workflow gap. Not just a cultural issue.

And Cloudflare making the case that non -human identity is now a core security story. Then in the lightning round, Azure DevOps server patches and Google Cloud OTLP metric support. Links and show notes are on shipitweekly .fm. You can also find video versions on YouTube. If this episode was useful, follow or subscribe wherever you listen and send it to the person on your team.

Who keeps having to explain that reliability problems are usually workflow problems long before they became on -call problems. I'm Brian, and I'll see you next week.

A lot of teams say they want better CICD. What they usually mean is they want fewer weird failures, less shared state nonsense, less tribal knowledge, and way less time wasted fighting the delivery system itself. Because the problem is not just that Jenkins is old.

The problem is when your pipelines are noisy, Fragile, hard to reuse, hard to observe, and tightly coupled to a bunch of infrastructure decisions developers should not have to think about. And when you finally fix the problem, you get a different problem. Success. Because once delivery gets fast and easy, people start using it for everything.

Bots, maintenance changes, dependency bumps, mass rollouts, and now the real question becomes, how do you keep the system smooth when everybody finally trusts it? Like and subscribe! Hey, I'm Brian. I work in DevOps and SRE, and I run Tellers Tech. Ship It Weekly is where I filter the noise and focus on what actually matters when you are the one running infrastructure and owning reliability.

Most weeks, it's a quick news recap. In between those, I do interview episodes with people who have actually built things, migrated real systems, and learned what works the hard way. Today is one of those conversations. I'm joined by Stefan Moser from Pipedrive.

He helped lead a big move from Jenkins to GitHub Actions, built a self -hosted runner platform on Kubernetes, moved delivery towards GitOps with Argo CD, and helped roll that model out across a large internal estate with hundreds of services. And what I like about this one is it's not just tool talk. We get into why Jenkins had become painful, from groovy fiction to noisy neighbor problems on shared VMs.

Why GitHub Actions ended up fitting better. How reusable workflows and custom actions helped. Why they chose Argo CD over other deployment options. And how they had to build better internal observability because GitHub alone was not enough at their scale. We also talk about the migration strategy. Which honestly is one of the best parts.

Dogfooding first, migrating in batches, using internal teams as the first proving ground, letting the process get polished before pushing it wider, and building something self -service enough that teams eventually started migrating on their own. And then there is the mobile story, which is its own thing.

Mac minis, messy runner drift, different toolchains and the surprisingly practical path they landed on after testing a few different options for stabilizing mobile CI. If you care about CI CD architecture, platform engineering, GitHub actions at scale, or how to do a migration like this without setting your org on fire, this one is worth your time. All right, let's jump in.

Today, I'm joined by Stefan Moser from Pipedrive. He helped lead a big move from Jenkins to GitHub Actions and built a self -hosted runner platform on Kubernetes, plus a GitOps CD flow with Argo CD. And we're going to talk about what worked, what broke, and what's worth copying. Stefan, thanks for joining me. Thank you. Thank you for the opportunity to talk about this adventure.

Basically, it's not the first time I'm talking about this. I already had... Meetup session that was recorded on YouTube, two blog posts, and basically just sharing again these adventures. But now with a little sparking because something already after one year already changed. So I have more stuff to add to these blog posts and meetup that we had. Awesome. Well, I'm excited to learn more.

So starting off, can you give me a thesis? Why did Jenkins stop working for you, and what were you trying to optimize with this new system? So basically, the first problem that we had with Jenkins is Jenkins decides being the pipelines being written in Groovy. That was the first problem. In Pipedrive, we are mainly working with TypeScript and Go. Groovy was really not very needed for us, so it was a bearer.

For the DevOps teams that we had at that time, plus the engineers trying to do writing something. That was the first issue. The second issue was basically the setup. So we had VMs, and that VMs were not isolated. So that means if a big pipeline lands in one VM, and we had other pipelines like to build the Docker containers, then we had the issue with the nice...

Noisy neighborhood so basically we get a starvation of the resource so it's really not predictable we tried to to improve that so we had a crazy idea first to build a internal ci engine so that was really the most crazy thing they did so basically copy the idea of yaml from github's gitlab ci we build the engine and then start to work on that but then we had another issues People need to learn and use syntax.

And after GitHub Land launched GitHub Actions, people started using it, some developers started using it in their app stores. And then we were thinking, why not experimenting? And the first idea was basically, besides Jenkins, we had another product, CodeChip, that was to do the pull request validation. So every time the pull request is open, we had a typical lint and test work.

So basically linting to building the... Code, leading the code, and doing the unit tests. They were running in the code chip. And I can tell it was worse than Jenkins because every project was individual. So imagine if I had to change or to share something, it's very difficult. So we had an idea. Well, let's try to win the GitHub Actions. And basically, I team up with a colleague, Gregor.

And basically, it was the idea that... Let's replace CodeChip with GitHub. Actually, let's see what happens. And that was really the kickoff. So we get a green card to export idea. So the first thing was thinking that is, okay, I need to run this somewhere. Probably GitHub, even we had a free package, will not be enough. So let's go to find solutions for self -hosting.

We found, after some ideas, we found a community project. It is ARC, Actions Run Controller. At this moment, it's already belonging to or maintained by Gita, but at that time was really purely community -based. One thing that it was very... Important for us, or very convenient for us, it was a Kubernetes controller. So we spent the last years working with Kubernetes.

We are understanding the Kubernetes API, these CRDs, all talking in the Kubernetes way. So it was more easy to work with that. So we define resource, get runners running, fine. One thing that is also awesome, we can grab a pre -built image and then... And put that customization that you want. So basically, developer needs HGK, IKH, or different tooling. Just put it inside of that custom image and then we ship it.

It was even for easy. Then we already know how to restrict and monitoring Kubernetes in production. Just apply the same idea in the...

CI environment so basically I did a thing that is normally not normal standard so basically I don't I didn't set the requests to be the smallest possible most necessary no I just was brute force so I say for example set the request and the limits to be almost the same so that I could have one thing that is pretty bit in the So that every time a developer gets a runner, it's always the same CPU and memory.

So that avoids the problem of noisy neighborhoods. Then we had already ideas how to, we already know how to see and control the resource in Kubernetes. So apply the same metrics. So basically we bring back the, of the really tools that we had in production to.

The ci cluster but then we also did want to be more straightforward in in the process to maintain the clusters basically instead of building the cluster from scratch we use ets to be more easy and then we want to to have scalability in terms of the nodes we decide then to go to the new project at that time was carpenter so basically the the magic way of AWS to spawn nodes faster than a cluster of auto -scaling.

So basically, then we get this solution. So a controller that was listing the GitHub app books to spawn jobs, spawn when the instance creates a new pod that represents a runner. If that runner doesn't have a space, the carpenter then creates a new node.

And that was really bring the flexibility okay I just need to set the max number of thoughts or the runners and the ecosystem for the cluster scale up and scale down when when necessary and this was really basically the why we we moved to uh two changes and first steps we then basically did the migration of code chip to github actions at that time also was released the reusable workflow so that means we are able to

Reusable to create a workflow and spread that workflow to the old repositories so reducing the the repetition that we had and the manual configuration we had with Jenkins with the code chip and Basically, after we dropped out of CodeChip, it was the time that we had the opportunity to revamp CI -CD.

So we bring a group of engineers, I think four engineers plus a developer experience product manager. So basically, we had a product manager dedicated for the developer experience. And then we decided to revamp.

Uh necessity and they basically said to put some kind of competition in in terms of tooling so I remember that the first contender was basically github actions in terms of ci but then also we bring argo workflows and tecton because were two projects that I was curious about using And in terms of deploying, I'm thinking Argo CD plus Flux.

We even tried with Spinnaker, but it was so messy to spin up that system that we just dropped it. And how we chose, basically, first was...

Readability or feasibility or the easiness to create the workflows and the specific customizations so and that is really shine for us the kit of actions basically people complain about the action but really the fact that the actions are written in javascript and then we are using typescript so it's natural to create custom logic with typescript even then we can go to creating composite actions with some more for some batch scripting.

Scripts, we just use that language to create the customization. And then it was really easy to pack up. So basically, we create action, we have a package, like a package, and then can put it a block that we can put in different place. And then the other point was basically about how we can expose or show the workflows to the developers. And that is... The less clicks the developer does, find lots better.

And that, of course, makes the GitHub action to be a big content. So the fact that the workflows was next to the repository, the execution was next to the repository, the developer didn't need to switch to the platforms to see was really a big plus. Yeah. And that's really the reason why we went to... To GitHub Actions. Even at some point after this migration, what we had, it's basically a monorep of GitHub Actions.

And I think we have 50 or 60 GitHub Actions. So basically, we just build our actions and even it's our monorep of all the code base that you want to reuse in the organization. In terms of development, of deployment, one thing that we want to make. Clear is that we want to avoid push code.

So one of the issues that we had always is that with Jenkins and this push idea is that fact that I need to have a runner in the clustered push code. That means that I need to give the Kubernetes credentials to a runner to be able to write stuff. And that means that in some way I can try to capture that runner and do malicious stuff.

Let's do the reverse so that is that the cluster is really in isolation and we get what we have it's someone inside that check something that it's really it's already trusted and use it to um to apply change in the cluster and that is really why you want to go to the to do githubs and of course in githubs we have two two tools to choose argo cd or flags yeah And in that case, it was basically one of the reasons why Argos CDF had a good UI comparing with Flux at that time.

It was like two, three years ago. So that was really the big reason that we picked up with Argos CDF Flux for the UI. And then we had already a way to show to developers what has happened in a more easy way.

Because again, that is that it's easy for developers to visualize doesn't mean that I allow them to change manifest with rcd yeah I know what I allow is to basically scope in inside of the application saying okay this is your application see your status you can go see the status of your resource you can see the logs if you need it you can see if the content the pod is it's restarting or not really um we added that idea.

Yeah, you can show. Yeah, after that idea, I think that picking up the GitHub Actions took me like two days to create MVP of the deployment flow, basically because I was reusing everything that I already did in the CodeChip migration, and we decided to have that MVP, and then... See what are the gaps.

So one thing that I forgot to mention is that initially in the first step or first iteration that we had with migrating to CodeChip, we had a problem with a lack of observability in terms of actions. GitHub at that time didn't provide any statistics about the actions. So if you want to know how long was the runner, we have any idea. You can just... See at the repository and was not good enough.

But especially in our organization that we already have, at that time we had like 700 service. We have architecture of microservice plus libraries. It was impossible to track at the repository level. So what we did is basically we create a service that lists these events and then starting putting everything inside of a database so that could build.

A source that we can then do queries and then find uh what are the the performance of of the the workflow because at some point a manager a director or even cto can comes to us it says what is the failure rate of these uh workflows where what are the workflows that are taking or these guys what are the deployment flows that are taking more time what are the steps that we need to to optimize and then basically this is really our gut feeling that we need that information and we are storing that information.

And until now, we're still using that service to track, even now to tracking more billing information, but it's really the source of truth. Okay, that was the phase one and phase two, phase three. It's basically making the product ready for the consumption. And one thing that we discovered that we lack, it's basically, again, organization -wide of the visibility.

GitHub actually is great to see your workflows, to see the organization. It's bad because we don't have anything. So we create a service. That is basically a registration service plus a UI that we have in our back office. So basically we have a UI in our back office that shows all the deployments, but they need to have some source.

And basically we create a service that consume events that we are producing in our workflows. And after consuming these events can do the mapping of what it's app. And basically we have basically our own interpretation of the events of. Because they have a different interpretation of what is deployment, what is a deploy, what is a region.

And then basically we add that internal logic or that internal understanding what is build, test, unit test, functional test, end -to -end test, deploy to a region, deploy to a... Deployment at a whole scale. And we have that understanding inside of a service.

And basically this was initial, the service that was consuming everything to create observability of what we need to do or what we have in terms of deployments to production. And after that, we have that, after having that UI, we are more confident to go. Now goes the question, how you do this kind of release? 700 service. I think we have five teams or 10 teams at that time. It was really a big migration.

How you start migrations? I think like any product should migrate or should be released, you first try it yourself. So basically... And we decided to migrate our own service. My team doesn't only produce CI, CD. We have a couple of service that supports the deployment flows and support the developer experience. So for example, we had one service that creates and do all the lifecycle of remote dev environment.

So basically we have remote dev environments that developers can use that it's a replica of production. And then we have a system that creates that. For example, this is one of these servers that we need to migrate. And that is really basically like this. So I have scripts, I have workflow, I have documentation to do it. So I deliver to my teammates so that my teammates can migrate. These bring two good things.

First, we test our process to someone that doesn't know about the process. And second, we allow... To share knowledge. So basically, knowledge that was restricted to that five people that create systems starting to be spread to the team because I'm sharing that knowledge by forcing the people to do the migration. And basically, it was basically this alpha grouping. Then we had a different team.

So basically, it was a platform team that it's more experienced than the normal developers around stuff with Kubernetes. And the deployment process and they have more out of the box or out of the standard service, then we go to them and say, okay, we have this Polish workflow. Let's now teach you, do a session and then allow them to migrate.

And then it was basically like the open beta of the system, of the migration. Again, the idea was to... Make everything more polished, more clear for the users. After they give us our feedback and then we improve the workflows, define, we start doing to think how we do all the role. In this case, we pick up the prioritization that SRE team define for the service. So basically SRE define tiers.

So tier one is a service that's very critical. Tier two, it's more or less. And then tier three is less critical. So we decided, okay, let's split this in batch. Let's go team by team, starting in tier three. Then they can move to tier two and then tier one. And that was the idea. And then the plan was basically split it in batch and assign a DevOps engineer to each batch, not to do the...

The migration but to be the assistant so basically the person who does the introduction of the new way ui the new workflow and then this is the the the interaction goal that we have to to do your mind and also to make them so basically if I want everyone one person that's responsible to achieve a goal of migrate x number of of service until the end of the month it will keep the momentum in the team and then we

Started doing that so team by team we started moving we have a schedule we have a limit number of of engineers so basically we have a queue until some moment we had guys from the middle of the queue at the end of saying okay I already saw the other guys doing migration.

I think I can do it alone. Okay, just check the recording that we had. Try yourself. Yeah, and they decide to try alone and starting doing migration alone. So basically, the system was already so well oiled and everything was moving smoothly. They were able to migrate alone. And that's really, I think, the real sex story of doing these migrations by batch.

Doing your dog food and all these things in a way that makes everything automatic or with less human intervention to allow people to use it. And then you also can think this is really the idea of maybe platform engineering, making everything self -service. When you have a product that's already usable for a normal product engineer that doesn't need to have the context of the CICD and how to deploy to Kubernetes.

It's a service that he can use and can work and can do his work autonomously. And yeah, that's then after five months, I think we migrate everything. Yes, we found what we get as issues, basically our success starting break things. So the process was so smoothly. The deployment flow was so good. That then we're starting an introduction of bots to look at deployment for the maintenance tasks.

So basically, the SRA team developed a service to create pull requests to adjust the resource usage in production and then automatically deploy that to production. So basically, it creates a pull request, we introduce a set of resources, and this pull request is already pre -approved, so it moves by all the normal pipeline flow and goes to production. Then we have the PandaBot.

Creating pull requests for the pendants and in some case the developers were already confident enough in the unit test they have in the functional test they have allowed that obligations or updates of of dependence to go without any approval from a human so we had then a situation that we have multiple um departments happen and then was struggling or doing impact in our system. Even we can grow, grow, grow.

Then we add some bottlenecks. At that point, we decide to improve that service that was the deployment registrar to have a queue. So the idea is that you then have a way that you, when you start the deployment, you send event, I have a deployment. And then what happens is that the deployment registrar register that deployment and put in the queue.

Evaluate the queue size so basically we define that we have the number of 50 deployments per available in parallel and also we do a tricky thing that is we only allow 10 of that queue to be used by bots so basically we always want to have some kind of free space for the moments to develop that develop features to ship and basically that's a and so we have a queue with all the idea to put a margin for humans.

And then if everything is okay, then we rerun the workflow that allows to execute everything. Yeah, it was basically that thing that we had to improve. Also, one of the bottlenecks that we had was we had issues with how we commit to an environment state. So basically, it was one of our... Really bottlenecks is when you have multiple deployments, you need to be, only can commit once, one at a time, the deployment.

So basically, we had a strategy that we create one commit per region. So we have a lot, a pile of commits to push to that, this environment site repository. And that was really a bottleneck. Fortunately, we were not very clever at that time to create the queue for that specific step. And then we had to re -engineering all the process because we had the issue that we didn't do like a FIFO.

So basically, we did implement a hot hobby. And that means that some guys was very unlucky that it was the first one to arrive and was not the first one to get their stuff deployed. Yeah. This is really the story that we have. What was missing at that time was in the presentation and blog post was really the migration to the mobile team. I don't know if you want to go to already that story.

It's a new story that I have. Or you want any other questions? Let's actually get to the mobile story in a second. So going back over your CICD process, I'm curious. First off, are you worried about The GitHub change to private self -hosted runners, they're changing the pricing model. So I guess they're going to charge for self -hosted as well now. Is that going to change your implementation at all?

That's the problem. I can think that if they're starting to charge that, then I need to start to thinking, if they start charging like that, then I need to really be very picky in their SLAs.

So I they how they can charge me a fee for the control plane when their control plane it's not really 99 or doesn't yeah meet the slas it certainly hasn't been lately that's for sure that that is really the thing even even we don't even we get that uh we more or less exclude that from our matrix of of um I already starting to think already seeing some page that shows the SLA in the last couple of months.

And then I'm starting to thinking if you charge me for this, I need to have a better SLA. I cannot stop working. And then for the size of what we have, it's basically or I need to find a different CI tooling and going to that path or even can go more crazy. Don't forget that, okay, we are a special case, or probably not a normal case. That is, we are already in enterprise level.

And we have GitHub hosted enterprise server. So basically, you can bring yourself. And if this is starting to be expensive, probably that's the thing I need to do. I bring down and then try to be less dependent of... The ability of the cloud version and then be my concern. And then I can then point to myself. Yeah, your GitHub is down because of myself and not because of some change in the cloud version.

But yeah, it will be really a concern. And if they force us, it's basically now I will go to my legal team and then check the SLA and let's see if they don't like the SLA. They're starting to get some notice from our legal team to get a recharge back or something. So also you had mentioned, that's a fair statement for sure. You had mentioned looking at Flux originally and then going with Argo.

Are you using anything like Crossplane with Argo yet or no? No. So basically the fact that we don't use Crossplane is basically because it's not in our domain of work. Okay. From what I understand, really, the idea is that cross -plane is a good way to provision a resource using Kubernetes as a native language.

Even I try to understand from Victor, Victor Werczek, that's working with cross -plane, the difference between Terraform and cross -plane, and that's really the reason that... We didn't go deep in crossfire because the main player of Terraform and provision infrastructure is the infrastructure team, not the engineering excellence department. Basically, we are the middle layer. Imagine it's like a lasagna or a burger.

Basically, the team is the lowest band and I am the lettuce. And then we have even up on me, we have the burger and then we have the tomato and they have the other. So basically we have.

These layers and I'm in that layer that I consume service from infrastructure team and then I deliver service to to to the to the platform team and to the to the developers so that's the reason that we don't look for crossplane that's thing that I would like to to experiment but then I need to really have a good case of why did a terraform to use crossplan yeah it's more I guess if you want the infrastructure

Definitions closer to the actual service right so if you want that all defined together and there's there's pros and cons for both ways I would say honestly even in our organization we kind of or I've worked in organizations where we've done both yeah even even listen uh what victor said about the idea of cross plan and how to use cross plan and that year for example is one of the examples that I can tell or I can get a model in terraform that's getting a postgres to be from Mother bless.

But probably if I want to talk with the developer, developer wants to have the minimal settings to change. So basically, I think the WinCross plan can abstract that with their internal resources and say, okay, define this YAML manifest or that YAML resource or, sorry, that CRD in YAML and then the controls and everything will set up everything for you. So at this moment, we don't add this kind of...

Need in terms of organization to have exposed so much the infrastructure to developers. And that's real. Without need, I don't have a way to force a tech to be used. Oh, that's fair. It's always a balance. Yes, it's cool, but I need to really, it makes sense to the case. So tell me about this, the new mobile deployments. How is that going? And how did you set that up?

The mobile team was using Jenkins, but even with more strange setup. So basically, they had a farm of Mac minis, and they basically reconnected with the Jenkins controller master. And they were using that. So basically, they were using groovy, pretty groovy with...

In their checking pipelines plus with fast line for one for the ios team the android team will use a different tooling and that was really that massive because part of the team needs to be operations to understand to upgrade the nodes to fix issues and also again the same issue of isolation when we had cases that the nodes were not very identical and a job was if lands in one machine was passing if lands in the other

Machine was failing or less force between artifacts between was screwing up with node versions hubi versions and was really a mess so lack of productivity and then that was really the idea We need to move to GitHub actions, but plus also find a way to make their compute power or compute resource be very stable, having the same identity that we had in the CI -CD for the microservice.

So that was a multiple deployment and isolate. And at that point, it was really a surprise for me, the end solution, but basically it was this case. So I went to research. That basically I put 4Ks on top of the table. First, using VMs inside of Mac. So basically a product from CircleCI. So basically a company that's already providing GitHub Action Runners for Macs and have the system to work.

And basically it's on top of the one tooling at the start. And basically it's a... A nice tooling to spawn VMs inside of Mac. Then the idea, okay, let's try to use Nix for some independence in isolation. And the other idea was basically also to use AWS. So why not spawn Macs in AWS and use it as a GitHub address? And at last I was thinking, I need to at least think in the way of outsourcing this to a company.

This guy is GitHub Actions, but... I could also think to other GitHub action providers or first to GitHub itself and then to other providers because you can see that Blacksmith and I think Depot are providers of GitHub action trends that you can offload and don't depend of GitHub to have the best performance in your machines. Basically, it was really a poor scientific research. I have three, four hypotheses.

I have one month to test it. And I set like one week for each hypothesis and then try to go. And this is really the first. It was in summer of last year. And also that culminate in the appearance of the... AI native mindset. So in this experimental research, after I collided with teachers, with tooling, I found that I had to use start to create VMs. So first hypothesis.

And then I would decide, well, if I try to create a controller, like I have the idea or already the use case that to have a control for Kubernetes, to run runners. So why I don't have an action runner control for that? And that was really the idea.

Starting to think I did a poc with shell script because it was a very easy command but then I decided okay I have a nice shell script let me do a really nice mvp and then I did my first specification development project so basically I used this idea I then signed to define my specifications in a markdown file what I want, what were the toolings, what were the constraints.

And then use, in this case, was already still using GitHub Copilot and say, I have this idea in this file, let's make a plan. And starting elaborating the plan, creating the plan. And then the process was really that way that after I have the plan, we need to have to -do lists or to -dos for each point. And then basically I force the Copilot to use, go for each to -do or each step. Do the implementation, I review it.

Okay, it's fine. Let's move to the next one. Fine. And then also ask to do a summary of each implementation. So basically to have a history of what I did. Just important for me, but also important to share with the team all this process. And then after one day and a half, I got a control. So basically I had a Mac mini in my desk. I put the control there and it was spawning. And doing the lifecycle of the VM.

So basically, a workflow, it's my runner, execute a niche, drop, tear down the VM, start a new one, register against GitHub, like normal flow. And I will say, yeah, nice. I just need to make some improvements. Then I moved to the Nix. AI, in this case, also it was Copilot, helped me a lot. How to build the recipes with Nix, but it fails tremendously just because of the way Xcode works. It was very annoying to work.

And then I had the issue of how to distribute Xcode. At that time, I was running out of time. AWS was not really an option to investigate. And then I started doing some calculations about cost if I use GitHub as a provider of headers. And surprise, surprise. It was cheap for our use case. So it was basically a matter of after spending three or four months collecting metrics in Jenkins, I say, yeah, we can use it.

And basically it was really the idea. I said, okay, this is the amount of money and comparing the working hours that it's necessary for an engineer to fix this, it's a good balance. And then after I convinced my director that it really makes sense in terms of financial terms.

You approve and then we move and then we move to migration and that this migration was I did with the junior the first thing that we did when this migration was really sit down with the developers and I asked anoint us to to to my junior in this case it was to internities sorry but it was really important to that yeah I asked him go for each um jenkins pipeline they have and start doing a flowchart so basically we

Had a flowchart for each Jenkins pipeline with steps and the steps in the way that what is supposed to do and what are the commands that are executed and then I sit down with the with each uh team from android and and ios and then let's go forward for each step and then try to understand as this makes sense this flow I don't care about what is the command that is executed.

Does this step make sense? Does this test make sense? Does this fork in the logic make sense? And then we also understand some good things that is some workflows or some checking job that we have was already redundant. We could refactor the input and combine in one single pipeline. That was really the idea. And then that was the second time we used the AI to speed up.

And this time I already added the session of agenting coding with my engineering department. So basically the engineering excellence create agenting sessions to teachers all to use AI tools in a more agentic way. So not to auto -complete features, but to give context, to give a goal. Have this guy, this AI, as really a partner to execute. And then at this time, already we was using cloud code.

And then, okay, let's bring these flowcharts, convert to something that is more digestible by AI. So I was able to export as a CSV file. And then, okay, these are CSV files, contains flowcharts of our workflows. Let's build GitHub Action workflows. And documentation. And he's starting doing the old workflow. Was not really exactly what we want, but was close enough.

Imagine that this was really a good best draft, a good first draft. This was really, now just adjust this step, this step, this step, and then we just starting building on that. And then, of course, this really make the work very easy. So we add like 17 workflows to migrate. What is the difference from this migration to the other migration? The other migration, we add like one or two workflows for all the process.

So basically, the issue was to replicate that to use workflows to 700 service. In this case, we have only two service, two repositories, the Android and iOS, but we have multiple workflows to migrate. And then we had to rewrite a lot of stuff. And that was the way that we use AI to basically at the pace of each day we migrate a workflow and then cloud code was able to digest some part of the code base.

So I had two issues. First, parts of the customization that they have in the iOS team was using FastLine that is written in Ruby. I don't know Ruby. So I used it to understand what was that.

What was the logic behind that ruby scripts and had extra features then I'm not very well versatile in the ios test and compiler so I never I'm not irs developer I don't know all the quirks about the the process to compile language compile ios app And I use protocol for that. So basically, it was already in a way that I had an issue in my pipeline. And I tell them, OK, I have problems like this.

I have an issue in my pipeline. This is the idea of the workflow. This is the idea of job. This is a step that's failing. Fetch.

Using github cli in that time even we using mcp probably it's best if you have a cli to tell the ai model to use that cli to fetch the information instead of uh bloat the contacts with mcps use the cli fetch the logs in that section and that let's go investigate what it's what it's failing and it was really able at some point I get get surprised because when I go in deep mode of troubleshooting the model in this case

Even the the agent that is called cloud cloud was able to go to the internet and find github issues about the problem pointing out I have found this issue probably it's about this let's double check and then I double check yeah probably makes sense let's try this change and it was really basically it was a more family language or I cannot what I can say it was like while we were in the past doing Googling.

So you put the problem, you try to find the issues. Here, I have the problem. Also find the issues in the internet and get me back the information and then validate with me and then explore. And basically, after four weeks, we migrate everything. We even add extra features they want. And they were very happy. And still, they are very happy. Of course, the initial costs failed because it was more than I expected.

But for one reason, developers were delivering more. So it was really a situation that the workflows and run are so stable, they can focus more in future and then increase the cost. But it's because they are shipping more features than they did in the past. That's a good cost problem to have, you know? Yeah. Yeah. I think that's the thing I have. Do you have any questions about that topic?

I did want to ask, wrapping up, if someone that's listening, they wanted to pull off a big CICD switch like you did, are there some lessons learned that you could give that they could follow so they don't set their org on fire? I mean, because this is a complex... Lift and shift, right?

Going from Jenkins to GitOps and introducing Argo and all the complexities around, you know, the Mac mini pipelines, like be interested. There's some like core lessons learned that you could impart. So the first thing is, I think you need to, even you have a big pipeline, I bet that you have a small part. So a niche. So try to find. A small part that you can replace and do it in isolation.

In my case, it was basically the pull request validation. It's detached in some way of the big flow. Try on that. If you cannot do that, try to find segmentations that you have in your organization in terms of teams. That's a good way to approach so that you can move parts of your team. Basically, if you have 10 teams or five teams, pick one team and try to go in that way.

So this is a way that you can try to reduce the buster radius. And of course, I think that was a thing that was very important for us, basically doing dogfood. I think it's very unfair for someone that is developing tools for developers not using that in their work. So I think this is really the most important thing. It's doing dogfooding.

And then if you want, try to find in each place if you don't have anything think do you have internal tooling that doesn't uh provide for your final customers that can be bad but for trying to apply this to your internal tooling yeah that makes sense cool where can people find your uh your posts and and where can they reach out to you okay so my posts are in the medium probably you can find them have the links in the description also I have that both was based in the talk that I did.

So also in YouTube, I will have that talk. Also, sometimes I do some publishing in LinkedIn. So you can go there, try to reach me in LinkedIn. Awesome. I'll leave the links for your Medium posts and your LinkedIn and anything else in the show notes. Stefan, thanks for coming on. Really appreciate it. Okay. Thank you. All right. That's my conversation with Stefan Moser.

My biggest takeaway from this one is that good CICD is not just about picking a newer tool. It is about building a delivery system that is predictable. Observable, isolated, and usable enough that engineers can trust it without needing constant help from platform teams. That is really the thread running through this whole episode. They did not just swap Jenkins for GitHub Actions.

They reduced noisy neighbor problems. They standardized runners. They leaned into reusable workflows. They moved deployment towards GitOps. They built their own visibility layer when the platform was not giving them enough, and they rolled it out in a way that let teams build confidence instead of forcing a giant overnight cutover. I also liked that he was honest about what happens when the new system works.

Once deploys get easier, people use them more. Bots start shipping changes, automation starts piling up, and then you discover the next bottleneck, whether that is queuing, fairness, or protecting enough room for humans to still get work out. That is the real platform lesson. Success creates new load.

And the better your self -service story gets, the more you have to think about throughput, guardrails, and the system behavior under trust. The other part I liked was his migration advice at the end. Start with a niche. Reduce blast radius. Dog food your own system first. And if you are building tools for developers, use them yourself first before asking everyone else to bet on them.

That is probably the cleanest takeaway from the whole episode. If you enjoyed this episode, follow Ship It Weekly wherever you listen to podcasts. If you want the show notes, links to Stefan, his write -ups, and the resources we talked about, head over to shipitweekly .fm. Thanks for listening, and I'll see you later this week. Thank you.

A lot of infrastructure work gets easier right around the moment it gets more opinionated. Private connectivity becomes a product. Cluster networking becomes a managed default. Ingress migrations stop being optional. Observability config starts acting like a real standard. And the old we'll patch that next sprint stuff still shows up in the lightning round waiting to ruin somebody's week. That's the theme this week.

The platform layer keeps absorbing work teams used to hand roll, babysit, or quietly postpone. Sometimes that is great. Sometimes it is the platform telling you that your old defaults are running out of runway. Hey, I'm Brian Teller. I work in DevOps and SRE, and I run Teller's Tech. This is Ship It Weekly, where I filter the noise and focus on what actually changes how we run infrastructure and own reliability.

Show notes and links are on shipitweekly .fm. If the show's been useful, follow it wherever you listen. Ratings help way more than they should. And if you want more signal between episodes, check out oncallbrief .com. The latest brief this week was heavy on breaking changes, security patches, and platform releases, and that definitely shaped this episode.

We have five main stories today, then the lightning round, and we'll wrap with the human closer.

We're starting with aws interconnect going generally available because this is a pretty clean signal that aws wants private connectivity to feel a lot more like a managed cloud primitive and a lot less like a telecom product then cloudflare mesh which feels like the private network for humans services and agents version of the same broader trend after that gitlab 19 .0 because the move away from bundled nginx ingress

And bundled data services is exactly the kind of breaking change platform teams actually have to plan around.

And that one came straight out of this week's on -call brief. Then we've got EKS auto mode networking and what it means when AWS owns more of the cluster networking path for you. And finally, OpenTelemetry declarative config reaching stability, which is quieter than the others, but honestly kind of foundational. Story one. AWS interconnect is cloud networking getting productized harder. Let's start there.

AWS interconnect is now generally available and the core pitch is pretty straightforward. AWS is offering managed network connectivity in two directions, multi -cloud connectivity between AWS and other cloud providers and last mile connectivity between AWS and your on -prem or enterprise sites. AWS says that the service is turnkey.

Private and designed to remove the usual infrastructure complexity from the customer side. The part that stood out to me is not just the feature list. It's the framing. This is AWS taking something that traditionally felt like network engineering plus vendor coordination plus waiting and trying to turn it into a cloud service with a cleaner control surface. Multicloud starts with Google Cloud.

Azure is coming later this year and the traffic stays on private backbones instead of bouncing over the public internet. The last mile option launches in the US with Lumen. That matters because private connectivity has always been one of those areas where cloud teams and networking teams can end up speaking slightly different languages while the delivery timeline drags on forever.

So when AWS starts saying, we'll manage more of this path for you, that is not just a product launch. That is a platform boundary shifting. And I think the practical takeaway is pretty simple. If you run multi -cloud, hybrid, branch -heavy, or edgish environments, this is the kind of thing worth paying attention to early.

Not because you need to adopt it tomorrow, but because it changes the default expectation of how painful private connectivity is supposed to be. Story 2. Cloudflare Mesh is basically saying private networking should include agents now. Next up, Cloudflare Mesh. Cloudflare launched Mesh as a private networking layer for users, nodes, agents, and workers.

Their pitch is that the client on private networks are no longer just humans and services. Now they're also autonomous agents that need scoped access to internal APIs, databases, and other private systems without exposing those systems to the public internet. And honestly, I think that framing lands.

Because this is one of the first times I've seen a big networking security vendor say out loud that agent traffic is not just a cute add -on to existing access models. Cloudflare is treating it as a first -class access pattern.

Mesh plugs into Cloudflare 1, applies existing gateway, access and posture policies and ties into workers vpcs so workers durable objects and agent workloads can reach private infrastructure directly the interesting part is that this is not being sold like a traditional vpn replacement story it is more like your private network should already know how to deal with services laptops remote servers and agent style

Workloads on the same fabric Cloudflare also says Mesh is many too many private networking over its global network, not a collection of one -off tunnels glued together.

So the bigger read for me is this. Networking vendors are starting to assume the future client is sometimes a human, sometimes a workload, and sometimes an agent acting semi -autonomously. If that assumption keeps spreading, private access tooling is going to look a lot more like policy -driven platform plumbing and a lot less like old -school remote access.

Story three, GitLab 19 .0 is what real platform migration pressure looks like. Now to GitLab. This week's on -call brief highlighted GitLab 19 .0, and I think it is a really good platform story because it is not flashy at all. It is just the kind of change that forces actual planning. GitLab says 19 .0 includes 15 breaking changes.

And one of the big ones for self -managed Helm users is that bundled Nginx ingress is being replaced by Gateway API with Envoy Gateway as the default. GitLab says Nginx ingress hit end of life in March 2026, though it can still be explicitly re -enabled until removal in 20 .0. That alone is enough to matter. But then there is the second punch.

GitLab is also removing bundled PostgreSQL, Redis, and MinIO from the Helm chart and operator path, with no replacement, citing licensing, maintenance, and image availability issues. This is the exact kind of story that feels just operational until you are the team that has to sequence the migration. Because these are not cosmetic defaults.

These are parts of the install path that people absolutely built around, especially in test, POC, or smaller self -managed setup. And now the platform is basically saying that convenience path was temporary. So the takeaway here is not just GitLab changed some things. It is that platform teams need to notice when a project's default architecture starts growing up past its old bundled assumptions.

Gateway API is not just a new thing to learn. For some teams, it is now the road forward whether they wanted a migration project or not. Story four, EKS Auto Mode networking. Is AWS trying to make cluster networking feel less handmade? Next up, EKS Auto Mode. AWS published a good breakdown this week of how EKS Auto Mode handles networking. And the story is really about ownership.

Auto Mode sets up the VPC CNI for you. Manages DNS as a core component, supports node level DNS caching, and lets you request ALBs and NLBs through native Kubernetes resources without a separate load balancer controller. AWS also says it handles CNI lifecycle management as part of the cluster maintenance. That is a pretty meaningful bundle of responsibility.

Because cluster networking is one of those areas where teams can spend years carrying around a bunch of, well, this is just how our cluster works, setup that is really a mix of controllers, tuning, upgrades, and networking assumptions nobody wants to touch during business hours. What AWS is doing here is making a stronger argument that for a lot of teams, that should stop being custom glue. Pods get VPC IPs directly.

Traffic follows normal VPC route tables. You use AWS native networking services and tools you already know. And AWS is taking more responsibility for keeping the CNI path current and sane. Now, that trade is not going to be for everybody. Some teams want the flexibility. Some teams need the knobs. Some teams have enough scar tissue to be suspicious anytime a managed mode says trust us. Fair.

But for a lot of orgs, the real risk is not lack of knobs. It is the pile of half -owned networking glue they already have. Story 5. OpenTelemetry declarative config getting stable is boring in the best way. Last main story. OpenTelemetry announced that key portions of its declarative configuration spec are now stable.

That includes the JSON schema, YAML file representation, in -memory model, parsing and creation operations, and the OTEL config file environment variable used to point SDKs at a config file. The blog also says implementations are currently available in C++, Go, Java, JavaScript, and PHP, with .NET and Python underway.

I really like this one because observability configuration has had a lot of it depends energy for a long time.

Environment variables everywhere slightly different mental models across languages lots of power not always a lot of consistency so when open telemetry says the declarative config model is stable and should increasingly be treated as a first -class ux surface that is real platform maturity they even say the future process should be declarative configuration first and that older environment variable patterns that do

Not interoperate well may get deprecated over time that does not make for a flashy headline But it does make life better for platform teams that want more consistency across languages and services without every team inventing its own telemetry setup philosophy.

And honestly, that is a very DevOps story. Not exciting in the launch demo sense, but very exciting in the maybe we stop relearning config drift in five languages sense. A few quick ones before we wrap. Container D pushed security releases across supported branches, including 1 .7 .31, 2 .0 .8, 2 .1 .7, and 2 .2 .3 to address CVE -2026 -35469.

The release notes describe it as a SPDY stream issue and call out hardening around sanitization errors before returning them over gRPC to prevent possible credential leaks in pod events.

Patch this one github also launched code security risk assessment for organizations github says org admins and security managers can run a free assessment to review code vulnerabilities across their org and the docs say it scans up to 20 repositories and reports severity and autofix eligible findings that is a pretty decent quick win tool if you want a fast read on exposure without building a whole campaign around it And AWS published guidance on secure AI agent access patterns using MCP.

Their framing is that IAM has to become the authorization layer for these non -deterministic systems because coding assistants and agents choose tools at runtime and can act at machine speed. That is a good reminder that it's just an assistant stops being true the second it has real permissions. I think the cleanest closer for this one is pretty simple. Infrastructure keeps moving from assembled to opinionated.

Private connectivity gets wrapped as a managed service. Private networking gets redesigned around workloads, agents, and not just people. Kubernetes install paths grow out of old bundled defaults. Cluster networking becomes more provider -owned. Observability config starts trying to look like an actual stable interface instead of a pile of toggles. And that shift cuts both ways. It can absolutely reduce toil.

It can also remove some of the comfortable ambiguity teams used to hide inside. Because once the platform gives you a clearer default, you have to make a more conscious decision if you want to keep doing things the old, harder way. That is the human part of ops that shows up over and over. A lot of teams say they want simplicity.

What they usually mean is they want someone else to own the painful complexity without taking away the escape hatches that they have grown attached to. Sometimes that works. Sometimes the industry just moves on and your migration project shows up whether you invited it or not. All right. That's it for this week of Ship It Weekly. Quick recap.

AWS interconnect going GA and pushing private connectivity further into managed service territory. Cloudflare Mesh building private networking for users, workloads, and agents. GitLab 19 .0 forcing some real migration planning around gateway API and bundled services. EKS auto mode networking making cluster networking less handmade.

And OpenTelemetry declarative config getting stable in the kind of boring, foundational way that usually matters a lot later. Links and show notes are on ShipItWeekly .fm. You can also find the video versions on YouTube. And if you want more signal before the episode, check out OnCallBrief .com. If this episode was useful, follow or subscribe wherever you listen.

And send it to the person on your team who keeps having to explain that managed does not always mean simple. But it usually does mean the default architecture is changing whether people noticed or not. I'm Brian, and I'll see you next week.

Picture this, not some vague future where AI might matter to security someday. I mean right now. What if the real shift is not that AI can write code? What if the real shift is that it can read a messy environment faster than your organization can even decide who owns half of it?

Old services, stale images, forgotten runners, broad IAM roles, weird sidecars, ancient dependencies, a dashboard nobody meant to leave exposed. A pipeline permission nobody wanted to untangle. That is the frame for today. Hey, I'm Brian from Tellers Tech and this is Ship It Weekly. You can find full show notes, links, and past episodes at shipitweekly .fm.

And if you've been getting value out of the show, subscribe wherever you listen and leave a rating or review. It really does help. Today is a special, just one topic, Claude Mythos Preview, Project Glasswing, and what this might mean for business, security, and the way we think about DevOps. Cloud, infra, and the platform risk if this is even half as meaningful as it looks. So let's start with the obvious question.

Why is this worth a special? Because this does not feel like a normal model launch story. Anthropic is not framing mythos like, hey, the coding benchmark went up again. They are framing it more like a threshold event for security. And that gets my attention.

Not because I want to do the usual AI doom spiral thing, but because if offensive capability really is getting faster and cheaper, then that changes the operating assumptions underneath a lot of how companies run software. That is the part I care about. Not the hype cycle, not the branding, not whether people on LinkedIn are going to overreact for two weeks.

I care about what this means if you run systems for a living. If you are the person dealing with cloud sprawl. Aging workloads, risky exceptions, strange identity boundaries, rollout safety, patch friction, and all the messy stuff real companies carry around for years. That is why this is worth a special. Because this is not really an AI story. It is an infrastructure risk story. It is a tempo story.

It is a story about what happens if exploit discovery starts moving at machine speed while most engineering organizations are still patching, approving, and modernizing at human speed. And before we go deeper, I want to do a callback to our OpenClaw special for a second. That episode was really about what happens when Agent Tolling stops being a toy and starts acting like a control plane. Real tokens.

Real permissions. Real blast radius. That was the point. This Mythos story is different, but it rhymes. OpenClaw was about what happens when we give software more autonomy inside messy environments. Mythos is about what happens when the capability to find weaknesses in those environments and chain those weaknesses together starts scaling faster too.

So I would not call this a sequel to OpenClaw, but I do think it lives in the same broader theme. Software is getting more capable. Attack surfaces are getting weirder. And the control planes around us are not always as sturdy as we pretend they are. So what actually happened? Anthropic released Claude Mythos Preview, not as some big open general release, not as a fun playground feature.

It paired it with something called Project Glasswing, which is clearly being positioned around defensive cybersecurity and critical software protection. That alone tells you a lot. Because release posture matters. Companies tell on themselves a little bit by how they ship things. And when the release pattern looks more like we should probably be careful with this, that is worth paying attention to.

Now, I want to be honest here. A lot of the detailed evidence right now still comes from Anthropic itself. So this is not one of those stories where we have 20 independent teardown articles and a full year of real -world data. We do not. And that means you should not just swallow every claim whole. But I also think the dumb response is to roll your eyes and call it marketing and move on.

Because even if you discount the loudest version of the story, the direction of the story is pretty clear. These models are getting better at the kind of patient, iterative, technical reasoning that matters a lot in security work. Reading code, testing ideas, trying paths, rejecting dead ends. Following weird clues, chaining small mistakes into bigger outcomes. That is not a toy capability.

That is exactly the kind of work defenders and attackers both care about. And this is where I think the smart posture sits. Do not get gullible, but also do not get lazy. Those are two different mistakes. Getting gullible means repeating every claim like it is already fully proven and independently validated. I do not think this is reasonable.

Getting lazy means hearing something uncomfortable and dismissing it because it sounds too dramatic or too early. I do not think that is smart either. The better middle ground is this. Maybe the headline version gets softened over time. Maybe some details get revised. Maybe some of the early framing turns out to be too strong. Okay, fine.

But if the core trend is real, and I think it probably is, then the implication is still serious. The implication is not that AI suddenly invented security problems. The implication is that it may be making parts of offensive security cheaper, faster, and more scalable. And once that starts happening, messy organizations pay for it first. That is the key. Messy pays first.

Not necessarily the company with the most code. Not necessarily the company with the fanciest stack. The company with the most weak assumptions, slow cleanup, fuzzy ownership, and chainable mistakes. That is who pays first. So let's talk about what changes if this is even half true. Not all true, half true. What changes? Well, for one, platform debt becomes attacker leverage.

That old base image you have not refreshed? That broad deploy role nobody wanted to narrow. That weird internal service that still trusts way too much. That forgotten runner. That weird path with too much access. That long -lived secret. That temporary exception that turns two years old. Those things do not just sit there as background ugliness anymore.

They become easier to discover, easier to reason about, and possibly easier to combine. And that combination part matters. Because most real incidents are not about one perfect movie villain bug. They are about a chain, a little access here, a weak boundary there, too much trust somewhere else, an old dependency, a stale assumption, a permission that never got revisited. That is how real environments break.

So if a model gets better at following those threads with patience and speed, the problem is not just vulnerable code. The problem is vulnerable operating models. That is the line that I keep coming back to, not vulnerable apps. Vulnerable operating models. Because a lot of companies do not have a pure software problem. They have a systems problem.

And this is also why this is not just for AppSec people or weird infra nerds like me. Business leaders should care too. Because if exploit discovery gets cheaper, then risk math changes. Vendor risk changes. Cyber insurance assumptions change. Patch service level exceptions change. Acquisition diligence changes. Third -party component trust gets more fragile. Legacy systems become more dangerous to leave alone.

And once you get into regulated industries or older enterprises, this gets even more interesting. Because a lot of those environments are not clean. They are layered. Modern things on top of old things. Cloud native things next to brittle things. Shared dependencies. Inherited architecture. Historical compromises. Basically, every mature company has some version of this.

Which means the real business question is not, should we be scared of mythos? The better question is, are we still operating like offensive capability is scarce, expensive, or relatively slow? Because if that assumption is weakening, then defensive speed becomes more valuable. Inventory becomes more valuable. Rollback confidence becomes more valuable. Privilege reduction becomes more valuable.

And isolation becomes more valuable. That changes how you think about maturity. It stops being polish. It becomes resilience. Now let's bring this back to our world. What does this mean for DevOps, cloud, platform, and infra teams? Honestly, it means a lot of the boring stuff just got more important. And I know that is not the sexy answer, but it is the real answer.

If you run platform or infra, your job is not just uptime and throughput and deployment velocity. Your job is also making it harder for mistakes to connect. That is the lens. Make it harder for mistakes to connect. That means looking hard at identity boundaries. Do we have roles that are broader than they need to be? Do our pipelines have access they should not have?

Can one compromise in one place turn into access somewhere much more sensitive? That means looking at credentials. Where do long -lived credentials still exist? Where are secrets exposed to more systems than necessary? Where are we still relying on convenience over containment? That means looking at old compute and images. How stale are our base images? How stale are our AMIs?

How many things are running because nobody wants to touch them? How many self -hosted utilities are basically just squatting in the environment and hoping nobody notices? That means looking at network boundaries and trust assumptions. What internal services are reachable too broadly? What dashboards, admin surfaces, or support tooling have drifted into being more exposed than anyone intended.

What old assumptions about internal means safe are still hanging around. And it means looking at rollout safety because patching speed only matters if patching is survivable. If every patch feels like a mini heart attack because rollout is fragile and confidence is low, then your organization is going to move too slowly exactly when slow movement starts costing more. That, to me, is the DevOps angle.

Not AI will replace security engineers, no. This is more like the value of disciplined infrastructure and safer defaults just went up. I keep coming back to this phrase because I think it is the practical takeaway. Boring work just got promoted. Inventory, golden paths, patch automation, safer defaults, ephemeral credentials, cleaner ownership. Faster rollback. Less standing privilege. Better dependency visibility.

Tighter separation between build, deploy, secrets, and runtime. That stuff is not glamorous. Nobody wants to do some chest -thumping keynote about cleaning up permissions in some ugly deployment path. But this is exactly the kind of work that matters if the cost curve on offense is moving. Because when offense gets cheaper, defenders do not win by being more dramatic. They win by being less sloppy. Less trust.

Less drift. Less ambiguity. Less hidden blast radius. This is not as fun to talk about as something big AI changes everything narrative. But it is the part that actually helps. So here's the uncomfortable takeaway. For years, security people have said defenders have the hardest job because defenders have to protect everything and attackers only need one path. That asymmetry is not new.

What might be changing is the cost of finding that path. That is the part worth sitting with. If finding and chaining exploitable paths gets cheaper, then the old enterprise habit of living with moderate mess becomes a worse bet. We'll get to it next sprint. We should clean that up this quarter. Nobody really owns that, but it still works. We know the role is broad, but tightening it would take time.

That thing is ugly, but touching it is risky. Those are all normal enterprise sentences, and they may be getting more expensive. That is really what this episode is about. Not panic, not hype, not AI fanfiction. Just a sober possibility. The exploit timeline may be collapsing faster than the enterprise cleanup timeline. And if that is true, then platform maturity is not cleanup work anymore.

It is frontline security work. That is the shift. So if I were talking to a platform team or engineering leadership right now, I would not say panic. I would say go look where your environment is embarrassingly chainable. That is the phrase. Embarrassingly chainable. Not just what has a critical CVE. Not just what your scanner painted red this week. Look for where mistakes connect too easily.

Can someone go from an internet -facing thing to an internal thing too easily? Can someone pivot from CI to secrets too easily? Can someone inherit cloud access they never should have had? Can one weird old service become a bridge into something much more important? Do you actually know what is exposed? Do you actually know what is privileged? Do you actually know what would be painful to patch fast?

That is where I would start. And then I would ask the harder question. What cleanup work have we been treating like optional that maybe is not optional anymore? Because let's be honest. Most companies do not need a frontier model to tell them their permissions are too broad, their dependencies are stale, their ownership is fuzzy, and their patching process is slower than it should be. They already know that.

The real question is whether the cost of leaving that mess alone is going up. I think it probably is. So to wrap this up, mythos might turn out to be a true inflection point. It might also turn out that some of the early narrative gets revised as more evidence comes out. Those things are not mutually exclusive.

But if you work in DevOps, SRE, cloud, or platform engineering, the useful takeaway is pretty much the same either way. Assume your messy edges matter more. Assume your platform maturity is security work. Assume rollback, repair, and isolation matter even more than before. And assume we'll get to it later is getting riskier. That is the lesson that I would keep. Not fear, not hype, just this.

If offensive capability is speeding up, then defensive maturity has to stop being treated like optional cleanup. The full show notes and links for this episode are on shipitweekly .fm. And if this episode got your brain moving a little bit, subscribe to the show wherever you listen. Follow, rate, review, share it with somebody who runs infra, security, or engineering. That stuff really does help.

I'm Brian from Tellers Tech, and this has been Ship It Weekly. Thanks for listening, and I'll see you next time.