Ship It Weekly Transcripts

Hey, I'm Brian and this is Ship It Weekly by Tellers Tech. This week is pretty stacked for anyone running Kubernetes or building internal platforms. We've got Kubernetes officially retiring Ingress Engine X, CNCF, tightening up what platform engineering actually means and a new Kubernetes AI conformance program that lines up with a bunch of SRE in the age of AI conversations.

Then I'll hit a few quick links worth bookmarking and we'll close with a short piece on fixation during incidents and how that messes with our thinking. Let's start with the big one if you run clusters Ingress Engine X. Kubernetes maintainers have announced that the community Ingress Engine X project is being retired. The official Kubernetes blog and follow -up posts spell it out like this.

Ingress Engine X is moving to best effort maintenance until March, 2026. After that, there will be no new releases, no bug fixes, and no security patches. The manifests and code will still be there, but you're on your own if any new vulnerability or bug shows up. The reason they give is pretty straightforward. SIG Network and the Security Response Committee want to prioritize the safety and security of the ecosystem.

Keeping such a widely used Ingress on life support without enough dedicated maintainers is a risk, so they're drawing a clear line and telling people to move before that date. If you're running Kubernetes, you probably already know why this matters. Ingress Engine X is one of the most common Ingress controllers out there. It shows up in old blog posts, helm charts, getting started guides, everywhere.

For a lot of teams, it's just assumed to be the default. So what do you do with this? First, don't treat March 2026 as we'll worry about it then. You need real runway for this migration. This is not just swapping an image tag. You're picking a new edge component and threading that change through your environments. There are a few broad paths you can take.

You can move towards Gateway API based solutions and treat this as a chance to modernize your traffic routing story. You can adopt one of the vendor controllers that support Gateway properly, or if you're already using a cloud provider's ingress, load balancer integration, you might lean further into that and simplify.

The right answer depends on whether you want portability, deep integration with your cloud, or something more like a full -blown API gateway at the edge. Either way, I'd frame it like this internally. This is a real time -boxed infra -migration with risk and planning required, not a background refactor.

You want a plan for where you're migrating to, how you're going to test behavior and rules, how you'll roll out by environment, and how you'll roll back if things go sideways. Also, if you have clusters owned by different teams or old clusters that nobody touches because they just work, those are exactly the places that are going to bite you when Ingress Engine X is out of maintenance.

It's a good excuse to inventory where you're using it and pull those into the same migration plan. The bigger lesson here is that Kubernetes is not just the API server and Kubelet. The ecosystem around it has a life cycle too. You can't treat community projects as permanent infrastructure. They age, they lose maintainers, and they get retired.

All right, let's zoom out from Ingress and talk about platform engineering in general. The CNCF just published a new What is Platform Engineering post that's worth a read, especially if your team already has platform in the name. Their definition is pretty close to how a lot of us have been using the term, but it's nice to see it formalized.

They describe platform engineering as a discipline focused on building and maintaining software development platforms that provide self -service for developer teams. In other words, your job is to give product teams a coherent way to provision infrastructure, deploy, test, observe, and operate their apps without each team reinventing that stack from scratch. A few things they emphasize.

Platform teams should be reducing developer cognitive load, not adding to it. They talk about internal developer platforms, golden paths, paved roads, and treating the platform as a product with clear users and feedback channels. Compliance and policy enforcement are built into the platform, not bolted on as a separate gauntlet that devs have to run at the end.

What I like about this writeup is that it gives you language you can point at when your platform team is really just a rebranded ops team doing tickets and fighting fires. If your day to day is mostly people open a Jira, we click buttons in the console. That's not what CNCF is doing. Describing here. They're talking about a team that builds and evolves a product. Internal APIs, templates.

Pipelines, and tooling that developers can use themselves. You can use this in a few practical ways. If you're trying to justify time to work on self -service, developer portals, or opinionated templates, you now have a reference that says, this is not a vanity project. This is literally what this discipline is meant to do.

If you're inheriting a mess of unstructured Kubernetes, Terraform, and CI builds, you can point at the CNCF definition and say, here's what platform engineering actually looks like, and here's the gap between that and what we have. And if leadership wants platform engineering because it's a buzzword, this is a nice way to align them on what that implies. Roadmaps.

UX, and internal customers, not just more infra people. They also recently published a top five hard earned lessons piece from Kubernetes experts that lines up nicely with this. It talks about life cycle pain, upgrade complexity, and how that lack of guardrails and policies burns teams over time. That's basically the problem space that platform engineering is trying to address.

Now let's take that platform story and connect it to where the industry is clearly heading. AI running on all of this. CNCF has now formally launched the certified Kubernetes AI conformance program. This came out of KubeCon North America and has been picked up in places like Forbes and other coverage. The idea is pretty simple.

Everyone is trying to run AI and ML workloads on Kubernetes now, but every environment is slightly different. Different operators, different GPU scheduling strategies, different ways of handling storage and networking for models. The conformance program defines a shared set of capabilities and configurations that a platform needs to meet to be considered. AI conformant on Kubernetes.

It's similar in spirit to the existing Kubernetes software conformance program, but focused on AI workloads. The goal is portability and predictability. If a vendor or platform is certified, you should be able to run common AI frameworks and workloads there without a ton of custom glue. For platform and SRE teams, there are a couple of implications.

First, AI and ML workloads are no longer pet projects off to the side. They're becoming first class citizens on your clusters and in your CI CD. You're going to deal with GPU capacity planning, noisy neighbors, model deployment pipelines, and data movement as real operational concerns. Second, standards like this give you something to anchor on.

Instead of every team building their own ad hoc pattern for running models, you can say, we want our platform to meet this conformance, or use the checklist as input to your own design. In parallel with that, there's a good article on DevOps .com titled SRE in the Age of AI, what reliability looks like when systems learn.

It talks about how SRE is shifting from guarding mostly deterministic systems to working with adaptive learning systems where behavior changes over time. Traditional SRE practices like SLOs, incident response, and postmortems still matter. But now you have extra dimensions, model drift, data quality, and feedback loops. You're not just measuring latency and error rates, you're worrying about correctness.

Bias, and how often the model does something unexpected. So if your org is doing AI, I'd be asking, who owns the reliability of those workloads? Is it the ML team, the platform team, or SRE? How are we observing model behavior, not just pod CPU? And are we going to align to something like this AI conformance program, or are we comfortable having a one -off AI setup for each team?

Big picture, all of this says the platform you build now has to support both normal services and AI workloads and the reliability story has to keep up with that. All right, that's the big three. Let's hit a few quick links worth saving. Quick lightning round. These are things you might want to throw into your read later queue or share with the team.

First one is from the CNCF top five hard earned lessons from the experts on managing Kubernetes. It's a short piece, but it reinforces what a lot of us already know. Most of the pain isn't in the initial cluster build, it's in upgrades, dependency sprawl, and lack of guardrails. Good thing to hand to leaders who think Kubernetes is set it and forget it.

From SRE Weekly, there's a nice database migration story from Tynes about zero downtime migrations. What I like there is they actually define what zero downtime meant for them and admit where they accepted some degradation. It's a solid template if you're trying to socialize realistic expectations around a big DB change.

There's also a piece about upgrading Postgres SQL with minimal downtime at something like 20 ,000 transactions per second. If you own Postgres in production, this is a good case study on planning, replication, and rollback.

And finally, there's a really interesting article on building a distributed priority queue on top of Kafka from the They talk about supporting different SLOs for different events and how they added a proxy layer to avoid head of line blocking within partitions. If you're using Kafka heavily and starting to bump into everything has the same priority, it's a worthwhile read.

I'll drop links to all of those in the show notes. Let's finish with something on the human side of incidents. In SRE Weekly issue 497, there's a piece by Lauren Hutchstein titled, Fixation, The Ever -Present Risk During Incident Handling. The core idea is simple, but important. During incidents, we tend to latch onto a single theory, a single plan. Or a single mental model of what's going wrong.

Once we're locked in, we ignore or downplay signals that don't fit that story. You've probably seen this, right? Someone says it's DNS or it's the database early in the incident. And from that point on, every log line and every graph gets interpreted as evidence for that. Even when it doesn't really fit, or the first mitigation we try becomes the plan and we keep pushing on it long after it's clear it's not working.

Lauren's point is that fixation is a normal human response under stress, but it's dangerous for incident handling. The way out isn't to pretend we won't do it, it's to structure our process so we catch it. A couple of simple things you can do. Make it someone's job in the incident to periodically ask, what else could this be? And what evidence would change our mind about the current theory?

It sounds basic, but actually saying that out loud gives people permission to raise alternate explanations. Also, be explicit when a hypothesis fails. Instead of silently shifting to a new theory, call it out. We tried X, it didn't change the symptoms, so we're going to park that and consider Y. That keeps the team from unconsciously dragging the old theory along.

The broader theme, which ties back to the outages and platform work we talked about earlier, is that incidents are not just technical events. They're also snapshots of how we think under pressure. How we reason about risk and how we handle uncertainty and whether our process helps or gets in the way.

I'll put a link to that fixation article and also the ongoing trade -offs and incidents as landmark piece from Fred Hebert in the notes if you want to go deeper on this. All right. That's it for this episode. We talked about the retirement of Ingress Engine X. And why that should trigger a real migration plan, not a to -do buried in confluence.

We walked through CNCF's updated framing for platform engineering and how it lines up with building actual internal platforms instead of just rebranding ops. And we looked at the new Kubernetes AI conformance program and some of the thinking around SRE in the age of AI and where models and data pipelines are part of the reliability.

In the show notes, I'll include links to the Ingress Engine X retirement posts and migration guides, the CNCF platform engineering blog, and the AI conformance program, the DevOps .com SRE in AI article, plus the SRE weekly links we talked about. If you found this useful, share it with someone on your team who's thinking about platform work or planning cluster changes.

I'm Brian and this is Ship It Weekly by Teller's Tech and I'll see you in the next one.

Hey, I'm Brian from Tellers Tech, and this is the first episode of Ship It Weekly, a quick rundown of what this show is supposed to be. Most weeks, this is going to be a short practical recap of what happened in the DevOps. SRE and platform engineering world. Stuff like big cloud changes, notable incident write -ups, useful new tools, and the occasional culture or burnout topic.

Think a couple of main stories, a quick tools and releases segment, and maybe one human thing at the end. This first episode is going to be a little different. The last stretch has had a few really big outages from providers we all depend on. Cloudflare had a global issue that broke a bunch of sites. AWS US East One had a long regional incident and GitHub had a major hiccup where core Git operations stopped working.

So instead of a grab bag of topics, this one is more of a focused special on those outages and what they say about how we design our own systems. Going forward, the default will be more typical news rundown, but I'll probably come back to this kind of themed episode whenever something big like this happens again.

All the links and source material will be in the show notes if you want the full timelines and technical details. Alright, let's start with Cloudflare. On November 18th, Cloudflare had a bad morning and took a decent chunk of the internet with it. If you were online, you probably saw it in one form or another.

Lots of sites and apps that sit behind Cloudflare started returning Cloudflare error pages instead of real responses. Big names, smaller sites, random government and financial pages, all showing variations of we can't reach the origin or something went wrong. Cloudflare has since shared enough detail to understand the shape of it. The issue started with a configuration file they generate to manage threat traffic.

That file grew much larger than they expected, and that pushed part of their internal traffic management software into a failure mode. Once that system went sideways, they couldn't reliably process requests for a lot of customers until they got it back under control. They were pretty clear there's no sign this was an external attack.

This was normal complex system at scale behavior, config and software interacting in a way that wasn't caught ahead of time. From our side, the interesting part isn't Cloudflare is unreliable because everybody at that scale has incidents. The part that matters is how many of us quietly treat our CDN and WAF layer as if it can't fail.

If all of your HTTP traffic goes through one provider and you have no way around them, then your uptime is effectively pinned to their uptime. A couple of questions this should raise for your own setup. If your CDN is down for two or three hours in the middle of the day, what happens to your users? Do you have a way to temporarily route some traffic directly to your origin?

Is that just a theoretical DNS change, or is it a documented, tested step you've actually walked through? And do you have monitoring that makes it obvious the CDN layer is broken but our origin is fine, or would you be piecing that together from user reports and a status page? Cloudflare frontends a huge percentage of the web. Incidents like this are not rare in the big picture.

They're good prompts for a realistic, CDN is down, tabletop exercise with your team. Let's move from the edge of the internet into the cloud itself and talk about AWS US East 1. Back on October 20th, AWS had a major incident in the US East 1 region. Depending on which analysis you read, it impacted well over 100 AWS services and lasted somewhere in the 14 -15 hour range.

That's not a brief blip, that's most of the workday. A lot of well -known companies reported knock -on effects, slow or failing requests, backends timing out, and internal tools misbehaving. If you look at AWS's own summaries and third party breakdowns, you see a combination of issues inside key subsystems. Services that monitor and manage other services had problems.

There were DNS resolution issues inside the region and the control plane APIs people depend on to manage resources were degraded or error prone for long stretches. The important bit for us is how that intersects with the way people talk about high availability. And disaster recovery. A lot of teams quite honestly stop at, we're spread across multiple availability zones in the US East one, so we're good.

That helps if a single data center has a power problem or a localized failure. It does very little for you when the whole region is unhealthy in ways that touch both data plane and control plane. The second pattern that shows up in postmortems and social posts is the backup plan. You see some version of, if US East 1 has trouble, we'll just redeploy to another region with Terraform or CloudFormation.

But in an event like this, the very APIs those tools rely on are also degraded. So your recovery plan assumes the control plane is perfectly usable at the exact moment AWS is saying, we're having issues with the control plane operations in this region. A few questions to think about in the context of your own systems.

Do you have anything running in another region right now, even in a scaled down form that could serve as a fallback, not theory, but actual workloads you can point to? Could you provide some kind of degraded experience from that secondary region without building it on the fly in the middle of an incident? And have you ever walked through a full failover?

And fail back end to end when things were calm so you know what breaks, how long it actually takes, and who needs to be involved. I'm not saying everyone should be active active everywhere. That's not realistic for most teams.

But if you call a system mission critical, and it only exists in a single AWS region with untested, we'll spin it up elsewhere docs, incidents like this are a pretty strong signal that's not a Now, let's shift from user -facing outages and regional issues to something closer to home. GitHub. On November 18th, the same day as the Cloudflare incident, GitHub had its own major problem.

According to their status updates and multiple tracking sites, they started investigating failures on all Git operations in the early evening of UTC. That meant push, pull, clone, over both HTTP and SSH were failing or timing out. A bit later, they also called out degraded availability for code spaces. It took roughly an hour before they reported recovery.

From a development and operations perspective, that touches a lot of things at once. CI systems that do a fresh clone from GitHub every run will fail. GitOps tools like Argo CD or Flux that continuously sync from GitHub will stop updating. Developers trying to push a fix for some other outage can't get their code up.

And if your only CI system is GitHub Actions, those workflows are either delayed or completely blocked. So while GitHub going down doesn't look like a classic production is down incident, it absolutely can turn into one because it stops you from changing production at the exact moment you might need to.

Most teams don't have a playbook for GitHub is unavailable in the same way they have a run book for ServiceX is unhealthy. A few things worth considering. Do you have read -only mirrors of your most important repositories anywhere else? Another Git provider, an internal mirror, anything? Even a simple periodic mirror of your infra and core app repos can make a difference.

Can your CI run from a cached copy of the repo and existing artifacts for some period of time? Or is every pipeline hardwired to always pull from GitHub's live API? And if GitHub Actions is your only pipeline engine, do you have any backup, even if it's slower and more manual, or is the default we simply wait.

None of this has to be perfect, but your critical repos and pipelines deserve to be treated as part of your reliability story the same way your databases and load balancers are. Now that we've looked at Cloudflare at the edge, AWS in the region, and GitHub in the development loop, let's zoom out and talk about the pattern. All three of these incidents point at the same basic reality.

We are heavily dependent on a small set of external providers that we treat like background infrastructure. The CDN or WAF in front of us, the primary cloud region we run in, and the platform that hosts our code and pipelines. When they have issues, they expose assumptions in our designs and in our runbooks. The root causes are also pretty typical for large complex systems.

A configuration file grows larger than expected and interacts badly with software that wasn't written for that case. Internal health or management services fail in surprising ways and drag down other components. Service operations inside GitHub stumble and suddenly Git operations don't reliably work. Nothing exotic. Just scale and complexity doing what they do.

For me, the practical move here is not to panic about the cloud being fragile. It's to get very explicit about where your external single points of failure are and then improve a couple of them in a concrete way. For each major provider you depend on, ask two questions. If this provider is impaired for a few hours in the middle of the day, what exactly breaks for our users and what breaks for our ability to respond?

And what is the specific set of steps we would take with the tooling and people we have right now? Then pick one of those areas and move it a step forward. That might be putting a minimal but real footprint in a second AWS region for your most critical services and exercising it on a schedule. Documenting and testing a simple path to temporarily bypass the CDN for some subset of traffic if the edge is misbehaving.

Or mirroring your key repos and adjusting CI so you're not entirely dependent on a single Git provider for every build and deployment. You don't need a perfect answer for every scenario, but these kinds of outages are useful pressure tests and they're good leverage when you're trying to justify reliability work to people who only see the cloud marketing slides.

That's it for this first episode of Ship It Weekly by Tellers Tech. Today was a bit of a special, mostly focused on three related outages and what they mean for reliability and architecture. Going forward, the normal format will be a little more mixed.

Most weeks, I'll cover a couple of main stories, some quick mentions of tools, or releases that are worth a look, and usually one culture or career topic that's relevant to DevOps and SRE work. We'll still do focused episodes like this when something big happens, but you can expect more variety week to week.

If this was useful, feel free to share it with someone on your team who thinks about incidents, DR, and platform design. Show notes will have links to the Cloudflare incident explanations, the AWS US EastOne analysis, and the GitHub outage history, so you can dig into the details if you want. I'm Brian. Thanks for listening, and I'll see you in the next one.