Hey, I'm Brian from Tellers Tech, and this is the first episode of Ship It Weekly, a quick rundown of what this show is supposed to be. Most weeks, this is going to be a short practical recap of what happened in the DevOps. SRE and platform engineering world. Stuff like big cloud changes, notable incident write -ups, useful new tools, and the occasional culture or burnout topic.
Think a couple of main stories, a quick tools and releases segment, and maybe one human thing at the end. This first episode is going to be a little different. The last stretch has had a few really big outages from providers we all depend on. Cloudflare had a global issue that broke a bunch of sites. AWS US East One had a long regional incident and GitHub had a major hiccup where core Git operations stopped working.
So instead of a grab bag of topics, this one is more of a focused special on those outages and what they say about how we design our own systems. Going forward, the default will be more typical news rundown, but I'll probably come back to this kind of themed episode whenever something big like this happens again.
All the links and source material will be in the show notes if you want the full timelines and technical details. Alright, let's start with Cloudflare. On November 18th, Cloudflare had a bad morning and took a decent chunk of the internet with it. If you were online, you probably saw it in one form or another.
Lots of sites and apps that sit behind Cloudflare started returning Cloudflare error pages instead of real responses. Big names, smaller sites, random government and financial pages, all showing variations of we can't reach the origin or something went wrong. Cloudflare has since shared enough detail to understand the shape of it. The issue started with a configuration file they generate to manage threat traffic.
That file grew much larger than they expected, and that pushed part of their internal traffic management software into a failure mode. Once that system went sideways, they couldn't reliably process requests for a lot of customers until they got it back under control. They were pretty clear there's no sign this was an external attack.
This was normal complex system at scale behavior, config and software interacting in a way that wasn't caught ahead of time. From our side, the interesting part isn't Cloudflare is unreliable because everybody at that scale has incidents. The part that matters is how many of us quietly treat our CDN and WAF layer as if it can't fail.
If all of your HTTP traffic goes through one provider and you have no way around them, then your uptime is effectively pinned to their uptime. A couple of questions this should raise for your own setup. If your CDN is down for two or three hours in the middle of the day, what happens to your users? Do you have a way to temporarily route some traffic directly to your origin?
Is that just a theoretical DNS change, or is it a documented, tested step you've actually walked through? And do you have monitoring that makes it obvious the CDN layer is broken but our origin is fine, or would you be piecing that together from user reports and a status page? Cloudflare frontends a huge percentage of the web. Incidents like this are not rare in the big picture.
They're good prompts for a realistic, CDN is down, tabletop exercise with your team. Let's move from the edge of the internet into the cloud itself and talk about AWS US East 1. Back on October 20th, AWS had a major incident in the US East 1 region. Depending on which analysis you read, it impacted well over 100 AWS services and lasted somewhere in the 14 -15 hour range.
That's not a brief blip, that's most of the workday. A lot of well -known companies reported knock -on effects, slow or failing requests, backends timing out, and internal tools misbehaving. If you look at AWS's own summaries and third party breakdowns, you see a combination of issues inside key subsystems. Services that monitor and manage other services had problems.
There were DNS resolution issues inside the region and the control plane APIs people depend on to manage resources were degraded or error prone for long stretches. The important bit for us is how that intersects with the way people talk about high availability. And disaster recovery. A lot of teams quite honestly stop at, we're spread across multiple availability zones in the US East one, so we're good.
That helps if a single data center has a power problem or a localized failure. It does very little for you when the whole region is unhealthy in ways that touch both data plane and control plane. The second pattern that shows up in postmortems and social posts is the backup plan. You see some version of, if US East 1 has trouble, we'll just redeploy to another region with Terraform or CloudFormation.
But in an event like this, the very APIs those tools rely on are also degraded. So your recovery plan assumes the control plane is perfectly usable at the exact moment AWS is saying, we're having issues with the control plane operations in this region. A few questions to think about in the context of your own systems.
Do you have anything running in another region right now, even in a scaled down form that could serve as a fallback, not theory, but actual workloads you can point to? Could you provide some kind of degraded experience from that secondary region without building it on the fly in the middle of an incident? And have you ever walked through a full failover?
And fail back end to end when things were calm so you know what breaks, how long it actually takes, and who needs to be involved. I'm not saying everyone should be active active everywhere. That's not realistic for most teams.
But if you call a system mission critical, and it only exists in a single AWS region with untested, we'll spin it up elsewhere docs, incidents like this are a pretty strong signal that's not a Now, let's shift from user -facing outages and regional issues to something closer to home. GitHub. On November 18th, the same day as the Cloudflare incident, GitHub had its own major problem.
According to their status updates and multiple tracking sites, they started investigating failures on all Git operations in the early evening of UTC. That meant push, pull, clone, over both HTTP and SSH were failing or timing out. A bit later, they also called out degraded availability for code spaces. It took roughly an hour before they reported recovery.
From a development and operations perspective, that touches a lot of things at once. CI systems that do a fresh clone from GitHub every run will fail. GitOps tools like Argo CD or Flux that continuously sync from GitHub will stop updating. Developers trying to push a fix for some other outage can't get their code up.
And if your only CI system is GitHub Actions, those workflows are either delayed or completely blocked. So while GitHub going down doesn't look like a classic production is down incident, it absolutely can turn into one because it stops you from changing production at the exact moment you might need to.
Most teams don't have a playbook for GitHub is unavailable in the same way they have a run book for ServiceX is unhealthy. A few things worth considering. Do you have read -only mirrors of your most important repositories anywhere else? Another Git provider, an internal mirror, anything? Even a simple periodic mirror of your infra and core app repos can make a difference.
Can your CI run from a cached copy of the repo and existing artifacts for some period of time? Or is every pipeline hardwired to always pull from GitHub's live API? And if GitHub Actions is your only pipeline engine, do you have any backup, even if it's slower and more manual, or is the default we simply wait.
None of this has to be perfect, but your critical repos and pipelines deserve to be treated as part of your reliability story the same way your databases and load balancers are. Now that we've looked at Cloudflare at the edge, AWS in the region, and GitHub in the development loop, let's zoom out and talk about the pattern. All three of these incidents point at the same basic reality.
We are heavily dependent on a small set of external providers that we treat like background infrastructure. The CDN or WAF in front of us, the primary cloud region we run in, and the platform that hosts our code and pipelines. When they have issues, they expose assumptions in our designs and in our runbooks. The root causes are also pretty typical for large complex systems.
A configuration file grows larger than expected and interacts badly with software that wasn't written for that case. Internal health or management services fail in surprising ways and drag down other components. Service operations inside GitHub stumble and suddenly Git operations don't reliably work. Nothing exotic. Just scale and complexity doing what they do.
For me, the practical move here is not to panic about the cloud being fragile. It's to get very explicit about where your external single points of failure are and then improve a couple of them in a concrete way. For each major provider you depend on, ask two questions. If this provider is impaired for a few hours in the middle of the day, what exactly breaks for our users and what breaks for our ability to respond?
And what is the specific set of steps we would take with the tooling and people we have right now? Then pick one of those areas and move it a step forward. That might be putting a minimal but real footprint in a second AWS region for your most critical services and exercising it on a schedule. Documenting and testing a simple path to temporarily bypass the CDN for some subset of traffic if the edge is misbehaving.
Or mirroring your key repos and adjusting CI so you're not entirely dependent on a single Git provider for every build and deployment. You don't need a perfect answer for every scenario, but these kinds of outages are useful pressure tests and they're good leverage when you're trying to justify reliability work to people who only see the cloud marketing slides.
That's it for this first episode of Ship It Weekly by Tellers Tech. Today was a bit of a special, mostly focused on three related outages and what they mean for reliability and architecture. Going forward, the normal format will be a little more mixed.
Most weeks, I'll cover a couple of main stories, some quick mentions of tools, or releases that are worth a look, and usually one culture or career topic that's relevant to DevOps and SRE work. We'll still do focused episodes like this when something big happens, but you can expect more variety week to week.
If this was useful, feel free to share it with someone on your team who thinks about incidents, DR, and platform design. Show notes will have links to the Cloudflare incident explanations, the AWS US EastOne analysis, and the GitHub outage history, so you can dig into the details if you want. I'm Brian. Thanks for listening, and I'll see you in the next one.
Scroll inside the box to read the full transcript, or expand for a larger view.