AI is making it easier than ever to create more software, more code, more diffs, more experiments, more changes moving through systems that were already complicated before everybody got a robot assistant in the editor. And that sounds great until you are the team responsible for production because production does not really care that the code was generated faster. It still cares about latency.
It still cares about overload. It still cares about dependencies, rollbacks, traffic routing, region failures, weird edge cases, and whether the people on call actually know what to do when the system starts acting strange. That is the part of the AI conversation that feels under-discussed to me. Not just can AI write code, it can. Not just can AI help debug, sometimes yes.
The better question is what happens to reliability when the volume of change goes up, the amount of generated code goes up and the context behind that code gets thinner. Because incidents are not just caused by bad code. They are caused by systems behaving in ways people did not expect, under conditions they did not practice, with dependencies they forgot were part of the critical path.
That is really what this conversation is about. Not just SLOs, not just incident reviews, not just AI agents. More like, what does reliability actually look like when change is accelerating? Systems are more connected, and recovery matters just as much as prevention. I'm Brian Teller from Teller's Tech, and this is Ship It Weekly.
Welcome back to Ship It Weekly, where I filter the noise and focus on what actually matters when you are the one running infrastructure and owning reliability. Most weeks, it's a quick news recap. In between those, I do conversation episodes with people who are building platforms, running infrastructure, organizing events, and thinking through where this industry is actually headed.
Today is one of those conversations. I'm joined by Francois Richard, an engineering director at Meta. We're talking about reliability at scale, how AI and automation are changing production risk, what teams actually learn from incidents, and why recovery practice matters just as much as prevention.
And I like this conversation because it gets past the neat, clean version of reliability that fits nicely in a dashboard. Reliability is not just a number. SLOs matter, dashboards matter, alerts matter, guardrails matter. But none of that means much if the team cannot use it to make better decisions, recover from real failures, and improve the system after something breaks. You start with the obvious goal.
Keep the system up. Keep the users happy. Avoid incidents. Do not page people for nonsense. Do not let every deployment feel like a coin flip. Then reality shows up. A region has problems. A dependency gets slow. A service gets overloaded and cannot restart because it is too overloaded to recover cleanly. Traffic spikes during a major event. A change rolls out faster than the team understands it.
Or an AI-generated pile of code technically works until it fails in a way nobody has enough context to explain quickly. And somewhere in the middle of that, reliability stops being just prevention and starts becoming practice. In this conversation, Francois and I talk about how Meta thinks about reliability across both reactive and proactive sides.
Incident response, incident reviews, SLOs, guardrails, validation, disaster recovery testing and what happens when you actually practice taking a region out instead of just assuming the failover plan works because the diagram looks good.
We also get into what teams learn during incidentsss, not just from the post postmortem afterward but inside the pressure cooker itself where engineers have to make decisions quickly build consensus fast understand the system under stress and figure out what is actually happening while the clock is running there is also a good thread in here around AI agents in incident response not the fantasy version where you hand production to an agent and hope it saves the day more the practical version where AI helps with investigation telemetry metrics logs relationships across services and narrowing down what might be happening faster than a human clicking around dashboards alone and towards the end we talk about recovery practice known failures versus unknown failures why teams should test the failure modes they claim they can survive how smaller teams can learn from Meta-scale reliability patterns and why not every system needs six nines on day one so if you work around DevOps, SRE, platform engineering, infrastructure, engineering leadership, incident response, or you are just trying to figure out what reliability looks like when AI is increasing the volume of change, this one should be worth your time.
All right, let's jump in. Today, I'm joined by Francois Richard. He is an engineering director at Meta, and we're talking about reliability at scale, how AI and automation are changing production risk, what teams learn from incidents, and why recovery practice matters just as much as prevention. Francois, thank you for joining me. Thank you, thank you. Happy to be here.
Okay, so when you think about reliability at Meta scale, where does the work actually start? I think for me, reliability is actually two things. What people kind of like know usually it's the reactive side is how do we manage incidentss? How do we respond to problemss? And how do we handle things and recover and fall back when something goes bad?
But at the same time, it's also everything that I call like, you know, the proactive side of the house. This is everything that has to do with, hey, how are we going to prevent reliability problems from happening? What sort of guardrails do we put in place? What sort of framework? What sort of tooling do we allow platform or do we give to platform services and application developers to use?
And also, what is the real validation that we are doing? There's a lot of papers available in the public and so on around like, you know, there's something that we call like the storm program.
And it's basically our way to test like, you know, disaster recovery, where once in a while we take an entire region out, out of the like, you know, like if there was an earthquake or there was like a giant electrical grid failure. And we test what happens with the system. How do they recover? How do they handle the spike in traffic and so on?
So it's both like for me, reliability is both like, you know, the reactive side and also the proactive side. Is that kind of like chaos engineering, the concepts? Yes, yes, yes, exactly. And, you know, we go as far as, you know, disconnecting a complete region from the map. Interesting. Okay, so speaking about reliability, what makes an SLO useful instead of just another dashboard number?
Like, how do you think about that? Yes, an engineer, like a senior engineer that I work with always says that the SLO of something, of a system, is actually the promise that you're making to your customer. That, you know, XYZ should work 99.99% of the time and you should expect that amount of latency on average at, you know, P90 or something like this.
So we use this, like, you know, most of the platform we operate at Meta and also the product services and so on. They will all, the big ones, will all have some form of SLO for the top APIs or the top services that they are doing. We have some alerts that will trigger on SLO, but usually this is too late. Like, you know, we want to catch problemss.
The SLO allows us in the long run to actually figure out like, hey, are we investing enough in reliability? Are we keeping the bar high enough? Or do we have wiggle room to take some risk, introduce new features? Because you all know that when you introduce or you do a giant migration, there's a period of time where it takes time to stabilize things. So the SLOs are helping us make these decisions in the long run.
And you can clearly see when you've been tracking them for a while that some systems are trending up and some systems are trending down. And for the ones that are trending down, it allows you, it gives you the argument and the data to start investing more.
So this is kind of like, you know, our way about deciding if we have enough investment in an area or if, you know, we are okay and we can take more, you know, progressive risk. So you had, we talked a little bit about incidents before the show, and then you had just mentioned incidents. Across the incident lifecycle, Where do teams actually learn the most?
So in terms of learning during incidentss, I think it happens in two phases. First, it happens while handling the incident during what I call like the pressure cooker. When you actually need to handle the incident, you actually need to make decisions quickly, build consensus fast with a small group of people about what's the next action to take.
And then what are the bypass, the restart process, which system is calling what. This is actually a time where a lot of people, like junior engineers and senior, they learn a lot. They learn a lot about the system, but they also learn a lot about team dynamics and themselves during that period. Then the next phase happens when kind of like you build the...
Incident review report or we call it the SEV report, where you basically, you know, once the incident is handled, mitigated, you know, a few days after, you basically sit down and then we have like a tool to do this. And then people sit down and they write a report around like, you know, hey, what exactly happened? What was the root cause of this?
What was the exact timeline, which is kind of like forcing people to go back into logs and all of this? Could the alerts have fired earlier? Did we have the right instrumentation? Did we diagnostic, like diagnose this thing, like, you know, the proper way? And, you know, what are the follow -up tasks, like to make sure that this doesn't happen again?
Or that we have some level of automation to be able to remediate this type of problem automatically. So you build that report. But then from there, you go into a series of reviews. You review it with your team, with your organization. And if the incident was severe enough, we actually reviewed them at the company level. And during that time, it's a big opportunity to get feedback from other teams.
From other senior engineers that, yeah, we see this pattern. Here's, you know, how we handle this type of thing. And you actually learn a lot during that process. So this is usually like, you know, the learnings happen in kind of like two phases. Yeah. And so given that learning, what should an incident review actually produce? Like once the incident's over? The incident review, like we have a report type of thing.
It's a template. It has like, you know, walks you to, for you to get like two. The root cause of it. But I think the most important thing that it produces is also like, you know, what are the follow-up actions? The follow-up could be like, hey, we need to implement more redundancy. We need to implement better diagnostics. We need to implement better ways of handling these exceptions or problems.
We need a different deployment process. This is usually like, you know, the best outcome we could get because, you know, the more you mature your system, the more you'll be able to cover a lot of these, like, kind of like unknown slash unexpected situationss. So this is usually like the good outcome of these. And what do you find is the difference between finding the cause versus improving the system?
The difference between the cause and improving the system. Or is there a difference? There is a bit. Because for me, finding the cause overall is, do I have the right tools to pinpoint the problem very quickly. And then I'll be able to find the cause. And then the improvements are more are more about eliminating this exact problem or that class of problem.
And the improvement can be that like you know, like from a code structure that this will always happen, but you can work around with either deployment strategy, automation of recovery and things like this so that it's... It's not visible to the end users with strategies of retries and things like that. So there's a big difference between finding the cause.
Like if you work in mobile apps, you know that there's going to be a couple of cases where like the cause is going to be like, you know, the mobile networks just drop on you. And the app needs over time to come up with ways to mask it. You'll never eliminate the cause, but you'll have apps that are a lot better at masking this type of problem. So it's slightly different. Okay, so let's switch gears to AI agents.
Are you using AI in incident response at all? Yes. In triaging? We use it for what I would call like investigation. So, you know, there's usually like two classeses of production incidentss. You have one class when it happens. You kind of know what this is. Like, you know, it's a pattern that you already recognize. The alarms or alerts are very clear about what's happening.
So you kind of know and you can narrow down and then you can just go focus on, hey, you know, how do I remediate that? There are some of the cases like where you kind of see like, oh, I'm starting to see transient errors. I'm starting to see latency creeped up and where it's kind of like, I'm kind of not sure. And then usually like, you know, we launch what's called like investigations and things like that.
Really, the AI agents are helping us in investigation because they can analyze and gather large swath of data very, very quickly and then pinpoint relationships between datasets. Like, you know, it's probably the same for everybody, but we have multiple layers of software on top of each other. They call each other and they call back.
Sometimes finding complex situation requires us to analyze a lot of data at the same time. Even myself personally, like, you know, we have like centralized, like, you know, monitoring system.
And then if you aggregate a lot of data with very small granularity, like, you know, at the 10 second or like at the minute level, and you have a lot of time series, not the data point, it gets a pain to actually like get the right query and the right graph for the problem that you're investigating at a point in time like so the AI agent has kind of like shifted this thing where they help us find these things and even like you know in some cases like what i do is like i will dump all the metrics locally in the SQLite database and then like do all sorts of things because it's local and my SSD can do it a lot faster And then you use also the AI agents to help you craft the queries.
I don't remember on top of my head all the various flavors of joins and all of that. So it's helping a lot. So with investigation, it helps a lot. It's helping a lot. It's not perfect. You still need to guide it very well because the topology of the system, the request... The request flow between the system is still not something that I will call it like the various AI agents are actually mastering.
They master code very well. But these flows are a little bit more difficult and understanding how they map into the application or the product flows is even more complicated. But it does help. That's the investigation side of it. We're doing a lot also into what I call like having a standard dataset across the company that kind of like represents the state of reliability.
Like we drive a couple of initiatives globally, like we call it like change safety. Do you have SLOs for something? Do you have guardrails?
And with these datasets being kind of official, then it's easy for each service team or product team to build their own customized kind of dashboard like right now the cost of making dashboards is almost free and really tailoreded to their use cases or to their product using the standard dataset and that's another place where it's been like oh, this has been great.
And how are you, I'm assuming you're still validating that that root cause though right if it comes up with a you know it uses MCP servers it's reaching out to Argo Kubernetes whatever it's gathering all the telemetry from Grafana and then it's putting that together and it's saying, hey, this pod was crash loop back off. This caused this issue. Are you still validating that?
And what is that process for validating that? Like, is there a standard SOP that you're following? No, I don't think we have like a standard evaluation.
The evaluation, like most of the time, the tools that you have in the moment from an incident is either you move away from the problem or you roll back the problem so we will try like you know first you know move away let's call it like a problem is a backend problem it's isolated to a region we'll move away from that region to kind of like prove that theory and then we'll probably during that time roll back the system or the backend or whatever it was that introduced that problem and then you can imagine that we can then now that the hypothesis is live then we can shift a portion of traffic back to that problem and see like oh is it still happening or not this is all manual we have some of these process that are automated but they really boxed into very very well known like you know they had like very well established runbook that have been like you know in place for the longest time that we know that when this happened do this like it's almost like if then else But the most like the investigation result, the validation is more to roll out the fix, you know, driven by human with like, you know, slow roll type of techniques.
Do you think that AI is increasing the speed of change, the volume of change or the type of failure that teams need to expect? But it's clear that it's increasing the volume of change. We have data internally that, you know, the number of diffss is through the roof. The number of lines of code changed is, again, through the roof.
The type of failure is like one thing that, because there's more code and it's going faster, it's the type of failure becomes more into the, what I would call like pseudo-unexpected. When a product developer, application developer, when they... Roll out a new feature, they do understand the business logic.
They do understand how it should show to the user, what they expect the user to do, where's a corner where a user could get stuck. They understand that. That's not a problem. But if you have to build on modern apps on mobile phones or on backend, A lot of these things are built on top of frameworks. There's a lot of boilerplate stuff, especially to handle like, you know, async callbacks and things like this.
And since a lot of that code gets auto-generated, people are losing context around these things. And when something doesn't work, we realize that it takes us more time to understand why it doesn't work. Because there was not that additional context that...
The developer or that team developing that feature took a lot of time to craft it and a lot of like internal reviews and things like that so so it goes faster out but that context on how it's built is actually lost along the way and this is basically what we need to reconstruct live and again like i said earlier a lot of our tooling is like you know in this case we're just like we'll roll back and you figure it out like bring back the system to a stable state and then you can figure it out on your own time without being on the pressure cooker.
Okay, so speaking along the same lines, you had mentioned too that recovery comes down to practice. So what does good recovery practice actually look like? You know, there's a set of standard failuress you expect your app to have. And it starts at, you know, if you're running, if you have data centers or region in the United States or in Europe or something like that, you get...
Tons of things like fiber cut in hurricanes and, you know, power grid and things like that. You need to understand how your system will react.
Like, you know, a simple thing, like if I'm a startup and I run into a AWS region, but I actually never test the fact that one of them will go down, I can guarantee you, like, as soon as you get a little bit of complexity in your system, something will not work as expected in there.
For us is actually to do, to validate it, to not just test, like we run, I would call it like data simulation and analysis in advance to figure out like, okay, if we lose that data center, we'll be okay. But it is always interesting. And then after we've done the validation and vetted that it all worked. We actually do the test and we discover a suite of other problems that the analysis had never found.
So exercising the failure for real at the lower scale allows you to find a lot of these problems. And it's the same thing like, you know, if you have very sensitive hotspot in your application and your product, injecting failure for real at a very low percentage of user will...
Allow you okay do i have the right detection like did that trigger properly did it get to the right person did this team know how to handle it like without this it's it all becomes like improv like you improv all the time versus when you have a portion of your infrastructure that's always tested in such a way first you keep on hardening and it gets better and it gets better and the muscle of handling it gets better also so i guess that I was going to ask about like how, how would teams practice known failures versus unknown failures?
But it sounds like it comes down to just that practicing and pulling out the regions. Okay. And there's, there's a bunch of types of failure. Like, you know, I, I made an allusion about worst-case failures versus worst-case failure, which is, um, you know, you lose a complete region.
And with the weather patterns in the U.S., you know, we have hurricane season, we have like fire season, tornado, like the cuts happen everywhere. So you got to be ready. But there's also other things like, you know, for meta, we are often, I would call it like we got, we get these traffic spikes. It could be because there's something in the news related. It could be a big event like the Super Bowl, New Year's Eve.
The World Cup's coming. So we know that every time there's going to be a goal, it's going to spike. What we realize over time with practice is that some system will be overloaded. And in some cases, you'll have to restart them. And during overload, the damn thing cannot restart. You cannot kill it. Like it's all completely dead. Like it's completely blocked. So we practice a lot of these cases too.
And it's something specific. To us, we get these spikes that are kind of unexpected. Most systems have some form of overload protection up to a point, but it's something we really had to invest in, into like, in the worst load scenario, can I restart? Do I have, like, is my process smart enough to just accept requests later down the road?
Or do I have enough control in the traffic routing to, like, okay, choke the traffic, bring back up, warm up your memory, and then, like, gradually. Like, we had to practice that a lot. Yeah, so you're dealing with, like, thrashing and having, like, no... Yeah. Yes, yes. Resources being completely exhausted. Yeah, yeah. Completely gone. And is that typically at the control plane Completely gone.
And is that typically at the control plane level? Or... Some of our control plane is pretty good you know, there's some backend. Like you can imagine like, you know, an Instagram feedback an Instagram feed backend or Facebook feed backend same type of things. And we have a couple of academic paper. People can look at it. It's called like Taiji, I believe.
And then when we discuss these things on how we can reroute traffic and then control and then remove it from an area to allow the system to reload. Okay. So with that in mind. I would imagine most listeners are not operating at Meta scale, so they're not dealing with every time there's a goal at the World Cup, you know, their servers go down.
What ideas can actually transfer for most like SRE DevOps that are listening? I think the example I gave earlier, like you often see like no blame on Amazon, but if US East go down, like, you know, we see the world of the internet like go like berserk. So I'm like, guys, like, you know, you got to start testing these cases.
Like you've seen it happen, like, you know, at least like, you know, three or four times in the last two years. You got to start like, you know, having your dual regions and really validating it. And maybe it's a question about like, you know, every Tuesday, I do not run a single thing in US East, you know, whatever the name is these days to make sure that it's, you know, it's performing.
And then it's kind of like tackling the problems in a way like, you know, when you look at your incident inside a small, a smaller company, medium sized startup or so on, like I get a list of incidents.
There's a point where you start having enough data that you could say, okay, most of them are caused by, it's because we launched that feature or because we're doing configuration change or because we have a billing problem. It's starting to.
Trend your your SEV data and figure out hey what are the top two or three then we can start attacking and really focusing because like you can focus on everything but you'll get nowhere it's really start bucketizing like you know in having the discipline of writing it down like it's not that hard to write like a SEV report there's a lot of example in there and having the discipline of going back and Now with LLMs, do the analysis is kind of like trivial.
You need to, we used to need to allow a lot of people to do this and now it's a lot easier. So I think that's a big opportunity to actually really focus on where reliability matters. Yeah, that's fair. You had mentioned too that reliability expectations should match the system and product life cycle. Yes. Can you talk about that? Like, because 100 % uptime is not always like the right goal. Yes, yes, yes, yes.
There is.
There's like you know when i say about the expectation versus life cycle we should always expect like if something is is has some level of complexity and some level of feature at the beginning when you roll it out like you know getting two nines or even three nines of reliability it's pretty good like like like not a simple system that's that that that's usually like you can get it easy but like if it's early in the life cycle of that product or that backend at the beginning, it's going to be rocky and it should be expected.
Like I see a lot of people start with the assumption that, Hey, everything should be like six nines, six nines guys. Like this is like insane. Like, and you invest a lot and then you, you kind of like mix, uh, miss your product market fit while you're doing that. Like there are some like Facebook and Instagram, there's a gazillion experiments that runs at a given point in time.
Some of them are not fully reliable like and it's okay because we're trying to figure out if that feature will be something that users enjoy and use or not but they are other part of the app which like needs to be like rock solid and this is the place where we invest so depending on the life cycle if you are early or not because there's a cost at investing and then when especially like either your SRE or production engineering and you have to also convince your PM, your product managers that this is the case, having that conversation from that standpointpoint is a lot more easier than just like being the one that says, no, I need six nines of reliability for everything.
That's not happening. Yeah. Well, six nines too, that's like what, 31 seconds a year of downtime? I mean, the amount of infrastructure and cost complexity to just keep a system at scale. Depending on the system, I mean, I assume if it's like a very important financial system, maybe it matters. But for the majority of systems, like do the users really care about 31 seconds of outage per year?
I mean, are they going to? They do care. You know, we can see from our own data that, you know, if we have too much problems in a row, we can see that, you know, engagement eroding. So they do care. They do show. But it tends to stabilize when we stabilize things. So there's a correlation there. But sometimes the investment to reach that extra nine is just so high.
And while you're doing that extra nine, you're also sacrificing not just the speed and velocity, but also your mitigation time for future unknowns. Because when you have invested so much complexity and then you get into an unknown situation and then now you have to untangle all of this, your mitigation time will be higher. So it's kind of like it's a delicate trade-off. Like it's not a win-win all along.
Yeah, for sure. It's a reality there. A real conversation that you have to have and a balance that you have to have there. Okay, so wrapping up, this conversation also connects to At Scale systems and reliability. Why does that program feel relevant right now? Yeah. So we have this suite of conferences, like it's called At Scale. And we have like four conferences a year.
And the next one that's coming is basically systems and reliability together. And what we want to target for this specific conference is, hey, how are we injecting? What are the systems that are like, you know, building reliability for AI? And what are the systems that are under the hood for AI? Because people tend to talk about, hey, we talk about the models, okay?
But we never talk about like the underlying plumbing that is required to train and serve the model and the insane amount of data that we need to move back and forth. And then on the other angle, we also, we will discuss like... All the other areas where we actually use AI to enhance reliability. So it's both like, you know, both cases that we want to do. This conference is going to happen in person.
It's going to be in Bellevue. And you can find the detail. The website is at scale conference. And we could put that at the show notes. Yes. So if someone works in infra SRE platform or engineering leadership. Why should this be on their radar? And what are you watching specifically with the conference? I'm just curious. Personally, I'm a big fan of this conference. I've been part of it for a while.
And one thing that I tell the speakers that are coming from Meta, and we have speakers from NVIDIA, from Microsoft, from Google, the goal here, I want to have the real technical discussion. With Meta, I'm not in the business of selling cloud services. Like I'm not the business of selling API or things like this.
So what I want to talk is like, you know, Microsoft, you have a Kubernetes cluster with one million servers. How the hell did you do that? Like, this is like, you know, the goal is really like, I want to really understand the story behind the system. I want to have like the real technical conversation. And I want to avoid the sales pitch of like, oh. Use that service, use that thing.
So this is really the focus of the conference is to really have that technical discussion and also the story behind the system. Like, I'm always fascinated. I've always been fascinated by, like, okay, a team starts something. Like, I've been involved in ZooKeeper and all sorts of other things in my days at Yahoo back then. You start something, it was to solve a problem.
Then you open source and you're like, oh, that became this? Oh my God. So I'm really interested in this and then like the ups and downs of the team because you always have like, you know, the hype at the beginning and then, oh, the reality hits and you're like, oh my God, like this will not work as expected. And how do you overcome that?
Like that, for me, that's the most interesting part of all of these stories and all of these presentations. So, okay, wrapping up, what...
Kind of reliability conversations does the industry need more of i think right now the industry does understand the the use of AI for code generation like i think we get that i think we can get that we can go all out i think we understand that everybody can build a custom app like for themselves and they will only use it for for them it's gonna be perfect for them and then it's okay if it goes down but i think the rest of the industry has not kept up with that rate of change and there's not enough investment in kind of like defense like we're able to generate code are we able to debug it faster are we able to understand it faster are we able to troubleshoot it faster like like that has not kind of followed.
And then I feel that we are catching up now. And then the hype seems to be like mostly on, on the model. And then there's amazing infrastructure that had to be built underneathneath. And I think you know, everybody needs to understand it a little bit more. Yeah, for sure. Awesome. So I will put links for At Scale Systems and Reliability reliability in the show notes. I can put your LinkedIn there as well.
Is there any other links or comments that you'd like to give to the audience before we wrap up? No, I think that's it. Don't give up. Awesome. Thank you so much, Francois, for coming on. Really appreciate your time. Thank you so much. Thank you. Bye-bye. All right. That was my conversation with Francois Richard from Meta. My biggest takeaway from this one is that reliability is not just about preventing failure.
The better question is, what happens when prevention fails? Because sometimes the answer is a rollback. Sometimes it is moving traffic. Sometimes it is draining a region. Sometimes it is restarting a service. Sometimes it is realizing that the service cannot restart cleanly because it is already overloaded, which is the kind of fun little production detail that does not usually show up in architecture diagrams.
That is the part that I think is worth paying attention to. A lot of teams talk about reliability like it is mostly a tooling problem. Get the right dashboards. Get the right alerting. Define the SLOs. Add some runbooks. Maybe sprinkle in some AI and pretend the incident lifecycle is solved. But reliability is not just the tools. It is the practice around the tools.
It is whether the SLO actually represents a promise to users. It is whether the alert fires early enough to matter. It is whether the incident review produces real follow-up work instead of just a nicer explanation of what broke. It is whether the team has practiced the failure mode before production forces them to learn it live. And honestly, that is the part of the conversation that translates really well.
Even if you are nowhere near Meta scale. Most of us are not dealing with World Cup traffic spikes or massive global systems, but a lot of us are depending on a cloud region more than we want to admit. A lot of us say we are multi -region, but have not actually run without the primary region on a boring Tuesday. A lot of us have runbooks that look reasonable until someone has to follow them under pressure.
A lot of us have services that should recover automatically, but only if the failure happens in the exact way we imagined. That is where the work is. Practice the recovery. Test the boring assumptions. Look at your incident data. Bucket the causes. Figure out what keeps showing up. Then go after the top patterns instead of trying to boil the ocean.
I also liked Francois' point about AI changing the reliability equation. AI can absolutely help with investigation. It can look across a lot of data quickly. It can help build queries, connect patterns, and speed up the part where humans are trying to figure out what changed and what is related. But AI is also increasing the volume of change.
More diffs, more generated code, more boilerplate, more systems moving faster, and sometimes less human context behind the code that just went out. That is a weird trade-off. Because if code moves faster than understanding, Reliability teams are going to feel that gap during incidentsss.
The system breaks, and now someone has to reconstruct not just what changed, but why it changed, what the generated code is actually doing, what assumptions it made, and how to get back to a stable state. That does not mean AI is bad. It means the defensive side has to catch up. Debugging has to get better. Observability has to get better. Incident response has to get better. Recovery practice has to get better.
And humans still need to be in the loop for judgment, especially when the system is too important to let a guess turn into the next mitigation. I also think the lifecycle point matters. Not every system needs the same reliability target. A brand new experiment probably should not get the same investment as a core production path that millions or billions of people depend on.
Six nines sounds impressive until you realize what it costs, what complexity it adds, and how much slower it can make future changes. But the reverse is true too. If a system becomes important and the reliability investment never catches up, you are just borrowing risk until production collects. So maybe the healthier conversation is not how do we make everything maximally reliable.
It is more like what promise are we making? Who depends on this system? What happens when it fails? And have we practiced the recovery enough to believe our own answer? That is probably where a lot of reliability conversations are heading. Not AI will fix incidents. Not SLOs solve reliability. Not just make everything multi-region and call it done. More like, what failures should we expect?
What failures have we practiced? And what are we learning every time production teaches us something? I'll have links to Francois, Meta's At Scale systems and reliability event, and anything else we mentioned in the show notes. If you enjoyed this conversation, follow or subscribe to Ship It Weekly wherever you listen to podcasts.
It helps the show and it makes sure you get both these conversation episodes and the weekly DevOps, SRE, platform, cloud, and security news recaps. You can also find the show notes and links over at shipitweekly.fm. Thanks for listening, and I'll see you later this week.
Scroll inside the box to read the full transcript.
