Ship It Weekly Host Commentaries

This episode is really about the control plane getting wider.

That sounds like a platform-engineering phrase, but it is becoming one of the more important ways to think about modern production systems.

A few years ago, when people said “control plane,” they usually meant something fairly specific. Kubernetes API server. Cloud API. CI/CD system. Maybe an internal deployment platform.

Now it is messier than that.

Your container runtime is part of the control plane because it decides how workloads actually start on the node.

Your database failover automation is part of the control plane because it decides whether recovery is safe or reckless.

Your AI incident-response agent is part of the control plane because it can inspect telemetry, summarize what changed, recommend action, and maybe someday trigger work directly.

Your Kubernetes API server egress path is part of the control plane because a stale route table or broken firewall path can stop admission webhooks, OIDC, and aggregated API calls from working.

Your credential revocation tooling is part of the control plane because compromised access has to be cut off fast.

Your cloud console is part of the control plane because operators still need a way to reach the environment during an incident.

Even object metadata starts to matter when data, AI, search, and agent workflows depend on understanding what an object is, not just where it lives.

That is the through-line in this episode.

containerd disclosed a batch of CRI plugin vulnerabilities, and the lesson is that Kubernetes security does not stop at pod specs, RBAC, admission control, or image scanning. Eventually the node runtime has to pull the image, unpack it, restore it, wire up devices, handle logs, and start the container. That runtime layer is not invisible plumbing. It is a trust boundary.

The Datadog PostgreSQL HA story is a different kind of control-plane lesson. Their gameday did not just ask whether PostgreSQL could fail over on Kubernetes. It exposed the harder question: can it fail over safely? If every standby is behind, promotion may be possible, but it may not be correct. And in databases, correct usually matters more than fast.

That is the part I love about the Datadog writeup. It is not the fantasy version of HA where automation magically fixes everything. It is the real version where replication lag, synchronous writes, RPO, RTO, and promotion safety all collide. Failover is only useful if the recovery path does not create a bigger problem.

The AWS DevOps Agent and Datadog MCP Server story pushes this same theme into AI operations. AI incident response is moving from demo to production workflow. That is exciting, but the question cannot just be “is the agent smart?” The better question is “what authority does it have?”

Can it only read?

Can it write?

Can it open tickets?

Can it trigger automation?

Can it roll back?

Can it page someone?

Can it make things worse quickly and very confidently?

That is the uncomfortable part. AI incident tooling can be genuinely useful, especially during the early chaos of an incident when everyone is jumping between dashboards, traces, logs, deploy history, and Slack threads. But once an agent sits near the operational control plane, it needs the same boring guardrails as any other production automation: least privilege, audit logs, approval boundaries, rollback rules, and a clear line between recommendation and execution.

Then there is EKS customer-routed control-plane egress, which is one of those features that sounds boring until you think through the failure modes. Routing Kubernetes API server outbound traffic through your own VPC is a real win for private and regulated environments. But it also means your route tables, security groups, NACLs, firewalls, and private connectivity can now become control-plane dependencies.

That is powerful.

It is also something you bring to a design review.

The lightning round kept hitting the same idea from different angles. GitHub credential revocation is incident-response infrastructure. AWS Console Private Access pulls more operator workflow behind private network boundaries. Vercel Connect points toward short-lived, task-scoped credentials for agents instead of long-lived secrets sitting around forever. S3 annotations make object metadata more directly attached, mutable, and queryable instead of living in another side table that drifts from reality.

Different stories, same shape.

Authority keeps moving.

Trust keeps spreading.

The blast radius keeps expanding.

And that is where Marc Brooker’s post on waiting fits so well as the closer.

Your dashboards may measure averages, but your users do not experience averages. They experience the time they spend waiting. A ten-minute outage and a ten-hour outage might both count as one incident in a tracker, but they do not feel the same to the people stuck inside them. A service with a decent average can still feel terrible if users keep landing in the tail.

That is the reliability lesson underneath the whole episode.

When the system breaks, users do not experience your architecture diagram. They do not care whether it was the runtime, the database, the AI agent, the credential system, the route table, the cloud console, or the metadata layer.

They experience waiting.

Waiting for a request.

Waiting for recovery.

Waiting for a deploy to stop failing.

Waiting for a credential to get revoked.

Waiting for the control plane to come back.

Waiting for someone to find the right context.

So the practical question is not just “is this system up?”

It is also:

Where is authority hiding?

What has to work before recovery can happen?

Which defaults are trusted?

Which control-plane paths are invisible?

Which tools can make changes?

Which systems can block deploys?

Which dependencies only show up when something breaks?

And most importantly, what does this feel like from the waiting side?

Because your dashboards measure the average.

Your users feel the wait.

---

Extra links worth including:

containerd CRI plugin vulnerabilities / AWS security bulletin
https://aws.amazon.com/security/security-bulletins/2026-046-aws/

Datadog: PostgreSQL high availability on Kubernetes
https://www.datadoghq.com/blog/engineering/postgresql-ha-kubernetes/

AWS DevOps Agent and Datadog MCP Server
https://aws.amazon.com/blogs/devops/production-ready-autonomous-incident-resolution-with-aws-devops-agent-now-ga-and-datadog-mcp-server/

Amazon EKS customer-routed control-plane egress
https://aws.amazon.com/blogs/containers/amazon-eks-now-supports-control-plane-egress-through-your-vpc/

GitHub self-service credential revocation for incident response
https://github.blog/changelog/2026-06-24-self-service-credential-revocation-for-incident-response/

AWS Management Console Private Access
https://aws.amazon.com/about-aws/whats-new/2026/06/aws-management-console-private/

Vercel Connect
https://vercel.com/blog/introducing-vercel-connect

Amazon S3 annotations
https://aws.amazon.com/blogs/aws/amazon-s3-annotations-attach-rich-queryable-context-directly-to-your-objects/

Marc Brooker: Waiting, latency, MTTR, and the inspection paradox
https://brooker.co.za/blog/2026/06/19/waiting.html

This week’s On Call Brief
https://www.tellerstech.com/on-call-brief-news/2026-W26/

More Ship It Weekly episodes
https://shipitweekly.fm/

For this Conversations episode, the part that stuck with me is how easy it is for backend and cloud people to bring the wrong mental model to mobile security.

In backend systems, we are used to control.

We control the runtime. We control the infrastructure. We control IAM. We control the network path. We control the deployment process. We control observability, logging, and a lot of the runtime environment around the application.

Mobile changes that.

You ship the app and now it is out there.

It is running on someone else’s device, in someone else’s environment, with software you do not control, under conditions you may never see directly. Maybe the device is rooted. Maybe it is jailbroken. Maybe there is malware nearby. Maybe someone is decompiling the app, running instrumentation, hooking values, bypassing checks, or trying to understand enough of the workflow to abuse the backend.

That is the trust shift Joel DeStefano kept coming back to in this conversation.

Mobile app security is not just “did we scan the app before release?”

Scanning matters. Dependency checks matter. TLS configuration matters. Not shipping secrets in the app matters. All of that is table stakes.

But it is point-in-time assurance.

It tells you something before release.

It does not tell you what happens after the app leaves the pipeline.

That is where I think a lot of teams get too comfortable. They assume the App Store review process is handling more than it is. They assume the OS is protecting the application logic. They assume that because the backend stores the sensitive data, the mobile app itself is not an interesting target.

That last one is probably the most dangerous assumption.

The sensitive thing is not always a secret string sitting in the app.

Sometimes the sensitive thing is the workflow.

How does the app authenticate? How does it talk to the backend? What does the backend trust? What values does the app send? What checks happen locally? What behavior changes when a condition flips from false to true?

If an attacker can learn the business logic, they may not need a database dump from the app. They may only need enough understanding to abuse the system around it.

That is especially true for apps tied to money, login, account recovery, medical workflows, retail transactions, subscriptions, loyalty points, streaming access, or anything else that represents real business value.

So the better question is not just, “Do we have secrets in the app?”

The better question is, “What can someone learn from this app, and how could they use that against us?”

That is a much more useful way to think about mobile security.

I also liked how Joel framed the layered approach.

Testing matters.

Hardening matters.

Runtime protection matters.

Monitoring matters.

Those are not interchangeable pieces. They solve different parts of the problem.

Testing and scanning help catch issues before release. Hardening and obfuscation make reverse engineering harder. Runtime checks help detect suspicious environments, debuggers, hooking frameworks, rooted devices, overlays, accessibility abuse, and other signs that something about the app execution is not right. Monitoring helps teams understand what is actually happening after the app is in the wild.

That monitoring piece is important because the threat landscape does not freeze when you ship.

Attackers change tools. Tactics change. Activity spikes in different regions. A new framework gets popular. A workflow starts being abused in a way nobody expected.

If you are not watching, you are guessing.

That does not mean every app needs the same level of protection. A tiny internal utility is not the same as a banking app, a healthcare app, or a payment flow. But a lot more apps are business-critical than people admit.

Mobile apps are not just “apps” anymore.

They are storefronts. Payment platforms. Account portals. Medical device managers. Subscription systems. Streaming platforms. Banking experiences. Identity and recovery paths.

If that is where users spend time and money, attackers will spend time there too.

The CI/CD part of this conversation is also worth paying attention to.

Security that destroys the release process usually does not last.

That is true in cloud. It is true in platform engineering. It is true in mobile too.

If the security tooling slows everything down, breaks builds constantly, creates stability issues, or forces teams into a painful release process, engineering teams will eventually route around it. Maybe not officially. Maybe not loudly. But they will find a way to keep shipping.

So the goal has to be security that lives inside the real delivery process.

High-severity findings should block. Critical issues should stop the release. But lower-severity issues may become tracked risk. Runtime protections still need QA. Performance constraints still matter. Stability matters. App size can matter. A few milliseconds might be acceptable. A crash is not.

That is the tradeoff.

Not “security versus shipping.”

More like, “how do we make security part of shipping without pretending delivery constraints do not exist?”

That is a healthier conversation for DevOps, SRE, platform, AppSec, and mobile teams to have together.

I also thought the AI angle was interesting because it fits the broader theme we keep seeing across security.

AI may make attackers faster.

It may help them understand app logic faster. It may help them reason about code faster. It may help them reverse engineer faster. It may compress the time between “I have this app” and “I understand where to poke.”

But Joel’s point was that the fundamentals do not disappear.

If you already have the right model, AI increases pressure.

If you do not have the right model, AI makes the gap more painful.

That seems broadly true across security right now. AI is not replacing the need for good fundamentals. It is raising the cost of not having them.

For mobile, those fundamentals look like understanding the trust model, protecting the app, making reverse engineering harder, adding runtime checks, validating that the app connecting to your backend is genuine, and monitoring what is happening in the field.

The biggest mistake is assuming someone else has solved the problem for you.

The App Store will not save you.

The OS will not save your business logic.

A scanner will not protect runtime behavior.

And “we do not store sensitive data in the app” does not mean the app has nothing valuable to reveal.

That is probably my main takeaway from this episode.

Mobile app security is not magic, but it is different.

It is a different trust boundary. It is a different runtime model. It is a different visibility problem. And it needs to be treated as part of the system, not as a separate thing that gets checked at the end.

Because once the app is shipped, the question is not just whether you built it safely.

The question is whether you can still trust what is connecting back.

This episode is about default trust getting punished.

That sounds a little dramatic, but it is the thread that connects all of these stories.

PeopleSoft was trusted because it was an enterprise system that had been around forever.

npm install scripts were trusted because that is just how the ecosystem worked.

GitHub agentic workflows were trusted with personal access tokens because that was the easy path.

AI models were trusted because the API worked yesterday.

Tekton events were trusted to keep flowing.

Triton defaults were trusted until they changed.

Homebrew taps were trusted because developer tooling has historically been pretty casual about running code.

Different stories, same pattern.

Production risk often hides inside the defaults nobody has revisited.

The PeopleSoft story is the loudest example. Oracle issued an emergency advisory for CVE-2026-35273 in PeopleSoft PeopleTools, a remotely exploitable unauthenticated vulnerability that Oracle says can lead to remote code execution.

And yes, PeopleSoft is not exactly the trendiest infrastructure topic in the world.

But that is kind of the point.

A lot of the systems that hold the most sensitive business data are not the shiny new systems. They are the old enterprise platforms that sit somewhere in the corner of the architecture diagram with an owner nobody wants to bother too much.

HR systems.

Finance systems.

Payroll systems.

Student record systems.

ERP platforms.

Admin portals.

The stuff that feels boring until someone steals data from it.

The mistake teams make is treating “legacy” as if it means “less important.” Sometimes legacy means high-value, hard-to-patch, full of data, and surrounded by years of network exceptions nobody wants to touch.

That is not low priority. That is a great place for attackers to dig.

The takeaway is not just “patch PeopleSoft,” although obviously, yes, patch PeopleSoft. The larger takeaway is that enterprise app tiers are still production systems. They still need ownership, logging, patching, access review, exposure review, and incident response coverage.

Even if everyone emotionally moved on to Kubernetes.

The npm v12 change is the same idea in a different place.

For years, npm install has been more powerful than people like to admit. You install a package, and that package can run code during install. Sometimes that code builds native bindings. Sometimes it downloads binaries. Sometimes it generates files. Sometimes it does the little bit of magic that makes a build work.

But that same magic is also a supply-chain attack surface.

A malicious package does not need your application to import it at runtime if it can execute during install. It only needs to land in the dependency tree. Then the developer laptop, CI runner, or build machine does the rest.

That is why npm v12 changing install-script defaults matters. Moving from implicit trust toward explicit trust is a good security direction. But it is also going to reveal how many pipelines quietly depended on install-time behavior that nobody documented.

That is the uncomfortable but useful part.

Some builds are probably going to break. Some packages will need explicit trust. Some internal assumptions will become visible. And while that is annoying, it is also an opportunity to ask better questions.

Which install scripts run today?

Which ones actually need to run?

Which ones do we trust?

Do our CI runners have secrets available during install?

Can install-time code reach internal systems?

Are developers running installs on machines with cloud credentials, SSH keys, or publishing rights?

The blast radius of an install script is not just the package. It is whatever the install environment can access.

GitHub’s agentic workflow change is another trust default shifting in the right direction.

Agentic workflows no longer needing long-lived personal access tokens sounds boring, but boring credential changes are often the ones that save you later.

Long-lived PATs are convenient, but they are also messy. They are tied to people. They get copied. They sit in secrets. They outlive their original purpose. They survive team changes, repo changes, workflow changes, and organizational memory.

That is already not ideal for normal automation.

It is worse for agentic workflows.

Once agents enter CI/CD, they are not just assistants. They are automation surfaces. They can read repo context, react to issues or pull requests, generate changes, open PRs, call tools, and potentially become part of release, remediation, or triage paths.

So the identity they run under matters a lot.

Moving toward scoped workflow permissions and GitHub Actions tokens is a better default than handing agents long-lived personal credentials. But it does not remove the need for a threat model.

You still need to ask what repos can run these workflows, what events trigger them, what permissions they get, whether issue or pull request content can influence them, and whether they can see secrets, publish artifacts, or affect production.

The update is good.

The bigger lesson is that agentic CI/CD belongs in the same production automation review process as anything else that can change code, trigger work, or spend money.

The Anthropic story is the weirdest one because it is not a traditional outage.

There was no bad deploy.

No regional failure.

No database incident.

No capacity problem.

But Fable 5 and Mythos 5 still became unavailable to customers after a U.S. export-control directive.

For anyone depending on those models, the operational effect is still real.

The dependency disappeared.

Just not for an uptime reason.

That is the part platform and reliability teams need to absorb.

AI dependencies are not only API dependencies anymore. They are also policy dependencies, geography dependencies, vendor dependencies, compliance dependencies, model availability dependencies, and probably a few more categories we have not named yet.

A fallback plan for AI cannot just say “use a different model” unless someone has actually tested that.

Can you swap the behavior?

The context window?

The latency?

The cost profile?

The tool-calling behavior?

The prompt format?

The eval results?

The moderation behavior?

The user experience?

Sometimes the fallback is a different model. Sometimes it is degraded mode. Sometimes it is cached output. Sometimes it is a human review queue. Sometimes it is a feature flag that disables a workflow before it breaks the product.

That sounds boring.

But boring fallback behavior beats waking up to discover your product depends on a model you can no longer access.

The lightning round kept reinforcing the same point.

Tekton CloudEvents moving to a dedicated events controller is not a giant story, but it matters if your CI/CD observability or automation depends on those events. Small platform changes can break the glue.

NVIDIA Triton changing defaults around shared memory and queue behavior is another reminder that AI inference servers are now normal production infrastructure. Defaults change. Resource behavior changes. Model control changes. Upgrade them like production systems, not experiments.

AWS Nitro Isolation Engine and formal verification is about cloud trust getting more explicit. Customers still cannot inspect everything underneath their workloads, but the platform is moving toward more clearly defined and mathematically checked isolation boundaries.

Homebrew 6.0 tap trust is another supply-chain default getting stricter. Third-party developer tooling can run code too, and developer tooling is part of the software supply chain whether teams admit it or not.

That is the bigger theme of the episode.

Defaults are not neutral.

They are decisions someone made for convenience, compatibility, usability, or speed.

Sometimes those defaults are still right.

Sometimes they are outdated.

Sometimes they are dangerous.

And sometimes nobody knows because nobody has looked at them in years.

That is the platform work hiding underneath this whole episode.

Not just building new systems.

Not just adopting new tools.

Not just patching the obvious CVE.

But asking what your organization trusts by default.

Which enterprise apps are still exposed?

Which package installs run code in CI?

Which agents still use long-lived tokens?

Which AI models sit in production paths?

Which developer tools can execute code before anyone reviews them?

Which event pipelines are assumed to keep working?

Which cloud isolation guarantees are you relying on?

Those questions are not flashy, but they are where a lot of real production risk lives.

The takeaway from this episode is simple:

Find the defaults.

Then decide whether they still deserve your trust.

Extra links worth including on the episode page:

Oracle PeopleSoft CVE-2026-35273 advisory
https://www.oracle.com/security-alerts/alert-cve-2026-35273.html

Oracle PeopleSoft CVE-2026-35273 risk matrix
https://www.oracle.com/security-alerts/cve-2026-35273verbose.html

npm v12 breaking changes
https://github.blog/changelog/2026-06-09-upcoming-breaking-changes-for-npm-v12/

GitHub Agentic Workflows no longer need PATs
https://github.blog/changelog/2026-06-11-agentic-workflows-no-longer-need-a-personal-access-token/

GitHub Agentic Workflows authentication docs
https://github.github.com/gh-aw/reference/auth/

Anthropic Fable 5 / Mythos 5 access statement
https://www.anthropic.com/news/fable-mythos-access

Tekton Pipelines releases
https://github.com/tektoncd/pipeline/releases

NVIDIA Triton Inference Server 26.04 release notes
https://docs.nvidia.com/deeplearning/triton-inference-server/release-notes/rel-26-04.html

AWS Nitro Isolation Engine
https://aws.amazon.com/blogs/compute/aws-nitro-isolation-engine-formally-verifying-the-hypervisor-in-the-aws-nitro-system/

Homebrew 6.0.0
https://brew.sh/2026/06/11/homebrew-6.0.0/

This week’s On Call Brief
https://www.tellerstech.com/on-call-brief-news/2026-W25/

More Ship It Weekly episodes
https://shipitweekly.fm/

For this Conversations episode, the part that stuck with me is that reliability is not really about whether something fails.

It is about what happens next.

That sounds obvious, but I think a lot of teams still treat reliability like it is mostly a prevention problem. Better deploy checks. Better alerts. Better dashboards. Better SLOs. Better review processes. Better guardrails. All of that matters, obviously.

But production still gets a vote.

A region has issues. A dependency slows down. A rollout behaves differently under real traffic. A service gets overloaded and then cannot restart cleanly because it is already too far gone. The system does something nobody expected because the conditions were never actually tested.

That is where reliability stops being a dashboard and starts becoming a practice.

That was the biggest thread for me in this conversation with Francois Richard from Meta. He framed reliability as both reactive and proactive work, which is probably the right split. You need the incident response muscle. You need people who can stay calm in the pressure cooker, make decisions quickly, build consensus, understand the system under stress, and recover.

But you also need the proactive side. Guardrails, validation, disaster recovery testing, and the uncomfortable work of actually exercising failure modes before they show up on their own.

That is the part most teams agree with in theory and avoid in practice.

Everyone likes the idea of multi-region. Fewer teams like the idea of turning off the primary region on a Tuesday and seeing what actually breaks.

Everyone likes the idea of graceful recovery. Fewer teams have tested whether the overloaded service can restart while it is overloaded.

Everyone likes the idea of runbooks. Fewer teams know whether the runbook still works when the person following it is tired, under pressure, and trying to make sense of five dashboards while Slack is melting.

That is why I liked the way Francois talked about practice. Not just tabletop exercises. Not just theoretical architecture reviews. Real validation. Real failure injection. Real regional testing. Real traffic and overload scenarios, at safe enough scale that you can learn before it becomes a customer-visible disaster.

For most of us, the lesson is not “copy what Meta does.”

That would be silly. Most teams are not dealing with World Cup traffic spikes, global-scale social products, or the same infrastructure footprint.

But the pattern transfers really well.

Take the failure modes you claim you can survive and test them. Take the incident patterns that keep showing up and bucket them. Take the systems that are critical and ask whether the recovery plan is something you have actually practiced, or just something you hope will work because it looks reasonable in a diagram.

The SLO discussion was also useful because it puts reliability in business terms without turning it into corporate fluff.

An SLO is not just a graph. It is a promise.

That is a much better way to think about it. What are we promising users? What are we promising internal customers? What are we promising product teams? And does the reliability investment match that promise?

This is where teams can get weird in both directions.

Sometimes teams underinvest in reliability because the system “mostly works,” until one day it becomes critical and the reliability model never caught up.

Other times, teams overinvest too early and try to make a young experimental system behave like a mature core production path. That can add cost, slow down learning, and introduce complexity before the product even proves it deserves that level of investment.

Six nines sounds great until you ask what it costs, how much complexity it adds, and whether the business actually needs that promise right now.

That does not mean users do not care about reliability. They absolutely do. Francois called that out too. If systems have too many problems in a row, engagement suffers. People notice. Trust erodes.

But the answer is not “make everything maximally reliable.” The better answer is to match the reliability target to the lifecycle, importance, and risk of the system.

A new experiment does not need the same reliability posture as login, feed, payments, messaging, or whatever your real critical path is.

That is a healthier conversation for SRE and platform teams to have with product and engineering leadership. Not “I need six nines for everything.” More like, “Here is the promise this system is making. Here is the cost of that promise. Here is the risk if we miss it. Is that the tradeoff we want?”

The AI part of this conversation is where things get more interesting.

Francois talked about AI helping with investigation. That makes sense. Incidents often involve too much data, too many dashboards, too many layers, and too many relationships between services. If AI can help gather telemetry, summarize patterns, generate queries, and point humans toward likely relationships faster, that is useful.

That is not the same thing as handing production to an agent and letting it freestyle the mitigation.

The useful version is more grounded. AI helps humans move faster during investigation. Humans still validate. Humans still decide. Humans still handle the judgment calls, especially when the system is important and the mitigation could make things worse.

But AI is also creating more change.

More diffs. More lines of code. More generated boilerplate. More changes moving through systems faster than before.

That creates a reliability gap that I think a lot of teams are going to feel.

The code can move faster than the understanding.

A product engineer may understand the user behavior and the business logic, but not every generated async callback, framework detail, or edge case buried in the generated implementation. Then when it breaks, the reliability or platform team has to reconstruct what happened under pressure.

What changed?

Why did it change?

What assumption did the generated code make?

What dependency is involved?

Is this a known pattern or something new?

Can we roll back?

Can we move traffic?

Can we isolate the failure?

That is not an anti-AI argument. It is just the operational reality.

If AI speeds up software delivery, the defensive side has to speed up too.

Observability has to improve. Debugging has to improve. Incident investigation has to improve. Recovery practice has to improve. The ability to understand generated code and generated changes has to improve.

Otherwise, teams are going to get more output without enough context, and that cost shows up during incidents.

That is why this episode pairs well with the At Scale Systems & Reliability conversation too. A lot of AI discussion is still stuck at the model layer or the code generation layer. But the infrastructure underneath matters. The systems that train, serve, move data, recover, and keep large AI workloads reliable matter. And the systems that use AI to improve reliability matter too.

That is the conversation I think SRE, DevOps, platform, and infrastructure teams need more of.

Not just “AI can write code.”

Not just “AI can summarize incidents.”

More like, what does the whole production system look like when AI increases the rate of change?

How do we preserve human understanding?

How do we validate what AI suggests?

How do we practice recovery?

How do we make sure the systems behind AI are reliable enough for the expectations being placed on them?

And how do we avoid pretending that a faster delivery loop automatically means a safer one?

The answer probably looks boring in the best way.

Write down the incidents. Review them honestly. Look for patterns. Practice the failures. Test the failover. Validate your assumptions. Make SLOs useful. Match reliability investment to product maturity. Use AI where it helps, but do not confuse investigation assistance with operational judgment.

That is not flashy, but it is the work.

And if there is one practical takeaway from this conversation, it is probably this: do not wait for production to be the first place your recovery plan gets tested.

Production will test it eventually.

The only real question is whether your team has already practiced.

This episode is about the hidden glue holding production together.

That phrase kept coming up while I was looking at this week’s stories. Not because the stories are all the same, but because the failure pattern underneath them feels familiar.

The thing that breaks production is not always the big obvious service.

Sometimes it is the recovery path.

The support workflow.

The generated file.

The tenant boundary.

The async event path.

The admin console.

The logs.

The thing between two systems that quietly decides what is allowed to happen next.

Coinbase’s May 7 outage is the cleanest SRE example. The starting point was physical: multiple chiller failures in a single AWS us-east-1 data hall, which led to a thermal safety shutdown affecting EC2 and EBS resources in one availability zone.

That part is important, but it is not the whole lesson.

Most teams already know an availability zone can fail. We all have the diagram. Three boxes, three AZs, clean arrows, nice labels, everyone nods.

The real question is whether the system can actually recover when one of those boxes disappears.

Coinbase’s postmortem is useful because it shows how quickly the conversation moves from “an AZ failed” to “what was actually coupled to that AZ?” Low-latency systems, stateful dependencies, managed Kafka recovery, ordering, offsets, recovery paths, and failover assumptions all start to matter very quickly.

That is the part worth taking back to your own environment.

Do not let “multi-AZ” become a comfort phrase.

Find the parts of the system that are still effectively tied to one zone, one stateful dependency, or one recovery path. Especially the low-latency pieces, Kafka pieces, databases, matching engines, and services with ordering or money-adjacent semantics.

Then ask the annoying questions.

What happens if that AZ disappears?

What has to recover in place?

What can actually fail over?

How long does recovery take?

And when was the last time anyone proved it?

That is where resilience becomes real.

The Meta AI support story is a different kind of glue, but the same theme. On the surface, it sounds like a consumer social media incident. Instagram accounts, password resets, account recovery.

But the platform angle is much more interesting.

AI support automation became part of an identity recovery control plane.

That matters because account recovery is privileged infrastructure. It can reset passwords, change emails, bypass parts of the normal login path, and help users regain access when the normal flow fails.

That means it has power.

So when AI is added to that workflow, the question is not just whether the answer is helpful. The question is what the workflow can actually do.

Can it only summarize?

Can it route tickets?

Can it recommend next steps?

Can it trigger actions?

Can it change trust?

Can it move an account recovery flow forward?

Those are very different risk profiles.

The takeaway is not “AI support is bad.” The takeaway is that support workflows can be production control planes, especially when they touch identity.

Password resets, MFA resets, email changes, support impersonation, admin ownership changes, and account recovery flows are not just customer service features. They are identity control paths.

If AI can influence them, they need real authorization logic, verification, rate limits, abuse detection, audit logs, escalation paths, and blast radius controls.

The scariest AI system in production might not be your coding agent. It might be the support flow that can reset passwords.

The AWS AgentCore CLI story is another version of the same problem. This time the glue is generated code.

The issue was CVE-2026-11393. In short, collaborator metadata from a Bedrock agent import could end up inserted into a generated Python file in a way that allowed code injection.

The important lesson is simple.

Generated code is still code.

That sounds obvious, but teams do not always treat generated code with the same suspicion as handwritten code. Generated files feel like artifacts. Outputs. Something official. Something the tool made, so maybe nobody needs to read it too closely.

But if metadata, instructions, templates, or agent definitions can become executable files, that path needs review.

This is not really a brand-new class of problem. It is the same old family as SQL injection, template injection, shell injection, CI config injection, and YAML templating weirdness.

Different costume, same villain.

The AI angle makes it feel new, but the secure engineering lesson is old: do not turn text into executable context without a hard boundary.

Do not assume metadata is safe because it came from an API.

Do not assume generated code is safe because it came from an official tool.

And do not assume “instructions” are harmless just because they are called instructions.

In agent systems, instructions can become operational inputs. They can shape behavior, move between systems, become config, become prompts, and sometimes become code.

So the practical takeaway is pretty direct. Keep the CLI updated. Inspect generated files before running them in trusted environments. Run agent tooling with limited permissions. And pay attention to what credentials are available if something goes wrong.

Because the blast radius is not just the bug.

The blast radius is what the generated code can access when it runs.

The Apigee story is about tenant isolation, and I kept this one in the episode because the lesson is still useful even though the specific cross-tenant issue comes from Google’s Apigee security bulletins rather than being a brand-new weekly disclosure.

Tenant isolation is one of the deepest assumptions in cloud and SaaS systems.

You have your environment. Other customers have theirs. The provider, or the platform team, keeps those boundaries in place.

That assumption is what lets anyone build on shared infrastructure at all.

But cross-tenant bugs often do not show up only in the obvious request path. They show up in the seams.

Logs.

Analytics.

Dashboards.

Exports.

Admin tools.

Support consoles.

Background jobs.

Search indexes.

The internal paths that still carry customer data.

That is why the Apigee issue is worth talking about. API management can sit close to access logs, analytics, client behavior, API paths, timing, errors, and operational metadata. Cross-tenant access to that kind of data is not just a privacy problem. It can become a reconnaissance problem, a compliance problem, and a customer trust problem.

The takeaway is not “never use managed services.” That would be ridiculous.

The takeaway is that tenant isolation has to be tested beyond the happy path.

Can tenant A see tenant B’s logs?

Can a support view cross the wrong boundary?

Can an analytics query forget tenant context?

Can an async job mix records that should never touch?

Can an export, dashboard, or search index leak data from the wrong tenant?

Those are platform questions.

The lightning round kept the same pattern going.

Cloudflare turning threat intelligence into real-time WAF rules is useful, but once threat intel enters the blocking path, it becomes production change management. Start in visibility mode, watch false positives, stage the rollout, and make sure the logs explain why something was blocked.

AWS Lambda tenant isolation with event source mappings is a reminder that tenant context has to survive async paths. Queues and event systems are where context gets lost, default tenants sneak in, retries behave differently, and batches mix things that should not mix.

OpenSearch Serverless scale-to-zero is a FinOps story, but also an operations story. Serverless does not mean operationally invisible. It means the operational questions moved to cold starts, indexing behavior, latency, cost patterns, and traffic spikes.

And GitHub Enterprise Managed Users IP allow list coverage is another reminder that source control is production infrastructure. Repo access is part of your security boundary, and that boundary keeps moving into identity, network policy, device posture, and enterprise governance.

Different stories.

Same pattern.

Modern reliability and security problems keep showing up in the seams.

Not always the app.

Not always the database.

Not always the obvious critical service.

Sometimes it is the thing between systems that nobody fully owns until it fails.

That is where a lot of real platform work lives now.

Not just making the app scale.

Not just adding another region.

Not just buying a managed service.

But asking where the authority actually sits.

Who can recover an account?

Who can generate code?

Who can access another tenant’s data?

Who carries tenant identity through a queue?

Who owns the failover path?

Who tested the recovery plan?

Who gets paged when the glue fails?

That work is not glamorous, but it is where a lot of production risk hides.

So the takeaway from this episode is simple:

Do not just review the big production systems.

Review the seams.

That is usually where the incident is waiting.

Extra links worth including on the episode page:

Coinbase May 7 outage postmortem
https://www.coinbase.com/blog/a-postmortem-of-our-may-7-2026-outage

Meta AI support / Instagram account recovery reporting
https://www.theverge.com/tech/945658/meta-ai-support-chatbot-exploit-instagram-accounts

AWS AgentCore CLI CVE-2026-11393
https://aws.amazon.com/security/security-bulletins/2026-040-aws/

AgentCore CLI GitHub advisory
https://github.com/aws/agentcore-cli/security/advisories/GHSA-m4x6-gwgp-4pm7

Google Apigee security bulletins
https://docs.cloud.google.com/apigee/docs/security-bulletins/security-bulletins

Cloudflare real-time threat intel WAF rules
https://blog.cloudflare.com/realtime-threat-intel-waf-rules/

AWS Lambda tenant isolation with event source mappings
https://aws.amazon.com/blogs/compute/integrating-event-source-mappings-with-aws-lambda-tenant-isolation-mode/

Amazon OpenSearch Serverless next generation
https://aws.amazon.com/about-aws/whats-new/2026/05/amazon-opensearch-serverless-next-generation-generally-available/

GitHub Enterprise Managed Users IP allow list coverage
https://github.blog/changelog/2026-06-08-ip-allow-list-coverage-for-emu-namespaces-in-general-availability/

This week’s On Call Brief
https://www.tellerstech.com/on-call-brief-news/2026-W24/

More Ship It Weekly episodes
https://shipitweekly.fm/