Ship It Weekly Host Commentaries
Host commentary is the written layer behind each episode: judgment calls, context the audio did not have time for, and links worth bookmarking. This archive collects every episode that ships with commentary so you can skim by week without opening the full player.
Commentary is distinct from show notes (RSS descriptions) and transcripts. Show notes summarize the episode; commentary is the host's editorial read on what mattered and why.
What this page is for
What host commentary is
Editorial context from the host — not a recap of the audio. Expect opinions, follow-up links, and the operational framing that does not fit in a headline.
Read inline or on the episode page
This archive shows full commentary text for browsing and search. Open any episode for audio, chapters, transcripts, and show notes in one place.
Pair with transcripts
Prefer the spoken word? The transcript archive lets you search episode dialogue without scrubbing audio. Episode transcripts →
Read host commentaries
This page is for you if…
- You want the host's take without listening to the full episode
- You are sharing operational context with your team in writing
- You prefer editorial framing over RSS show-note summaries
- You bookmark links and references from weekly news roundups
Ship It Conversations: Evan Phoenix of Miren on Deployment Pain, Terraform, Waypoint, and Better Defaults for Small Teams
This conversation with Evan hit on something I think a lot of platform and DevOps teams feel, even if they do not always say it directly: deployment is still weirdly painful.
That sounds almost ridiculous at this point. We have had so many waves of tools that were supposed to make this easier. Cloud platforms. Containers. Kubernetes. Terraform. GitOps. Internal developer platforms. PaaS products. CI/CD systems. Now AI agents that can generate a bunch of code and infrastructure glue before you have even had enough coffee to regret the prompt.
And yet, for a lot of teams, the actual act of taking an application and safely getting it into production still feels harder than it should.
I liked Evan’s framing of deployment as the “final boss” of software delivery, because it really does sit at the intersection of everything else. Deployment is not just “run the app.” It is the app, the runtime, the container image, the registry, the network path, the secrets, the config, the database, the rollback, the logs, the thing that wakes somebody up, and the thing the next person has to understand when something breaks.
That is why deployment tools tend to get complicated so quickly.
Every app is shaped a little differently. One HTTP app is not the same as another HTTP app. A Ruby app is not the same as a Go service. A JavaScript app is not the same as a Java service. A background worker is not the same as a public API. Then you multiply that by every cloud provider, every orchestration model, every security requirement, every team structure, every compliance need, and suddenly the “simple” deployment platform has to make a lot of choices.
And that is really where the conversation got interesting to me.
A lot of engineers say they want flexibility. And honestly, we do. We like having knobs. We like knowing that we can tune something if we need to. We like not being boxed in.
But there is a cost to that.
Every knob is a decision. Every decision is something somebody has to understand. Every exception is something the team has to support. And eventually the platform that was supposed to make delivery easier becomes another product the team has to maintain.
That is not always bad. Sometimes you really do need the big flexible platform. Sometimes Kubernetes is exactly the right answer. Sometimes Terraform gives you the control and consistency you need. Sometimes the team has enough scale, enough complexity, or enough operational maturity that the extra surface area is worth it.
But that is not every team.
Evan kept coming back to small teams. Teams that are application-focused. Teams that just want to ship. Teams that do not have a full platform engineering group sitting around waiting to build and maintain abstractions on top of abstractions. Teams that maybe need deploys, logs, rollbacks, config, HTTP apps, and a path to production that does not require them to become experts in every layer underneath it.
That is the part I think a lot of platform conversations miss.
The goal is not always to build the most powerful platform. Sometimes the better question is: how little platform can we get away with while still shipping safely?
I also liked the Terraform Enterprise and Terragrunt thread because it gets at the same idea from a different angle.
The hard part of Terraform was never only the language. It was the shape around it. How do you organize repos? How do you manage state? How do you split modules? How do you reduce blast radius? How do you avoid one giant infra repo where every change feels like it is poking a thousand resources?
That is why tools like Terragrunt became useful for a lot of teams. Not because everyone loves another layer, but because people wanted an opinion. They wanted guardrails. They wanted someone to say, “put this here, structure it like this, and stop inventing a new pattern for every environment.”
That desire for opinionated tooling is not a weakness. It is usually a sign that the team has better things to do than debate folder structure forever.
The OpenTofu discussion was interesting too. The license change obviously mattered, but Evan’s point was that OpenTofu also had a backlog of features people had wanted for a while. Some of those features were things Terraform had intentionally avoided, and in some cases for valid operational reasons. Encrypted state was the example that stood out. It sounds good, and in many ways it is good, but it also creates another availability and recovery problem if the keys become the thing blocking your infrastructure operations.
That is a very real infrastructure tradeoff. Security, recoverability, simplicity, and operability are always pulling on each other.
The Waypoint section may have been my favorite part of the conversation.
I remember Waypoint being one of those tools that seemed like it should make a ton of sense. A deployment workflow engine from HashiCorp? That sounds like something a lot of teams would want.
But Evan’s takeaway was pretty blunt: a pure workflow engine was not enough product.
That stuck with me.
Because if a tool makes you do a ton of homework before it gives you value, you are going to lose people. If the user has to set up the registry, wire the backend, understand where images go, configure ECS or Kubernetes or whatever else, and then finally maybe get to deploy something, that is a lot to ask.
Especially if the person’s actual job is not “become an expert in deployment plumbing.” Their job is to ship the application.
That also maps directly to the internal platform problem. How many companies have a deployment system someone started building because the team needed something better, then the boss told them to stop working on it because, technically, the company is not in the deployment platform business?
So now everyone is stuck with a half-finished platform that mostly works, but only if you know the weird ritual.
That Jenkins comment in the episode was painfully real. I think a lot of us have had some version of that. The job works, nobody knows why, nobody wants to touch it, and the entire deployment process is held together by fear and one person’s ancient Groovy choices.
The AI angle makes all of this even more important.
AI can absolutely help with infrastructure and deployment. It can generate Terraform. It can wire together services. It can produce config. It can read documentation and maybe map an app to a platform faster than a person could.
But Evan’s question was the right one: is the AI on call?
When the generated infrastructure breaks, who owns it? Who understands why those services were wired together that way? Who knows which parts are safe to change? Who knows whether the AI made a reasonable long-term operational choice or just produced something that worked once?
That is the risk I keep seeing with AI-generated infrastructure. It can make the first draft easier, but the first draft is not the same thing as operational ownership.
And maybe that means good abstractions matter more, not less.
If the platform surface area is small and obvious, AI has fewer ways to get it wrong. Deploy. Roll back. Get logs. Set config. Check status. That is very different from telling an agent to wire together a dozen AWS services and hoping it invents a sane deployment platform your team can operate for the next three years.
So the real future might not be AI replacing deployment platforms.
It might be AI making the difference between good and bad platform abstractions even more obvious.
The practical takeaway for me is this: be careful which opinions you adopt, and be even more careful which opinions you accidentally create.
If you choose Kubernetes, you are adopting opinions. If you choose Terraform, you are adopting opinions. If you choose Terragrunt, OpenTofu, ECS, Heroku, Waypoint, Miren, or an internal deployment system, you are adopting opinions there too.
The question is whether those opinions match your team.
Do they match your application? Do they match your operational maturity? Do they match the amount of infrastructure ownership you actually want? Do they help your team ship safely, or do they just move the pain somewhere else?
That is why I liked this conversation. It was not really a “this tool versus that tool” conversation. It was more about the shape of deployment work, and why smaller, clearer, more opinionated systems may be the right answer for more teams than we usually admit.
Sometimes the best platform is not the most powerful one.
Sometimes it is the one that lets the team ship, understand what happened, roll back when needed, and then get back to building the thing customers actually care about.
Additional Links Mentioned
Miren:
https://miren.dev
Miren Discord:
https://miren.dev/discord
Miren on GitHub:
https://github.com/mirendev
Evan Phoenix:
https://evanphx.dev
Evan on Bluesky:
https://bsky.app/profile/evanphx.dev
Puma:
https://puma.io
Rubinius:
https://github.com/rubinius/rubinius
Terraform:
https://www.terraform.io
Terraform Enterprise:
https://developer.hashicorp.com/terraform/enterprise
Terraform Cloud:
https://developer.hashicorp.com/terraform/cloud-docs
Terragrunt:
https://terragrunt.gruntwork.io
OpenTofu:
https://opentofu.org
HashiCorp Waypoint:
https://github.com/hashicorp/waypoint
Kubernetes:
https://kubernetes.io
Docker:
https://www.docker.com
AWS ECS:
https://aws.amazon.com/ecs/
Heroku:
https://www.heroku.com
Knative:
https://knative.dev
HashiCorp Vault:
https://developer.hashicorp.com/vault
Scroll inside the box to read the full commentary.
Amazon Q CVEs, Hijacked npm and Go Packages, AWS WAF HTTP/2 Issues, Lambda MicroVMs, and Why Execution Is the Boundary Now
This episode is about execution boundaries.
That sounds like a security phrase, but it is really a platform phrase now.
Because modern systems are full of places where something gets permission to run, inspect, transform, block, or decide.
That “something” might be an IDE extension.
It might be a language server.
It might be an AI coding assistant.
It might be a VS Code task.
It might be a WAF parser.
It might be a sandbox running user-generated code.
It might be an advisory database feeding alerts into a vulnerability management process.
The common thread is execution.
Before something runs, trust is still theoretical.
After something runs, trust becomes a blast radius.
That is why the Amazon Q Developer and AWS Language Server CVEs matter. The scary part is not just that there were bugs in an IDE plugin. The scary part is where those tools sit.
They sit next to source code.
They sit next to local credentials.
They sit next to AWS profiles.
They sit next to project configuration.
They sit next to terminals, repo context, MCP configuration, and sometimes production access.
So when a workspace can cross into command execution, that is not just a weird local developer bug. It can become a cloud credential problem, a source control problem, or a CI/CD problem.
And yes, the user still had to trust the workspace.
But that is exactly the uncomfortable part.
Developers trust workspaces constantly.
You clone a repo, open it, click through the prompt, and get back to work. That is normal behavior. But normal behavior becomes risky when the repo is no longer just source code. The repo is config. The repo is tasks. The repo is language server behavior. The repo is MCP setup. The repo is local execution context.
That is a much bigger boundary than most teams treat it as.
The JFrog story pushes that same idea even harder.
The hijacked npm and Go packages were not relying on the usual npm lifecycle-script path. They used hidden VS Code tasks configured to run when a folder opens.
That is the part that should make people pause.
The ecosystem is getting better at package-manager security. npm v12 is changing install-script defaults. CI pipelines are starting to pay more attention to dependency execution. Teams are getting more suspicious of install-time code.
That is good.
But attackers adapt.
If the package manager gets harder, they move into the IDE. And the IDE is a great execution surface because developers open folders all day.
Example repos.
Proofs of concept.
Vendor samples.
Issue reproductions.
AI-generated apps.
Random repos from Slack, GitHub issues, Discord, or wherever else work happens.
The editor is not passive anymore. It runs extensions. It loads config. It starts language servers. It runs tasks. It reads environment variables. It can access terminals and local secrets.
So “don’t run random code” is not enough advice anymore.
Sometimes “opening the folder” is the thing that runs code.
The AWS WAF HTTP/2 story is different, but it still fits.
A WAF is supposed to inspect requests and block bad ones. But that only works if the WAF sees the request the same way the rest of the stack sees it.
If HTTP/2 request bodies are split across frames, and the inspection layer only evaluates part of the body before making a decision, then the security control is operating on an incomplete view.
That is the whole problem.
Security controls are not magic shields.
They are parsers, policies, and defaults.
And if the parser has a blind spot, the policy may be correct but still not protect you.
That is why “do we have WAF?” is not the right question.
The better question is: what does the WAF actually see?
Where does TLS terminate?
Is HTTP/2 involved?
Are request bodies accumulated before inspection?
Does ALB behave the way the protection model assumes?
Do the proxy, load balancer, WAF, app server, and backend framework all agree on what the request means?
Because when those layers disagree, attackers get room to move.
Then AWS Lambda MicroVMs are the other side of the same episode.
The first few stories are about unexpected execution paths. Lambda MicroVMs are about building a better place for execution to happen.
That matters because running untrusted code is becoming normal.
AI coding assistants need sandboxes.
Users upload code.
Agents generate scripts.
Security tools analyze unknown payloads.
Data platforms run user-defined logic.
Developer platforms execute test cases.
Plugin systems need isolation.
The question is no longer “will we run untrusted code?”
For a lot of teams, the question is “where are we already running it, what can it reach, and how badly can it hurt us?”
Containers are useful, but shared-kernel isolation is not always the right boundary for untrusted code.
Full VMs are stronger, but heavier.
Serverless is convenient, but not every sandbox use case fits the traditional request-response Lambda model.
So the product direction here is interesting even if you do not adopt this specific AWS feature. Sandboxing is becoming a first-class platform primitive.
That is the larger trend.
Execution is becoming something platform teams need to design, not something they inherit accidentally.
The lightning round keeps reinforcing it.
GitHub’s advisory database hitting record volume is a reminder that vulnerability management is not just about getting more feeds. It is about deciding what matters before the queue buries the team.
Git 2.55 is a reminder that developer platforms are infrastructure too. At enough scale, Git performance becomes engineering productivity.
Valkey 9.1 on ElastiCache is a reminder that cache engines are production systems. Memory efficiency, throughput, command behavior, and isolation matter when the cache sits directly in the request path.
And the Fable 5 callback is a reminder that AI dependency risk does not disappear just because the model came back online. If a model can go away for policy reasons and then return with new access rules or behavior, that is still something reliability teams need to track.
That is why the closer lands where it does.
Execution is where trust becomes real.
A repo is not just source code if opening it can trigger an assistant, language server, or task runner.
An IDE is not just an editor if it can run commands, load tools, and inherit credentials.
A WAF is not just a checkbox if it only inspects part of the request.
A sandbox is not optional if your product runs code from users, agents, or plugins.
An advisory feed is not useful just because it is complete. Someone still has to decide what matters.
So the practical questions this week are pretty simple:
Where does code run?
What can it reach?
What can it read?
What can it write?
What authority does it inherit?
What logs prove what happened?
Who owns that boundary?
And if the answer is “we’re not sure,” that is the work.
Because before execution, trust is a policy.
After execution, trust is a blast radius.
Extra links worth including:
AWS security bulletin: Amazon Q / AWS Language Server CVEs
https://aws.amazon.com/security/security-bulletins/2026-047-aws/
JFrog: Hijacked npm packages using VS Code tasks
https://research.jfrog.com/post/hijacked-npm-vscode-tasks-blockchain/
AWS security bulletin: AWS WAF HTTP/2 inspection issues
https://aws.amazon.com/security/security-bulletins/2026-048-aws/
AWS Lambda MicroVMs
https://aws.amazon.com/blogs/aws/run-isolated-sandboxes-with-full-lifecycle-control-aws-lambda-introduces-microvms/
GitHub Advisory Database record volume
https://github.blog/security/supply-chain-security/inside-the-advisory-database-and-what-happens-when-vulnerability-volume-breaks-records/
Git 2.55 highlights
https://github.blog/open-source/git/highlights-from-git-2-55/
Amazon ElastiCache Valkey 9.1
https://aws.amazon.com/blogs/database/announcing-valkey-9-1-for-amazon-elasticache/
Claude Fable 5 and Mythos 5 model docs
https://platform.claude.com/docs/en/about-claude/models/introducing-claude-fable-5-and-claude-mythos-5
This week’s On Call Brief
https://www.tellerstech.com/on-call-brief-news/2026-W27/
More Ship It Weekly episodes
https://shipitweekly.fm/
Scroll inside the box to read the full commentary.
Ship It Conversations: Kat Traxler of Vectra AI on AI Security, the Zero-Day Clock, IAM, and Cloud Risk
For this Conversations episode, the part that stuck with me is that AI security is not really one conversation.
It is a speed conversation.
It is a context conversation.
It is a fundamentals conversation.
And honestly, it is still very much a people conversation.
That is what I liked about talking with Kat Traxler from Vectra AI. She is not dismissing the AI security concern. There are real changes happening. Models are getting better at finding certain classes of bugs. Researchers can move faster. Bug hunters can narrow huge codebases down more quickly. Attackers can also use the same kind of tooling to move faster.
That part matters.
But the part that gets lost in a lot of the panic is that speed is not the same thing as understanding.
AI can help find suspicious code paths. It can help with injection bugs. It can help with RCE patterns. It can help generate queries, first drafts, summaries, and research direction. It can make the first pass through a messy problem less painful.
But it does not automatically know the threat model.
It does not always understand privilege.
It does not know what your users expect.
It does not know what your business logic means.
It does not know whether something is actually exploitable in your environment, with your data, your permissions, your deployment shape, and your weird internal assumptions.
That is where human judgment still matters.
And that is probably the most important theme of this episode.
The model can help find needles, but the human still has to know which needles matter.
That came up a few different ways. Bug hunting, writing, cloud analysis, IAM, prompt injection, insecure-by-design flaws. The pattern is pretty consistent. AI can accelerate parts of the work, but if you do not have the expertise to challenge it, constrain it, and validate it, it can make you faster in the wrong direction.
Kat’s example with VPC flow logs is a good one. If a model tells you that you can infer packet-level details from data that does not actually contain those fields, that answer might sound useful if you do not know better. But if you do know the system, you can push back and say, no, that is not possible. Here are the fields we actually have. Here is what can and cannot be inferred.
That is the difference between AI as leverage and AI as a confidence machine.
And that matters a lot in security.
Because security does not reward confident guesses.
The other thing that stood out to me is Kat’s pushback on the full zero-day apocalypse narrative.
There is a real issue underneath it. The time between vulnerability disclosure and exploitation has been shrinking. AI may compress that even further for certain classes of vulnerabilities. That is not fake. Security teams should pay attention to that.
But the jump from “attackers may weaponize faster” to “global technology collapses in a few months” is a much bigger leap.
And Kat’s point was basically, let’s spend more time on the “and then what?”
If the zero-day clock keeps shrinking, what actually happens?
Which teams are most exposed?
Which systems are most at risk?
Which attackers change behavior?
Which organizations need to respond differently?
And which problems are still the same boring problems they were yesterday?
Because that is the part that I think a lot of DevOps, SRE, platform, and security teams need to hear.
The scary future does not erase the current fundamentals.
Credentials still matter.
IAM still matters.
Misconfigurations still matter.
Known vulnerabilities still matter.
Patch management still matters.
Source code secrets still matter.
Human fatigue still matters.
Attackers are practical. They are going to take the lowest-friction path that works. A lot of the time, that path is not a cinematic AI-generated zero-day chain. It is a leaked key. A reused credential. An over-permissive role. A service account nobody remembered. A server that should have been patched six months ago.
That does not mean AI risk is overblown.
It means we need to hold both ideas at the same time.
AI may make certain security problems move faster.
But if your IAM is already a mess, your secrets are everywhere, your patching is inconsistent, and nobody owns the exposed attack surface, the future AI threat is not an excuse to ignore the thing already sitting on fire.
The IAM part of the conversation felt especially true.
IAM is hard because it looks like a technical service, but it is really where people, process, and technology collide.
It is credentials. It is authentication. It is access. It is human behavior. It is shortcuts. It is tired people. It is teams under pressure. It is old roles. It is permissions nobody wants to remove because something might break. It is attackers knowing that credential abuse is often the easiest way in.
That is why IAM stays painful.
Not because nobody understands least privilege as a concept.
Everyone understands the concept.
The hard part is making it work in a real organization where people are trying to ship software, troubleshoot production, rotate keys, grant access, clean up old permissions, and not break the thing that pays the bills.
So when Kat says IAM is really about people, I think that is right.
The insecure-by-design discussion was another useful thread.
A lot of AI security focus is on code-level vulnerabilities. Can the model find injection bugs? Can it find RCE? Can it find XSS? Can it narrow the search space?
That work matters.
But insecure-by-design flaws are different.
Those are not always bad lines of code. They are bad assumptions.
The AirTag example is a good way to think about it. The device may function as designed, but the design missed a real-world abuse case. The issue is not just technical correctness. It is whether the product accounts for how people can misuse it, who can be harmed, what customers expect, and what happens when the system is used in a way the builders did not intend.
That is a different class of security work.
And I suspect it is going to stay valuable, even as AI gets better at finding some code-level bugs.
Maybe especially then.
If AI makes some of the easier code-level searching faster, more of the important work may shift toward context, abuse paths, design flaws, threat modeling, and the uncomfortable question of what the system allows someone to do that we did not intend.
That is harder.
It is also probably more important.
So my takeaway from this episode is not “panic about AI security.”
It is also not “everything is fine.”
It is more like, keep your head.
Use AI where it helps. Let it speed up the annoying parts. Let it help with first passes, rough drafts, code search, query generation, pattern matching, and research workflows.
But do not outsource judgment.
Do not let the AI output become the security conclusion.
Do not confuse a confident answer with a correct one.
And do not let the excitement around future AI-driven attacks distract from the fundamentals that are still hurting teams right now.
Threat model the new stuff.
Patch the known stuff.
Clean up IAM.
Protect credentials.
Own your exposed attack surface.
Write down what your system actually promises.
And when AI gives you an answer, treat it like a starting point, not the finish line.
Because the future may move faster.
But the work still has to be done.
Scroll inside the box to read the full commentary.
containerd CRI Vulnerabilities, Datadog PostgreSQL HA on Kubernetes, AWS DevOps Agent with Datadog MCP Server, EKS Control Plane Egress, and Why Users Feel the Wait
This episode is really about the control plane getting wider.
That sounds like a platform-engineering phrase, but it is becoming one of the more important ways to think about modern production systems.
A few years ago, when people said “control plane,” they usually meant something fairly specific. Kubernetes API server. Cloud API. CI/CD system. Maybe an internal deployment platform.
Now it is messier than that.
Your container runtime is part of the control plane because it decides how workloads actually start on the node.
Your database failover automation is part of the control plane because it decides whether recovery is safe or reckless.
Your AI incident-response agent is part of the control plane because it can inspect telemetry, summarize what changed, recommend action, and maybe someday trigger work directly.
Your Kubernetes API server egress path is part of the control plane because a stale route table or broken firewall path can stop admission webhooks, OIDC, and aggregated API calls from working.
Your credential revocation tooling is part of the control plane because compromised access has to be cut off fast.
Your cloud console is part of the control plane because operators still need a way to reach the environment during an incident.
Even object metadata starts to matter when data, AI, search, and agent workflows depend on understanding what an object is, not just where it lives.
That is the through-line in this episode.
containerd disclosed a batch of CRI plugin vulnerabilities, and the lesson is that Kubernetes security does not stop at pod specs, RBAC, admission control, or image scanning. Eventually the node runtime has to pull the image, unpack it, restore it, wire up devices, handle logs, and start the container. That runtime layer is not invisible plumbing. It is a trust boundary.
The Datadog PostgreSQL HA story is a different kind of control-plane lesson. Their gameday did not just ask whether PostgreSQL could fail over on Kubernetes. It exposed the harder question: can it fail over safely? If every standby is behind, promotion may be possible, but it may not be correct. And in databases, correct usually matters more than fast.
That is the part I love about the Datadog writeup. It is not the fantasy version of HA where automation magically fixes everything. It is the real version where replication lag, synchronous writes, RPO, RTO, and promotion safety all collide. Failover is only useful if the recovery path does not create a bigger problem.
The AWS DevOps Agent and Datadog MCP Server story pushes this same theme into AI operations. AI incident response is moving from demo to production workflow. That is exciting, but the question cannot just be “is the agent smart?” The better question is “what authority does it have?”
Can it only read?
Can it write?
Can it open tickets?
Can it trigger automation?
Can it roll back?
Can it page someone?
Can it make things worse quickly and very confidently?
That is the uncomfortable part. AI incident tooling can be genuinely useful, especially during the early chaos of an incident when everyone is jumping between dashboards, traces, logs, deploy history, and Slack threads. But once an agent sits near the operational control plane, it needs the same boring guardrails as any other production automation: least privilege, audit logs, approval boundaries, rollback rules, and a clear line between recommendation and execution.
Then there is EKS customer-routed control-plane egress, which is one of those features that sounds boring until you think through the failure modes. Routing Kubernetes API server outbound traffic through your own VPC is a real win for private and regulated environments. But it also means your route tables, security groups, NACLs, firewalls, and private connectivity can now become control-plane dependencies.
That is powerful.
It is also something you bring to a design review.
The lightning round kept hitting the same idea from different angles. GitHub credential revocation is incident-response infrastructure. AWS Console Private Access pulls more operator workflow behind private network boundaries. Vercel Connect points toward short-lived, task-scoped credentials for agents instead of long-lived secrets sitting around forever. S3 annotations make object metadata more directly attached, mutable, and queryable instead of living in another side table that drifts from reality.
Different stories, same shape.
Authority keeps moving.
Trust keeps spreading.
The blast radius keeps expanding.
And that is where Marc Brooker’s post on waiting fits so well as the closer.
Your dashboards may measure averages, but your users do not experience averages. They experience the time they spend waiting. A ten-minute outage and a ten-hour outage might both count as one incident in a tracker, but they do not feel the same to the people stuck inside them. A service with a decent average can still feel terrible if users keep landing in the tail.
That is the reliability lesson underneath the whole episode.
When the system breaks, users do not experience your architecture diagram. They do not care whether it was the runtime, the database, the AI agent, the credential system, the route table, the cloud console, or the metadata layer.
They experience waiting.
Waiting for a request.
Waiting for recovery.
Waiting for a deploy to stop failing.
Waiting for a credential to get revoked.
Waiting for the control plane to come back.
Waiting for someone to find the right context.
So the practical question is not just “is this system up?”
It is also:
Where is authority hiding?
What has to work before recovery can happen?
Which defaults are trusted?
Which control-plane paths are invisible?
Which tools can make changes?
Which systems can block deploys?
Which dependencies only show up when something breaks?
And most importantly, what does this feel like from the waiting side?
Because your dashboards measure the average.
Your users feel the wait.
Extra links worth including:
containerd CRI plugin vulnerabilities / AWS security bulletin
https://aws.amazon.com/security/security-bulletins/2026-046-aws/
Datadog: PostgreSQL high availability on Kubernetes
https://www.datadoghq.com/blog/engineering/postgresql-ha-kubernetes/
AWS DevOps Agent and Datadog MCP Server
https://aws.amazon.com/blogs/devops/production-ready-autonomous-incident-resolution-with-aws-devops-agent-now-ga-and-datadog-mcp-server/
Amazon EKS customer-routed control-plane egress
https://aws.amazon.com/blogs/containers/amazon-eks-now-supports-control-plane-egress-through-your-vpc/
GitHub self-service credential revocation for incident response
https://github.blog/changelog/2026-06-24-self-service-credential-revocation-for-incident-response/
AWS Management Console Private Access
https://aws.amazon.com/about-aws/whats-new/2026/06/aws-management-console-private/
Vercel Connect
https://vercel.com/blog/introducing-vercel-connect
Amazon S3 annotations
https://aws.amazon.com/blogs/aws/amazon-s3-annotations-attach-rich-queryable-context-directly-to-your-objects/
Marc Brooker: Waiting, latency, MTTR, and the inspection paradox
https://brooker.co.za/blog/2026/06/19/waiting.html
This week’s On Call Brief
https://www.tellerstech.com/on-call-brief-news/2026-W26/
More Ship It Weekly episodes
https://shipitweekly.fm/
Scroll inside the box to read the full commentary.
Ship It Conversations: Guardsquare’s Joel DeStefano on Mobile App Security, Runtime Protection, App Hardening, and Why Scanning Isn’t Enough
For this Conversations episode, the part that stuck with me is how easy it is for backend and cloud people to bring the wrong mental model to mobile security.
In backend systems, we are used to control.
We control the runtime. We control the infrastructure. We control IAM. We control the network path. We control the deployment process. We control observability, logging, and a lot of the runtime environment around the application.
Mobile changes that.
You ship the app and now it is out there.
It is running on someone else’s device, in someone else’s environment, with software you do not control, under conditions you may never see directly. Maybe the device is rooted. Maybe it is jailbroken. Maybe there is malware nearby. Maybe someone is decompiling the app, running instrumentation, hooking values, bypassing checks, or trying to understand enough of the workflow to abuse the backend.
That is the trust shift Joel DeStefano kept coming back to in this conversation.
Mobile app security is not just “did we scan the app before release?”
Scanning matters. Dependency checks matter. TLS configuration matters. Not shipping secrets in the app matters. All of that is table stakes.
But it is point-in-time assurance.
It tells you something before release.
It does not tell you what happens after the app leaves the pipeline.
That is where I think a lot of teams get too comfortable. They assume the App Store review process is handling more than it is. They assume the OS is protecting the application logic. They assume that because the backend stores the sensitive data, the mobile app itself is not an interesting target.
That last one is probably the most dangerous assumption.
The sensitive thing is not always a secret string sitting in the app.
Sometimes the sensitive thing is the workflow.
How does the app authenticate? How does it talk to the backend? What does the backend trust? What values does the app send? What checks happen locally? What behavior changes when a condition flips from false to true?
If an attacker can learn the business logic, they may not need a database dump from the app. They may only need enough understanding to abuse the system around it.
That is especially true for apps tied to money, login, account recovery, medical workflows, retail transactions, subscriptions, loyalty points, streaming access, or anything else that represents real business value.
So the better question is not just, “Do we have secrets in the app?”
The better question is, “What can someone learn from this app, and how could they use that against us?”
That is a much more useful way to think about mobile security.
I also liked how Joel framed the layered approach.
Testing matters.
Hardening matters.
Runtime protection matters.
Monitoring matters.
Those are not interchangeable pieces. They solve different parts of the problem.
Testing and scanning help catch issues before release. Hardening and obfuscation make reverse engineering harder. Runtime checks help detect suspicious environments, debuggers, hooking frameworks, rooted devices, overlays, accessibility abuse, and other signs that something about the app execution is not right. Monitoring helps teams understand what is actually happening after the app is in the wild.
That monitoring piece is important because the threat landscape does not freeze when you ship.
Attackers change tools. Tactics change. Activity spikes in different regions. A new framework gets popular. A workflow starts being abused in a way nobody expected.
If you are not watching, you are guessing.
That does not mean every app needs the same level of protection. A tiny internal utility is not the same as a banking app, a healthcare app, or a payment flow. But a lot more apps are business-critical than people admit.
Mobile apps are not just “apps” anymore.
They are storefronts. Payment platforms. Account portals. Medical device managers. Subscription systems. Streaming platforms. Banking experiences. Identity and recovery paths.
If that is where users spend time and money, attackers will spend time there too.
The CI/CD part of this conversation is also worth paying attention to.
Security that destroys the release process usually does not last.
That is true in cloud. It is true in platform engineering. It is true in mobile too.
If the security tooling slows everything down, breaks builds constantly, creates stability issues, or forces teams into a painful release process, engineering teams will eventually route around it. Maybe not officially. Maybe not loudly. But they will find a way to keep shipping.
So the goal has to be security that lives inside the real delivery process.
High-severity findings should block. Critical issues should stop the release. But lower-severity issues may become tracked risk. Runtime protections still need QA. Performance constraints still matter. Stability matters. App size can matter. A few milliseconds might be acceptable. A crash is not.
That is the tradeoff.
Not “security versus shipping.”
More like, “how do we make security part of shipping without pretending delivery constraints do not exist?”
That is a healthier conversation for DevOps, SRE, platform, AppSec, and mobile teams to have together.
I also thought the AI angle was interesting because it fits the broader theme we keep seeing across security.
AI may make attackers faster.
It may help them understand app logic faster. It may help them reason about code faster. It may help them reverse engineer faster. It may compress the time between “I have this app” and “I understand where to poke.”
But Joel’s point was that the fundamentals do not disappear.
If you already have the right model, AI increases pressure.
If you do not have the right model, AI makes the gap more painful.
That seems broadly true across security right now. AI is not replacing the need for good fundamentals. It is raising the cost of not having them.
For mobile, those fundamentals look like understanding the trust model, protecting the app, making reverse engineering harder, adding runtime checks, validating that the app connecting to your backend is genuine, and monitoring what is happening in the field.
The biggest mistake is assuming someone else has solved the problem for you.
The App Store will not save you.
The OS will not save your business logic.
A scanner will not protect runtime behavior.
And “we do not store sensitive data in the app” does not mean the app has nothing valuable to reveal.
That is probably my main takeaway from this episode.
Mobile app security is not magic, but it is different.
It is a different trust boundary. It is a different runtime model. It is a different visibility problem. And it needs to be treated as part of the system, not as a separate thing that gets checked at the end.
Because once the app is shipped, the question is not just whether you built it safely.
The question is whether you can still trust what is connecting back.
Scroll inside the box to read the full commentary.