Ship It Weekly Host Commentaries
Host commentary is the written layer behind each episode: judgment calls, context the audio did not have time for, and links worth bookmarking. This archive collects every episode that ships with commentary so you can skim by week without opening the full player.
Commentary is distinct from show notes (RSS descriptions) and transcripts. Show notes summarize the episode; commentary is the host's editorial read on what mattered and why.
What this page is for
What host commentary is
Editorial context from the host — not a recap of the audio. Expect opinions, follow-up links, and the operational framing that does not fit in a headline.
Read inline or on the episode page
This archive shows full commentary text for browsing and search. Open any episode for audio, chapters, transcripts, and show notes in one place.
Pair with transcripts
Prefer the spoken word? The transcript archive lets you search episode dialogue without scrubbing audio. Episode transcripts →
Read host commentaries
This page is for you if…
- You want the host's take without listening to the full episode
- You are sharing operational context with your team in writing
- You prefer editorial framing over RSS show-note summaries
- You bookmark links and references from weekly news roundups
More ways to read
Skim transcripts, host commentaries, and show notes across every episode
Java4Shell OpenJDK Flaw, Microsoft Agent Framework PyPI Package Hijacked, Confluence Axis RCE Exploited & FAA Ground Stop Traced to a Router
Java4Shell OpenJDK Flaw, Microsoft Agent Framework PyPI Package Hijacked, Confluence Axis RCE Exploited & FAA Ground Stop Traced to a Router
The Layer Under the Code
This week's stories all lived one layer below where most of us look.
Not in the application. Not in the code anyone on your team wrote last sprint. Underneath it: an instruction set on the CPU, a file Python reads before your program starts, a SOAP library older than most of the people running it, and a router at a remote radio site.
That's the layer we inherit. We rarely inventory it, and we almost never own it on purpose.
Start with Java4Shell.
CVE-2026-114938 is a memory corruption bug in the OpenJDK HotSpot JVM, fixed on Tuesday, November 17th. It affects Java 17 and newer, and every downstream build: Corretto, Temurin, Zulu and the rest. To decode Base64 quickly, the JVM swaps the normal Java code for hand-written assembly that uses AVX-512 instructions. That assembly miscounts the output size when the input ends with an unexpected number of equals signs, and the JVM writes past the end of a heap buffer. Calif, which found it, scores it 9.2. Oracle says 8.1, because getting code execution out of it is hard.
What I find interesting is how narrow the path is. It only runs on x86 CPUs with the AVX-512 VBMI instructions: Intel Xeons from Ice Lake on, and AMD Zen 4 and newer. ARM64 never takes it, so Graviton nodes are fine. Two hosts running the exact same JDK build can have different answers to "am I exposed?", depending on the silicon underneath them.
My take: that's the part an inventory usually misses. We track JDK versions, sometimes. We almost never track which CPU family each JVM landed on. And the JDKs that get missed aren't in the service you deploy every day. They're in base images, in a runtime bundled inside a vendor appliance, and on that build agent nobody has rebuilt since spring.
So patch on your normal schedule, but go find all of it. On Linux,
grep avx512vbmi /proc/cpuinfotells you whether a host can take the vulnerable path. If one can't be patched this week, Wiz says-XX:-UseBASE64Intrinsicscloses the path at some cost in performance. As of Wednesday, GreyNoise had seen exploit attempts but no successful break-ins.The bug is in Java. The inventory problem is yours.
Then Microsoft's agent-framework package on PyPI.
On the evening of November 17th, UTC, two malicious versions, 1.26.1 and 1.26.2, went up 33 minutes apart. Both were gone within an hour of the first. The package gets about 190,000 downloads a month, and the last clean version is 1.26.0. How the attacker got publishing access hasn't been disclosed.
The mechanism is the reason I put it second. Neither version needed you to import anything. Each dropped a
.pthfile into site-packages. Python reads those files every time the interpreter starts, and it runs any line in them that begins withimport. So the stealer ran with your tests, your app, any script in that environment, even pip itself.It also didn't stop at what was stored. It collected environment variables and credential files, and then it asked Azure for fresh managed identity tokens, and for Azure DevOps OIDC tokens. The second version added GitHub Actions OIDC. Those tokens were requested as you. Rotating your stored secrets doesn't touch them. Wiz says they last up to 24 hours.
That changes what cleanup means. Search lockfiles, CI logs and images for 1.26.1 and 1.26.2, and pin 1.26.0. Look for
webhook.sitein proxy and DNS logs. And if it ran, rotate what the environment could reach, then go read what your managed identities and federated credentials actually did that day.Then Confluence.
CVE-2026-90436 is a 9.8 critical bug in Apache Axis 1, the old SOAP engine behind Confluence's remote API. A request gets to choose which Java class is loaded, and no login is needed. Axis 1 is end of life, and Apache isn't planning a fix.
Rapid7 saw it used against a customer on November 13th. Within seconds, the Confluence Java process was starting bash and curl. About three minutes in, the attacker was in MySQL with Confluence's own database credentials, searching pages and attachments for credentials, tokens, VPN configs and private keys. About four hours in, a production data warehouse. About twelve hours in, Microsoft 365, through legacy authentication.
The setting is the story. The remote API has been off by default for new installs since Confluence 3.0. The exposed servers are the ones where someone turned it on for an integration, maybe years ago, and nobody turned it off. Check General Configuration, then Further Configuration, then Remote API. Atlassian fixed it in 9.2.28, and 10.x doesn't have the feature. Cloud isn't affected. N-able says attacks on N-central strongly correlate with this flaw, and its fix is 2026.4 Hotfix 3.
A wiki is only as safe as the credentials people have pasted into it. My guess is most of us have never searched ours.
Then the FAA.
At about 3:30 PM Eastern on Friday, November 13th, the FAA stopped flights nationwide until 8:15 that evening. There's no sign of a cyberattack. The preliminary story is a network change: at a radio site in El Dorado, Arkansas, L3Harris was installing a new router as part of a copper-to-fiber move. Officials say it flooded the network with about a thousand connection attempts every four to five seconds, and the legacy equipment behind it went unstable. Seventeen of the 22 high-altitude centers were affected.
The part that stuck with me: the FAA Administrator said they spent about 45 minutes finding the cause, partly because that maintenance was logged by hand. They went through sheets of paper to find out what had changed. The FAA has since disabled more than 60 other Juniper routers, and says it will rethink when this kind of work gets done. This one happened midday.
So ask the plain questions. When something breaks, can you list every change from the last hour? Can you search that list, or is it somebody's notes? Does your maintenance window match your quietest traffic, or the contractor's calendar?
The human closer is where this week comes together for me.
Oakley Hall's piece, "Why Your RCAs Aren't Working," gives four reasons, and two of them fit. The first: we collect documents, not data. There's no table you can query across incidents by change type, platform or hour, so spotting a pattern depends on who has read the most RCAs. The FAA's paper log is that, about as literally as it gets.
The second: systemic problems get assigned to whichever team hit them first. That team can fix its own service. It usually can't change a shared base image, a JDK fleet, an end-of-life library inside Confluence, or how a package installs. So it builds a workaround, the RCA closes, and the next team finds the same problem.
Every story this week was work for whoever owns the layer underneath. That only works if someone actually does.
A corrective action needs an owner who can actually finish it.
One last thing: thank you for listening, for sharing episodes, and for sending stories my way. If the show's helped you this year, the DevOps Dozen vote is open through December 31st.
Additional Links
Attackers compromise Microsoft's agent-framework (The Hacker News)
CVE-2026-90436 (Atlassian)
phpAfterShell (depthfirst)
The GitHub Actions secret stealer and its indicators (StepSecurity)
Java4Shell (Calif)
SRE Weekly #543
Scroll inside the box to read the full commentary, or expand for a larger view.
Ship It Conversations: Ted Julian of Flux on the AI Code Visibility Gap, the Code Review Bottleneck, Measuring What AI Changes, Who Owns AI-Generated Code in Production, Guardrails Before Loops, and Why Disruption Is a Career Opportunity
Ship It Conversations: Ted Julian of Flux on the AI Code Visibility Gap, the Code Review Bottleneck, Measuring What AI Changes, Who Owns AI-Generated Code in Production, Guardrails Before Loops, and Why Disruption Is a Career Opportunity
The line I can't shake from my conversation with Ted Julian is about a cat. I asked him who owns AI-generated code once it's running in production. The engineers? The CIO? Some new AI director? His answer: "it's who feeds the cat, right? If multiple people feed the cat, the cat starves."
Ted is the founder and CEO of Flux, an engineering intelligence platform. Quick note for this audience: not Flux CD, the GitOps tool. This Flux lives at askflux.ai. Ted has been building security and infrastructure companies for more than twenty years, including Resilient Systems, Arbor Networks and @stake, and that security background shows up all over this conversation.
Earlier this year, Flux put out a report called the AI Code Generation Reality Check. Two numbers jumped out at me: 44.7% of the organizations surveyed already have AI-generated code running in production, and 35% are writing it but not shipping it.
That second number is the interesting one. Ted was upfront that some of it is plain experimentation: a throwaway prototype somebody built because it was a chance to try Claude. But the survey also asked what was holding teams back, and the answers were about trust. Complexity that could spiral into an outage. A new library with a vulnerability that came along with the prototype. Regulated data nobody has really scrutinized. A prototype can look great right up until a healthcare company realizes it's about to land in production.
He also said something I think gets lost in these conversations: "the models are the worst they will ever be." They'll get better. But the more regulated you are, and the more you care about five nines, the more the smart leaders are putting guardrails and visibility in place now.
From the SRE side, my question was what breaks first in review and release when code gets cheap to produce. Ted didn't hesitate: code review. He describes an AI transformation as a play. Act one introduces the characters and everything looks promising. Act two is where the villain shows up. Code review is the beginning of act two.
And it isn't a bottleneck you can throw more compute at. Using AI for code review is "probably a very, very good idea," he said, but in sensitive environments he still wants humans in the loop. So it becomes a people problem: who reviews, how much, and whether the team is set up for it.
His best story was about Sally. Seniors are jumping into the review breach. Maybe Sally did reviews part time before. Now it's most of her week, she's great at it, she loves it, and through all those reviews she's mentoring every junior on the team. Ted said he had goosebumps talking about it. His point is that this is shadow work. It has never really been in a ticket, and if you aren't looking at things like commits versus code reviews per developer, you're blind to one of the most important dynamics on your team.
Then came the part I'd send to every engineering manager. I asked how you measure what AI is actually doing, since everyone reaches for bigger PRs and more frequent PRs. Ted called it a "once-in-a-career hack." Start measuring features versus maintenance versus bug fixes, programmatically, not the end-of-quarter guess people make for R&D tax credits. Tag it in Jira or GitHub, use Flux, use anything. Backfill it so you have a before. Then show the after.
At one customer, that turned into a chart in their Q1 board meeting. The board had pushed the team hard to lean into AI and thought they were stagnant. Feature work had gone from 9% to 43% by the end of the year. The curve will flatten over time, so document the spike now, give the team the credit they're due, and show senior management the results.
On security, Ted described something every one of us has seen. It's so easy to go tab, tab, tab, accept, accept, accept. A new library comes in with a vulnerability, the thing works, nothing breaks functionally, and it lands in production. "We'll just have experienced developers" isn't enough. It's belt-and-suspenders time.
And it isn't only developers anymore. Marketing is coding. Sales is coding. Support is coding. Ted thinks that's great, managed properly. The last thing an engineering team should be is "the no squad." You want to be the yes squad, with guardrails. If it's easy for a developer to miss a bad dependency, a marketer with the best intentions in the world doesn't stand a chance.
He came at this from security, and security has lived this tension through the commercialization of the internet, mobile and cloud: manage the risk, but if you just say no all the time, people go around you. I said on the show that these are the same guardrails we've always had, and AI just forces us to take them seriously because everything moves faster. Ted's take is that engineering itself is being upended for the first time. Sales went through it with Salesforce, marketing with HubSpot. Now it's our turn.
The bigger bill, he thinks, is tech debt. Code review will look like a small problem next to the day somebody has to understand generated code in detail, and "nobody really wrote it." Maybe it was outsourced and that vendor is gone. Maybe Sally left for a great opportunity, and now it's "Joe, good luck." He expects that bill to come due for most of us over the next year to eighteen months.
Back to the cat. Ted doesn't think anyone can say canonically who owns AI-generated code yet, but his own answer was clear: one owner, the head of engineering, and ultimately the CIO. Governance without crushing velocity is "the sixty million dollar question." Every organization is different, but this isn't anybody's first rodeo. We've done it in every big transformation; the controls are just in a new domain.
Looking ahead, he says we're in the first inning. Loops and continuous development sound great, but loops have a dark side, and they accelerate the bad just as much as the good. Static dashboards and weekly snapshots won't keep up once the code is writing itself. He expects a control plane above all of it, with agentic systems that can tell an engineering leader in real time that churn in one part of the code base is slowing a team down, and why.
When I asked what high-performing teams do differently, his answer was that a lot of them are coding their own visibility tools, because that's easier than retooling legacy systems built on tickets and tags. Fine as a first step, he said, but you're not going to write your own software for everything. At some point it's better to make it somebody else's problem. That's the case for Flux: a code-first approach, so you get the visibility without changing how your teams use Jira or GitHub.
What stuck with me most was his last thought. When he mentors college students, he tells them tech disruption is the best opportunity to advance your career, "because in the early days there are no experts." Go deep and become the expert. Engineering is being upended for the first time, and the first time only happens once.
You can find Ted and Flux at askflux.ai, and the full report and Flux's Cobalt case study are in the show notes.
Scroll inside the box to read the full commentary, or expand for a larger view.
Ship It Conversations: Tristan Handy of Fivetran + dbt on AI Agents as Data Consumers, Why Dashboards Stop Being the Product, Context Layers, Evals, Operating on the System, Software Factories, and Why Text-to-SQL Isn't the Hard Part
Ship It Conversations: Tristan Handy of Fivetran + dbt on AI Agents as Data Consumers, Why Dashboards Stop Being the Product, Context Layers, Evals, Operating on the System, Software Factories, and Why Text-to-SQL Isn't the Hard Part
The line I keep coming back to from my conversation with Tristan Handy is one he dropped two minutes in: dashboards are "a hack for the human visual cortex."
Tristan is the founder of dbt Labs and president and co-founder of Fivetran. If you've worked anywhere near a data team, you've run into dbt, and his team likes to call it Terraform for data. That framing made this one easy to follow from the SRE side, and it's a big part of why I wanted him on.
Dashboards exist because our brains are the ones looking for patterns. Pipe a lot of information in fast, and a person spots what matters. But more and more, the thing reading your company's data isn't a person. It's an agent, and agents don't want to read your data as pictures.
So Tristan thinks the asset moves down the stack. Agents will work from three places: the gold layer, which is well-modeled data in your lake or warehouse; a semantic layer that captures how your organization calculates its metrics; and a context store that helps the agent navigate everything it can see and make meaning of it. Dashboards don't disappear. They become the translation layer for when an agent notices something and you ask it to explain.
The semantic layer part stuck with me. He says the way you define your metrics "is not primarily a technical artifact, it's primarily a decision-making artifact. It's a reflection of how your organization wants to see itself."
That's not a data problem. That's an org problem, and anyone who has argued about what "available" means in an SLO will recognize it.
Then I asked the question I care about most. In SRE, we can tell you whether a request succeeded. An agent can get a successful response and still make the wrong decision. So what does reliability look like at the data layer?
His answer is the one I put in the hook. Agents are very good at writing SQL. "The problem today is not text to SQL." The SQL compiles and runs almost every time. The hard part is whether the agent understood the question you asked and the context of the data, and whether it got you the semantically correct answer.
He pointed to benchmarks like Spider 2.0, which try to model how messy real enterprise data is. In the results he's seen, an agent turned loose on raw enterprise data gets something like 30% of its answers semantically right. Give it context and a semantic layer, and it's north of 90%.
That's the whole game for his world now: take what people know, either in their heads or because they know where to look, and put it in the hands of agents so they come back with answers you can trust.
I pushed on governance. If an agent can discover data and act on it, where's the line between useful autonomy and access you'll regret? Tristan was honest that most data teams aren't wrestling with that yet. Data has mostly been how a business looks at itself, not part of how it operates. For thirty years, data people have been building "highly refined nervous systems" that collect, refine and present information for a human to act on. He thinks those systems are about to get hooked up to action, and agents are what will do it.
On wrong answers, he described a spectrum I've seen in ops too. Some organizations "run and gun": trust the answer by default, and "you certainly can move real damn fast that way." Enterprises usually keep a final human check. He compared it to the choice you make in a software factory today: who merges the pull request, the human or the agent?
The career part is where this conversation goes beyond data. Tristan said all of these knowledge-work fields are going to be systematized, analytics engineering, SRE and software engineering alike. You'll "either be operating inside the system or you'll be operating on the system." The more interesting work is on the system: designing the loops and the evals, and seeing the needs in the organization that nobody has written down yet.
One of dbt's values, which he wrote ten years ago, is that "we believe in moving up the stack." He called this "a big moving up the stack moment." I think he's right, and it isn't comfortable. A lot of us tied our identity to technical skills that took years to build.
We also talked about how dbt earned trust. It helped that dbt, like Terraform, isn't in the request-response path, and that there wasn't much else like it. But my favorite story was about one of the first reasonably sized companies to use it. They showed up in dbt's Slack and asked, "You guys don't have any tests. Is this a real thing? Or is this just some little side project?" dbt was about six weeks old. The team promised to build tests if the company would use it. That kind of pressure sharpens you.
When I asked what developer-first companies get wrong selling to enterprises, he answered for himself instead: "I think that I was too willing to say yes to take the revenue." Fortune 500 pull meant about two years of building enterprise features, like multi-cloud deployment and role-based access control, instead of moving the product vision forward. You have to build those things. You just want to be careful about when.
The open source line they settled on is a useful one: state and user interface. If a feature needs a database or a GUI, it goes in the cloud product. dbt itself still runs on top of files. When I brought up HashiCorp and IBM, he was quick to say this is pretty different: since the merger, they've only become more permissive with licenses, and the paid pieces are modules that plug into what you already run rather than a separate enterprise edition.
His first step for a platform or data team about to connect agents to enterprise data: "eat your spinach." All the good ideas from the last decade are still good ideas. A well-governed data lake, well-modeled data and a semantic layer. The genuinely new piece is a context layer. Big companies are building their own, the way they used to build their own orchestrators, and some are starting to buy them. Then evals, so accuracy goes up and not down as people change the stack.
He said a context layer is "the difference between a data agent that you want to come back to and ask more questions of and the one that you don't." And when it works, adoption looks nothing like a BI rollout. He's seen query volumes go up by 50x over a few months, because people finally ask the questions that used to cost too much friction.
The end of the conversation is the part I'd send to anyone running a platform team. Tristan has been building his own personal agent, and he's moved his own development from sitting at a terminal asking Claude Code to do things to a software factory, where he files tickets and comments on pull requests. He thinks "most software maintenance will end up needing to be turned into a software factory."
His team did that for dbt 2. About twenty people were working on conformance, making sure dbt 2 supports everything dbt 1 does, and progress had flattened. So they paused for about two months and built an agent loop that finds conformance defects, writes tests for them, and writes and validates the fixes. Conformance went from something like 45% in January to 93%, and he said, "I don't think we would have been able to ship V2 without it."
The pause is the part I'd underline. Tristan admitted it was stressful: he wanted the line to keep going up, and his team told him no, pause, and then it'll go up fast. That's operating on the system, and it worked.
You can find dbt at getdbt.com, Fivetran at fivetran.com, and Tristan's Analytics Engineering Roundup and Analytics Engineering Podcast in the show notes.
Scroll inside the box to read the full commentary, or expand for a larger view.
AWS Roles Trusting Qubole Abused to Steal Secrets, Claude Code's Don't Ask Again File Leaks Credentials, KubeCon Agent Sandboxes for Kubernetes & WorkOS Breach Traced to an AI Agent
AWS Roles Trusting Qubole Abused to Steal Secrets, Claude Code's Don't Ask Again File Leaks Credentials, KubeCon Agent Sandboxes for Kubernetes & WorkOS Breach Traced to an AI Agent
Every main story this week came down to one question: what can it touch?
Not what it's supposed to do. What it can actually reach, today, with the access it already has.
A vendor role nobody used anymore but nobody removed. An approval button that quietly saved the secret along with the command. An AI agent running code nobody reviewed, on a kernel it shares with everything else. And a legacy internal tool that was reachable from the internet with a credential meant for someone's laptop.
None of those were decisions anyone made. They were access that outlived the reason for it.
Start with Qubole.
Wiz reported that between October 14th and October 26th, an attacker used IAM credentials from an AWS account tied to Qubole, a data lake platform, to assume the cross-account roles customers had created to trust it. Once inside, it went straight for credentials: Lambda environment variables, Secrets Manager values and SSM parameters. It also downloaded S3 data, sometimes entire buckets in a single session. How it got Qubole's credentials hasn't been disclosed.
The part I keep thinking about is the External ID. Every assume role call Wiz tied to the attacker included it, and every one succeeded. That's not a bug. The External ID was never meant to stop this. It protects against the confused deputy problem, where another of the vendor's customers tricks the vendor into acting on your account. It isn't a secret either: it sits in plain text in CloudTrail. If the vendor's own credentials are stolen, the External ID comes along for free.
And here's the detail that should bother all of us. In most of these environments, the attacker's calls were the only assume role activity from that account in October. The integration was idle. The trust policy was still live.
My take: a cross-account role is a standing grant, and standing grants don't get re-reviewed unless something forces it. Nobody files a ticket to remove a vendor integration they stopped thinking about.
So go find every IAM role whose trust policy names an outside account, and ask whether you still use that vendor. If you don't, delete the role. If you do, narrow the trust to the vendor's specific role where they support it, and add a source IP or VPC endpoint condition. Wiz says a source IP condition alone would have blocked every role assumption they saw. And if you ever connected Qubole, check CloudTrail from October 1st, rotate everything those roles could read, and revoke active sessions.
Then Claude Code.
When Claude Code asks to run a shell command and you pick "Yes, and don't ask again," it saves an allow rule to
.claude/settings.local.jsonin your project folder. That rule is the full command. So if the command had a password or a token in it, that secret now sits in a file in plain text.Apiiro found about 104,000 public copies of that file on GitHub, and 1,122 of them held at least one secret, including 379 database passwords and 23 AWS keys. GitGuardian counted more than 31,000 secrets pushed in that file since January 2025, and more than 5,300 still worked when it found them.
The Docker part is the one I'd look at first. GitGuardian found 613 secrets in public Docker Hub images, 99 of them valid. If your Dockerfile does
COPY . ., it copies your working tree, not your git index. A file git ignores still ships in the image unless.dockerignoreexcludes it. Claude Code does add the file to your global git excludes, but that only covers your own laptop, not a teammate's machine, a CI runner or an image build.My take: any feature that remembers an approval is storing something. When the thing it remembers is a command, it can be storing a credential you never meant to save.
So rotate first, because git history, forks and image layers all keep copies. Then untrack the file, add
.claudeto both.gitignoreand.dockerignore, and delete any allow rule with a literal secret in it. Going forward, keep credentials in environment variables and approve command patterns instead of exact commands.KubeCon was the hopeful story.
The CNCF took in two new Sandbox projects built for agents. Agent Substrate, contributed by Google, is a zero-trust runtime for AI agents on Kubernetes. OpenShell, led by NVIDIA, enforces which files, networks, credentials and model endpoints an agent can reach. gVisor was contributed too. The same week, AWS launched Nitro Sandboxes for EKS: set a pod's runtime class to
isolated-sandbox, and that pod gets its own lightweight virtual machine on Nitro, with its own kernel. It needs Kubernetes 1.36 or later. AWS calls it available today, without saying GA or preview.Here's why it matters. A normal container shares the host's kernel, and that's been fine for code your team wrote and reviewed. An agent runs code nobody reviewed. A container escape puts it on the host, next to everyone else's pods. A sandbox moves the boundary off that shared kernel.
Two cautions. Sandbox stage at the CNCF means early, so evaluate Agent Substrate and OpenShell rather than deploy them. And isolation limits what an agent can do to the host. It doesn't limit what the agent can reach with the credentials you hand it.
If you run coding agents or untrusted CI on Kubernetes, my guess at the combination to fix first is the default runtime, plus a broad service account, plus open egress.
WorkOS is the same question from the other side.
On November 2nd, WorkOS disclosed unauthorized access to an internal sales tool. The first access was October 25th. Nobody noticed until records were pulled in bulk on October 30th. On November 6th, a security researcher came forward: they'd been running benchmark evaluations with ARTEX, an AI offensive security framework, when their agent escaped its test environment and ended up at WorkOS. WorkOS and an independent forensics firm matched the researcher's evidence to its logs. The escape itself is the researcher's account, and nobody has said how it happened.
What I find most useful is WorkOS's own list of what let it in. The tool should not have been reachable from the internet. Its API accepted a weak credential meant for local development. There was no rate limiting. And it took five days to notice. None of that needed a clever agent.
WorkOS has since cut all its internal tools off from the internet, and it now runs ARTEX against its own systems. That's the right instinct. Agents, whoever runs them and whether or not anyone meant to, never get tired of trying the dev credential you forgot about.
The human closer is where this lands for me.
Brent Chapman's piece argues that incident resolution time isn't something you should target. It mostly depends on the incident, not on your process, and once it's a target, the number gets managed instead of the system. His fire department example stuck with me: targets for dispatch, turnout and travel time, but none for time on scene, because a grease fire and a warehouse fire aren't the same job. His test for what to target: if you can name two or three specific things you'd change to improve a step, it's controllable enough to target.
Run this week through that test.
You don't control whether a vendor you connected years ago gets breached. You do control whether that role still exists, and which addresses it accepts. You don't control what a coding tool writes to disk. You do control your
.dockerignore, and what your scanners check. And WorkOS couldn't control an agent showing up, but five days to notice is something a rate limit and a volume alert can change.The parts that are yours to change are usually smaller, and more boring, than a number on a dashboard.
Target what you can change, and track the rest.
Additional Links
SRE Weekly #542
N-able restores N-central cloud services (Cyber Press)
Scroll inside the box to read the full commentary, or expand for a larger view.
Ship It Conversations: Mallory Haigh of PlatformEngineering.org on Why Context Isn't Enough for AI Agents, Harness and Governance, Probabilistic and Hybrid Paths, Agent Identity, and Building Platform Trust Before Agents
Ship It Conversations: Mallory Haigh of PlatformEngineering.org on Why Context Isn't Enough for AI Agents, Harness and Governance, Probabilistic and Hybrid Paths, Agent Identity, and Building Platform Trust Before Agents
The line I keep coming back to from my conversation with Mallory Haigh is one she says is about 90% of her job: telling people "they need to ask why more."
Mallory is Head of Platform Education and Advocacy at PlatformEngineering.org. She teaches in Platform Engineering University, and she works directly with platform teams through Platform Engineering Consulting, which she calls "Mallory as a service." She also gets called a platform therapist, and after this conversation I understand why.
We started with a line from her pitch: context delivery alone isn't enough.
Her example stuck with me. Drop someone in the middle of New York City, tell them they're in the middle of New York City, and give them nothing else. No idea where they're supposed to go or what they're supposed to do. Chances are they get lost or get themselves into trouble.
That's what a lot of us do to agents. We hand over some context, maybe an agents.md file, and call it done.
Mallory says the agent also needs to know what exactly it's supposed to be doing, how it should do it, and how it'll know the work was done, and done correctly. She splits that into two layers. The harness is how the agent does the work. Governance is who and what is allowed, what's safe, and how it's watched. Together, she calls them agent infrastructure.
The watching part is where she thinks teams go wrong fastest. A good platform is auditable. You can see what an actor did, human or agent, and if the platform makes a decision itself, that's part of the audit trail too. For agents, she's looking at token spend, time to completion, which gates passed or failed, and how many times it went around a retry loop before it met the acceptance criteria.
And all of it has to tie back to why you brought the agent in. Reduced downtime is nice. But why did you need to reduce it? Did you have a problem?
I pushed on definition of done, because that assumes the ticket says what done means. Her answer made me laugh: "The definition of done sucks. It doesn't matter whether or not it's a human or a machine, it still sucks." The difference is that the machine can create a lot more chaos, a lot faster.
Then I asked about something I've been hearing a lot, including on this show: IDPs becoming context layers for agents.
Mallory thinks that's partly right. An IDP is an abstraction, and a golden path is a way to direct human behavior. You can extend that path to agents. The IDP is the foundation the agentic platform gets built on, and it has to work for the humans first.
What's missing, she says, are path specifications. She described three kinds.
Probabilistic paths are agent-driven. The agent decides what to do at each step, and the output isn't deterministic. Her example is a PR review agent that posts comments against your standards. It can be read-only and touch nothing.
Deterministic paths are what platform engineers already know: pipelines. A CI build runs the same way every time.
Hybrid paths combine the two. The agent opens a PR, a deterministic gate like CI validates it, and if it fails, the agent goes through a retry loop. At the stage most teams are at, it can't do the merge. A human makes the final call.
On that retry loop, she was practical. Set a limit. "Three to five, I would say, is your magic number to start with." Why? "I don't really know. Just seems to work." And make every retry teach something, either to the agent or to the person who built the platform.
She also said something I hadn't thought about. If an agent always does the same thing with the same outcome, maybe it shouldn't be an agent. Make it more of a checklist, use fewer tokens, and stop paying for free rein.
The part I keep thinking about is trust.
I asked how governance changes with agents, since platform teams already struggle with it when humans do the work. Mallory said it absolutely gets harder, because agents make us confront our lack of trust in ourselves and in the systems we've built. Platform engineering, agents or not, is "a practice about building trust."
So she's been telling teams to slow down. In the three weeks before we recorded, she'd told multiple platform teams to pump the brakes on putting an agent in, because they couldn't get human governance right yet. Her question for them is a fair one. How can you trust several machines running in parallel if you can't trust a person to go through a process without someone standing over their shoulder?
She also reminded me that developer experience isn't the only stakeholder. Security and governance teams need a say in how these platforms are designed.
For identity, she doesn't treat agents very differently from people. Start with the RBAC and role bindings you already have and narrow them. Build specialists, not generalists: an agent for Terraform inside one product team, not an agent for everything. "You can't control what you can't see and what you can't understand."
On the SRE side, she told me about one of the biggest financial services companies in the US, which runs agents over terabytes of logs a day. The agents spot problems before they surface and get the alert to the right person. Don't page Bob, who's on PTO. Page Dave, who's on call. They don't do auto-remediation yet, and she thinks that's the right call. And if your humans see thousands of alerts a day that mean nothing, fix that for the humans first.
Then there's tribal knowledge. To give agents what they need, teams have to write down every assumption they make. Who to ask about Terraform. Where the real answers live. What the new junior engineer will and won't get assigned. Mallory says teams are now doing "anthropology on their own organizations." It's hard, and it's worth it either way, because it makes the platform better for humans too.
So where does a platform team start?
Ask why. If the answer is velocity, then start small. Her first path is the read-only PR review agent: low risk, a real time saver, and a good check that people can explain code they've generated with Claude or anything else. Once you're comfortable, move to the hybrid validate-change path, with a human reviewing the evidence. Then security scanning, then running it in dev, maybe staging. Training wheels first. As she put it, you don't start by jumping in the Formula One car.
She left me with a line from one of her PlatformCon New York workshops. An attendee asked how they're ever going to write an agents.md file for their agents when they can't even write a README for their repos.
That's the takeaway I'm keeping. Ask why before you add the agent. Fix human governance first. And write down what your organization knows, because, in Mallory's words, "it won't just help agents, it will help the people that work there."
Scroll inside the box to read the full commentary, or expand for a larger view.