Ship It Conversations: Mallory Haigh of PlatformEngineering.org on Why Context Isn't Enough for AI Agents, Harness and Governance, Probabilistic and Hybrid Paths, Agent Identity, and Building Platform Trust Before Agents
Try reloading the player or open this episode directly on YouTube.
In this episode, Mallory Haigh discusses the limitations of providing context alone for AI agents, emphasizing the need for harness and governance layers.
Now Playing
Ship It Conversations: Mallory Haigh of PlatformEngineering.org on Why Context Isn't Enough for AI Agents, Harness and Governance, Probabilistic and Hybrid Paths, Agent Identity, and Building Platform Trust Before Agents
Ship It Weekly
0:00Current chapter: Cold open39:20
Chapters
Jump to a section in this episode.
Speed & share
Transcript
Drop someone in the middle of New York City with
no directions and they'll get lost. Lots of teams
do that to AI agents. They hand over some context,
maybe an agents.md file, and call it done. My
guest today says that's not enough. The agent
needs to know what to do, how to do it, how it'll
know it's done, and what it's allowed to touch.
And if your platform can't do that for humans
yet, she'll tell you to pump the brakes. As one
of her workshop attendees put it, if we can't
even write a README for our repos, how are we
going to write one for our agents? I'm Brian
Teller from Teller's Tech and this is Ship It Weekly.
Welcome back to Ship It Weekly, where
I filter the noise and focus on what matters
when you're the one running infrastructure and
owning reliability. Most weeks, it's a quick
DevOps, SRE, platform, cloud, and security news
recap. In between those, I do conversation episodes
with people building and operating these systems.
Ship It Weekly turns one this November. To celebrate
through mid-December, you are getting two conversations
a week, Mondays and Wednesdays, with the news
every Friday. Then we're back to our usual rhythm.
Today, I'm joined by Mallory, Head of Platform
Education and Advocacy at PlatformEngineering.org.
Mallory works with platform and DevEx teams
on how platforms actually succeed or fail inside
engineering organizations. And lately, on what
changes when AI agents enter the software delivery
workflow. We talk about... why context delivery
alone isn't enough, the harness and governance
layers that actually run agents on a platform,
the three kinds of paths agents work on, why she
tells some teams to fix human governance before
adding agents, and where a platform team should
actually start. All right, let's jump in.
Mallory, thank you for joining me. Thanks for having me,
Brian. This is really exciting. So I'm interested
to hear one line that you had mentioned was context
delivery alone isn't enough. What do you mean
by that? I try to think of a lot of the things
with AI in general in the same way we think about
interacting with humans. So context delivery
with a human isn't alone. I can put you in the
middle of New York City and say you're in the
middle of New York City. But if I don't give
you any clear direction on where I'm expecting
you to go or what I'm expecting you to do, and
I just took you out of the middle of your day
and put you there chances are you might get
yourself into some trouble or you're going to
get lost. It's the same thing with agents. So
when I talk about context delivery on its own
just not being enough, there's other things we
have to actually give that agent to. It's things
like, what exactly are you supposed to be doing?
How exactly should you be looking at doing it?
And how are you going to know whether or not
it was done and done correctly? Then there's
also this other piece, which comes to governance
and security, which sits alongside that. That
says, okay, here's how you do your work, and
here's how you're allowed to do it, where you're
allowed to do it, and the constraints of that
entire operation. That's what I mean. You just
give something context. You give it an agent.md
file. That's not enough. There needs to be
a lot more. And what is that more? Is it a Kyverno
rule? What do you mean by that? Definitely part
of it. We talk about this agent infrastructure
layer. And that's really what runs agents inside
of a platform. And there's two pieces. So there's
the harness. That's how the agent does the work.
So that's an execution component. There's context,
capability, that evaluation rubric, basically.
So like, did I do a good job? Am I done? But
the second piece is that governance layer. So
what's allowed? What's safe? What is actually
being observed here? So that's things like identity.
So whether or not that is something like... OPA,
Kyverno, just regular kind of RBAC roles applied
in a slightly different way. It's security parameters
and context too. So things like codified security
standards, whatever that might look like for
you, or even just some sort of vulnerability
scanning and testing. And then there's also just
its unique requirements for observability. Agent
observability is something that people are talking
about a lot. And that's a big part of that governance
piece. But both of those harness and governance
layers make up agent infrastructure as a whole.
So when you talk about observability with agents,
what does that mean? Like what kind of analytics
are we trying to get out of the agent or what's
valuable to pass into like, I don't know, Grafana,
whatever? Yep. And I think there's a lot of different
ways people are talking about this. But again,
I come back to like, how do we think about observability
with humans? A good platform at the end of the
day is very auditable. So I can see what actions
were taken either by an agent or a human, so
an actor inside of that platform, or the platform
making a decision itself. And what we like to
think about is if the platform makes a decision,
that should be part of the audit trail as well.
So you're observing actions, but you're also
observing outcomes and system health and performance.
You know, there's always that discussion too
about observability isn't just looking at system
health. It's also that system performance aspect.
So for agents, you know, you're looking at token
spend, you're looking at, you know, the time
to completion, to getting to that evaluation
piece. So if there is a loop, for instance, that
it goes through a retry loop, how many times
is it going through that loop? You should, by
the way, set a limit on that. But how many times
did it go through it before it got to that acceptance
criteria? There's always pieces too with just
sort of monitoring the step-by-step process.
So was this gate passed? Was this gate failed?
What triggered this particular event? So it's
that bigger picture observability that we talk
about in platform engineering as one of the pieces
people tend to get wrong really quickly because
they just focus in on like... performance metrics
or, you know, system performance, like, oh, we have
a whole bunch of 400s. That's a problem. Um, it's
a little bit different and a little bit more
nuanced than that. So it's efficiency, but it's
also, like, correctness? Are we actually validating
for correctness, and how are we doing that? Or
is the user providing that input, or how are we
determining the correctness from the agent? I
think that's a good question because it really
depends on the circumstance in which you're using
that agent to begin with. And this is something
that I work a lot with teams on is sort of deciding
when are you going to bring an agent into here
and why? What are you trying to specifically
improve? So with anything in platform engineering,
you want to have a clear why. Why are you doing
it? And that should guide you towards what it
is you actually need to measure to explain that
you're successful. So whether or not that's some
sort of throughput metric, whether or not it's
something around uptime, you know, mean time recovery,
whatever other DORA metric you want to use. That
stuff is all really important, but you have to
be able to actually articulate that against your
why. So why are you even doing this to begin
with? It's nice to say, great, you reduced our
downtime, but why did we need to do that? Did
we have a problem? And it's the same with the
agentic pieces too. So we want to put that agent
in to actually make something better, make something
faster, be more innovative, reduce toil on developers
or operations folks, move at a speed that regular
humans just can't do. So we have to be able to
say to ourselves, what do we need to observe, and
what do we need to actually think about in order
to say, hey, is this actually working? That is on
top of the definition of success for the particular
task that the agent is asked to do. So whether
or not that's, you know, here's a ticket in Jira,
go through this, the success criteria defined
at the end, and when you meet all of that success
criteria, plus you meet these testing success
or fail points, then you're done. Then you can
consider this shipped and you can hand it off
agent to the next step. That's the kind of stuff
we're thinking about. Makes sense. That's also
assuming, I guess, the definition of done is
well articulated in the ticket. Well, this is
why I think agentic integration as a whole is
making people very uncomfortable. And it's not
just because of the AI bad conversation that
people like to have. It's also the Whole thing
with AI is it's forcing us to confront our own
humanity in a very strange way. And that is a
good example of it. The definition of done sucks.
It doesn't matter whether or not it's a human
or a machine. It still sucks. It's just the machine
is a lot more capable of creating a lot more
chaos than the human is because they can move
that much quicker. When people say IDPs are becoming
context layers for agents, what are they getting
right? What they're getting right is that An
IDP is an abstraction. And in a way, a context
layer is an abstraction. It's an abstraction
on knowledge and practice and, you know, behavior
expectation. And what IDPs are at their core
is engineering behavior. The idea of the golden
path as a whole is to direct human behavior in
a certain way. And we can extend that path to
agents to direct their behavior. The IDP is the
foundation in which the agentic platform is built,
regardless as to what direction your agentic
platform goes or whether or not, you know, you're
only bringing a few autonomous pieces in, really
gradually scaling up, whatever that looks like.
That IDP is the foundation and that has to be
right. And it has to work for the human users
in your organization because they are the ones
that will determine whether or not it's built
contextually, to overuse that word, correct for
the organization. And the things that they have
to do inside of it and their day to day. So that's
like, it's partially right. And it kind of scratches,
I think, at multiple layers. But there's a lot
more to it. Are you seeing that they're missing
on the flip side of that, conversely? Yeah. And
that's a good question. This is the question
I think everybody asks me. So that's one layer.
in an agentic platform. And really that's sort
of your tooling layer. It's your foundational
substrate that everything else is built on. But
there's other pieces too. So there's that agent
infrastructure piece that I talked about earlier.
So that harness, how the agent does the work,
it's the governance piece, who's allowed, what's
allowed to do it, what's safe, how is it watched.
But then there's this other piece that's at the
middle of these that allows them to interact.
And this is the path specifications. So you can
think of this as looking at golden paths and
sort of, you know, 100x-ing them. What this
really is, is the repeatable actions and the
rules around when agents are called and when
they aren't. So there's three kinds, realistically,
that we are primarily thinking about. The first
is that probabilistic set of paths that are very
agent-driven. There's also the deterministic
ones that are pipeline-driven. And then there's
hybrids that are that agent and a gate kind of
combined together. You know, if I think of a
probabilistic path that's agent driven, that's
an agent deciding what to do at each step where
the output is fairly non-deterministic. So it's
like reviewing a PR, posting comments against
whatever your standards are that it has access
to. And, you know, an easy one to start with
is that PR review piece because it's fairly risk
averse. It could just be read only. Doesn't touch
anything. The deterministic piece is what we're
fairly used to in platform engineering. That's
pipeline driven. So it's something like a CI
build. It runs same way every time it's called.
There's no agent involved in its execution. Maybe
it's triggered by an agent, but the path itself
just, it runs. It's the way it is. You know,
and it's incredibly predictable in that way.
The hybrid would look more like a validate change
type path where there's this agent and gate piece
in there. So agent generates output. Let's say
it's PR. It's the common one. But then there's
that deterministic gate that validates it's a
CI build, essentially. And if it fails, the agent
has a retry loop it can go through. And let's
just say at the stage you're at, a human review
on that pass of the loop. It can't do the merge,
the agent. The CI says whether or not it is appropriate
to be merged, and the human does the final deciding.
But that's a hybrid path at its core. So those
are the three layers that we really need to think
about. The IDP is just a piece. So is that agent
infrastructure. But it's these paths that have
to be almost redesigned with agents in mind.
Sometimes you take the human path and you're
augmenting it and you're also making sure it
can work at much faster speeds and in parallel
at the same time. How do you keep an agent from
just looping forever or optimizing the wrong
thing in that model? Yeah. So that's a good question
too. The really important thing is setting limits.
If you saw a developer that was just going and
repushing a thousand PRs with similar messaging
and eventually get frustrated and their messaging
says things like, please work, oh my God, make
it stop. You'd step in and you'd say, whoa, whoa,
whoa, let's look at what we need to change. Same
thing with an agent, but put a limit in, put
in three to five, I would say is your magic number
to start with. Why is it the magic number? I
don't really know, just seems to work. I think
humans just like those numbers allow the retries
to also be learning experiences. So you can make
sure when you're setting it up that each retry
gives that agent information that it can learn
from, or that, you know, comes back as feedback
to the human that's built this platform to say,
hey, maybe the kind of agent infrastructure layer
needs to be adjusted here, or for lack of better
terms, your system prompt needs reconfigured.
There's something not quite right. Because often...
The failure is a learning experience for either
the human or the machine, and you might as well
take advantage of that. So what should the platform
score or evaluate in that loop, in that process?
Yeah, it does depend on the path type. So let's
just look specifically at a probabilistic path.
What we really want to be evaluating here is
the decisions that the agent took. Maybe how
many decisions it had to take, how many tokens
it used in order to do this, whether like how
many retry loops it had to go through in order
to get to that definition of success. And, you
know, there's time components here. You could
be looking at costs as well as tokens. You can
look at it either way. And honestly, just kind
of the measure of success, the measure of the
behavior that the human's job at this stage becomes
monitoring behavior and monitoring outcomes.
You know, one thing I'll talk to teams about
quite a bit is if you're running an agent constantly
and it's always doing the same thing with the
same outcome, maybe you need to revisit whether
or not an agent is appropriate there. So you
need to keep track of outcomes. Do you have to
be calling through a crazy amount of tokens every
single time you run this when it's a fairly simple
process that's readily executed in the same way?
You could probably reduce the number of tokens
you're using and get the agent to be a lot more
checklisty because you have it so standardized.
And that's the things to watch. You know, if
you're always getting the same output, you're
probably wasting money if you're just letting
an agent, you know, kind of free reign through
everything. So we talked a little bit earlier
about governance. So one thing that I've definitely
realized in the industry is platform teams already
struggle with governance when humans are doing
the work. How do we... change that with agents?
And it seems like it gets harder with agents,
at least in my opinion, what I've come across.
How do we handle that? Yeah, it absolutely does
get harder. And I think it really comes down
to the reason why is because we're confronting
our own lack of trust in ourselves and in a system
we've built. Fundamentally, at the end of the
day, platform engineering, regardless of whether
or not there's agents in this loop, is a practice
about building trust. It's organizational trust
in the system. It's organizational trust in the
people who have access to the system or the actors
who have access to it. So that foundation has
to be good and it has to be trustworthy. And,
you know, I, in the last three weeks alone, have
told multiple platform teams to pump the brakes
on putting an agent in because they can't even
get human governance correct. They are still
struggling. And this is where, you know. We'll
really sit down and understand where this is
happening. Where is this trust breaking down?
And is it just because we haven't codified the
governance yet, that identity security and observability
piece? Or is it unclear what you're supposed
to be doing? In which case you have to work with
security. You have to work with the governance
pieces of your org to really make sure you're
getting this right. And, you know, if their trust
in the system isn't there and they're able to
sort of pull the plug on this stuff, that's where
you need to focus. We focus so much on developer
experience, we forget that there are other stakeholders
we have to care about. And that often is governance
security teams, you know, having to feel like
they have a say in how these platforms are designed.
And yeah, it gets even messier when agents come
in, especially if that foundational trust just
isn't there. How can you trust a machine, multiple
machines in parallel, if you can't even trust
a person to go through a process without someone
standing over their shoulder constantly? So how
should teams think about agent identity versus
human identity? Yeah, I don't often think of
them that differently. The reason why, I think,
is because it does separate that trust a little
bit. If you trust your system to allow humans
to do the right kind of thing, agents can probably
work within the same boundaries. The difference
is the parallelization piece, so you can have
multiple running at the same time. And in that
case, my recommendation is always, you're not
looking at generalists, you're looking at specialists.
So you need to constrain around that speciality.
Rather than, we used to constrain a lot, or not constrain a lot, we used to split people into, you were a front
-end engineer, you were a back-end engineer.
Never shall the two meet. We probably spent a
good 10 years doing this when it was unnecessary.
Full stack is very useful. And that's really
not how we necessarily want agents to think,
especially when we're trying to do this in the
enterprise at large scale. We want to be specialized.
Have an agent for Terraform specifically, maybe
inside of this particular product team. Don't
allow the scope to go beyond that. It's way easier
to control the smaller the scope is, and it's
also way easier to monitor. And, you know, you
can't control what you can't see and what you
can't understand. So the smaller the chunk with
identity, the easier it is to manage. But start
with the foundational layer you have for humans.
You just have to augment it. So you probably
have RBAC in place already. You probably have
some level of role binding or an inheritance
pattern in place. Base it off that just now into
the specificity of whatever that agent's specialty
is. Do you have like the same like time bound
access that you do for humans or? I don't think
there's a standard set for this yet. And it depends
on the context. Like I'm a big fan of lower environments,
a little more relaxed. It's also where you should
start really in terms of building in governance
and security because it's easier to codify those
standards in a more relaxed environment and then
harden them as you go. But yeah, I would approach
it at the same kind of way you approach the human
side of things, just obviously considering that
these things can move quicker. If you're relying
on an external model, there's always the questions
of what happens if that model is somehow compromised.
And obviously that happens internally too with
internally hosted models, but it's that sort
of thing you have to consider. But time constraint,
definitely have it in place. You can make it
shorter. It's not the end of the world. These
things run quickly. But start somewhere, measure
and see how you feel about things and evaluate
the organizational trust in those things at that
base level and then make changes as you need
to. So we talked about agents involved in generating
code or being on the developer side of things.
How do you view using agents on the SRE incident
response side of a platform in an IDP? What are
your thoughts on that? It's one of my favorite
places to put these things because you have the
ability to have a very powerful, essentially
like emergency dispatch system or early warning
system. You know, a really good client of mine
talked about how they've brought an agent into
kind of their log ingestion and they're getting
terabytes of logs per day. This is one of the
biggest financial services companies in the US.
Terabytes of logs. I don't think a human can
go through that, but their agents do. And they're
able to recognize potential problems before they
even surface. And they don't do auto remediation
just yet. I mean, yeah, they aren't quite there
yet. You know, we're still working with them
to build in the foundations of what auto remediation
needs to look like. But it's making sure that
it's cataloging the issue. It's alerting the
right people in the right channels at the right
time. So don't go send a PagerDuty page to Bob
who's on PTO. Send it to Dave who's in the office
and on call right now. And it knows that. And
that's really helped them get ahead of some really
large issues. But also when there is something
that is major happening that maybe wasn't seen
or it's kind of that instant everything's on
fire, they have a much cleaner picture of where
it is. And they're not sifting through terabytes
of logs to decide what's relevant and what isn't.
Agents are fairly able to recognize those patterns
based on how they've built that context layer,
that governance layer, that agent architecture
around that get them the information they need.
Yeah. I think identifying the gaps in metrics
or alarms that you have is just as important
as identifying the alert fatigue patterns that
you may have. And yeah, I'm sure agents can help
with both of those. Yeah. And an agent fatigue
pattern, this is something your base platform
should be handling too. If you want to do platform
engineering right and really make your SREs have
good days, or anybody that has to see 6,000
alerts that don't mean anything every single
day, consider how the platform can help that.
Because you can make sure things are routed.
You can make sure things are relevant to that
person's context window. You could probably get
my opinion on what I think about AIs in terms
of how they exist in the world. But it's really
important to address that for humans. Because
alert fatigue is real. I cannot tell you the
amount of times, even on my own stuff, And I'm
probably the worst for actually fixing this because
it's just me on my own projects that I get an
alert. I'm like, yeah, whatever. And then I'll
get a bunch of users pinging me like, dude, everything's
broken. So fix that for the humans first. Okay,
so I guess speaking to the human side, do platform
teams need to focus more like on the design for
agents as another kind of user? Or philosophically,
do we think about that differently when we're
designing the platform? I love that question
because it's really the foundational thesis of
what I'm talking about right now. What we're
talking about, what we're writing books about,
conference main stage talks and everything. And
it is that at the end of the day, if your underlying
platform, that IDP is not strong, is not mature,
is not delivering value, is not actually reducing
toil, isn't adopted, good luck putting an agent
in there, or any, you know, kind of AI at all. And
don't. I wouldn't do it, honestly. So you need to
get that foundational design for humans first.
Then your design has to shift to really thinking
about codifying the things that make us understand
how to do our job. You know, that's that's that
part of that agent infrastructure piece. It's
so important, and it's something that I've been
talking a lot about recently, is like, you have
to now codify every single assumption you make
as an organization. Every single cultural piece
of working on that team. Like, I know I need
to go talk to Andrew because he knows exactly
how to write Terraform and I'm an idiot at it.
It's like, I keep breaking it. Or I know that
if I need this information, I jump into Confluence
to go get it. Or the assumption that I'm a junior
engineer, so I'm not going to get all the...
really, you know, hardcore tickets. I'm going to
get kind of the grunt work. Those are things that
we've never written down, generally, and now we
have to. So it does, often. One of the most interesting
things I think from like a psychological perspective
to see is teams are now going through this process
of doing anthropology on their own organizations
and writing, like, an ethnography on, you know, how
they work, how they approach problems. And having
to document that's really hard. So that is the
next step. But honestly, it's a good exercise
to do regardless because it'll make your platform
better for humans, but also make it the right
place to throw agents. Yeah, I think anyone that's
onboarded with a new company just realize it might
even be outside of the engineering organization,
but there's this tribal knowledge that like,
what does this acronym mean? You know, you have
this imposter syndrome where it's like, wait, is
this an acronym I should know? No, this is... I don't
want to ask someone. Yeah, yeah, no, this is an internal
acronym that, you know, it's just tribal knowledge
that people have. Or even just like, how, who do
you go to when you have a question about something
specific? Or, well, I need access to this, but you
got to go ask, like, John in finance. He'll get
that for you. Well, who was going to tell me that
until I ran into it? Um... Yeah, like just think
about onboarding a new human. How long does it
take to get them comfortable and actually doing
stuff? But there's things you don't even think
about in that too. So it sounds like engineers
trusting agents less. It's not so much because
they're risky, but maybe because the platform
around them is unclear. Does that sound right?
I mean, there is going to be that element of
humans not trusting AI. And you know what? I
think it's valid. But if they trust the platform,
to have constraints because they've seen it with
people they've seen it not only help people work
faster but work safer they're going to feel more
comfortable about agents coming into that system
and working within the same guardrails and working
within the same patterns the patterns are clear
they're observable and they are value generating
to begin with and it's clear to see how they
do that I try to avoid saying value generating
for a lot of engineering conversations because
often the eyes start glazing over and they look
at me like, how dare you? But it's true. It's
what we're doing. And it really is important
to start thinking that way when you want to go
down the agentic route anyway. Yeah, we have
to build that foundation and we have to have
that foundational trust there before we can introduce
something that runs a lot faster. Before you
can ride a bike, you start with training wheels,
then you take the training wheels off. Look, now
you're steady enough and stable enough, you can
go mountain biking, or you can get on a race bike,
or, um, heck, you're crazy enough, you want to go
do motocross or something. You know, it's it's
you don't just start by jumping in the Formula
One car. Okay, so I'm a platform engineer. I have
an IDP. I'm running Backstage. I have a service
catalog. I, I've been told by my director of engineering
that I need to have an agentic workflows built
into my platform. Where do I start? What do I
start with? First thing you do is ask why. It's
like 90% of my job, I think, is telling people
they need to ask why more. Okay, why? What are
we trying to achieve by doing this? Are we doing
AI for the sake of AI? Engineers move faster,
velocity, right? That's the answer I get, right?
That's better than just because someone on LinkedIn
said I should, which believe it or not happens.
You know, okay, so we've established why. We
want velocity. So we have a platform. It's mature.
It's adopted. Why are we talking about velocity
being slow? Is it just because we want to unlock
the potential of AI? Then great. We will look
at the platform that we currently have in front
of us and say, what makes the most sense for
an agent? And this is where there's three paths
I want people to usually start with here. And
the first is that... very, very probabilistic
agent-driven PR review task where it's that
read-only output. You know, they go through,
they add comments based on the standards of the
organization, your ADRs, whatever you have in
place. There's no modification of code happening,
but that is the first place to start because
let's be honest, code review is a... very time
consuming process. A lot of the time, it's one
of those things where I have 25 PRs to review
and I'll get to them when I get to them. Or I
have to set aside a specific day to go ahead
and do that. But what if that PR could happen
really quickly, like within a couple minutes?
That would speed us up exponentially. So that's
a nice one to do first because to be honest,
it's very low risk. You as a human can go in
and make sure that what it is saying, it's not
hallucinating, that it is correct against the
standards that you have. You can review the behavior.
But at the end of the day, like that's still
way faster because if it's let's say it's four
or five hundred lines of code, that PR review
would take a human a decent amount of time. It's
also a good exercise to make sure that teams
are doing PRs in a way that they are able to
explain the code that they've generated, whether
or not they've had. Claude do it for them you
know I think you still need to be able to say
this is the thing that I've built or that my
agent and I have built and this is why it's important
to this ticket and this is why we think it's
the right approach. So I think that's a good place
to start. After that, you know, you could really
start looking at that hybrid piece, so that validate
change component, once you're comfortable with
that PR review on its own. So, you know, the agent
does its PR, its own PR, sorry, and then there's
some sort of, um, CI or deterministic gate of another
type that's called to validate it, and there's
that retry loop until it gets it right. That's
another one to start with because I think it's
quite safe. Um, the, you know, the the baseline you
can move from with both of those is very strong.
You know, have a human review the evidence that
it was successful. And then once you're happy
with that, go into security scanning. Then, hey,
we're confident running this in dev all the time,
maybe even staging. That's an easy velocity game.
And it's a fairly quick one to put in place as
long as you have the foundational platform there.
Makes sense. So you had recently talked at PlatformCon.
Is there anything you'd like to leave our audience
with? Any thoughts from that conference or your
talks in general? Yeah, I think there's two pieces.
And it's at the end of the day, the kind of overarching
story is that as a platform engineer, you're
so valuable to your organization, technically,
not just when it comes to building that IDP,
but building that agentic development platform
or that agentic finance, marketing, communications
platform, whatever it might be, because you are
good at taking requirements and translating them
into value and being able to talk between different
traditionally siloed teams and build a product
that actually works for them. So that's kind
of the overarching thing, but there's two really
key components here. And the first one is, remember
this, and this is a quote, actually. Person in
one of the workshops I did at PlatformCon New
York this year said, and it's possibly my favorite
thing I've heard in a long time, and it's that
if we can't even write a readme.md file for
our repos, how the hell are we ever going to
write an agent.md for our agents? So keep that
in mind. You know, that is so important foundationally
to what we're doing here. And the other piece
that I just want to kind of give people is that,
you know, change is scary sometimes that scary
change puts us on the precipice of something
really special. And I think this is that time
for platform engineers to be at that gateway
into something really impactful. And AI has a
lot of ethical questions around it and has a
lot of pluses and minuses, but you're in the
place where you can actually not just implement
this cool technology, but build something really
powerful. And that is this base of knowledge.
This is putting on paper the culture of your
organization, and it it won't just help agents,
it will help the people that work there. I promise
you it will. So yeah, that's like, that's my big
things right now. And and I'm really excited in
this space. I think it's really, really exciting
to be a part of and to be able to speak about
it and work with teams directly on what they're
doing, um, and kind of everything I say comes from
doing that and actually working with organizations
so I'm not just kind of making it up as I go
along. Well said. Where can people find out more
about you, follow you, that sort of thing? Yeah,
so just find me on LinkedIn, sort of the easiest
place to find me. And I tend to post a lot of
stuff there, whether I'm writing on Substack,
stuff on YouTube, podcasts like this one, the
stuff my colleagues are working on. So some colleagues
of mine recently. Released a book, which is called
Thinking in Platforms, and it's really about
taking platform engineering as an ideology and
expanding it beyond software, outside of IT even,
into the rest of the world and sort of the philosophy
on what guides us and how we do things. PlatformEngineering.org,
that's where you can also find me, not
just in the community abroad, but inside of
the Platform Engineering University. You can
see me in a lot of the courses. I do teach quite
a few of them. So if you want to get certified,
you get to see my face a whole bunch and hear
my voice. And then also Platform Engineering
Consulting. So pec.platformengineering.org.
And that is how you can get what we like to jokingly
call Mallory as a service. Awesome. What all
do you provide in that consulting? I'm just curious.
Yeah. It started jokingly getting called a platform
therapist because I spent most of my time feeling
like I was giving people therapy or group therapy.
But it's not really a joke anymore. A lot of
what we're doing there is really helping teams
be successful with the frameworks that we've
developed for platform engineering. You know,
I will go train teams on the best practices.
And it's not, by the way, like, here's the tools
you're going to use here. And here is how you're
going to write a Pulumi. And here's how you're
going to work with Kubernetes. It's not that
at all. The cultural, the human side, but it's
also this business case, metrics evaluation,
roadmap development, building golden paths. It's
all these pieces. There's this agentic stuff
now as well. So I'm just a coach. There is the
element of therapy. We do joke about that quite
a bit because, like I said earlier, it's so much
of me asking people, but have you asked why?
And yeah, building their standard of practice
for platform engineering, really. That's awesome.
Thank you, Mallory, so much for coming on. Really,
really appreciate your time. No problem, Brian.
Thanks so much. It's been great. And yeah, I
look forward to chatting with you again. That
was my conversation with Mallory from PlatformEngineering.org.
The biggest takeaway for me is that context
alone isn't enough. An agents.md file is like
dropping someone in the middle of New York City.
The agent also needs to know what to do, how
to do it, how it'll know it's done and what
it's allowed to touch. I also liked how she breaks
down paths. Probabilistic paths are agent-driven
like a read-only PR review. Deterministic paths
are your pipelines like a CI build. Hybrid paths
pair an agent with a gate. The agent opens the
PR, CI checks it, the agent retries, and a human
still makes the call. And cap those retries.
Three to five is her magic number. The part that
stuck with me most is that platform engineering
is really about trust. She's told teams to pump
the brakes on agents because they can't get human
governance right yet. Start with the identity
and RBAC you already have and give agents narrow
specialties. So where do you start? Ask why.
Then try a read-only PR review agent, move to
that hybrid path, and write down the tribal knowledge
your org never documented. Your agents need it,
and so do your humans. You can find Mallory on
LinkedIn, plus PlatformEngineering.org, through
the links in the show notes. Follow or subscribe
to Ship It Weekly wherever you listen, find previous
episodes at shipitweekly.fm. Thanks for listening,
and I'll see you Friday for the news.
Ship It Conversations: Mallory Haigh of PlatformEngineering.org on…
The line I keep coming back to from my conversation with Mallory Haigh is one she says is about 90% of her job: telling people "they need to ask why more."
Mallory is Head of Platform Education and Advocacy at PlatformEngineering.org. She teaches in Platform Engineering University, and she works directly with platform teams through Platform Engineering Consulting, which she calls "Mallory as a service." She also gets called a platform therapist, and after this conversation I understand why.
We started with a line from her pitch: context delivery alone isn't enough.
Her example stuck with me. Drop someone in the middle of New York City, tell them they're in the middle of New York City, and give them nothing else. No idea where they're supposed to go or what they're supposed to do. Chances are they get lost or get themselves into trouble.
That's what a lot of us do to agents. We hand over some context, maybe an agents.md file, and call it done.
Mallory says the agent also needs to know what exactly it's supposed to be doing, how it should do it, and how it'll know the work was done, and done correctly. She splits that into two layers. The harness is how the agent does the work. Governance is who and what is allowed, what's safe, and how it's watched. Together, she calls them agent infrastructure.
The watching part is where she thinks teams go wrong fastest. A good platform is auditable. You can see what an actor did, human or agent, and if the platform makes a decision itself, that's part of the audit trail too. For agents, she's looking at token spend, time to completion, which gates passed or failed, and how many times it went around a retry loop before it met the acceptance criteria.
And all of it has to tie back to why you brought the agent in. Reduced downtime is nice. But why did you need to reduce it? Did you have a problem?
I pushed on definition of done, because that assumes the ticket says what done means. Her answer made me laugh: "The definition of done sucks. It doesn't matter whether or not it's a human or a machine, it still sucks." The difference is that the machine can create a lot more chaos, a lot faster.
Then I asked about something I've been hearing a lot, including on this show: IDPs becoming context layers for agents.
Mallory thinks that's partly right. An IDP is an abstraction, and a golden path is a way to direct human behavior. You can extend that path to agents. The IDP is the foundation the agentic platform gets built on, and it has to work for the humans first.
What's missing, she says, are path specifications. She described three kinds.
Probabilistic paths are agent-driven. The agent decides what to do at each step, and the output isn't deterministic. Her example is a PR review agent that posts comments against your standards. It can be read-only and touch nothing.
Deterministic paths are what platform engineers already know: pipelines. A CI build runs the same way every time.
Hybrid paths combine the two. The agent opens a PR, a deterministic gate like CI validates it, and if it fails, the agent goes through a retry loop. At the stage most teams are at, it can't do the merge. A human makes the final call.
On that retry loop, she was practical. Set a limit. "Three to five, I would say, is your magic number to start with." Why? "I don't really know. Just seems to work." And make every retry teach something, either to the agent or to the person who built the platform.
She also said something I hadn't thought about. If an agent always does the same thing with the same outcome, maybe it shouldn't be an agent. Make it more of a checklist, use fewer tokens, and stop paying for free rein.
The part I keep thinking about is trust.
I asked how governance changes with agents, since platform teams already struggle with it when humans do the work. Mallory said it absolutely gets harder, because agents make us confront our lack of trust in ourselves and in the systems we've built. Platform engineering, agents or not, is "a practice about building trust."
So she's been telling teams to slow down. In the three weeks before we recorded, she'd told multiple platform teams to pump the brakes on putting an agent in, because they couldn't get human governance right yet. Her question for them is a fair one. How can you trust several machines running in parallel if you can't trust a person to go through a process without someone standing over their shoulder?
She also reminded me that developer experience isn't the only stakeholder. Security and governance teams need a say in how these platforms are designed.
For identity, she doesn't treat agents very differently from people. Start with the RBAC and role bindings you already have and narrow them. Build specialists, not generalists: an agent for Terraform inside one product team, not an agent for everything. "You can't control what you can't see and what you can't understand."
On the SRE side, she told me about one of the biggest financial services companies in the US, which runs agents over terabytes of logs a day. The agents spot problems before they surface and get the alert to the right person. Don't page Bob, who's on PTO. Page Dave, who's on call. They don't do auto-remediation yet, and she thinks that's the right call. And if your humans see thousands of alerts a day that mean nothing, fix that for the humans first.
Then there's tribal knowledge. To give agents what they need, teams have to write down every assumption they make. Who to ask about Terraform. Where the real answers live. What the new junior engineer will and won't get assigned. Mallory says teams are now doing "anthropology on their own organizations." It's hard, and it's worth it either way, because it makes the platform better for humans too.
So where does a platform team start?
Ask why. If the answer is velocity, then start small. Her first path is the read-only PR review agent: low risk, a real time saver, and a good check that people can explain code they've generated with Claude or anything else. Once you're comfortable, move to the hybrid validate-change path, with a human reviewing the evidence. Then security scanning, then running it in dev, maybe staging. Training wheels first. As she put it, you don't start by jumping in the Formula One car.
She left me with a line from one of her PlatformCon New York workshops. An attendee asked how they're ever going to write an agents.md file for their agents when they can't even write a README for their repos.
That's the takeaway I'm keeping. Ask why before you add the agent. Fix human governance first. And write down what your organization knows, because, in Mallory's words, "it won't just help agents, it will help the people that work there."
Notes
Show Notes
This is a guest conversation episode of Ship It Weekly, separate from the weekly news recaps.
In this Ship It Conversations episode, I talk with Mallory Haigh about why context alone isn't enough for AI agents, the harness and governance layers that actually run them, and why she tells some platform teams to pump the brakes before adding agents at all.
Mallory is Head of Platform Education and Advocacy at PlatformEngineering.org. She teaches in Platform Engineering University and coaches platform teams through Platform Engineering Consulting, where the job gets jokingly called "platform therapist".
We start with her New York City analogy. Drop someone in the middle of the city with no directions and they'll get lost, and an agents.md file on its own does the same to an agent. It also needs to know what to do, how to do it, how it'll know it's done, and what it's allowed to touch. Mallory splits that into the harness, which is how the agent does the work, and governance: who and what is allowed, what's safe, and how it's watched.
Then we get into IDPs as context layers. Mallory says that's partly right, because the IDP and its golden paths are the foundation. What's missing are path specifications. Probabilistic paths are agent-driven, like a read-only PR review. Deterministic paths are your pipelines, like a CI build. Hybrid paths pair an agent with a gate: the agent opens the PR, CI validates it, the agent retries, and a human makes the final call. Cap those retries at three to five, and if an agent always does the same thing with the same outcome, ask whether it should be an agent at all.
On governance, Mallory says platform engineering is a practice about building trust, and that in the last three weeks alone she's told multiple platform teams to pump the brakes on agents because they can't get human governance right yet. We cover agent identity (specialists, not generalists, built on the RBAC you already have), agents watching terabytes of logs a day for SRE teams, and why every team now has to write down the tribal knowledge it never documented.
Her advice on where to start: ask why. Then try a read-only PR review agent, move to a hybrid validate-change path, and remember the line from one of her PlatformCon New York workshops: if we can't even write a README for our repos, how are we going to write one for our agents?
Highlights
Why context delivery alone isn't enough for AI agents
The harness and governance layers that make up agent infrastructure
Observability for agents: token spend, time to completion and retry loops
Probabilistic, deterministic and hybrid paths
Retry limits, and when an agent should be a checklist instead
Pumping the brakes when human governance isn't there yet
Agent identity, RBAC and specialist agents
Where a platform team should start: ask why, then a read-only PR review agent
Mallory Haigh is Head of Platform Education and Advocacy at PlatformEngineering.org. She teaches in Platform Engineering University and works directly with platform and DevEx teams through Platform Engineering Consulting, on how platforms succeed or fail inside engineering organizations and what changes when AI agents join the software delivery workflow.
25 years in production: DevOps, SRE, platform, and cloud. Host of Kube Signals on KubeFM. Grafana Champion; Confluent Community Catalyst; IBM Influencer 2026; DevOps Institute & ITIL Ambassador.
The line I keep coming back to from my conversation with Mallory Haigh is one she says is about 90% of her job: telling people "they need to ask why more."
Mallory is Head of Platform Education and Advocacy at PlatformEngineering.org. She teaches in Platform Engineering University, and she works directly with platform teams through Platform Engineering Consulting, which she calls "Mallory as a service." She also gets called a platform therapist, and after this conversation I understand why.
We started with a line from her pitch: context delivery alone isn't enough.
Her example stuck with me. Drop someone in the middle of New York City, tell them they're in the middle of New York City, and give them nothing else. No idea where they're supposed to go or what they're supposed to do. Chances are they get lost or get themselves into trouble.
That's what a lot of us do to agents. We hand over some context, maybe an agents.md file, and call it done.
Mallory says the agent also needs to know what exactly it's supposed to be doing, how it should do it, and how it'll know the work was done, and done correctly. She splits that into two layers. The harness is how the agent does the work. Governance is who and what is allowed, what's safe, and how it's watched. Together, she calls them agent infrastructure.
The watching part is where she thinks teams go wrong fastest. A good platform is auditable. You can see what an actor did, human or agent, and if the platform makes a decision itself, that's part of the audit trail too. For agents, she's looking at token spend, time to completion, which gates passed or failed, and how many times it went around a retry loop before it met the acceptance criteria.
And all of it has to tie back to why you brought the agent in. Reduced downtime is nice. But why did you need to reduce it? Did you have a problem?
I pushed on definition of done, because that assumes the ticket says what done means. Her answer made me laugh: "The definition of done sucks. It doesn't matter whether or not it's a human or a machine, it still sucks." The difference is that the machine can create a lot more chaos, a lot faster.
Then I asked about something I've been hearing a lot, including on this show: IDPs becoming context layers for agents.
Mallory thinks that's partly right. An IDP is an abstraction, and a golden path is a way to direct human behavior. You can extend that path to agents. The IDP is the foundation the agentic platform gets built on, and it has to work for the humans first.
What's missing, she says, are path specifications. She described three kinds.
Probabilistic paths are agent-driven. The agent decides what to do at each step, and the output isn't deterministic. Her example is a PR review agent that posts comments against your standards. It can be read-only and touch nothing.
Deterministic paths are what platform engineers already know: pipelines. A CI build runs the same way every time.
Hybrid paths combine the two. The agent opens a PR, a deterministic gate like CI validates it, and if it fails, the agent goes through a retry loop. At the stage most teams are at, it can't do the merge. A human makes the final call.
On that retry loop, she was practical. Set a limit. "Three to five, I would say, is your magic number to start with." Why? "I don't really know. Just seems to work." And make every retry teach something, either to the agent or to the person who built the platform.
She also said something I hadn't thought about. If an agent always does the same thing with the same outcome, maybe it shouldn't be an agent. Make it more of a checklist, use fewer tokens, and stop paying for free rein.
The part I keep thinking about is trust.
I asked how governance changes with agents, since platform teams already struggle with it when humans do the work. Mallory said it absolutely gets harder, because agents make us confront our lack of trust in ourselves and in the systems we've built. Platform engineering, agents or not, is "a practice about building trust."
So she's been telling teams to slow down. In the three weeks before we recorded, she'd told multiple platform teams to pump the brakes on putting an agent in, because they couldn't get human governance right yet. Her question for them is a fair one. How can you trust several machines running in parallel if you can't trust a person to go through a process without someone standing over their shoulder?
She also reminded me that developer experience isn't the only stakeholder. Security and governance teams need a say in how these platforms are designed.
For identity, she doesn't treat agents very differently from people. Start with the RBAC and role bindings you already have and narrow them. Build specialists, not generalists: an agent for Terraform inside one product team, not an agent for everything. "You can't control what you can't see and what you can't understand."
On the SRE side, she told me about one of the biggest financial services companies in the US, which runs agents over terabytes of logs a day. The agents spot problems before they surface and get the alert to the right person. Don't page Bob, who's on PTO. Page Dave, who's on call. They don't do auto-remediation yet, and she thinks that's the right call. And if your humans see thousands of alerts a day that mean nothing, fix that for the humans first.
Then there's tribal knowledge. To give agents what they need, teams have to write down every assumption they make. Who to ask about Terraform. Where the real answers live. What the new junior engineer will and won't get assigned. Mallory says teams are now doing "anthropology on their own organizations." It's hard, and it's worth it either way, because it makes the platform better for humans too.
So where does a platform team start?
Ask why. If the answer is velocity, then start small. Her first path is the read-only PR review agent: low risk, a real time saver, and a good check that people can explain code they've generated with Claude or anything else. Once you're comfortable, move to the hybrid validate-change path, with a human reviewing the evidence. Then security scanning, then running it in dev, maybe staging. Training wheels first. As she put it, you don't start by jumping in the Formula One car.
She left me with a line from one of her PlatformCon New York workshops. An attendee asked how they're ever going to write an agents.md file for their agents when they can't even write a README for their repos.
That's the takeaway I'm keeping. Ask why before you add the agent. Fix human governance first. And write down what your organization knows, because, in Mallory's words, "it won't just help agents, it will help the people that work there."