Ship It Conversations: Mat Ryer of Grafana Labs on AI Observability, Agents, Evals, and Operating AI in Production

Transcript

AI demos are easy. That is kind of the problem.

You can wire an LLM into a tool, give it a few

prompts, let it call an API, and suddenly it

looks like magic. It writes queries, it explains

dashboards, it investigates issues. It might

even suggest a fix. And in a demo, that feels

great. But production does not care about demos.

Production cares if the answer was right. If the

agent used the right tool. If it leaked data. If

it got confused by logs. If it made things worse

after a deploy. If the cost graph quietly turned

into a crime scene. And that is where this gets

interesting because the hard part is not proving

AI can do something impressive in a controlled

demo. The hard part is figuring out what happens

after people start depending on it. When it is

connected to real tools, real telemetry, real

production systems, real costs, real failure

modes, at that point, AI is not just a feature

anymore. It is another thing you have to operate.

I'm Brian Teller from Teller's Tech, and this

is Ship It Weekly. Welcome back to Ship It Weekly,

where I filter the noise and focus on what actually

matters when you are the one running infrastructure

and owning reliability. Most weeks, it's a quick

DevOps, SRE, platform, cloud, and security news

recap. In between those, I do conversation episodes

with people building and operating the systems we

all end up depending on. Today, I'm joined by

Mat Ryer from Grafana Labs. Mat is Senior

Director of AI at Grafana, a longtime Go developer,

author, open source contributor, and podcast

host. And in this episode, we talk about what

happens when AI moves out of the demo booth and

into production. We get into Grafana Assistant,

AI observability, evals, LLM-as-judge patterns,

telemetry cost, OpenTelemetry, agent guardrails

and why a chat interface is not enough if the

system underneath is making decisions people

need to trust one of the threads i liked most

in this conversation is Mat's point that AI

gives us a new primitive that sounds big and

abstract but his example is pretty simple a conversation

plus a loop suddenly becomes an agent Or as Mat

put it, for loops are back. But once people rely

on that agent, the problem changes. Now you need

to know whether it helped, whether it answered

the question, whether it picked the right tools,

whether a prompt change improved one workflow

while breaking another, whether your AI feature

is actually observable enough to operate like

the rest of your production stack. We also talk

about why AI observability is not just latency,

logs, and HTTP 200s. Those still matter, obviously.

But now you also care about behavior, cost, tool

selection, user feedback, eval results, model

versions, prompt changes, and whether the agent

is producing something useful. or just confidently

wandering around your telemetry matt also gets

into UX which i think is more important here

than people give it credit for because if the

AI gives you a wall of text you still have to

trust that text but if it can show you the graph

deep link you into the right Grafana view apply

the filters and let you inspect the source data

yourself that is a very different experience

and near the end We talk about where AI can actually

help operations teams today, not some giant six

-month AI transformation project. More like pick

something small. Use it to enhance the workflows

you already have. Let it write the query. Let

it help with the first pass. Let it automate

the boring parts, but build the guardrails and

feedback loops around it, which honestly fits

the name of the show pretty well. Ship something

small. learn from it, then ship again. All right,

let's jump in. Today, I'm joined by Mat Ryer

from Grafana Labs. Mat is a senior director

of AI at Grafana, where he's focused on how AI

fits into observability and production systems.

He's also a longtime Go developer, author, open

source contributor, and podcast host. And we're

talking about what happens when AI moves from

demos into production, why observability for

AI systems is not the same as basic service monitoring,

and what teams should be thinking about as telemetry

volume, cost, and operational complexity keep

climbing. Mat, thank you for joining me. Thank

you, Brian. Pleasure to be here. So starting

out, I'm curious, as AI systems move from experiments

into production, What are teams underestimating?

Yes. Well, first of all, I think it's very exciting,

this whole space. And I think that's where I

always start with this. We have a new primitive

now. We have a new capability, something that

we couldn't do before. And that, I think, is

just a very exciting thing. There are concerns

with AI and valid concerns. alongside them are

i think is the fact that we are now in a new

world and this is a very exciting thing you see

people making little agents to solve all kinds

of little problems we see people building big

agents and big complex ones as we are doing as

well at Grafana Labs to solve more complex and

deeper problems and just now like having a a

new primitive like a conversation Being able

to now, even just putting that in a loop, by

the way, which was when things became agentic.

That's just a for loop. That's just like, oh,

for loops are back. They're so back. They're

cool again. And I think agents are just amazing.

Just a simple primitive put together with something

that we're already very familiar with can produce

something new. So we're still in that phase.

And I love to see all the little things that

people are building. At some point you do end

up with, like we have a product in production,

Grafana Assistant, and people are relying on

it. They're using it as part of their day-to

-day. Some people have leaned very much so into

it and it's really unlocked them for all the

observability stuff they're trying to do. So

now suddenly we can't just vibe it. We can't

just feel it out. We've got to grow up and make

sure that this is going to work for them. And

I think people underestimate how hard that is.

It is quite easy to wire up an LLM to anything,

do a few simple tools, and suddenly you unlock

this new capability. But how do you really know

that's good? How do you know when you change

it, you aren't making it worse in some other

place? You know, we used to have unit tests,

test suites protecting us from all this. We didn't

really have that now. So how do we make sure?

when we make changes to it it is getting better

and how do we make sure it's even like correct

in the first place you know it demos very well

like you say we've kind of i think checked off

that from the mission like has to demo well for

people to care to it demos pretty well uh but

beyond the demo yeah how do you how do you make

sure it does that and we we have we had to solve

this problem in assistant which we built our

Grafana Assistant we launched that I think it

was last year now. And yeah, we've had to solve

that problem. So at a high level, how do you

start with that problem statement? How do you,

do you build special testing around that? How

do you actually solve that? Yeah, so I'd say,

right in the beginning, I think people still

vibe testing, manual testing, I still think is

the thing to do in the beginning. Because it's

the really when you're just feeling it out, you

need to know. Is this even possible? Is this

feasible? Does this make sense? And there's so

much innovation and there's so much opportunity

to learn that actually you still have to have

that same attitude of build things and ship it.

That's why I came on the Ship It Weekly podcast

because it's all about shipping it. So I would

say don't wait for there to be good evals. Don't

wait for you to have this problem solved before

you progress. But it is something to think about

in the back. in the background and yeah when

when we did assistant we essentially had to build

we had lots of different tools of course there's

like the normal telemetry that we're very good

at Grafana Labs metrics for measuring latencies

and things like this and we even have metrics

for costs measuring the the cost of different

operations and things uh logs of course kind

of makes sense i think tracing becomes more interesting

now because you could think of a conversation

really as a as a sort of trace you could imagine

it in that you know in that model um and and

i think so so you get a lot already just of the

basics but there are new things you care about

suddenly uh you know it's not just getting a

200 back from an LLM but now how do we how do

we make sure that the content the the way it

behaved the tools it chose to use how do we make

sure that stuff is right and we did it um partly

By vibing tools internally, literally we'd have

like, we've got a handful of them and various

dashboards and things and different databases

and different ways of storing things, you know,

just trying to sort of cobble together the solution

of this. And again, I think that was the right

approach in the beginning because it's a playground

where we can experiment and we can try things.

We can solve problems without having to think

about solving the entire problem. of this we

can just pick one piece of it and just solve

it with a standalone thing so there's a lot of

freedom in that and i think the culture that

we had in the engineering team when we're building

this played a lot into that because yeah we had

to make sure that people could try things just

solve that particular problem not go for months

and have big design sessions and just do this

enormous project to try and solve the whole thing

After we'd done all that, of course, you then

end up in a situation where you do have quite

a complex little set of internal tools that are

all slightly overlapping and doing things in

different ways. They don't all agree with each

other. Some of them are agentic themselves. So

that's even more getting like Inception, where

the agents are monitoring the agents to make

sure that they're behaving properly. And so then

we were able to take all that learning and put

that together into our AI observability product,

which is a new feature that we have in Grafana

Cloud. And that's our answer to this. And we

think we've picked the most important things

that people should care about and built that

into the product. And you can see we're different

to some kind of competitive products. We're different

in some interesting ways because I think some

of them were imagined. Whereas we built ours

from the real day-to-day of actually operating

the assistant at scale. Yeah, very cool. What

does production level observability for AI actually

look like? Well, you do need evals. You need

to be able to test whether what's actually being

asked for is being answered. And we have a...

We have a technique inside where we actually

have an LLM as a judge looking at a conversation.

And you can just do this. You can try this. It

works. Just ask it, did you answer the user's

request? Because you can see their first message.

You can see the path you went on and where you

ended up. Did you answer it? Did the user have

to intervene in any way? Were you ambiguous?

You can kind of ask it these things in, and you're

dealing with natural language now. So it's like

you write these tests also in natural language.

And then we can measure that. We can start to

measure and start to put some data and some science

behind the behavior of agents. So evals, I think,

are going to be increasingly more and more and

more important. We want these AIs, we want these

technologies to be trusted and to be used. Some

people write them off just because they know

that they can be wrong. So they can't trust it,

and so therefore they're not going to use it

at all. I don't think that's the right attitude,

personally. You know, the internet, the network

connections aren't reliable. They error all the

time. And we kind of paper over those errors

so you don't really notice it. Same kind of thing

with AI. We have these non-deterministic systems.

In some ways, when it comes to doing investigations

and things, for like digging around your telemetry,

that can actually be turned into an asset where...

The fact that multiple agents take random routes

through things can actually be good. That's what

humans might do if they all take a different

approach. But then you've got an even bigger

observability problem because now this one thing,

suddenly it's multiple conversations with models,

not even the same model, multiple models often,

certainly different system prompts, different

sets of tools. All this complexity now is kind

of a big... mess that you have to deal with and

we think yeah simple observability simple uh

and new signals as well as part of this is the

solution to so that you can just answer those

questions and make sure that your agents are

on the right track so kind of following up with

that a lot of teams instrumented everything then

they realized they were drowning in telemetry

costs uh what changed yes I think we're going

to go through that as well for AI. So I think

you're right. People would just monitor everything.

It feels really cheap. Like when you're writing

something, you just add a counter in code. It

feels like the cheapest thing you can do. As

soon as you then hit that scale, suddenly you're

counting. You're just counting at scale is hard.

So, yeah, you kind of create these new kind of

problems when you hit scale. The same thing is

definitely true for AI. I think a lot of people

aren't really monitoring much at the moment.

But when they do, and as this increases, we'll

have the same kind of problem. And, you know,

Grafana Labs has this technology called adaptive

telemetry, which is basically what they do is

they look at all the telemetry that you've used.

They look at all the queries you're making. And

then they identify bits where you've never queried

this metric. So we're going to automatically

sample it down. So we'll just take a sample of

it. We won't capture all of it. And, of course,

you can tune it and tweak it and things. And

this was really an answer to that problem, the

fact that people's telemetry was just growing

and growing and growing, and not just in the

history of it, but even just the now, because

people's systems and complexity is going up.

So I think that we'll end up with another kind

of adaptive telemetry. We have already in our

solution sampling, so you can choose how often

the evaluators will run on conversations. So

we're kind of thinking about that already. And

I think this is the advantage of building this

solution at Grafana Labs, because we had to solve

a lot of the observability problems already.

And a lot of the thinking applies directly, which

is very cool. We have these drill down apps,

which... basically present the data and you can

just click and drill into it. That turns out

to be good for humans, but actually quite good

for agents as well, because it's the same problem.

There's just too much stuff there and to look

through it all and find it is hard. So how do

we, yeah, the fact that we solve that for humans

turns out it's quite good to solve for agents.

Where does OpenTelemetry fit into this AI observability

story? I think it's going to have a... big part

of it we actually are talking already with um

ted young who was one of the co-founders of

OpenTelemetry yeah and we're kind of um i've

seen some proposals and i've seen some things

where the conversation data gets put into trace

data so that they they are stitched together

in that way we didn't take that approach because

of scale issues um that we saw so we haven't

got that approach in our solution but It could

well end up being some new standard like that.

But I just think with OpenTelemetry, it's going

to just take some time. You know, we like to

move very fast in the AI department in Grafana

Labs and in Grafana Labs in general. In some

places, you do have to go slowly. Where you're

dealing with standards or foundational things

that you really need to get them right, you're

going to go a bit slower. So I think it'll take

some time. But I do think OpenTelemetry should...

embrace this new world and and see which new

signals are common enough that we are going to

then capture them we got to make those decisions

ourselves when we built our solution so of course

we capture you know the the basic things you

expect metrics for latency and things we add

cost into there using metrics and stuff that

that's also turns out to be quite complicated

to do but very powerful if you have accurate

cost data um And with the way that we have labels

and things, kind of like the Prometheus model,

you can use labels for the different versions

of your app or different agent names or something.

So now you can compare performance, not just

cost and metrics, but the other dimensions as

well. And if you've got evals running, you can

compare behavior between different models as

well. So you can say, version one of our model.

Took a long time to answer this kind of question.

And it's because we hadn't prompted it much or,

you know, we just gave it one tool that it could

use. We did some work. We added a couple of tools.

We tweaked some prompts and our new version is

now much faster. It solves this problem. It's

much more accurate. The user's happy. We capture

ratings as well of users, you know, thumbs up,

thumbs down, as well as comments. All the sort

of practical things you need to actually have

some kind of feedback loop where you can improve

your agents. Is that what you have found to be,

I guess, the best feedback loop is like providing

like right in the chat, the ability to give feedback

immediately like, hey, this was not the right

response or hey, this was not detailed enough

or it went off on a tangent that was not really

related to the question. Or are you feeding that

in with other information as well? And then how

does that work? Yeah, I think user feedback is

the most valuable. And it is nice that you can,

even just a thumbs up, thumbs down is enough

to give you something to go on. I think actually

typing in a comment is even better, right? Because

we can genuinely have LLMs consider that commentary

and even suggest PRs. into our repo to improve

prompts and things like this um so mostly though

we don't get uh we don't get as much of that

as we would like so we can't rely on that and

the other thing is we're very sensitive about

learning from people's data without them realizing

that's what we're doing so yeah we are we are

very um yeah we want to make sure that we are

uh not yeah just being very trustworthy basically

if we're going to build AI that we expect people

to use I think the standards have to be very,

very high for things like that. Make sure that

your data is protected. It's easier than ever

to leak data. We had a case where, luckily it

was test data, but we set up a GitHub permission

for an MCP server. We didn't realize that you

don't need write permission to open an issue

in a public repo. So we didn't give it write

permission, but still it started to open an issue.

it was a test box, so it was no problem. It was putting data that

it saw in there into the issues very cool feature

of course and extremely useful for like IRM and

incident management and things especially useful

when it opens a PR for you very nice but we've

got to do that carefully and safely and i think

knowing that your systems are safe and aren't

behaving and that they will react uh properly

and appropriately to prompts that are kind of

probing and trying to do malicious things. I

think that all is a very important part of this

that we're still scratching the surface on. Yeah.

I think it also kind of highlights the need for

guardrails because LLMs are over-eager, right,

to solve the problem, get to the root cause and

provide a solution. And if it has the means to

do that, it will. Yeah. It's true for tools as

well. It's true for like, if you give it lots

of tools, it's just going to be like a kid in

a candy shop. It's going to be so excited to

just go and use all the tools. And given that

it's non-deterministic, you can easily end up

in a situation where it's using quite random

tools to try and solve problems. And it's trying

to use a hammer to open a door, for example,

which I'm not a DIY expert, but I don't think

that's how you do that. But yeah, so. So how

do you sort of tackle that? I think that's kind

of another interesting angle on this. Yeah, so

from a guardrails perspective, I'm sure at the

IAM level, like you're providing read-write

permissions or scoped permissions based on what's

needed. But are you doing like pre-prompting

as well? Or are you not relying on pre-prompting

at all? Because that's just too close to the

LLM and, you know, it could be influenced. uh

well yeah so we have how do you build that system

yeah yeah we've got a few different things one

of the so our AI observability solution lets

us uh write kind of security type tests and assert

them basically on the model on the model so that

kind of gets us we can do some kind of testing

there just to set test the end point to see what

how are you going to behave if a user asks for

this um you know it's also dealing with customer

data so it's also dealing with data in logs which

can just be any text so you have to be careful

to make sure that it doesn't just load some logs

and think oh i'm just going to execute something

in in here um and the real answer in in assistant

is we just have lots of different checks and

different gates and things happening because

there's so much it's a very complex um solution

i think it looks simple when you use it and the

UX there's another thing i will get to talk about

hopefully is the the user experience of this

is paramount here more than ever and it's always

been important i think in tech but the um you

know the fact that you you the fact that you

need to uh just just have well the fact that

we have this complex system we use multiple models

we've got like specialist tools inside assistant

all running in Grafana Cloud there's preemptive

work that happens around discovering the infrastructure

And so we put that into a semantic vector database.

So it's the agents have access to kind of learning

that it's had before. When incidents and investigations

are happening, that also is a great source of

information for agents. But again, how do you

make sure that's all safe? So I think at any

point when you're integrating and bringing some

data in, you have that security question of in

what ways could this be malicious? What protections

can we do here to help and make sure it's not

going to be a risky system for people to use?

And how have you thought about the UX side of

the user experience side of the AI agents? How

is that architected or how do you think about

providing that experience in a quality environment

that is actually useful? Yeah, this is somewhere

where I think the team have excelled. And we

have heard this feedback. One of the big labs

told us that at the time, this was the best looking

version of something that they'd seen using their

models, which was a big, big compliment, of course.

I think the key is we on the team just got really

annoyed by walls of text. It was a kind of a

byproduct of this is that, you know, the fact

that you are dealing in conversation means it's

generating text as what kind of how the whole

thing works. And it's just so much text. Now,

when it comes to trusting LLMs, you know, in

that text generation, that's where they can hallucinate

things. So we, instead of using text wherever

we can in the assistant. We'll use a Grafana

visualization. So, you know, luckily we have

this library of these beautiful tools. Grafana

has been around a while. So anything we needed

to represent from logs or from traces or metrics

or profiles or other ways with graph views and

Mermaid diagrams, things like this, we would

ask the assistant to do that and to prefer that.

And so it's... It's really about, I think, having

people with excellent taste, kind of designer

engineers in one, really, having them obsessed

with that mission of we're going to fight this

wall of text. Because if, you know, a picture

tells a thousand words. So if we can show you,

here's what the data is doing, not only can you

see it as a user, but that is still better as

an experience for users than reading what someone

tells you about that graph. Someone might say,

oh, it's a flat graph. Is it flat but high? Is

it flat but low? You can describe it. It's actually

easier. Just show me the graph. This is why I

don't think dashboards are going to go away in

the AI world because we're always going to care

about what's true. And if the agents are correct,

we're going to want to see that data. So the

Grafana Assistant is very visually beautiful

for that reason because it piggybacks a lot on

Grafana. Yeah, it's important to, it's going

to make a big difference. And I think you can

separate yourself if you put a bit more effort

into making sure that that experience is excellent.

I think you can separate yourselves from similar

capabilities. It turns out for some reason, and

we have a few theories on this, but it turns

out that LLMs are quite good at Grafana by default.

And we think it's because of so much of its open

source. Grafana. The whole code base is open

source, but also all the community stuff, all

the discussions around there, like community

dashboards out there. And a lot of the connected

ecosystem, Prometheus and OpenTelemetry, all

these other things are also kind of very open

source. And I think the LLMs have hoovered all

of that up, so they naturally have quite good

context. So when we first stitched together the

LLM... with Grafana and gave it a few tools and

had the UI in the Grafana front end. It was surprisingly

already quite good. And this is what people are

finding when they wire up Claude Code with our

gcx, which is a CLI tool that we released. That

thing, you know, for certain tasks, it's very

good. And if you've got your code right there

as well, like it's very handy to be working alongside

with production code. with telemetry you know

and having the having them considered at the

same time but all the work we then did in assistant

with the prompting the fine tuning of certain

little pieces of it the specialist tools and

lots of other kind of secret sauce that's gone

in created this uh complexity that we have to

deliver in a simple way so i think if you use

assistant it'll look hopefully just looks and

feels very intuitive you know it does cool things

like it can build deep links so it knows how

to jump you to the right page in Grafana but

not just the right page because all the your

because all the filters are in the URL parameters

it will apply those filters as well so it takes

you straight to the right view um and so that

was another good thing i think we did was We

aren't going to reinvent everything and give

you an AI-only experience here. We are going

to use all the capabilities that you're already

familiar with and some of the amazing tools that

already exist in Grafana as part of this too.

And the fact that the assistant knows what page

you're looking at. Little things like that. If

you go to a page and ask and say, tell me about

this, it knows what you're talking about. So

that really matters. Those kinds of little details,

that experience that the user's having, I think

is what makes it feel like excellent, helps you

trust it. You see the data itself as well. So

you can see if it's made a mistake. And I think

UX is more important than ever. Very true. Okay,

so for a team that is starting out wanting to

introduce AI, where... Where do you think AI

can actually help operations teams today? Like

where should they start? What's the most important

telemetry or piece of data to start using to

analyze? Well, it will depend probably on each

of their cases. The cool thing is they will know,

they'll have an instinct already for things that

they already care about. So the thing is, the

nice thing is probably people already are using

some kind of telemetry. They're using some kind

of observability. And really, AI is there to

enhance that experience. It's there to make that

easier to do. Since it writes the queries for

you, you don't have to learn PromQL and LogQL.

And I've used those languages before, of course,

but I would always have to go and look up again

how to do something or prefer the query builder,

honestly. Now you don't even need to do it at

all because genuinely, through natural language,

it can generate those things. And it generates

very complex ones as well. It's far out, you

know, surpassed me, my abilities. But I would

say like, use it initially to enhance what you're

already doing and see ways that you can automate.

In some ways, that's really what we're doing.

It's just automating it, but automating it in

a trustworthy way. And I would say from those

few little early things and do stuff, ship it,

Ship It Weekly. I like that actually as a motto,

Ship It Weekly. Not just the fact that this podcast

runs every week. But I actually ship things every

week. I like that because you're kind of forced

to focus on scope. Well, the scope that's most

important. You know, if you have a short window

for shipping things, then you've got to really

focus on what's important. And I think that applies

to people picking up AI things. Don't do some

enormous AI project that's going to take months

and months. Because honestly, everything's different

by then anyway. Pick something small, do it,

action it, and talk about it and share it. and

see what see if there's excitement you can drum

up it's what happened at Grafana Labs and then

now you see people all over the company building

all sorts of things and doing amazing things

with agents some of it will eventually be a product

i'm sure but it's it comes from play and that's

something that we do all right at Grafana Labs

we we have a lot of space for people to play

and get creative and try things and fail for

things to fail um there's a great analogy that

i love where they they did a study where they

got these two groups and they told they were

making pottery you know on the pottery wheel

but they were all amateurs they told one group

you've got to make the best pot you can and they

told the other group just just make as many pots

as you can doesn't matter what they're like just

make lots of them and then by the end of it the

the group that were making as many pots as they

could were making better pots than the group

that was trying to make the best pot and i think

That tells you a lot, I think. It's the trial

and error that you learn. You know, there was

a lot more waste on the floor of this other team

where pots hadn't worked and they had to throw

it away or smash it up and tear it down and turn

it into something else. So it feels wasteful,

I think, to some. And they might be tempted to

kind of optimize that away. But honestly, that's

often where the learning is. So I think people

should pick up something, tackle it. something

small have a play see what you can do and from

there yeah you create new problems for yourself

but they're good problems to have where where

do you think AI is going with observability this

year i mean obviously it's it's the talk at every

conference but is there overarching areas in

observability that you think that AI um that

we're just at the forefront like we're able to

do now with larger token uh more tokens more

uh better models um faster models is there anything

that like lends itself now that maybe we couldn't

have done six months ago or we could do in six

months um based on these these rapid changes

to AI i think seeing seeing the models get better

and get smarter is nice we are finding that we

have to tell it less which is interesting we're

also finding that we We are basically tuning

our prompts to a specific model. We noticed with

Claude 3.5 to 3.7 even, 3.7 didn't perform

as well as 3.5 by default. We had to make it

work by changing it and tweaking it because it

was a kind of different thing. And so I think

we'll get to the point where agents are, yeah,

you have to tell them a lot less. I think their

instincts would be good. We already have plenty

of tools to help with skills and things. In Assistant,

you can write skills, which lets you basically

control how the agents behave. This is nice if

you've got expert SREs where they've spent years

building these patterns and practices and they've

got these high standards. You can then give those

same high standards to the agents and they'll

follow your behaviors. So you can already customize

and tune it like that. But I think... that's

going to just improve. And I sort of expect that

to happen. Bigger context windows, not necessarily

seeing that improve things too much. We already

have kind of too much context. A big challenge

that we're doing at Grafana Labs is, and with

Assistant when it does an investigation, it's

about going and finding the right context and

keeping the right context and discarding the

things that's not right. So big context windows,

kind of they're tempting. You just think, oh,

we could just fill it all up with everything.

I don't think that's going to help it at all.

It's just more for it to think about. But I do

think we are going to see more things automated.

And one example I think is an easy one is rolling

back a deploy. Let's say you kick off a deploy

through GitHub, whatever your process is, at

the end, once it's out, Maybe, you know, there's

some, you do this in staging and things, whatever

you already have, all the great stuff. In production,

have agents actually go and do a series of kind

of critical tests, check things, check all the

dashboards, you know, they know how to use Grafana,

so they know how to use all your telemetry. So

how can we get all of that and make good decisions

from it? And then a decision might be that. that

deploys not i'm not happy with it for whatever

reason and i'm i as the agent i'm just going

to decide to roll it roll it back so i think

that's a safe quite a safe operation because

it's you should always be able to roll back a

release that should be a safe operation um it's

not always so it's you know it's not without

risk but um yeah that feels like something like

that i think we're going to get we're going to

see more and more of that and then i want to

go further what else are we happy to let it do

can it can it deal with scale issues based on

unexpected events can it um what else can it

do where else can it take action and i think

uh we need to get a bit braver sometimes uh on

some of these things it's on us to build the

tools and make sure that they are behaving well

and you can see what they're doing and you know

people will some people will always have the

the human in the loop to give the approval and

hit the button and i actually think that's also

fine but but all the work that leads up to that

point Because there's still a big space there

to explore. And we are exploring it. It is very

exciting, some of the things that can happen.

But I do think we're going to have agents doing

more of this operating things for us. At least

the simple stuff. And that frees us up to then

focus on the cases that aren't simple and that

are more complicated. Or frees us up to build

more things and do more other things. Very true.

Okay, so wrapping up, is there any belief about

AI or observability that you think is just wrong?

I think one thing that's wrong is because it's

not 100%, you therefore can't trust it. As we

mentioned earlier, yes, it's not perfect. It

gets things wrong. So do people, but you still

hire people, hopefully, for now. So I think that,

I think, but that's not just AI. I think people

often struggle with that. I remember, I remember

like in COVID times, I was wearing a mask and

a guy in the elevators like thought I was silly.

Like I thought I just believed some hype or whatever.

So he's like, why are you wearing that? They're

not even effective. They only like, only prevents

20% of spread or something. So it's like, oh,

is it? So it's only 20% fewer people going to

be. But because it wasn't 100%, he dismissed

it. I see people doing the same thing with AI.

Or they think with coding, it's not as good as

me because I prompted it, asked it to do something,

it didn't do it as well as I would have done

it. And so I did that last year and I'm not going

to touch it. AI is not for me. That's also a

mistake. Keep trying them. It's usually you haven't

prompted it well enough if it's not working.

You can prompt it differently and there's different

things you can try. And you can also be much

more specific. In fact, the more context you

give it, the better. So, yeah, it's not 100%.

It's not perfect, but it's good. It's changing

the world. Yeah, for sure. Mat, any closing

thoughts for our audience? No, but I just want

to say keep shipping it weekly because, you know,

that's the only real way to know you're doing

anything useful. Iterate, yeah. And keep listening

to this great podcast as well with Brian. Appreciate

it. Thanks, Mat. Thank you so much for your

time. Thank you for coming on. Really, really

appreciate it. Pleasure. All right. That was

my conversation with Mat Ryer from Grafana

Labs. The thing that stuck with me the most is

that AI in production has to grow up pretty fast.

It is fun when it is a prototype. It is fun when

someone wires up a model. gives it a few tools

and suddenly it can answer questions that used

to require bouncing between dashboards logs traces

docs and half a dozen slack threads that part

is genuinely cool but once people start depending

on it the bar changes now it needs observability

it needs evals it needs feedback loops it needs

cost awareness It needs guardrails. It needs

some way to tell whether the agent actually helped

or just produced a convincing answer. And that

is where I think this conversation gets useful

for platform and SRE teams. Because the trap

is thinking AI observability just means watching

the AI service like any other service. Did it

return a 200? How long did it take? How much

did it cost? Those are good signals. but they

are not enough. With agents, you care about the

path it took, the tools it used, the context

it kept, the context it threw away, whether it

answered the actual user request, whether the

user had to correct it, whether the model changed

behavior after a prompt update or model upgrade.

That is a much messier kind of production system.

I also liked Mat's take on trust. A lot of people

treat AI like it has to be perfect or it is useless.

And I get the instinct, especially when we are

talking about production systems. But we already

operate plenty of imperfect systems. Networks

fail. APIs time out. Humans miss things. Dashboards

lie by omission. Runbooks rot. The answer is

not blind trust, but it is also not refusing

to use the thing because it is not 100%. The

answer is instrumentation, feedback, constraints,

and judgment. It's the same as most other production

problems, honestly. The other big takeaway for

me was the UX side. AI should not just bury operators

in a wall of generated text. If the system can

show the graph, link to the right view, apply the

right filters, and expose the evidence behind

the answer, That is way more useful than a paragraph

that sounds right. Because at the end of the

day, operators still want to see what is true,

not just what the model said. So my takeaway

is pretty simple. Use AI to make the work easier.

Let it help search through telemetry. Let it

summarize patterns. Let it assist with the first

pass of an investigation. But do not skip the

production discipline. Measure it. Test it. Watch

the cost. Build evals. Keep humans in the loop

where the blast radius is real. And when you

give an agent tools, remember that it may actually

use them. That sounds obvious, but it is going

to matter a lot. I'll have links to Mat, Grafana

Labs, and anything else we mentioned in the show

notes. If you enjoyed this conversation, Follow

or subscribe to Ship It Weekly wherever you listen

to podcasts. It helps the show and it makes sure

you get both these conversation episodes and

the weekly DevOps, SRE, platform, cloud, and

security news recaps. You can also find everything

over at shipitweekly.fm. Thanks for listening

and I'll see you later this week.

Ship It Conversations: Mat Ryer of Grafana Labs on AI Observability, Agents, Evals, and Operating AI in Production

Watch this episode here

Chapters

Transcript

Catch This Episode

Host Commentary

Show Notes

👤 Guest

Episode Contributors

More from Ship It Weekly

EKS Rollbacks, GitHub Actions Supply Chain Attacks, AI Agentjacking, CloudWatch Log Alarms, and Why Safety Nets Don’t Replace Ownership

containerd CRI Vulnerabilities, Datadog PostgreSQL HA on Kubernetes, AWS DevOps Agent with Datadog MCP Server, EKS Control Plane Egress, and Why Users Feel the Wait

PeopleSoft Zero-Day Exploited, npm v12 Install Script Changes, GitHub Agentic Tokens, Anthropic Model Risk, and Default Trust Breaking

CISA’s GitHub Leak, AI Root Cause Analysis, Copilot Agents, Claude Code in CI/CD, and Kubernetes Seccomp Risk

Get the next episode in your inbox