Ship It Conversations: Tristan Handy of Fivetran + dbt on AI Agents as Data Consumers, Why Dashboards Stop Being the Product, Context Layers, Evals, Operating on the System, Software Factories, and Why Text-to-SQL Isn't the Hard Part
November 16, 2026 37:44Episode 84Conversation
devopsSREDevOps learningAI agentsdbtfivetrandata engineeringanalytics engineeringsemantic layertext to SQL
Try reloading the player or open this episode directly on YouTube.
In this episode, Tristan Handy discusses the shift towards AI agents as primary data consumers, moving beyond traditional dashboards. He explains the importance of context layers and how agents can achieve higher accuracy in data interpretation.
Now Playing
Ship It Conversations: Tristan Handy of Fivetran + dbt on AI Agents as Data Consumers, Why Dashboards Stop Being the Product, Context Layers, Evals, Operating on the System, Software Factories, and Why Text-to-SQL Isn't the Hard Part
Ship It Weekly
0:00Current chapter: Cold open37:44
Chapters
Jump to a section in this episode.
Speed & share
Transcript
Dashboards are a hack for the human visual cortex.
That's how my guest today puts it. They pipe
a lot of information into our brains fast so
we can spot the patterns. But more and more,
the thing reading your company's data isn't a
person. It's an AI agent. And agents are already
great at writing SQL. It compiles and it runs.
almost every time. The hard part is whether the
agent understood the question and the business
behind it. In the benchmarks he's seen, an agent
turned loose on raw enterprise data gets about
30% of its answers semantically right. Give
it context and a semantic layer, and it's north
of 90. I'm Brian Teller from Teller's Tech, and
this is Ship It Weekly.
Welcome back to Ship
It Weekly where I filter the noise and focus
on what matters when you are the one running
infrastructure and owning reliability. Most weeks,
it's a quick DevOps, SRE, platform, cloud, and
security news recap. In between those, I do conversation
episodes with people building and operating these
systems. Ship It Weekly turns one this Friday.
To celebrate through mid-December, you are getting
two conversations a week. Mondays and Wednesdays,
with the news every Friday. Then, we're back
to our usual rhythm. Today, I'm joined by Tristan
Handy, co-founder of dbt Labs, and now president
of Fivetran and dbt. If you've worked anywhere
near a data team, you've probably run into dbt.
His team likes to call it Terraform for data.
We talk about what changes when AI agents become
the main consumers of your data, why the hard
part isn't text to SQL, what a context layer
is, and why you need one, and why he thinks the
most interesting work is operating on the system
instead of inside it. He also gets into what
dbt got wrong selling to enterprises, and the
agent loop his team built to ship dbt 2. All right,
let's jump in.
Tristan, thank you for joining me. Thanks for having me. So starting off, I'm
curious. We've spent years building data platforms
around people looking at dashboards. What changes
when an AI agent becomes the thing consuming
that data? Oh, gosh. Yeah, we're just diving
right in, aren't we? Maybe the first and most
kind of obvious thing is that dashboards aren't
the product anymore. Dashboards are a hack for
the human visual cortex. They are a way to pipe
lots of information into our brains very quickly
and allow us to notice patterns. And that makes
a ton of sense when our brains are the ones that
are looking for the patterns and applying the
intelligence. But it is becoming true that agents
are replacing us as the primary form of intelligence
that is looking at our data and trying to find
patterns and the meaning of those patterns. And
they don't really want to consume data via images.
And so dashboards go away. Dashboards will stay,
I think, as a translation layer so that if agents
notice something interesting, the next step will
be, help explain to me what that thing is that
you noticed. I think often a dashboard will be
a component of that answer. But so the asset
shifts lower in the stack and agents will consume
some combination of your gold layer, which is
essentially like well-modeled data in tables
in your data lake or data warehouse or whatever.
There will be a semantic layer which will actually
capture, here is how our organization calculates
metrics. We can talk about why that's important,
but the organizational design of your metrics
is not primarily a technical artifact. It's primarily
a decision-making artifact. It's a reflection
of how your organization wants to see itself.
And then a context store, so that your agent
has the ability to understand how to navigate
all the data that it has available to it and
make meaning out of it. So I think that those
are the primary three places that agents will
interface with your data stack. And when you
say gold, you're talking like a medallion
setup? Okay. Just so our listeners that may not
understand the gold, bronze, silver, yeah. Totally.
I understand that the world of the data practitioner
is a little bit different than the world of the
SRE or DevOps practitioner. So I'm happy to go
as deep or not deep as relevant. But yeah, assume
that all of your data comes into your data lake
in a pretty raw form. And oftentimes, it actually
takes many stacked... functions, transformations
in a DAG in order to actually get to a place
where it represents some meaningful unit of business
activity. And that's typically what you call
your gold layer. And for this data, are we looking
at this differently for an agent than we would
a person like a practitioner that's actually
analyzing this data? Are agents looking at it
the same way? Do we have to reason about it differently?
Do we have to structure it differently? Yeah, okay, that's a great question. I, I think that the core
way that you structure data can be very similar.
It's, you know, whenever it was the 90s or something
when Ralph Kimball wrote the first book on, you
know, how do you structure relational data for
analytics. That approach... And there's several
competing approaches, but whatever. The approach
that you've taken to data modeling historically,
I think, can persist pretty well. From an agent's
perspective, really, they don't really care that
much about exactly how you structure your data
as long as they can understand how you have structured
it. The constraints on how you structure your...
data are much more imposed by the query engine.
So everybody's really very focused these days
on optimizing your spend on tokens. In my world,
for several years, the runaway cost was the compute
on top of your data lake. A very blunt instrument
and scans can be very expensive. So your data
modeling is primarily done to make sure that
you're not spending an arm and a leg in your
data lake compute costs. Makes sense. Okay,
so in SRE, we can tell you whether a request
succeeded, but an agent can get a successful
response and still make the wrong decision. What
does reliability look like at the data layer?
Wait, is your question about reliability in the
context of agents, or is it just data generally?
on the agent side specifically, if you can answer
that. Yeah, because it's a somewhat hard question
to answer when humans are in the loop too. That's
fair. But there are benchmarks when it comes
to evaluating agents and their accuracy for data
-related tasks. I think that the one that comes
up most often is called Spider. I think there's
one, Spider 2. Benchmarks like this attempt to model
the complexity of real world enterprise data,
which has, it's very messy. It has a tremendous
number of different entities modeled in it. Join
pathways are not clear, et cetera, et cetera.
And you can then ask questions of this data set.
and use an agent to go get you the answers to
these questions. And you can experiment with
how you ask the questions, what tools you give
the agents, what kind of context you give them,
and measure the difference in the results on
the benchmark. There are pretty stark differences
in the performance you will see. If I could summarize
the kind of general industry wisdom on these
evals today, it is that agents are very good
at writing SQL. They can write SQL that compiles
and executes against your database with almost
100% accuracy. The problem today is not text
to SQL. The challenge today is, did the agent...
fully understand the question that you are asking?
Do they understand the context of the data? And
do they actually get you the semantically correct
answer to your question? And that one is much,
much harder. The most recent benchmarks that
I've seen on this are that if you just turn an
agent loose on enterprise data with the Spider 2
benchmark, you're at something like 30% accuracy
for this semantic correctness. But if you supply
it with context and a semantic layer, then you
will be able to get that accuracy north of 90%.
So really, the world of data today is all about
how do we construct systems that take the knowledge
that humans either have in their brains or know
how to get access to via whatever mechanisms
and put that at the hands of agents so that they
can come up with correct, trustworthy answers.
Yeah, no, it makes 100% sense. I guess kind
of pulling on that thread, now talking about
like the governance side of it, if an agent can
discover data and act on it, where do we put
the boundary between useful autonomy? and access
that you'll end up regretting. Say more about
that. Where's the regret coming from? Well, I'm
more thinking the difference between reading
data and taking action. And that taking action
step can obviously devolve into some crazy modifications
of your application, your infrastructure. So
where do we draw that line? I think that it is
very interesting to hear you ask that. And the
reason that I scratched my head at first is that
I don't think that's generally a question that
data practitioners are wrestling with yet. I
think it's one where they will absolutely have
to, but the kind of organizational history of
data is that it is used not as a part of the...
operational flow of the business, but as a part
of the ability of the business to introspect
itself. And so the result is always prepared
for some human brain to look at and potentially
take action on. But I completely believe that...
So what data people and data technologists have
been doing for 30 years now is essentially building
these highly refined nervous systems whereby
data is collected and refined and eventually
presented. And I think that those nervous systems
are going to be quickly hooked up to action taking
and agents are what's going to do that. And so
we will quickly have to wrestle with the question
of like, okay, how do we assess the seriousness
of a bad outcome here? And how do we put the
right checks in place and all of that stuff?
But that's largely not where the industry is
at yet. What about even on the PII side of that
data or incorrect assumptions from the data that
the agent may be making? Is that a concern from
the data pipeline side? So I'm not a security
expert, but what I would suggest is that there
was, in the very early days of the... AI data
conversation. There was a lot of concern with,
well, can I let my model, the models have access
to customers' PII? And I think that we've largely
moved past that conversation because it's such
an obviously important commercial thing. And
so at this point, from my perspective, letting
AWS store customer's data in S3 versus letting
them process it using bedrock. It's like kind
of all the same. AWS has a mechanism for doing
that. We already trust our cloud providers, et
cetera. So I don't think that's a big thing absent
the fact that certainly it is possible to design
agents in a, you know, we're all too aware of
this now in the post-hugging face world. It's
possible to design agents that can do inappropriate
things with data. And so you just have to make
sure you're staying really far away from anything
like that. I think that was half of your question.
What was the other half? Oh, incorrect results
of decisions. Okay, this is an interesting one
because different organizations think about this
differently. There are organizations today where,
I don't know, I might call it run and gun. Where
it's just like, this is the answer and I'm just
going to by default trust it. And we generally
have pretty high accuracy rates. So like, let's
go. And you know, maybe you'll get some decisions
wrong in that modality, but I don't know. Maybe
it's appropriate if velocity is the thing that
you're optimizing for. Because you certainly
can move real damn fast that way. In an enterprise
context, typically you move more slowly and that
final check is done in the same way that when
you build a software factory today, you have
this decision to make who merges the pull requests.
And some people say the human does and some people
say that the agent does. And so you get to make
that decision. Okay, so let's move on to analytics
engineering. So dbt helped make data transformation
look more like software engineering. As agents
write more SQL and consume more data, how does
the analytics engineer's job change? My general
guidance here is really not about analytics engineers
or SREs or software engineers. It is all of these
very technology knowledge work fields are going
to be systematized in a way that they haven't
been before. And you as a professional will either
be operating inside the system or you'll be operating
on the system. And from my perspective, the more
interesting work to do is to be operating on
the system, to be designing the loops the evals,
et cetera, as opposed to having tickets assigned
to you. I think that this is true of all of the
different types of jobs that I just mentioned
and many more. And so for analytics engineers,
that really looks like not, how do I really cleverly
model this data to solve this kind of thorny
quantitative problem? It turns out that, you know, Astra 6 is going
to be able to solve that problem as well as you
can, if not better. And instead, you should be
thinking about the system holistically and saying,
what evals do I have? What evals do I need? How
do I see the things that are unseen, like the
latent business needs in the organization that
are not being reflected in the loop that we have
today, etc. And that type of work is going to
be important for a long time to come. Yeah, so
it's moving up a layer, essentially. 100%. One
of our values that I now authored 10 years ago
is we believe in moving up the stack. And I think
this is a big moving up the stack moment. We've
all been very identified with our own set of
technical skills because we spent a long time
building those technical skills. And it turns out that the technical
skills are not going to be the differentiators
for us moving forwards. Yeah, I think you're
right. When you were building dbt, what made
practitioners trust it enough to put it into their
production workflow? How did you make that happen?
How did you build that trust with the community?
That's a good question. I think that there were
a couple different factors. One is that it is,
dbt is a lot like Terraform. We've often thought
about it as Terraform for data. And Terraform
is not really in the production request response
flow. And so while it's certainly a critical
piece of infrastructure, it doesn't have the
requirements on it that something in the request
response flow has. It had so much, it was very
unique and did not have a ton of substitutes.
And so our early adopters felt like they were
willing to do the work to look at the code, engage
with the community and say, you know, like literally
our first, you know, reasonably sized company
that started using dbt showed up in our Slack
community and said, you guys don't have any tests.
is this a real thing or is this just some side
project? And, you know, with dbt at that point,
it existed for like all of six weeks or something.
And yeah, we didn't, we hadn't built a test suite
yet. And so we were like, Oh no, no, no, no.
If, if you'll use it, we'll, we promise to build
tests. So like that kind of pressure over time
eventually sharpens you. That's fair. Do you
think that there's something that developer-first
companies get wrong when they start selling
to enterprises? Oh my God. That was a painful process.
I'll answer a slightly different version of the
question, which is, what, what do I think we got
wrong? Um, I, I think that I was too willing to
say yes to take the revenue. And, um, I think that
in part that is because I didn't have some grand
vision of what our commercial product was going
to become. I had a pretty clear vision of the
role for our open source product, but the commercial
product was really just, let's see what things
people are willing to pay for. And in the early
days, we experienced some real Fortune 500 pull.
We essentially, as we built up our engineering
team, because, you know, back in 2020, you had
to actually hire engineers to write code. And
we built up our engineering team and we built
enterprise features. And for like, I don't know,
two years, we weren't able to make meaningful
forward progress on the product vision because
we were just so focused on building enterprise
crap, enabling multi-cloud deployment, role
-based access control, all of these different
things that you have to build, but you want to
be real mindful of when you sequence the work
in. How did you decide what stayed open source
versus became that commercial product? We ended
up, after maybe two years, we kind of solidified
an idea of of that dividing line. The answer
effectively became state and user interface.
And so if this was a feature that required a
database, then it was better for a cloud product.
I mean, our installed product, it operated purely
on top of files. It still does operate on top
of files. There's no database persistence then
if you wanted a GUI associated with it, and there
ended up being a pretty meaningful group of users
who, because of where data practitioners' careers
kind of come from, the GUI was a useful bridge
for them. Has the Fivetran combination changed
how you think about practitioner choice at all?
Like between open source and commercial? Yeah.
I think that the... The merger that we have is
a real journey getting through this. It's a year
-long process. We closed the merger about five
months ago at this point. There is a lot that
it changes about our world. It doesn't change
what we are delivering to our open source community.
In fact, it enables us to... be more open because
we have more options to monetize across a broader
set of platform features. But I think that we
are now very focused on innovating in a way that
looks more like Lego bricks, as opposed to historically,
we thought about really like a... And it's very similar to a Confluent or
a Mongo or whatever. You think about like, here's
our community offering and here's our enterprise
offering, and you get to make a choice. Which one
of those do you want? And the way that we are
constructing the world now is that we have, as
a combined company, we have multiple different
open source offerings. dbt is our biggest open
source offering. And then there are modules that
you can adopt a la carte. They don't ask you to move
into a different version of the software. They
plug into, uh, the, what you're already using. And
rather than adopting some kind of enterprise
version, you say like, I want, I want dbt state
or I want lake compute or whatever, and you can
uh, on a consumption basis, pay for that specific
thing. Okay, yeah, that makes sense. I was, in my
mind, I'm thinking about it, going back to that
Terraform example, HashiCorp, IBM, you know, acquisition
and how that kind of changed licensing, and yeah.
So, yeah. Yeah, pretty different. Yeah, no, no, it's
uh, we've, we've only become more permissive with
licenses, uh, since, since the merger. And, um, we
and Fivetran are super different than HashiCorp
and IBM in that both of, both companies, uh, kind
of grew up together and are directly complementary
pieces of technology. And so they together form
a very nicely packaged platform offering. Like
symbiotic relationship more than, yeah, gotcha.
Okay, so if I run a platform or data team and
people are asking me to connect agents to our
enterprise data, what is the first thing you'd
have me do? Oh my goodness. Well, the first thing
is actually, it's like eat your spinach. All
of the things that were good ideas in the past
are still good ideas. If you have a well-governed
data lake and the data in it is well-modeled
and you have a semantic layer, and these are
all things that have been useful for a decade,
then you're already a long way down the path.
What you still need to do is you need to figure
out an answer to the context layer, which is
a legitimately new thing. Now, it's not rocket
science. I mean, these are the kinds of things
that, I mean, like I remember 10 years ago, it
was very trendy to build your own data tooling.
Uber built their own orchestrator. Airbnb built
their own orchestrator. And this is like a thing
that people did. Airbnb built their own data
visualization tool. Nobody builds that kind of
stuff anymore because they're like, whatever,
there's good enough off the shelf stuff. But
so like these same types of companies are currently
building their own context layers. And so you
can do that. There are starting to be context
layers that you can buy. We have a context layer.
It's very new. It's like, I don't know, a couple
dozen companies using it. We use it internally.
It's freaking awesome. The difference between
not having a context layer and having a context
layer is the difference between a data agent
that you want to come back to and ask more questions
of and the one that you don't. And then you need
to implement evals to make sure that as people
on the data team implement changes throughout
the entire stack, that the accuracy of answers
goes up and not down. And then you probably are
thinking about real time because you're going
to want to start having agents act on behalf
of the intelligence that they get and not just
serve it up to you as a human. And real time
is important for action taking. So there's this
whole set of things that you will do. But really,
step one is. Make sure your house is in order
from the best practices. And two is context layer.
And then you need to get adoption across your
company. And this is very different than the
adoption of a BI tool. People know what a BI
tool is. Most people have lived their entire
careers in a context where there's always been
some type of way to get a dashboard for something.
But this is a new type of modality. And you don't
just want somebody to go through a training and
say like, okay, cool, I know how to make a dashboard.
What people who have deployed these things in
production find is that there is a real depth
of engagement that occurs with analytics agents
where you can experience over the course of several
months of adoption, you can experience Query
volumes go up by 50x just because there were
all these latent questions that people throughout
your organization had that the friction was just
too high to get them answered previously. Yeah,
now that makes sense. Okay, so wrapping up, we
were talking before we started recording. I'm
just curious, what have you been building lately
with AI tools? Oh gosh. So I have for the past
month ever since, personal agents have become
the focus of things in the Valley. It's either
personal agents like Muse or Jev, where everybody's
talking about one of the two. And I don't know
if you are like me at all, but I really just,
after two decades of consumer tech, I just really
don't believe that... My personal incentives
are aligned with any of the companies that are
building personal agents today. And so I've been
building my own personal agent. And it's actually
not like, it's not rocket science. It's a bunch
of integrations and it's a bunch of, you can
build really clear permissions into it, blah,
blah, blah. But the thing that we were talking
about before hopping on is I recently decided
to get a little more serious about the development
workflow and have refactored everything from
a, like, Claude Code, sit at my terminal and ask
it to do stuff to a software factory where I
file tickets and comment on pull requests and
this kind of thing. And that transition has been
super, super fun for me because it's just much
more effective. And I think that most people
are still experiencing the world of agentic development
in a Claude Code terminal. And I know that that
only really started working well a year ago,
a little less than a year ago. But I don't think
that's the primary modality that we should be
interacting with these things anymore. So do
you think building these software factories,
do you think that this will have an impact on
dbt open source? the way you run an organization,
or are you thinking about this just as like small
side project efficiencies? It's a good question.
I think that most software maintenance will end
up needing to be turned into a software factory.
And so that absolutely will. I mean, here's one
of the ways that we implemented this internally.
We've just gone through a big upgrade from dbt 1
to dbt 2. We do not increment major versions very
often. That's been five years and coming. And
because of the way that dbt's product surface
area looks, there's a non-trivial migration
effort dbt 2 has much tighter language constructs,
and you have to validate users' codes more effectively.
We care a lot about conformance. How do you make
sure that dbt 2 supports all the functionality
of dbt 1? And we have a metric that we track,
which is our conformance metric. And we just
weren't making progress fast enough. We have
a team of something like 20 people that were
just focused on conformance all day, every day.
And a year ago, they made some early wins and
then it kind of flattened off a little bit. And
so we kind of took almost like a two-month pause
and we implemented something called Conforman.
And it was stressful because I'm like, hey, everybody,
I want the line to keep going up. And they're
like, no, no, no, we're going to pause and then
it's going to go up fast. And Conforman is essentially
a loop that identifies conformance defects. It
writes tests for them. It writes the code to fix
them and validate them, etc. And we drove up conformance
from, I can't remember, something like 45% in January
to now 93%. And, you know, every successive percent
gets harder. You have to fight for it harder. So
it's been a dramatic, dramatic... Uh, I don't think
we would have been able to ship V2 without it.
That's awesome. Where can people find more out
about dbt, Fivetran, and yourself? dbt is getdbt.com.
Fivetran is fivetran.com. And I, these
days, primarily, I've hidden from most forms
of social media, but I still have a podcast at
the Analytics Engineering Roundup and the Analytics
Engineering Podcast. Cool. I'll link all of those
in the show notes. Tristan, thank you so much
for coming on. Really, really appreciate your
time. Thanks, it's been fun. That was my conversation
with Tristan Handy from Fivetran and dbt. The
biggest takeaway for me is that the hard part
of pointing agents at your data isn't the SQL.
Context and a semantic layer are the difference
between about 30% right and north of 90. So
his first step for a platform or data team is
kind of boring. Eat your spinach. A well-governed
data lake, well-modeled data, and a semantic
layer are still the foundation. Then add the
piece that's genuinely new, a context layer,
plus evals. So accuracy goes up and not down
as people change the stack. I also liked his
take on careers. As more of this work gets systematized,
you are either working inside the system, or
on it. The interesting work is on the system, designing
the loops and the evals, and spotting the needs
nobody has written down yet. He calls that moving
up the stack. And his team lives it. Their dbt 2
conformance work had stalled. So they paused for
two months and built an agent loop that finds
the gaps, writes the tests, and fixes the code.
It went from something like 45% to 93%. He doesn't
think they could have shipped version 2 without
it. You can find dbt at getdbt.com, Fivetran
at fivetran.com, and Tristan's Analytics Engineering
Roundup newsletter and podcast all linked in
the show notes. Follow or subscribe to Ship It
Weekly wherever you listen, and find previous
episodes at shipitweekly.fm. Thanks for listening,
and I'll see you later this week.
Ship It Conversations: Tristan Handy of Fivetran +…
The line I keep coming back to from my conversation with Tristan Handy is one he dropped two minutes in: dashboards are "a hack for the human visual cortex."
Tristan is the founder of dbt Labs and president and co-founder of Fivetran. If you've worked anywhere near a data team, you've run into dbt, and his team likes to call it Terraform for data. That framing made this one easy to follow from the SRE side, and it's a big part of why I wanted him on.
Dashboards exist because our brains are the ones looking for patterns. Pipe a lot of information in fast, and a person spots what matters. But more and more, the thing reading your company's data isn't a person. It's an agent, and agents don't want to read your data as pictures.
So Tristan thinks the asset moves down the stack. Agents will work from three places: the gold layer, which is well-modeled data in your lake or warehouse; a semantic layer that captures how your organization calculates its metrics; and a context store that helps the agent navigate everything it can see and make meaning of it. Dashboards don't disappear. They become the translation layer for when an agent notices something and you ask it to explain.
The semantic layer part stuck with me. He says the way you define your metrics "is not primarily a technical artifact, it's primarily a decision-making artifact. It's a reflection of how your organization wants to see itself."
That's not a data problem. That's an org problem, and anyone who has argued about what "available" means in an SLO will recognize it.
Then I asked the question I care about most. In SRE, we can tell you whether a request succeeded. An agent can get a successful response and still make the wrong decision. So what does reliability look like at the data layer?
His answer is the one I put in the hook. Agents are very good at writing SQL. "The problem today is not text to SQL." The SQL compiles and runs almost every time. The hard part is whether the agent understood the question you asked and the context of the data, and whether it got you the semantically correct answer.
He pointed to benchmarks like Spider 2.0, which try to model how messy real enterprise data is. In the results he's seen, an agent turned loose on raw enterprise data gets something like 30% of its answers semantically right. Give it context and a semantic layer, and it's north of 90%.
That's the whole game for his world now: take what people know, either in their heads or because they know where to look, and put it in the hands of agents so they come back with answers you can trust.
I pushed on governance. If an agent can discover data and act on it, where's the line between useful autonomy and access you'll regret? Tristan was honest that most data teams aren't wrestling with that yet. Data has mostly been how a business looks at itself, not part of how it operates. For thirty years, data people have been building "highly refined nervous systems" that collect, refine and present information for a human to act on. He thinks those systems are about to get hooked up to action, and agents are what will do it.
On wrong answers, he described a spectrum I've seen in ops too. Some organizations "run and gun": trust the answer by default, and "you certainly can move real damn fast that way." Enterprises usually keep a final human check. He compared it to the choice you make in a software factory today: who merges the pull request, the human or the agent?
The career part is where this conversation goes beyond data. Tristan said all of these knowledge-work fields are going to be systematized, analytics engineering, SRE and software engineering alike. You'll "either be operating inside the system or you'll be operating on the system." The more interesting work is on the system: designing the loops and the evals, and seeing the needs in the organization that nobody has written down yet.
One of dbt's values, which he wrote ten years ago, is that "we believe in moving up the stack." He called this "a big moving up the stack moment." I think he's right, and it isn't comfortable. A lot of us tied our identity to technical skills that took years to build.
We also talked about how dbt earned trust. It helped that dbt, like Terraform, isn't in the request-response path, and that there wasn't much else like it. But my favorite story was about one of the first reasonably sized companies to use it. They showed up in dbt's Slack and asked, "You guys don't have any tests. Is this a real thing? Or is this just some little side project?" dbt was about six weeks old. The team promised to build tests if the company would use it. That kind of pressure sharpens you.
When I asked what developer-first companies get wrong selling to enterprises, he answered for himself instead: "I think that I was too willing to say yes to take the revenue." Fortune 500 pull meant about two years of building enterprise features, like multi-cloud deployment and role-based access control, instead of moving the product vision forward. You have to build those things. You just want to be careful about when.
The open source line they settled on is a useful one: state and user interface. If a feature needs a database or a GUI, it goes in the cloud product. dbt itself still runs on top of files. When I brought up HashiCorp and IBM, he was quick to say this is pretty different: since the merger, they've only become more permissive with licenses, and the paid pieces are modules that plug into what you already run rather than a separate enterprise edition.
His first step for a platform or data team about to connect agents to enterprise data: "eat your spinach." All the good ideas from the last decade are still good ideas. A well-governed data lake, well-modeled data and a semantic layer. The genuinely new piece is a context layer. Big companies are building their own, the way they used to build their own orchestrators, and some are starting to buy them. Then evals, so accuracy goes up and not down as people change the stack.
He said a context layer is "the difference between a data agent that you want to come back to and ask more questions of and the one that you don't." And when it works, adoption looks nothing like a BI rollout. He's seen query volumes go up by 50x over a few months, because people finally ask the questions that used to cost too much friction.
The end of the conversation is the part I'd send to anyone running a platform team. Tristan has been building his own personal agent, and he's moved his own development from sitting at a terminal asking Claude Code to do things to a software factory, where he files tickets and comments on pull requests. He thinks "most software maintenance will end up needing to be turned into a software factory."
His team did that for dbt 2. About twenty people were working on conformance, making sure dbt 2 supports everything dbt 1 does, and progress had flattened. So they paused for about two months and built an agent loop that finds conformance defects, writes tests for them, and writes and validates the fixes. Conformance went from something like 45% in January to 93%, and he said, "I don't think we would have been able to ship V2 without it."
The pause is the part I'd underline. Tristan admitted it was stressful: he wanted the line to keep going up, and his team told him no, pause, and then it'll go up fast. That's operating on the system, and it worked.
You can find dbt at getdbt.com, Fivetran at fivetran.com, and Tristan's Analytics Engineering Roundup and Analytics Engineering Podcast in the show notes.
Notes
Show Notes
This is a guest conversation episode of Ship It Weekly, separate from the weekly news recaps.
In this Ship It Conversations episode, I talk with Tristan Handy about what changes when AI agents, not people, become the main consumers of your company's data, why the hard part isn't text-to-SQL, and why he thinks the most interesting work is operating on the system instead of inside it.
Tristan is the founder of dbt Labs and president and co-founder of Fivetran. If you've worked anywhere near a data team, you've probably run into dbt. His team likes to call it Terraform for data.
Correction: Tristan's title changed after we recorded. He is now the founder of dbt Labs and president and co-founder of Fivetran.
We start with his line that dashboards are a hack for the human visual cortex. As agents take over the pattern-finding, the asset moves down the stack: a well-modeled gold layer, a semantic layer that captures how your organization calculates its metrics, and a context store that helps the agent make sense of all of it.
On reliability, Tristan says agents already write SQL that compiles and runs almost every time. The hard part is whether they understood the question and the business behind it. In the benchmarks he's seen, an agent turned loose on raw enterprise data gets about 30% of its answers semantically right. Give it context and a semantic layer, and it's north of 90%.
We also get into where autonomy should stop once agents can act on data, why analytics engineers (and SREs) should be operating on the system rather than inside it, how dbt earned practitioners' trust, what he got wrong selling to enterprises, and where dbt draws its open source line after the Fivetran merger.
His first step for a platform or data team is to eat your spinach: a well-governed data lake, well-modeled data and a semantic layer, then a context layer and evals. And his team paused dbt 2 work for two months to build an agent loop that took conformance from about 45% to 93%.
Highlights
Why dashboards stop being the product when agents read the data
Gold layer, semantic layer and context store
Text-to-SQL vs semantic correctness: about 30% vs north of 90%
Where useful autonomy ends and regret begins
Operating on the system instead of inside it
How dbt earned trust, and what it got wrong selling to enterprises
State and UI as the open source line
Eat your spinach, then a context layer and evals
A software factory that got dbt 2 conformance from 45% to 93%
Founder of dbt Labs, President and Co-founder of Fivetran
Tristan Handy is the founder of dbt Labs and president and co-founder of Fivetran. dbt is the open source data transformation framework his team calls Terraform for data, and he still writes The Analytics Engineering Roundup and hosts The Analytics Engineering Podcast.
25 years in production: DevOps, SRE, platform, and cloud. Host of Kube Signals on KubeFM. Grafana Champion; Confluent Community Catalyst; IBM Influencer 2026; DevOps Institute & ITIL Ambassador.
The line I keep coming back to from my conversation with Tristan Handy is one he dropped two minutes in: dashboards are "a hack for the human visual cortex."
Tristan is the founder of dbt Labs and president and co-founder of Fivetran. If you've worked anywhere near a data team, you've run into dbt, and his team likes to call it Terraform for data. That framing made this one easy to follow from the SRE side, and it's a big part of why I wanted him on.
Dashboards exist because our brains are the ones looking for patterns. Pipe a lot of information in fast, and a person spots what matters. But more and more, the thing reading your company's data isn't a person. It's an agent, and agents don't want to read your data as pictures.
So Tristan thinks the asset moves down the stack. Agents will work from three places: the gold layer, which is well-modeled data in your lake or warehouse; a semantic layer that captures how your organization calculates its metrics; and a context store that helps the agent navigate everything it can see and make meaning of it. Dashboards don't disappear. They become the translation layer for when an agent notices something and you ask it to explain.
The semantic layer part stuck with me. He says the way you define your metrics "is not primarily a technical artifact, it's primarily a decision-making artifact. It's a reflection of how your organization wants to see itself."
That's not a data problem. That's an org problem, and anyone who has argued about what "available" means in an SLO will recognize it.
Then I asked the question I care about most. In SRE, we can tell you whether a request succeeded. An agent can get a successful response and still make the wrong decision. So what does reliability look like at the data layer?
His answer is the one I put in the hook. Agents are very good at writing SQL. "The problem today is not text to SQL." The SQL compiles and runs almost every time. The hard part is whether the agent understood the question you asked and the context of the data, and whether it got you the semantically correct answer.
He pointed to benchmarks like Spider 2.0, which try to model how messy real enterprise data is. In the results he's seen, an agent turned loose on raw enterprise data gets something like 30% of its answers semantically right. Give it context and a semantic layer, and it's north of 90%.
That's the whole game for his world now: take what people know, either in their heads or because they know where to look, and put it in the hands of agents so they come back with answers you can trust.
I pushed on governance. If an agent can discover data and act on it, where's the line between useful autonomy and access you'll regret? Tristan was honest that most data teams aren't wrestling with that yet. Data has mostly been how a business looks at itself, not part of how it operates. For thirty years, data people have been building "highly refined nervous systems" that collect, refine and present information for a human to act on. He thinks those systems are about to get hooked up to action, and agents are what will do it.
On wrong answers, he described a spectrum I've seen in ops too. Some organizations "run and gun": trust the answer by default, and "you certainly can move real damn fast that way." Enterprises usually keep a final human check. He compared it to the choice you make in a software factory today: who merges the pull request, the human or the agent?
The career part is where this conversation goes beyond data. Tristan said all of these knowledge-work fields are going to be systematized, analytics engineering, SRE and software engineering alike. You'll "either be operating inside the system or you'll be operating on the system." The more interesting work is on the system: designing the loops and the evals, and seeing the needs in the organization that nobody has written down yet.
One of dbt's values, which he wrote ten years ago, is that "we believe in moving up the stack." He called this "a big moving up the stack moment." I think he's right, and it isn't comfortable. A lot of us tied our identity to technical skills that took years to build.
We also talked about how dbt earned trust. It helped that dbt, like Terraform, isn't in the request-response path, and that there wasn't much else like it. But my favorite story was about one of the first reasonably sized companies to use it. They showed up in dbt's Slack and asked, "You guys don't have any tests. Is this a real thing? Or is this just some little side project?" dbt was about six weeks old. The team promised to build tests if the company would use it. That kind of pressure sharpens you.
When I asked what developer-first companies get wrong selling to enterprises, he answered for himself instead: "I think that I was too willing to say yes to take the revenue." Fortune 500 pull meant about two years of building enterprise features, like multi-cloud deployment and role-based access control, instead of moving the product vision forward. You have to build those things. You just want to be careful about when.
The open source line they settled on is a useful one: state and user interface. If a feature needs a database or a GUI, it goes in the cloud product. dbt itself still runs on top of files. When I brought up HashiCorp and IBM, he was quick to say this is pretty different: since the merger, they've only become more permissive with licenses, and the paid pieces are modules that plug into what you already run rather than a separate enterprise edition.
His first step for a platform or data team about to connect agents to enterprise data: "eat your spinach." All the good ideas from the last decade are still good ideas. A well-governed data lake, well-modeled data and a semantic layer. The genuinely new piece is a context layer. Big companies are building their own, the way they used to build their own orchestrators, and some are starting to buy them. Then evals, so accuracy goes up and not down as people change the stack.
He said a context layer is "the difference between a data agent that you want to come back to and ask more questions of and the one that you don't." And when it works, adoption looks nothing like a BI rollout. He's seen query volumes go up by 50x over a few months, because people finally ask the questions that used to cost too much friction.
The end of the conversation is the part I'd send to anyone running a platform team. Tristan has been building his own personal agent, and he's moved his own development from sitting at a terminal asking Claude Code to do things to a software factory, where he files tickets and comments on pull requests. He thinks "most software maintenance will end up needing to be turned into a software factory."
His team did that for dbt 2. About twenty people were working on conformance, making sure dbt 2 supports everything dbt 1 does, and progress had flattened. So they paused for about two months and built an agent loop that finds conformance defects, writes tests for them, and writes and validates the fixes. Conformance went from something like 45% in January to 93%, and he said, "I don't think we would have been able to ship V2 without it."
The pause is the part I'd underline. Tristan admitted it was stressful: he wanted the line to keep going up, and his team told him no, pause, and then it'll go up fast. That's operating on the system, and it worked.
You can find dbt at getdbt.com, Fivetran at fivetran.com, and Tristan's Analytics Engineering Roundup and Analytics Engineering Podcast in the show notes.