Dashboards are a hack for the human visual cortex. That's how my guest today puts it. They pipe a lot of information into our brains fast so we can spot the patterns. But more and more, the thing reading your company's data isn't a person. It's an AI agent. And agents are already great at writing SQL. It compiles and it runs. Almost every time.
The hard part is whether the agent understood the question and the business behind it. In the benchmarks he's seen, an agent turned loose on raw enterprise data gets about 30% of its answers semantically right. Give it context and a semantic layer, and it's north of 90. I'm Brian Teller from Teller's Tech, and this is Ship It Weekly.
Welcome back to Ship It Weekly where I filter the noise and focus on what matters when you are the one running infrastructure and owning reliability. Most weeks, it's a quick DevOps, SRE, platform, cloud, and security news recap. In between those, I do conversation episodes with people building and operating these systems. Ship It Weekly turns one this Friday.
To celebrate through mid-December, you are getting two conversations a week. Mondays and Wednesdays, with the news every Friday. Then, we're back to our usual rhythm. Today, I'm joined by Tristan Handy, co-founder of dbt Labs, and now president of Fivetran and dbt. If you've worked anywhere near a data team, you've probably run into dbt. His team likes to call it Terraform for data.
We talk about what changes when AI agents become the main consumers of your data, why the hard part isn't text to SQL, what a context layer is, and why you need one, and why he thinks the most interesting work is operating on the system instead of inside it. He also gets into what dbt got wrong selling to enterprises, and the agent loop his team built to ship dbt 2. All right, let's jump in.
Tristan, thank you for joining me. Thanks for having me. So starting off, I'm curious. We've spent years building data platforms around people looking at dashboards. What changes when an AI agent becomes the thing consuming that data? Oh, gosh. Yeah, we're just diving right in, aren't we? Maybe the first and most kind of obvious thing is that dashboards aren't the product anymore.
Dashboards are a hack for the human visual cortex. They are a way to pipe lots of information into our brains very quickly and allow us to notice patterns. And that makes a ton of sense when our brains are the ones that are looking for the patterns and applying the intelligence.
But it is becoming true that agents are replacing us as the primary form of intelligence that is looking at our data and trying to find patterns and the meaning of those patterns. And they don't really want to consume data via images. And so dashboards go away.
Dashboards will stay, I think, as a translation layer so that if agents notice something interesting, the next step will be, help explain to me what that thing is that you noticed. I think often a dashboard will be a component of that answer.
But so the asset shifts lower in the stack and agents will consume some combination of your gold layer, which is essentially like well-modeled data in tables in your data lake or data warehouse or whatever. There will be a semantic layer which will actually capture, here is how our organization calculates metrics.
We can talk about why that's important, but the organizational design of your metrics is not primarily a technical artifact. It's primarily a decision-making artifact. It's a reflection of how your organization wants to see itself. And then a context store, so that your agent has the ability to understand how to navigate all the data that it has available to it and make meaning out of it.
So I think that those are the primary three places that agents will interface with your data stack. And when you say gold, you're talking like a medallion setup? Okay. Just so our listeners that may not understand the gold, bronze, silver, yeah. Totally. I understand that the world of the data practitioner is a little bit different than the world of the SRE or DevOps practitioner.
So I'm happy to go as deep or not deep as relevant. But yeah, assume that all of your data comes into your data lake in a pretty raw form. And oftentimes, it actually takes many stacked... Functions, transformations in a DAG in order to actually get to a place where it represents some meaningful unit of business activity. And that's typically what you call your gold layer.
And for this data, are we looking at this differently for an agent than we would a person like a practitioner that's actually analyzing this data? Are agents looking at it the same way? Do we have to reason about it differently? Do we have to structure it differently? Yeah, okay, that's a great question. I, I think that the core way that you structure data can be very similar.
It's, you know, whenever it was the 90s or something when Ralph Kimball wrote the first book on, you know, how do you structure relational data for analytics. That approach... And there's several competing approaches, but whatever. The approach that you've taken to data modeling historically, I think, can persist pretty well.
From an agent's perspective, really, they don't really care that much about exactly how you structure your data as long as they can understand how you have structured it. The constraints on how you structure your... Data are much more imposed by the query engine. So everybody's really very focused these days on optimizing your spend on tokens.
In my world, for several years, the runaway cost was the compute on top of your data lake. A very blunt instrument and scans can be very expensive. So your data modeling is primarily done to make sure that you're not spending an arm and a leg in your data lake compute costs. Makes sense.
Okay, so in SRE, we can tell you whether a request succeeded, but an agent can get a successful response and still make the wrong decision. What does reliability look like at the data layer? Wait, is your question about reliability in the context of agents, or is it just data generally? On the agent side specifically, if you can answer that.
Yeah, because it's a somewhat hard question to answer when humans are in the loop too. That's fair. But there are benchmarks when it comes to evaluating agents and their accuracy for data -related tasks. I think that the one that comes up most often is called Spider. I think there's one, Spider 2. Benchmarks like this attempt to model the complexity of real world enterprise data, which has, it's very messy.
It has a tremendous number of different entities modeled in it. Join pathways are not clear, et cetera, et cetera. And you can then ask questions of this data set. And use an agent to go get you the answers to these questions. And you can experiment with how you ask the questions, what tools you give the agents, what kind of context you give them, and measure the difference in the results on the benchmark.
There are pretty stark differences in the performance you will see. If I could summarize the kind of general industry wisdom on these evals today, it is that agents are very good at writing SQL. They can write SQL that compiles and executes against your database with almost 100% accuracy. The problem today is not text to SQL. The challenge today is, did the agent... Fully understand the question that you are asking?
Do they understand the context of the data? And do they actually get you the semantically correct answer to your question? And that one is much, much harder. The most recent benchmarks that I've seen on this are that if you just turn an agent loose on enterprise data with the Spider 2 benchmark, you're at something like 30% accuracy for this semantic correctness.
But if you supply it with context and a semantic layer, then you will be able to get that accuracy north of 90%. So really, the world of data today is all about how do we construct systems that take the knowledge that humans either have in their brains or know how to get access to via whatever mechanisms and put that at the hands of agents so that they can come up with correct, trustworthy answers.
Yeah, no, it makes 100% sense. I guess kind of pulling on that thread, now talking about like the governance side of it, if an agent can discover data and act on it, where do we put the boundary between useful autonomy? And access that you'll end up regretting. Say more about that. Where's the regret coming from? Well, I'm more thinking the difference between reading data and taking action.
And that taking action step can obviously devolve into some crazy modifications of your application, your infrastructure. So where do we draw that line? I think that it is very interesting to hear you ask that. And the reason that I scratched my head at first is that I don't think that's generally a question that data practitioners are wrestling with yet.
I think it's one where they will absolutely have to, but the kind of organizational history of data is that it is used not as a part of the... Operational flow of the business, but as a part of the ability of the business to introspect itself. And so the result is always prepared for some human brain to look at and potentially take action on. But I completely believe that...
So what data people and data technologists have been doing for 30 years now is essentially building these highly refined nervous systems whereby data is collected and refined and eventually presented. And I think that those nervous systems are going to be quickly hooked up to action taking and agents are what's going to do that.
And so we will quickly have to wrestle with the question of like, okay, how do we assess the seriousness of a bad outcome here? And how do we put the right checks in place and all of that stuff? But that's largely not where the industry is at yet. What about even on the PII side of that data or incorrect assumptions from the data that the agent may be making? Is that a concern from the data pipeline side?
So I'm not a security expert, but what I would suggest is that there was, in the very early days of the... AI data conversation. There was a lot of concern with, well, can I let my model, the models have access to customers' PII? And I think that we've largely moved past that conversation because it's such an obviously important commercial thing.
And so at this point, from my perspective, letting AWS store customer's data in S3 versus letting them process it using bedrock. It's like kind of all the same. AWS has a mechanism for doing that. We already trust our cloud providers, et cetera.
So I don't think that's a big thing absent the fact that certainly it is possible to design agents in a, you know, we're all too aware of this now in the post-hugging face world. It's possible to design agents that can do inappropriate things with data. And so you just have to make sure you're staying really far away from anything like that. I think that was half of your question. What was the other half?
Oh, incorrect results of decisions. Okay, this is an interesting one because different organizations think about this differently. There are organizations today where, I don't know, I might call it run and gun. Where it's just like, this is the answer and I'm just going to by default trust it. And we generally have pretty high accuracy rates. So like, let's go.
And you know, maybe you'll get some decisions wrong in that modality, but I don't know. Maybe it's appropriate if velocity is the thing that you're optimizing for. Because you certainly can move real damn fast that way.
In an enterprise context, typically you move more slowly and that final check is done in the same way that when you build a software factory today, you have this decision to make who merges the pull requests. And some people say the human does and some people say that the agent does. And so you get to make that decision. Okay, so let's move on to analytics engineering.
So dbt helped make data transformation look more like software engineering. As agents write more SQL and consume more data, how does the analytics engineer's job change? My general guidance here is really not about analytics engineers or SREs or software engineers. It is all of these very technology knowledge work fields are going to be systematized in a way that they haven't been before.
And you as a professional will either be operating inside the system or you'll be operating on the system. And from my perspective, the more interesting work to do is to be operating on the system, to be designing the loops the evals, et cetera, as opposed to having tickets assigned to you. I think that this is true of all of the different types of jobs that I just mentioned and many more.
And so for analytics engineers, that really looks like not, how do I really cleverly model this data to solve this kind of thorny quantitative problem? It turns out that, you know, Astra 6 is going to be able to solve that problem as well as you can, if not better. And instead, you should be thinking about the system holistically and saying, what evals do I have? What evals do I need?
How do I see the things that are unseen, like the latent business needs in the organization that are not being reflected in the loop that we have today, etc. And that type of work is going to be important for a long time to come. Yeah, so it's moving up a layer, essentially. 100%. One of our values that I now authored 10 years ago is we believe in moving up the stack.
And I think this is a big moving up the stack moment. We've all been very identified with our own set of technical skills because we spent a long time building those technical skills. And it turns out that the technical skills are not going to be the differentiators for us moving forwards. Yeah, I think you're right.
When you were building dbt, what made practitioners trust it enough to put it into their production workflow? How did you make that happen? How did you build that trust with the community? That's a good question. I think that there were a couple different factors. One is that it is, dbt is a lot like Terraform. We've often thought about it as Terraform for data.
And Terraform is not really in the production request response flow. And so while it's certainly a critical piece of infrastructure, it doesn't have the requirements on it that something in the request response flow has. It had so much, it was very unique and did not have a ton of substitutes.
And so our early adopters felt like they were willing to do the work to look at the code, engage with the community and say, you know, like literally our first, you know, reasonably sized company that started using dbt showed up in our Slack community and said, you guys don't have any tests. Is this a real thing or is this just some side project?
And, you know, with dbt at that point, it existed for like all of six weeks or something. And yeah, we didn't, we hadn't built a test suite yet. And so we were like, Oh no, no, no, no. If, if you'll use it, we'll, we promise to build tests. So like that kind of pressure over time eventually sharpens you. That's fair.
Do you think that there's something that developer-first companies get wrong when they start selling to enterprises? Oh my God. That was a painful process. I'll answer a slightly different version of the question, which is, what, what do I think we got wrong? Um, I, I think that I was too willing to say yes to take the revenue.
And, um, I think that in part that is because I didn't have some grand vision of what our commercial product was going to become. I had a pretty clear vision of the role for our open source product, but the commercial product was really just, let's see what things people are willing to pay for. And in the early days, we experienced some real Fortune 500 pull.
We essentially, as we built up our engineering team, because, you know, back in 2020, you had to actually hire engineers to write code. And we built up our engineering team and we built enterprise features.
And for like, I don't know, two years, we weren't able to make meaningful forward progress on the product vision because we were just so focused on building enterprise crap, enabling multi-cloud deployment, role -based access control, all of these different things that you have to build, but you want to be real mindful of when you sequence the work in.
How did you decide what stayed open source versus became that commercial product? We ended up, after maybe two years, we kind of solidified an idea of of that dividing line. The answer effectively became state and user interface. And so if this was a feature that required a database, then it was better for a cloud product. I mean, our installed product, it operated purely on top of files.
It still does operate on top of files. There's no database persistence then if you wanted a GUI associated with it, and there ended up being a pretty meaningful group of users who, because of where data practitioners' careers kind of come from, the GUI was a useful bridge for them. Has the Fivetran combination changed how you think about practitioner choice at all? Like between open source and commercial? Yeah.
I think that the... The merger that we have is a real journey getting through this. It's a year -long process. We closed the merger about five months ago at this point. There is a lot that it changes about our world. It doesn't change what we are delivering to our open source community. In fact, it enables us to... Be more open because we have more options to monetize across a broader set of platform features.
But I think that we are now very focused on innovating in a way that looks more like Lego bricks, as opposed to historically, we thought about really like a... And it's very similar to a Confluent or a Mongo or whatever. You think about like, here's our community offering and here's our enterprise offering, and you get to make a choice. Which one of those do you want?
And the way that we are constructing the world now is that we have, as a combined company, we have multiple different open source offerings. Dbt is our biggest open source offering. And then there are modules that you can adopt a la carte. They don't ask you to move into a different version of the software. They plug into, uh, the, what you're already using.
And rather than adopting some kind of enterprise version, you say like, I want, I want dbt state or I want lake compute or whatever, and you can uh, on a consumption basis, pay for that specific thing. Okay, yeah, that makes sense. I was, in my mind, I'm thinking about it, going back to that Terraform example, HashiCorp, IBM, you know, acquisition and how that kind of changed licensing, and yeah. So, yeah.
Yeah, pretty different. Yeah, no, no, it's uh, we've, we've only become more permissive with licenses, uh, since, since the merger. And, um, we and Fivetran are super different than HashiCorp and IBM in that both of, both companies, uh, kind of grew up together and are directly complementary pieces of technology. And so they together form a very nicely packaged platform offering.
Like symbiotic relationship more than, yeah, gotcha. Okay, so if I run a platform or data team and people are asking me to connect agents to our enterprise data, what is the first thing you'd have me do? Oh my goodness. Well, the first thing is actually, it's like eat your spinach. All of the things that were good ideas in the past are still good ideas.
If you have a well-governed data lake and the data in it is well-modeled and you have a semantic layer, and these are all things that have been useful for a decade, then you're already a long way down the path. What you still need to do is you need to figure out an answer to the context layer, which is a legitimately new thing. Now, it's not rocket science.
I mean, these are the kinds of things that, I mean, like I remember 10 years ago, it was very trendy to build your own data tooling. Uber built their own orchestrator. Airbnb built their own orchestrator. And this is like a thing that people did. Airbnb built their own data visualization tool. Nobody builds that kind of stuff anymore because they're like, whatever, there's good enough off the shelf stuff.
But so like these same types of companies are currently building their own context layers. And so you can do that. There are starting to be context layers that you can buy. We have a context layer. It's very new. It's like, I don't know, a couple dozen companies using it. We use it internally. It's freaking awesome.
The difference between not having a context layer and having a context layer is the difference between a data agent that you want to come back to and ask more questions of and the one that you don't. And then you need to implement evals to make sure that as people on the data team implement changes throughout the entire stack, that the accuracy of answers goes up and not down.
And then you probably are thinking about real time because you're going to want to start having agents act on behalf of the intelligence that they get and not just serve it up to you as a human. And real time is important for action taking. So there's this whole set of things that you will do. But really, step one is. Make sure your house is in order from the best practices. And two is context layer.
And then you need to get adoption across your company. And this is very different than the adoption of a BI tool. People know what a BI tool is. Most people have lived their entire careers in a context where there's always been some type of way to get a dashboard for something. But this is a new type of modality.
And you don't just want somebody to go through a training and say like, okay, cool, I know how to make a dashboard.
What people who have deployed these things in production find is that there is a real depth of engagement that occurs with analytics agents where you can experience over the course of several months of adoption, you can experience Query volumes go up by 50x just because there were all these latent questions that people throughout your organization had that the friction was just too high to get them answered previously.
Yeah, now that makes sense. Okay, so wrapping up, we were talking before we started recording. I'm just curious, what have you been building lately with AI tools? Oh gosh. So I have for the past month ever since, personal agents have become the focus of things in the Valley. It's either personal agents like Muse or Jev, where everybody's talking about one of the two.
And I don't know if you are like me at all, but I really just, after two decades of consumer tech, I just really don't believe that... My personal incentives are aligned with any of the companies that are building personal agents today. And so I've been building my own personal agent. And it's actually not like, it's not rocket science.
It's a bunch of integrations and it's a bunch of, you can build really clear permissions into it, blah, blah, blah.
But the thing that we were talking about before hopping on is I recently decided to get a little more serious about the development workflow and have refactored everything from a, like, Claude Code, sit at my terminal and ask it to do stuff to a software factory where I file tickets and comment on pull requests and this kind of thing.
And that transition has been super, super fun for me because it's just much more effective. And I think that most people are still experiencing the world of agentic development in a Claude Code terminal. And I know that that only really started working well a year ago, a little less than a year ago. But I don't think that's the primary modality that we should be interacting with these things anymore.
So do you think building these software factories, do you think that this will have an impact on dbt open source? The way you run an organization, or are you thinking about this just as like small side project efficiencies? It's a good question. I think that most software maintenance will end up needing to be turned into a software factory. And so that absolutely will.
I mean, here's one of the ways that we implemented this internally. We've just gone through a big upgrade from dbt 1 to dbt 2. We do not increment major versions very often. That's been five years and coming. And because of the way that dbt's product surface area looks, there's a non-trivial migration effort dbt 2 has much tighter language constructs, and you have to validate users' codes more effectively.
We care a lot about conformance. How do you make sure that dbt 2 supports all the functionality of dbt 1? And we have a metric that we track, which is our conformance metric. And we just weren't making progress fast enough. We have a team of something like 20 people that were just focused on conformance all day, every day. And a year ago, they made some early wins and then it kind of flattened off a little bit.
And so we kind of took almost like a two-month pause and we implemented something called Conforman. And it was stressful because I'm like, hey, everybody, I want the line to keep going up. And they're like, no, no, no, we're going to pause and then it's going to go up fast. And Conforman is essentially a loop that identifies conformance defects. It writes tests for them.
It writes the code to fix them and validate them, etc. And we drove up conformance from, I can't remember, something like 45% in January to now 93%. And, you know, every successive percent gets harder. You have to fight for it harder. So it's been a dramatic, dramatic... Uh, I don't think we would have been able to ship V2 without it. That's awesome. Where can people find more out about dbt, Fivetran, and yourself?
Dbt is getdbt.com. Fivetran is fivetran.com. And I, these days, primarily, I've hidden from most forms of social media, but I still have a podcast at the Analytics Engineering Roundup and the Analytics Engineering Podcast. Cool. I'll link all of those in the show notes. Tristan, thank you so much for coming on. Really, really appreciate your time. Thanks, it's been fun.
That was my conversation with Tristan Handy from Fivetran and dbt. The biggest takeaway for me is that the hard part of pointing agents at your data isn't the SQL. Context and a semantic layer are the difference between about 30% right and north of 90. So his first step for a platform or data team is kind of boring. Eat your spinach.
A well-governed data lake, well-modeled data, and a semantic layer are still the foundation. Then add the piece that's genuinely new, a context layer, plus evals. So accuracy goes up and not down as people change the stack. I also liked his take on careers. As more of this work gets systematized, you are either working inside the system, or on it.
The interesting work is on the system, designing the loops and the evals, and spotting the needs nobody has written down yet. He calls that moving up the stack. And his team lives it. Their dbt 2 conformance work had stalled. So they paused for two months and built an agent loop that finds the gaps, writes the tests, and fixes the code. It went from something like 45% to 93%.
He doesn't think they could have shipped version 2 without it. You can find dbt at getdbt.com, Fivetran at fivetran.com, and Tristan's Analytics Engineering Roundup newsletter and podcast all linked in the show notes. Follow or subscribe to Ship It Weekly wherever you listen, and find previous episodes at shipitweekly.fm. Thanks for listening, and I'll see you later this week.
Scroll inside the box to read the full transcript, or expand for a larger view.