Ship It Conversations: Ted Julian of Flux on the AI Code Visibility Gap, the Code Review Bottleneck, Measuring What AI Changes, Who Owns AI-Generated Code in Production, Guardrails Before Loops, and Why Disruption Is a Career Opportunity
Try reloading the player or open this episode directly on YouTube.
In this episode, Ted Julian discusses the challenges of AI-generated code in production, focusing on trust and code review bottlenecks. He highlights that nearly half of organizations use AI code, yet many hesitate to ship it due to concerns over complexity and security.
Now Playing
Ship It Conversations: Ted Julian of Flux on the AI Code Visibility Gap, the Code Review Bottleneck, Measuring What AI Changes, Who Owns AI-Generated Code in Production, Guardrails Before Loops, and Why Disruption Is a Career Opportunity
Ship It Weekly
0:00Current chapter: Cold open39:53
Chapters
Jump to a section in this episode.
Speed & share
Transcript
If more than one person is responsible for feeding
the cat, the cat starves. My guest today used
that line for a question a lot of teams are dodging
right now. Who owns AI-generated code once it's
running in production? His company surveyed engineering
teams. Almost half already run AI-generated
code in production. And about a third write it
but won't ship it. He says, Code review is only
act two. The bigger bill comes later, when somebody
has to understand code that nobody really wrote.
And the leaders who come out ahead won't be the
no squad. They'll be the yes squad, with guardrails.
I'm Brian Teller from Teller's Tech, and this
is Ship It Weekly.
Welcome back to Ship It Weekly,
where I filter the noise and focus on what matters
when you're the one running infrastructure and
owning reliability. Most weeks, it's a quick
DevOps, SRE, platform, cloud, and security news
recap. In between those, I do conversation episodes
with people building and operating these systems.
Ship It Weekly turns one this Friday. To celebrate
through mid-December, you are getting two conversations
a week, Mondays and Wednesdays, with the news
every Friday. Then... we're back to our usual
rhythm. Today, I'm joined by Ted Julian, founder
and CEO of Flux, an engineering intelligence
platform. Ted has been building security and
infrastructure companies for more than 20 years,
including Resilient Systems, which IBM acquired,
Arbor Networks, and @stake. Earlier this year,
Flux put out a report on the gap between AI-generated
code getting written and teams actually trusting
what it changes. We talk about why so much AI
-generated code gets written but never shipped,
why code review is the first thing to break,
a simple way to measure what AI is actually doing
for your team, who should own AI-generated code
in production, and why he thinks this is the
best career opportunity engineers will ever get.
Alright, let's jump in.
Ted, thank you for joining me.
It's a pleasure. Thanks for the invitation.
So I've read the report. I'm interested. Give me your thesis.
AI-generated code is clearly being used now, but what are teams still not ready for?
I think the surprise to me was to see the the gap between the code
generation and the code actually in production.
Okay. And then what the survey users did tell
us, like, it's like every survey, you kind of
want to do it again, because you get answers
to questions, and then you want to know, oh,
wow, it'd be great to know more about that. But
anyway, they did give us some examples of like,
what was holding them back in terms of security
concerns and other things that I'm sure we'll
talk about. Okay, so high level, what are some
concerns that you've seen? For sure, it's There's
a lack of trust, especially in more enterprise
class, more regulated environments, about taking
this generated code that maybe was for a prototype
and it went super quick and very excited, legitimately
excited about that. But when you pause to think
about putting it in production, if it's like
a HIPAA environment or if it's a highly sensitive,
like every minute costs us some huge amount of
dollars environment around the reliability of
that code broadly defined and then they did enumerate
some specific concerns whether it was complexity
that might be spiraling out of control and could
lead to an outage or the introduction of a new
security vulnerability based on a library that
was put in you know a pull request to to sort
of generate this prototype that we're now going
to make real um or other concerns around regulated
data? And has this code really been scrutinized
sufficiently for how it's handling this data
that looked great in the prototype? But again,
we're a regulated healthcare company, and now
this is winding up in production. Yeah, for sure.
Your report says 44.7% have AI-generated code
in production, and 35% are writing it but not
shipping it. What does that tell you? This is
an example where I'd love to do the survey, to
like really double click on that, because I just
talked about some of the hesitations, but I sure
would love to know more about them. And there's
some things we didn't ask that I suspect honestly
could be behind that too. So for example, if
you think about the time in which the survey
was done, there were probably a lot of people
that were still in the early era of experimentation,
right? So it literally could have been like some
throwaway something, but you know what? It was
a chance for us to, Try Claude because we haven't
done that yet. And, you know, it was just a little
prototype. Maybe you turn into something, maybe
not. So honestly, like just things like that,
just tests, proof of concepts could easily explain
some percentage of that. But we do know based
on what they did tell us that it that is was
not all that was going on here. You know, there
were those other concerns that we alluded to
that were the things that were giving them pause.
And then also just highlighted a general visibility
need, I think it's very safe to say, around what's
happening in our teams as they're embracing these
tools that would be required to sort of close
that gap and increase that percentage. Yeah,
that's for sure. So do you think there's like
a code quality problem? Or do you think it's
more like operational model problem? There is
a code quality problem. I mean, we could use
something like the Amazon outage from it was
probably six months, eight months ago now. Right.
And that was kind of a shot heard around the
world because that was the one where there's
the memo that leaked that basically said you're
going to sign off on all PRs, which I don't know
how that's going to work in practice with the
team that big, but they're smarter than me. But
anyway, like we're joking about it, but an outage
at a place like Amazon is a really big deal.
And for somebody that is so focused on moving
fast like Amazon is to put that kind of countermeasure
in place goes to show you that this is not theoretical,
that it is happening, but it's early. I think
in some respects you could say that's one of
the early canaries in the coal mine, and it's
probably a function of leaning in hard and leaning
in early. But it's a cautionary tale for the
rest of us that the models are the worst they
will ever be. In other words, they're going to
get better. And so maybe this problem isn't quite
as bad for those that are coming along behind
it. But that the more you lean in and the more
that you are regulated, care about... five nines
and so on, then the more that the smart leaders
are going to be putting guardrails, looking to
get that visibility to avoid those kinds of things
happening to them. Yeah. For me, I think of the
review side of things. So if AI makes code faster
to produce, like what breaks first in the review
and release process, right? 100%. We see this
at all our customers and we're starting to build
kind of a journey map for organizations as they
go through their AI transformation. And without
a doubt, that's kind of act two. And it's like
any good play. Act one, you're introducing characters
and there's all sorts of promise and things start
to look really interesting. But act two, that's
when some things happen. There's a villain or
there's a tragedy or something like that because
it makes it a great story. It'd be boring otherwise.
But I'd say this is maybe the beginning of act
two where, yeah, code review becomes... a real
problem. And what I think is interesting about
that is it gets into the people dynamics, right?
Where this isn't just a technical bottleneck.
And so we're going to throw more computer storage
or something at it. And I think in this case,
it's also not as easy as just using AI to do
code review, which is probably a very, very good
idea. But if you use Amazon as an example, whether
or not that's the right conclusion, I think there
still has to be humans in the loop when we're
talking about really sensitive environments.
So it isn't a simple, it isn't just More tech,
different tech. And so now there's a different
way of... It's more important to look at the
team dynamics and measure those team dynamics
because progressive improvement is going to require
solving that problem around a people. And whether
that's more people or different people or swapping
them around or whatever, until you solve that
problem, that continuous improvement will be
gated on that. How do we measure the... efficiency
gains of AI? Is it bigger PRs, more frequent
PRs? I mean, that's what everyone likes to use
as metrics. There is a once in a career hack
available to you now, which is to just do, this
is what we see smart customers do. They sort
of start to measure, they find a way to measure
features versus maintenance versus bug fixes,
this type of thing. Flux does this, but there's
lots of ways you could do this. You could tag
it in Jira. You could tag things in GitHub. You
could use some other tool, some way or another.
Try to find a more programmatic way of getting
this measurement than... What maybe people have
done in the past, which often is for R&D tax
credits, and it's sort of at the end of the quarter,
and you look at the team, and the leader will
be like, yeah, looking at GitHub, I'd say 30
% were features, and so that's what we'll put
in the spreadsheet, and that determines R&D
tax credits. Something that is a little bit more
real-time to try to get that measurement. Anyway,
so you backfill, and you define it as before,
right? So as of, let's just say, the beginning
of this calendar year. We've backfilled our PRs
and we know that we believe that our features
as a percentage of all engineering was 10%. Now,
if you're even remotely like leaning into code gen
and maybe you're migrating from GitHub Copilot
to Claude or you're rolling it out across more
teams, I mean, that should grow quite a bit.
Because if you simply say that the number of
the amount of code that we generate, let's just
keep the number simple, goes up by 10x. Well,
then probably the features go up by 10x or something
like this. Right. And so now you've got this
before and you got this after, which not only
shows velocity improving, but it shows features
in particular growing by, let's say, 5x. Keep
it that simple. I can tell you based on our customers,
like that one customer that became a board chart,
like their Q1 board meeting. That was a chart
in the board meeting. And I know somebody in
the meeting and the board was thrilled. They're
like. We were pushing you guys so hard last year
to lean in the AI. We thought the team was kind
of stagnant. They went from 9% to 43% by the
end of last year. So they were like, this is
tremendous. So anyway, I don't care how you do
it. This is kind of a once-in-a-lifetime career
hack. We're not going to have the curve like
it is now, right? it's only going to flatten
a little over time. So you're kind of crazy not
to avail yourself if you are leaning in the code
gen to document this spike in feature delivery
to both give the team the credit that they're
due for driving this transformation because let's
face it, like it's a little scary and it's a
new tech. And so people have to feel comfortable
about that, what that means for their careers,
their jobs, that kind of thing. So you absolutely
want to ground this in data and to give them
the well-deserved pat on the back. But at the
same time, you'd be crazy not to leverage that
with senior management to show like, hey, we're
doing it. We're doing exactly what you asked
us to do. And look at these results. They're
tremendous. Okay, so shifting gears a little
bit. Let's say we figured out how our AI efficiency
gains are. We've nailed all that down. We've
nailed down the FinOps side of it. How do we
deal with the security side? Your report calls
out security dependency changes and performance
impact as some of the hardest week-to-week
changes to detect. Those are exactly the things
that become incidents, at least in what I've
been through. Why are those so easy to miss?
I mean, we'll start with the less scary example,
which is simply you got teams leaning really
hard into generated code because there's an enormous
amount of pressure and they want to do the right
thing. And so they're prototyping stuff and it's
going great. And it is so easy to just keep,
you know, tab, tab, tab, you know, or accept,
accept, accept. And if you're not careful, like
there's a new library and that new library has
a vulnerability. Thing works. It doesn't create
a functional issue. And that winds up into production.
I mean, this is absolutely a very real concern
without some guardrails. It's just human nature.
So I don't think it's sufficient to say, oh,
we're just going to have really experienced developers
and they're going to know better. I mean, of
course, you want to have that as part of the
best practice. But it's belt and suspenders time.
I mean, there have to be some other guardrails
on this process or else you're just asking for
trouble. Now that all assumes the traditional
dev team, you know, but operating in a new way.
But you and I know, like, that's not the new
reality. The new reality is everybody's coding,
right? Marketing coding, sales is coding, supports
coding. And it's actually great, you know, managed
properly. So then the last thing you as an engineering
team was be the no squad and like shutting all
this down. I mean, you want to be the yes squad,
but... You know, if you think it's easy for a
developer to miss that, like a marketing person
with all the best intentions in the world trying
to do the right thing, like forget it. So it's
you can't have one without the other, you know,
like you can't say we're going to lean really
hard into this organization wide. It's not just
the engineering team. We're encouraging everybody
to lean into this opportunity to maximize productivity
gain and to innovate and to have fun and all
those fabulous things while not acknowledging
that these are absolutely side effects that come
with that. You can't have the one without the
other. And so the smart leaders are the ones
that are going to be the, not the no squad, they're
going to be the yes squad, but they're going
to find ways to put those guardrails in to architect
processes and so forth that help protect the
organization. Yeah. Where do you think those
performance regressions hide? when changes are
coming faster? We talked about code review, where
I think we're early is in, it's basically dealing
with the tech debt and, and dealing with the
complexity. And I think it's only the early adopters
that are living this on a day-to-day basis.
I think this is a bill that's going to come due
for maybe, like, the rest of us over the next year,
18 months, depending on where you are in your
journey. Um, but this is inevitable, and I do think
it's, it's going to be, uh, very much an eye-opener,
like the code review problem or that bottleneck
will look like a small problem than when this
Code complexity, tech debt issue starts to rear
its head because that will be the first time
that we really have to figure the code out, if
I'm honest, right? And there's going to be no
choice. It's not going to be just click accept
that looks reasonably okay. Somebody's going
to have to understand that code in a detailed
fashion and nobody really wrote it. It was generated.
And God knows, like, it was outsourced and we
dropped them. We're on another thing. Yeah, Sally
was amazing, but she found a great opportunity.
She's not here anymore. So, Joe, good luck. Yeah,
and it's not documented anywhere. No, no, no.
And again, the tooling's all going to get better.
None of this is cast in stone, right? And so,
yeah, there will be AI-driven improvements in
these various areas that are probably going to
make it better. But I don't think at the end
of the day, it's not a smart career decision
to say that it'll be fine. I think, again, the
smart leaders are going to be the ones that are
putting some guardrails, thinking about process
that will avert these problems is the one side
of the coin. And I guess the other side of the
coin is proactively identifying the bottlenecks
and getting out ahead of them so that they continue
to go through the various phases of their journey
and they're unlocking improvements at every step
of the way. OK, we need to take this seriously,
especially as we're moving quicker. Right. It
matters even more. Right. Please correct me because
I'm coming. I bring a cyber background to this
and a cyber point of view. And so what I'm about
to say may be wrong, may be offensive. So I'd
encourage to be challenged. But this to me is
very much a security mindset. I mean, security
for effectively forever has had this dilemma
of we're the no squad. Right. You name it. Commercialization
of the Internet, mobile, cloud, whatever, every
step of the way, security has to walk this tightrope
between, you know, managing risk, but also you
cannot be the no squad. Like if you just are
saying no all the time, like they're going to
find somebody else who has a way of saying, totally.
That's all that stuff. Yes. Well put. Right.
And I just, I'm just not sure engineering's quite
been in this position before to the same level.
And maybe to me, this is, It's part of this unprecedented
territory we're in right now. I mean, engineering
is being upended completely for the first time
ever. It's never happened. It happened with sales,
like with Salesforce. It happened with marketing,
with HubSpot, where those processes digitized.
And that was really painful and shook things
up, but it also was an enormous opportunity.
And I think that's at a very, very high level.
That's kind of where we are with engineering
right now. This whole thing is being completely
upended. And yes, it's absolutely fabulous, but
it's also scary and risky. And it's like I said,
you can't have one without the other. Yeah. Okay,
so let's stay, I guess, on that thread a little
bit with security and the compliance legal side
of it or engineering leadership side. One thing
that stood out to me is that this is not just
an engineering concern anymore. Security, compliance,
CTO, CIO and legal all showing concern. What
changed? I mean, we've seen this movie before.
This is what always happens. Again, I kind of
just talked about the big transformations, right?
Like this happened with the commercialization
of the Internet, right? Like, just get the website
up. For God's sakes, you know, X, Y, Z competitors
are all doing it. We need a website. And everybody
in IT or in the security department is like,
oh, hold on a second. Like, you're taking a network
that has never been connected to anything outside.
You know, it's just been completely internal.
And now you're connecting it literally to the
entire world. Like, there's some things we need
to do to be smart about how we do this. Ditto
cloud, ditto mobile. So I think that's the same
thing is that the organizations have leaned so
hard into this AI transformation, but eventually
these other functions realize that it is their
job to force the organization to be smart about
these kinds of things. And so that's why you
kind of go through these waves of like this initial
elation, but then things cool a little bit as
all these other functions weigh in and figure
out how to do this in a responsible way. So that's
like we've seen this movie before. And this is
maybe, again, the early part of act two, where
these teams sort of go from latent concern to
tooling up and figuring out like, this is how
we believe the organization needs to tackle this,
what the procedures are, they discover maybe
some tooling to help. So who owns the behavior
of AI generated code and production then? Is
it the engineers? Is it the... CIO? Is it the
AI director? What level? I mean, I guess everybody
does to an extent, but like at a traditional company,
who owns that? I don't know that that's been discerned.
I can't say that, like, I don't think we can canonically
say, like, the universe has voted, 62% say.
But, like, it's who feeds the cat, right? If multiple
people feed the cat, the cat starves. So it's
got to be one person that's responsible for feeding
the cat. And I think it's got to be the head
of engineering and ultimately the CIO. I mean,
it's the code. Let me ask it a different way
then. What does governance look like without
crushing velocity? That is the $60 million question.
No, I mean, it always is. Yeah. And I don't think
there's any easy answer because I think every
organization is different. And it's a function
of how regulated they are and how sensitive their
industry, their business is, how important reliability
is, where they are in their own maturity for
how they handle these things. I guess what I
would say is this isn't the first rodeo for anybody.
And so you just got to have to look back to these
past transformations to figure out what your
map is for going forward. It's just it's in a
very different domain. And so the controls are
different and maybe even not completely understood.
But that it's not like if you've it's not like
most modern organizations of any size haven't
seen this before. They've seen it before. It's
just figuring out how to make that muscle work
in a new way, you know, where these new exercises
or something. I don't know. OK, let's switch
gears to safeguards. So the report says teams
are buying code quality analysts automated code
review training, which safeguards actually move
the needle and which ones can become checkbox
theater? What should block a release versus what
should create a tracked risk? I really do think,
no, I mean, I think that's an individual, I mean,
that determines like how regulated you are or
how important uptime is or what have you. And
again, I really think that we're still figuring
out like gross best practices. Right. Like I
think if we just use code review as an example,
what we see all the time was organizations are
they get really dynamic as they're leading in
the code gen. Right. People who were just front
end developers, back end developers, whatever,
sort of becoming full stack developers because
they kind of can. And so and code review needs
are, you know, skyrocketing and seniors are just
like jumping into the breach, which in a way
is amazing, right? That Sally sort of maybe done
this part time, but now there's just so much
need for it. And she loves doing it. And she's
super good at it, that that's where she's spending
the bulk of her time. And it's a fabulous thing.
She's doing great. She's by the way, by virtue
of all these code reviews, she's mentoring like
all these more juniors and making them better
at what they're doing. I honestly have goosebumps
just talking with you about this, Brian. This
dynamic is incredible to the development of an
engineering team. And people are certainly adding
in tech there too, like using CodeRabbit for
code reviews, things of this nature, any part
of the CI/CD pipeline. But per the Amazon comment
earlier, you're never going to take humans out
of the loop. I don't think it would be smart.
And I also just don't think it would even fly
regulatory wise, right? Like you need to have
a trusted independent party like reviewing this
stuff and a human should be involved at some
point in the process. But I guess what I was
getting at is I think that these people dynamics
are under discussed and critically important
to success. And so much of this is shadow work
that has never been in a ticket, really. I mean,
it's the kind of thing that it comes up because
you're doing something else related to a pull
request and the person just spends some time
on it and then the pull request gets done. But
what this transformation is unveiling is there's
examples of this that are not trivial, like code
review. And if you aren't looking at like developer
participation ratios around commits versus code
reviews or that kind of thing, then you're going
to be blind to a very important dynamic to the
success of the engineering team in maximizing
this transformation. But just as importantly,
like the team dynamics that were underneath that,
right? Like this opportunity for Sally to shine
in a way that like she didn't even see coming
and she's loving it. And the impact that that's
having on the rest of the team. I guess the reason
I get goosebumps talking about it is like, this
is what any manager just, it's what makes the
job awesome. Like truly awesome. It makes you
like never want to stop doing it. Like I'll take
that dopamine hit and the team dynamics like
all day, every single day. So anyway, that opportunity
is there, but you have to be tooled to look for
it and to find it. Yeah. It just makes me think
like, where will we be in five years even? You
know, just five years, which isn't a long time,
but is in the current climate. It's happening
everywhere. Like, in our prep, we were talking a
little bit about, so we can maybe save this for
later, but it's like, what do teams look like in
the future? And I was just thinking about how,
you know, we're going back to, like, MRDs and PRDs
in a way because that's what we've learned is
like the next unlock. Um, and here too, what does
that mean for the people you need to have on
the team and for what their skill sets are as
a leader what does it mean for How you're tracking,
you know, people's contributions and how are
you sort of measuring them? And based on that,
how are you fostering continuous improvement
and development of the team? And here again is
where it's like unprecedented territory that
we're in. Like these are not things that were
part of like just day in, day out blocking and
tackling engineering team conversations. They're
things you talk about once a quarter, once a
year, when you're thinking about your teams and
how they're developing. But that is not nearly
fast enough right now if you want to be the best
of the best. Yeah, I even see companies iterating
over their OKRs continually now, which is interesting.
Okay, so I guess let's talk about that. Regarding
the report, what did you glean as far as the
future of AI? development? Yeah, I mean, to me,
it felt like we're in the first inning, you know,
these are just like, honestly, such basic concerns.
That if you want to think about that future,
where we're just verbosely prescriptive, and
we're in like this continuous loops of development,
and so on and so forth, like that ain't possible.
Right? Unless we have done a lot of spade work
on all of these guardrails and controls, because
Here again, you can't have one without the other.
You can't say you love loops and what that does
for your velocity without understanding that
there's a dark side. It will accelerate just
as much the bad as it does the good, so you better
have some guardrails around that, or else you'll
find yourself worse off. That, I suppose, is
maybe the cautionary tale for engineering leaders,
is that there's an exacerbating effect here that...
If you lean really hard into this stuff without
thinking carefully about the guardrails or about
what the processes and controls are, it can spiral
out of control really quickly. There's going
to be some smoking craters here. The Amazon thing,
I think, will seem small. I've read the Medium
articles on the Ralph Wiggum loops. I've read
Claude's program where they had the internal
vending machine that went rogue. There's just
countless examples of people just open claw,
even just what people were doing with open claw
early on. There's just so many examples of, yes,
if you let AI just loose, there are going to
be some interesting consequences that you couldn't
have even fathomed really before letting that
go. So guardrails, obviously, just goes back
to that. So important. It seems like the boring
side of the conversation, but like... It's paramount
to everything that comes after that. Yeah. I
mean, it's an engineering process. So every time
you'd make something better, that's going to
create a new bottleneck and you're going to have
to figure out how to tackle that bottleneck.
I mean, just thinking back on this question,
I think it's safe to say that in each of the
discrete functions, there will emerge progressively
better. AI driven solutions for that thing, whatever
it is, code review, generating documentation,
finding and fixing bugs, you name it. And so
of course, you're going to want to be tracking
all of that to try to maximize your productivity.
But every time you do that, there's going to
be a change in process and that's going to require
considering what the controls are, right? Like
does this, now that we've improved the way that
we're finding and fixing bugs, does that Does
the guardrails we had or the human control that
we had, is that still appropriate? Does that
need to change? Does that need to be updated?
I think that's the nature of the work that'll
need to happen. And I think it does also suggest
that there's some sort of a control plane that's
going to have to sit above all of this. Because
I do think what this will bring to a head is
the stovepiped nature of the way a lot of these
things happen today, right? A lot of these things
like... in your environment you'd know far better
than me like it's probably people looking at
like network stuff and application level stuff
and cloud level stuff and you're trying to get
some signal from all that noise to like figure
out what's going on in terms of performance of
reliability outages all the rest but that's just
as true in all of these other domains And at
the end of the day, the engineering leader is
going to need to have some visibility across
all of this in order to be effective. If them
being effective is going to require them to look
discreetly at all these different tools or to
manually... Sorry, but this is really very funny.
I mean, right now, these are pretty static dashboards
is our best attempt at doing this, right? So
how the hell are you going to do that when we
move into loop world? It's just not going to work.
So you're going to need to have, you know, some
triggers or controls and this sort of management
plane that in real time is kind of looking at
what's happening that ultimately gives that engineering
leader like that insight, because it's probably
often going to be blended, you know, or that's
that would be the ideal scenario, right, where
security is a little tricky. So maybe we'll talk
about there's a there's some churn happening
in a portion of the code base and that's slowing
your velocity for that that team and that part
of the application oh and by the way we think
that's because there was um A new library with
a vulnerability and the team is figuring out
like how to solve for this while preserving the
functionality of the application or something
like this. That's my best way off the top of
my head I can think of to talk about. Like there's
multiple things that are involved in understanding
that problem. And if you're in loop world where
the code is just writing itself and you're trying
to like... after it's already into production,
try to figure this out with a weekly snapshot
or something like that. It's just not going to
fly. We're going to need agentic systems is what
I think I'm using a lot of words to say. And
those agentic systems are going to have to be
multidisciplinary in order to be effective and
to support this vision that we're talking about
of like kind of autonomous, what are they calling
it? Dark factories, you know, just cranking the
out. For sure. Okay. So wrapping up, your
conclusion says the code isn't the hard part
anymore. Visibility and controls are. What are
high performing teams doing differently? It's
really interesting. What we're seeing is a lot
of people coding their own stuff, actually, to
try to solve this problem. And to be honest,
I think that's a function of there's a lot of
legacy tooling that people have used in the past.
And it's all been based on like derivatives of
work, right? You're, you've got to Ticketing
system, and that's great. I'm not saying get
rid of that or anything like that, but you have
this ticketing system and people are using it
a certain way, and so things are being tagged
a certain way, and maybe you've got similar things
happening in GitHub, and then you're building
a dashboard maybe off of all of that to try to
get the visibility that you want. And the problem,
of course, is that there's no way that can keep
up as we move to loops and as we move to dark
factories. Or why would you? You're just going
to have the AI working as hard to keep the tickets
up to date as it is creating the code. That seems
silly. So there's got to be a better way. And
this does get to what we do, but forget whether
it's Flux or anything else. We have to move to
a code-based approach. And we can move to a
code-based approach because we have AI now,
and AI understands code. But we need a better
mousetrap, I guess is what I'm trying to say.
Whether it's Flux or anything else, we need a
better mousetrap that's better suited to the
pace of development that we've now... unleashed
in the organization, to provide that, that, to close
that gap, to close that visibility gap. And the
first step that we're seeing people do because
they can, because it's easy to try, is coding their
own thing. Um, because they're finding those, like,
it's like easier to do that than it is to retool
your legacy solution, to be honest, to get the
visibility that you desire in this new AI environment.
So I think we'll see where that goes. Like, is
that sufficient? Maybe. But this, I think, is the
the canonical question for, like, coding your own
anything. Um, it's easy to try it, and I'm not saying
it can't work, but like, that argument writ large
means you're writing all your own software for
everything, and that just on its face doesn't
make any sense. So people are thinking we'll have
to choose, like, where that's good enough and where,
you know what, like, this is important enough, there's
enough change, like, we don't need to own this,
like we're better off making this somebody else's
problem. So how can Flux help? What does Flux offer?
I mean, the whole reason for starting the company
was seeing that starting with code generation,
that there was a complete upending of engineering
and that to manage that process would require
a completely different approach than everything
that we've tried so far. And so that's why we
built Flux, is to do a code-first approach to
how you're understanding how your code is changing
and how your teams are changing as you go through
your AI transformation. In a nutshell, That's
what we're here to do is to help make it easy
for engineering leaders to prove the efficacy
of their AI transformation and to continuously
improve it going forward with a code-first approach
because it's more accurate. It's better scalable.
Oh, and by the way, it removes all the toil of
all those legacy systems. So you get the benefits
of this approach without having to put more burden
on your teams to change how you're doing ticketing
or change how things are happening in GitHub
or any of that. Very cool. Awesome. Any last
thoughts you'd like to leave our audience with,
Ted? I guess it's mainly like, what a fun time
this is. Yeah, no, I agree. Right on, you know,
I mean, the transformation that's happening in
engineering has never happened before and it
never will happen again. And so that's both scary,
but it's also what an incredible opportunity
for everybody that's in this field and their
careers, wherever you sit in the stack. When
I mentor... people, like in college, this, what
I have said forever is a tech disruption, that
is your best opportunity to advance your career,
right? Is to go deep on that, to become an expert
in that, because in the early days there are no
experts. And that is, that has always been the,
like the silver bullet for really accelerating
the growth in your career. And that's exactly
where we are with engineering. And so I try to
focus on the upside of that because I think there
is enormous upside there for people who do lean
in and who do become the experts. And so that's,
I guess, what I would take away is wherever you're
in the stack, step back and just realize this
ain't gonna happen again. There'll be other stuff.
It'll be interesting. It'll be fascinating. But
the first time's the first time. And that's what
we're in for engineering with code. Absolutely.
Well, where can people find more about Flux and
you? askflux.ai. We've completely revamped the
website. We have the research survey on there.
We have a great case study about effective AI
transformations for engineering leaders that
I think would be a fun read for anybody to get
sort of a case study. That's what it is for like
how you could do it or maybe see where you're
at in your journey and what you can learn from
it. Very cool. Ted, thank you so much for your
time. Thank you for coming on. Really appreciate
it. Thanks a ton, Brian. It was a pleasure. That
was my conversation with Ted Julian from Flux.
The biggest takeaway for me is that writing the
code isn't the hard part anymore. Seeing and
trusting what it changes is. When Ted's survey
asked what was holding teams back, the answers
were about trust. Complexity that spirals into
an outage. A new library with a vulnerability.
Regulated data nobody has really scrutinized.
Code review breaks first. Ted calls it act two.
and it's a people problem as much as a tooling
problem. AI review helps, but he doesn't think
you take humans out of the loop. And the bigger
bill comes later. When someone has to understand
code nobody really wrote. His once-in-a-career
hack is to start measuring features versus maintenance
versus bug fixes now, backfill it, and show the
before and after. One customer went from 9% features to 43%.
And it ended up as a chart in their board meeting.
On ownership, it's who feeds the cat. Ted says
it has to be the head of engineering and ultimately
the CIO. And for your career, tech disruption
is the best chance you'll get to move up. Because
in the early days, there are no experts. You
can find Ted and Flux at askflux.ai, along with
the full report and a case study on AI transformations
for engineering leaders through the links in
the show notes. Follow or subscribe to Ship It
Weekly wherever you listen, and find previous
episodes at shipitweekly.fm. Thanks for listening,
The line I can't shake from my conversation with Ted Julian is about a cat. I asked him who owns AI-generated code once it's running in production. The engineers? The CIO? Some new AI director? His answer: "it's who feeds the cat, right? If multiple people feed the cat, the cat starves."
Ted is the founder and CEO of Flux, an engineering intelligence platform. Quick note for this audience: not Flux CD, the GitOps tool. This Flux lives at askflux.ai. Ted has been building security and infrastructure companies for more than twenty years, including Resilient Systems, Arbor Networks and @stake, and that security background shows up all over this conversation.
Earlier this year, Flux put out a report called the AI Code Generation Reality Check. Two numbers jumped out at me: 44.7% of the organizations surveyed already have AI-generated code running in production, and 35% are writing it but not shipping it.
That second number is the interesting one. Ted was upfront that some of it is plain experimentation: a throwaway prototype somebody built because it was a chance to try Claude. But the survey also asked what was holding teams back, and the answers were about trust. Complexity that could spiral into an outage. A new library with a vulnerability that came along with the prototype. Regulated data nobody has really scrutinized. A prototype can look great right up until a healthcare company realizes it's about to land in production.
He also said something I think gets lost in these conversations: "the models are the worst they will ever be." They'll get better. But the more regulated you are, and the more you care about five nines, the more the smart leaders are putting guardrails and visibility in place now.
From the SRE side, my question was what breaks first in review and release when code gets cheap to produce. Ted didn't hesitate: code review. He describes an AI transformation as a play. Act one introduces the characters and everything looks promising. Act two is where the villain shows up. Code review is the beginning of act two.
And it isn't a bottleneck you can throw more compute at. Using AI for code review is "probably a very, very good idea," he said, but in sensitive environments he still wants humans in the loop. So it becomes a people problem: who reviews, how much, and whether the team is set up for it.
His best story was about Sally. Seniors are jumping into the review breach. Maybe Sally did reviews part time before. Now it's most of her week, she's great at it, she loves it, and through all those reviews she's mentoring every junior on the team. Ted said he had goosebumps talking about it. His point is that this is shadow work. It has never really been in a ticket, and if you aren't looking at things like commits versus code reviews per developer, you're blind to one of the most important dynamics on your team.
Then came the part I'd send to every engineering manager. I asked how you measure what AI is actually doing, since everyone reaches for bigger PRs and more frequent PRs. Ted called it a "once-in-a-career hack." Start measuring features versus maintenance versus bug fixes, programmatically, not the end-of-quarter guess people make for R&D tax credits. Tag it in Jira or GitHub, use Flux, use anything. Backfill it so you have a before. Then show the after.
At one customer, that turned into a chart in their Q1 board meeting. The board had pushed the team hard to lean into AI and thought they were stagnant. Feature work had gone from 9% to 43% by the end of the year. The curve will flatten over time, so document the spike now, give the team the credit they're due, and show senior management the results.
On security, Ted described something every one of us has seen. It's so easy to go tab, tab, tab, accept, accept, accept. A new library comes in with a vulnerability, the thing works, nothing breaks functionally, and it lands in production. "We'll just have experienced developers" isn't enough. It's belt-and-suspenders time.
And it isn't only developers anymore. Marketing is coding. Sales is coding. Support is coding. Ted thinks that's great, managed properly. The last thing an engineering team should be is "the no squad." You want to be the yes squad, with guardrails. If it's easy for a developer to miss a bad dependency, a marketer with the best intentions in the world doesn't stand a chance.
He came at this from security, and security has lived this tension through the commercialization of the internet, mobile and cloud: manage the risk, but if you just say no all the time, people go around you. I said on the show that these are the same guardrails we've always had, and AI just forces us to take them seriously because everything moves faster. Ted's take is that engineering itself is being upended for the first time. Sales went through it with Salesforce, marketing with HubSpot. Now it's our turn.
The bigger bill, he thinks, is tech debt. Code review will look like a small problem next to the day somebody has to understand generated code in detail, and "nobody really wrote it." Maybe it was outsourced and that vendor is gone. Maybe Sally left for a great opportunity, and now it's "Joe, good luck." He expects that bill to come due for most of us over the next year to eighteen months.
Back to the cat. Ted doesn't think anyone can say canonically who owns AI-generated code yet, but his own answer was clear: one owner, the head of engineering, and ultimately the CIO. Governance without crushing velocity is "the sixty million dollar question." Every organization is different, but this isn't anybody's first rodeo. We've done it in every big transformation; the controls are just in a new domain.
Looking ahead, he says we're in the first inning. Loops and continuous development sound great, but loops have a dark side, and they accelerate the bad just as much as the good. Static dashboards and weekly snapshots won't keep up once the code is writing itself. He expects a control plane above all of it, with agentic systems that can tell an engineering leader in real time that churn in one part of the code base is slowing a team down, and why.
When I asked what high-performing teams do differently, his answer was that a lot of them are coding their own visibility tools, because that's easier than retooling legacy systems built on tickets and tags. Fine as a first step, he said, but you're not going to write your own software for everything. At some point it's better to make it somebody else's problem. That's the case for Flux: a code-first approach, so you get the visibility without changing how your teams use Jira or GitHub.
What stuck with me most was his last thought. When he mentors college students, he tells them tech disruption is the best opportunity to advance your career, "because in the early days there are no experts." Go deep and become the expert. Engineering is being upended for the first time, and the first time only happens once.
You can find Ted and Flux at askflux.ai, and the full report and Flux's Cobalt case study are in the show notes.
Notes
Show Notes
This is a guest conversation episode of Ship It Weekly, separate from the weekly news recaps.
In this Ship It Conversations episode, I talk with Ted Julian about the gap between AI-generated code getting written and teams trusting it enough to ship it, why code review is the first thing to break, and who should own AI-generated code once it's running in production.
Ted is the founder and CEO of Flux (askflux.ai), an engineering intelligence platform, not Flux CD the GitOps tool. He has spent more than twenty years building security and infrastructure companies, including Resilient Systems, Arbor Networks and @stake.
We start with Flux's AI Code Generation Reality Check: 44.7% of the organizations surveyed already run AI-generated code in production, and 35% write it but don't ship it. What holds them back is trust: complexity that could spiral into an outage, a new library with a vulnerability, regulated data nobody has really scrutinized.
Ted calls code review act two of an AI transformation, and he says it's a people problem as much as a tooling problem. His once-in-a-career hack: measure features versus maintenance versus bug fixes now, backfill it, and show the before and after. At one customer, feature work went from 9% to 43%, and it became a chart in their board meeting.
We also get into why engineering has to be the yes squad, with guardrails, the bill that comes due when someone has to understand code nobody really wrote, who feeds the cat, what loops mean for visibility, and why tech disruption is the best career opportunity you'll get.
Highlights
Written but not shipped: 44.7% in production, 35% holding back
Code review breaks first, and it's a people problem
Measure features vs. maintenance vs. bug fixes, then backfill
From 9% to 43% feature work, as a board chart
The yes squad, with guardrails
The bill for code nobody really wrote
Who feeds the cat: one owner for AI-generated code
Ted Julian is the founder and CEO of Flux (askflux.ai), a code-first engineering intelligence platform that shows engineering leaders how AI is changing their code and their teams. He has spent more than twenty years building security and infrastructure companies, including Resilient Systems (acquired by IBM), Arbor Networks and @stake.
25 years in production: DevOps, SRE, platform, and cloud. Host of Kube Signals on KubeFM. Grafana Champion; Confluent Community Catalyst; IBM Influencer 2026; DevOps Institute & ITIL Ambassador.
The line I can't shake from my conversation with Ted Julian is about a cat. I asked him who owns AI-generated code once it's running in production. The engineers? The CIO? Some new AI director? His answer: "it's who feeds the cat, right? If multiple people feed the cat, the cat starves."
Ted is the founder and CEO of Flux, an engineering intelligence platform. Quick note for this audience: not Flux CD, the GitOps tool. This Flux lives at askflux.ai. Ted has been building security and infrastructure companies for more than twenty years, including Resilient Systems, Arbor Networks and @stake, and that security background shows up all over this conversation.
Earlier this year, Flux put out a report called the AI Code Generation Reality Check. Two numbers jumped out at me: 44.7% of the organizations surveyed already have AI-generated code running in production, and 35% are writing it but not shipping it.
That second number is the interesting one. Ted was upfront that some of it is plain experimentation: a throwaway prototype somebody built because it was a chance to try Claude. But the survey also asked what was holding teams back, and the answers were about trust. Complexity that could spiral into an outage. A new library with a vulnerability that came along with the prototype. Regulated data nobody has really scrutinized. A prototype can look great right up until a healthcare company realizes it's about to land in production.
He also said something I think gets lost in these conversations: "the models are the worst they will ever be." They'll get better. But the more regulated you are, and the more you care about five nines, the more the smart leaders are putting guardrails and visibility in place now.
From the SRE side, my question was what breaks first in review and release when code gets cheap to produce. Ted didn't hesitate: code review. He describes an AI transformation as a play. Act one introduces the characters and everything looks promising. Act two is where the villain shows up. Code review is the beginning of act two.
And it isn't a bottleneck you can throw more compute at. Using AI for code review is "probably a very, very good idea," he said, but in sensitive environments he still wants humans in the loop. So it becomes a people problem: who reviews, how much, and whether the team is set up for it.
His best story was about Sally. Seniors are jumping into the review breach. Maybe Sally did reviews part time before. Now it's most of her week, she's great at it, she loves it, and through all those reviews she's mentoring every junior on the team. Ted said he had goosebumps talking about it. His point is that this is shadow work. It has never really been in a ticket, and if you aren't looking at things like commits versus code reviews per developer, you're blind to one of the most important dynamics on your team.
Then came the part I'd send to every engineering manager. I asked how you measure what AI is actually doing, since everyone reaches for bigger PRs and more frequent PRs. Ted called it a "once-in-a-career hack." Start measuring features versus maintenance versus bug fixes, programmatically, not the end-of-quarter guess people make for R&D tax credits. Tag it in Jira or GitHub, use Flux, use anything. Backfill it so you have a before. Then show the after.
At one customer, that turned into a chart in their Q1 board meeting. The board had pushed the team hard to lean into AI and thought they were stagnant. Feature work had gone from 9% to 43% by the end of the year. The curve will flatten over time, so document the spike now, give the team the credit they're due, and show senior management the results.
On security, Ted described something every one of us has seen. It's so easy to go tab, tab, tab, accept, accept, accept. A new library comes in with a vulnerability, the thing works, nothing breaks functionally, and it lands in production. "We'll just have experienced developers" isn't enough. It's belt-and-suspenders time.
And it isn't only developers anymore. Marketing is coding. Sales is coding. Support is coding. Ted thinks that's great, managed properly. The last thing an engineering team should be is "the no squad." You want to be the yes squad, with guardrails. If it's easy for a developer to miss a bad dependency, a marketer with the best intentions in the world doesn't stand a chance.
He came at this from security, and security has lived this tension through the commercialization of the internet, mobile and cloud: manage the risk, but if you just say no all the time, people go around you. I said on the show that these are the same guardrails we've always had, and AI just forces us to take them seriously because everything moves faster. Ted's take is that engineering itself is being upended for the first time. Sales went through it with Salesforce, marketing with HubSpot. Now it's our turn.
The bigger bill, he thinks, is tech debt. Code review will look like a small problem next to the day somebody has to understand generated code in detail, and "nobody really wrote it." Maybe it was outsourced and that vendor is gone. Maybe Sally left for a great opportunity, and now it's "Joe, good luck." He expects that bill to come due for most of us over the next year to eighteen months.
Back to the cat. Ted doesn't think anyone can say canonically who owns AI-generated code yet, but his own answer was clear: one owner, the head of engineering, and ultimately the CIO. Governance without crushing velocity is "the sixty million dollar question." Every organization is different, but this isn't anybody's first rodeo. We've done it in every big transformation; the controls are just in a new domain.
Looking ahead, he says we're in the first inning. Loops and continuous development sound great, but loops have a dark side, and they accelerate the bad just as much as the good. Static dashboards and weekly snapshots won't keep up once the code is writing itself. He expects a control plane above all of it, with agentic systems that can tell an engineering leader in real time that churn in one part of the code base is slowing a team down, and why.
When I asked what high-performing teams do differently, his answer was that a lot of them are coding their own visibility tools, because that's easier than retooling legacy systems built on tickets and tags. Fine as a first step, he said, but you're not going to write your own software for everything. At some point it's better to make it somebody else's problem. That's the case for Flux: a code-first approach, so you get the visibility without changing how your teams use Jira or GitHub.
What stuck with me most was his last thought. When he mentors college students, he tells them tech disruption is the best opportunity to advance your career, "because in the early days there are no experts." Go deep and become the expert. Engineering is being upended for the first time, and the first time only happens once.
You can find Ted and Flux at askflux.ai, and the full report and Flux's Cobalt case study are in the show notes.