0:00
Deployment was supposed to get easier. We got
0:03
cloud platforms, containers, Kubernetes, Terraform,
0:07
GitOps, internal developer platforms, and now
0:11
AI agents that can generate code faster than
0:15
most teams can review it. And yet, for a lot
0:18
of teams, shipping software still feels weirdly
0:21
painful. Not because people do not know how to
0:25
deploy. Because deployment is where all the hidden
0:28
complexity shows up. The app stack, the runtime,
0:32
the registry, the secrets, the networking, the
0:36
database, the cloud provider, the rollback path,
0:40
the weird internal Jenkins job nobody wants to
0:43
touch. And for smaller teams, that can be brutal.
0:47
They do not always need every knob Kubernetes
0:51
gives them. they do not always need a giant platform
0:54
engineering program. Sometimes they just need
0:58
a clean way to build, deploy, get logs, roll
1:02
back, and move on. So maybe the future of deployment
1:06
is not more flexibility. Maybe it is fewer decisions,
1:10
better defaults, and picking the right opinions
1:14
to outsource. I'm Brian Teller from Teller's
1:17
Tech, and this is Ship It Weekly. Welcome back
1:38
to Ship It Weekly, where I filter the noise and
1:40
focus on what actually matters when you are the
1:43
one running infrastructure and owning reliability.
1:46
Most weeks, it's a quick news recap. In between
1:50
those, I do conversation episodes with people
1:53
who are building platforms, running infrastructure,
1:56
organizing events, and thinking through where
1:59
this industry is actually headed. Today is one
2:02
of those conversations. I'm joined by Evan Phoenix,
2:06
CEO of Miren. Evan previously worked on Terraform
2:10
Enterprise and Waypoint at HashiCorp. And he
2:13
also built Puma and Rubinius. We talk about why
2:16
deployment is still painful. What teams keep
2:19
getting wrong when they try to simplify it. And
2:22
why small teams may not need more knobs. They
2:26
may need better opinions. There is also a good
2:29
thread in here around Terraform. OpenTofu, Terragrunt,
2:33
Waypoint, platform abstractions, and what happens
2:37
when tools make users do too much homework before
2:40
they get value. We also get into AI because yes,
2:44
AI can generate infrastructure code, but when
2:47
that generated deployment setup breaks, who is
2:50
on call for it? And does anyone actually understand
2:54
why it was wired together that way? So if you
2:58
work in DevOps, platform engineering, infrastructure,
3:01
developer experience, or you have ever stared
3:05
at a deployment system and thought, why is this
3:08
still so annoying? This one should be worth your
3:11
time. All right, let's jump in. Today, I'm joined
3:21
by Evan Phoenix. He is the CEO of Miren, previously
3:24
worked at Terraform Enterprise and Waypoint at
3:27
HashiCorp. and he built Puma and Rubinius. We're
3:30
talking about why deployment is so painful and
3:33
what teams keep getting wrong when they try to
3:36
simplify it. Evan, thank you for joining me.
3:38
Oh, my pleasure. Glad to be here. So walk me
3:42
through your history in tech. It's kind of interesting.
3:45
I'm curious. How did you land where you are now?
3:48
Well, I'm old enough that I graduated from college
3:51
right after the bubble burst. The dot-com bubble
3:55
burst. And so a lot of my career is actually
3:58
informed by lack of opportunity post bubble bursting.
4:02
I did all kinds of crazy interviews trying to
4:05
just get any job. I almost wrote web Perl
4:08
for many interviews and stuff like
4:12
that. I couldn't get anything. And so I end up
4:14
working for like a tiny ISP in Seattle. And then
4:17
from there, I kind of just. was sort of scrapping
4:22
i was like you know i met a guy in a hallway
4:24
and he told me about this company he's working
4:27
at and so i went to lunch and the lunch turned
4:30
into a job interview and i worked at that company
4:33
for a number of years and so like i didn't have
4:35
any a moment where it was like oh there's this
4:37
big thing and so i kind of was jumping around
4:39
and the other out the other part of it was i
4:42
started doing open source work in the late 90s
4:46
at this point. And I did it all the way through
4:49
college and then into my first few jobs. My first
4:52
few jobs were, I was working at a three -person
4:54
ISP. So I wasn't programming. I was like, open
4:57
source is what I had for programming. And so
4:59
I did that for a long time. And then I actually,
5:04
the company I was working for in Seattle at the
5:06
time was spinning down. I had a Ruby project
5:09
and I actually got. hired by Engine Yard, for
5:13
people who remember Engine Yard from the Ruby
5:14
days, and to basically work on Rubinius, my Ruby
5:18
implementation, and then also help out with sysadmin
5:23
work. And I kept ignoring the sysadmin work and
5:26
just working on Rubinius. And so they were like,
5:28
I'll just do that. So I got to work on my open
5:29
source project for like five years. They just
5:32
basically paid me. It's like living the dream.
5:34
And so like that. sort of balance like working
5:38
on open source and then finding a job and finding
5:40
interesting things and working on open source
5:41
has been this sort of constant thing and then
5:44
it it carried me through many jobs it carried
5:46
me into uh a startup that i had before HashiCorp
5:50
and then HashiCorp is a great sort of like job
5:52
and open source thing that carried me all the
5:54
way through that and then my current venture
5:56
Miren is a similar sort of thing it's like job
5:58
and open source and all kinds of things and so
6:00
yeah it's been that's the that's the short long
6:04
version Very cool. So my first job was, to date
6:09
myself to a degree, was doing dial -up tech support.
6:12
Yeah. Me too. That was the ISP. Yeah. I did dial
6:17
-up tech support. I did DSL tech support. And
6:20
then we were putting wireless antennas on people's
6:23
houses around Seattle. For long range. Yeah.
6:26
It was sort of long range. But yeah, directional.
6:29
Yes, yes. But big antennas. That was actually
6:31
how I got the, how that guy in the hallway talked
6:33
to me was because I was carrying a giant antenna.
6:36
And he was like, what is going on here? And I
6:38
explained it to him. He's like, we should have
6:40
lunch. And then, yeah, one thing or another.
6:42
But yeah. That's awesome. So where does Miren
6:44
fit in? What is Miren and how does it fit into
6:46
the ecosystem? So, I mean, it's a software deployment
6:49
system. It comes out of me constantly feeling
6:53
like we are always struggling with deployment
6:55
systems, right? Like people, I have... People
6:59
have lots of great ideas and especially now with
7:01
coding agents, you know, getting an idea out
7:04
there and getting it like, oh, my gosh, I got
7:06
this idea and I want to put the code together.
7:07
And then you have to figure out, like, how do
7:09
I give this to somebody? How do I let somebody
7:11
else use this? It's always like it breaks you
7:13
completely out of your rhythm. And so the like
7:15
one of the original ideas was like, how do we
7:17
integrate deployment? actually into sort of the
7:20
software development life cycle right so that
7:22
it's like it's not this thing where like i spent
7:24
all this time and now like oh god now i got to
7:26
deal with deployment but it's like oh deployment
7:28
is part of that i go through a deploy i can get
7:30
the logs it feels like it's part of the process
7:32
and so i i've worked with Kubernetes i worked
7:35
with Nomad i worked with lot most of the AWS
7:38
products to do deployment and i felt like They're
7:41
always missing something. And I wanted to feel
7:44
like I had my own things to say about where that
7:47
fit in. And the first cut of it is a self -hosted
7:53
piece of software. So it's open source. You go
7:54
and you run it wherever you want to run it. And
7:56
so that makes it very accessible. That opens
7:58
the doorway to lots of different people who have
8:00
different needs. You want to data sovereignty.
8:03
You want, oh, I'm really cost conscious. I want
8:06
to run it in one particular way. And one of the
8:09
big things. to do it for us was like, hey, you've
8:13
got this open source. You can run the deployment
8:16
system on your laptop. You can just deploy to
8:17
your laptop. And then when you're going and you're
8:19
deploying to production, it's the same. It's
8:21
exactly the same. And so you've got all of these,
8:23
all of your rhythm about how do I do it and how
8:26
do I work is exactly the same. And so now you
8:28
become that much more comfortable with the system
8:30
and then you can sort of grow with it. That's
8:33
awesome. That's great to hear. Okay, so. Why
8:35
is deployment still such a mess after years of
8:39
platforms and abstractions? Why? What's going
8:42
on? Yeah, I mean, it really is the final boss
8:45
in some ways of software. I think that it has
8:49
to do with a few different things. I think that
8:51
if we all were working in sort of one stack.
8:55
one system, I think it would be a lot simpler.
8:58
I don't think we'd be as happy, but I think that
9:00
deployment would be simpler. And because then
9:03
there would be just fewer knobs. There'd be,
9:05
you know, like all the, you know, if Java, maybe,
9:08
maybe not. I mean, it's not like Java beans really
9:11
solved all of the problems of the, you know,
9:13
like the late 90s, early 2000s. But there are
9:17
so many different shapes that an application
9:19
comes in. And some of that are just within the
9:22
stack. One JavaScript app could be completely
9:26
different and have a completely different profile
9:28
than the next one. But then you multiply that
9:31
matrix by all the different stacks and all the
9:33
different things. And now you've got sort of
9:34
a giant 3D cube of different ways that an app
9:38
can behave. And I think that deployment systems
9:41
try to say, OK, I want to do I want to give the
9:46
power of all of these things all at once? And
9:48
if so, I have to let you configure every possible
9:52
thing on here so that I can have every possible
9:55
stack here, right? And sometimes they're like,
9:58
well, that's really too difficult. Most people
10:00
are in this one quadrant. Let's actually focus
10:02
on the kind of apps that look in this shape,
10:05
your HTTP apps, request times under a second,
10:08
whatever it might be. And great, let's sort of
10:10
focus on that and try to distill down the needs
10:15
and the function and that kind of thing and make
10:17
it better. I think that, I do actually think
10:21
it's better now than it used to be. I actually
10:24
think there was a time where it was. better than
10:27
it is now also but like it's just because of
10:31
the number of variables that come into it and
10:33
then what each platform each platform wants to
10:35
have a unique say on it right so they want to
10:37
say like oh well we've got this cool load balancer
10:40
and so you should lean on the load balancer and
10:41
now i'm like okay i gotta learn load balancer
10:43
and i gotta learn all this kind of stuff right
10:44
and so or uh you know AWS is like great for like
10:48
oh we've got these AWS specific services to like
10:51
make your app work in this one way of these different
10:54
shapes and so it's the the amount of configuration
10:56
the amount of of of functionality that you want
11:00
to sort of eek from the platform too you think
11:03
ai helps us bridge that gap better i mean in
11:07
some ways yes in some ways because it's really
11:10
good ai is really good at if you could if a platform
11:13
can explain itself The AI is really good at being
11:16
like, okay, great. The platform works like this.
11:18
I'm going to basically match the app to the platform,
11:21
right? So the platform can have a bunch of different
11:23
things it does well, things that it doesn't do
11:25
well. And the AI can say like, okay, I'm going
11:27
to make the app work. You wanted this platform.
11:29
Great. I'm going to plug you into it. The inverse
11:32
is also true, although I don't think we're seeing
11:35
it that much yet. The app exists as this thing,
11:39
and the AI is like, great, let me find the deployment
11:41
platform that fits this. I don't think we're
11:43
seeing that quite yet, maybe eventually. I do
11:46
think it's making it better. Okay, so you've
11:52
lived through multiple waves of this will fix
11:54
it. What did people misunderstand about Terraform
11:58
Enterprise early on? You know, we were constantly
12:01
trying to figure out the shape of it. There's
12:06
a whole Conway's Law situation with Terraform
12:09
Enterprise, right? So we were constantly trying
12:12
to help people figure out the shape of like,
12:13
what should their repos look like? So in other
12:17
words, like, should it be like there's a repo
12:19
called Infra and it has like a thousand modules
12:23
that are basically going and configuring all
12:25
these different things. And every time anybody
12:28
changes anything, we have to run a plan that
12:31
basically touches like, you know. 1 ,500 resources
12:34
or something like that, right? Do you want that?
12:37
Or do you want like, oh, well, I want my, like,
12:41
should it be tiered? So it's like, okay, the
12:43
load balancers are a module, a thing, there's
12:46
like infra -loadlb, and that's just the load
12:49
balancers. And so it was, and I don't know that
12:54
Terraform really ever figured this out. even
12:57
to this day, if I'm honest. But people were constantly,
13:01
that was the big question. We had white glove
13:04
service at HashiCorp at one point that was
13:06
like, how do I organize my Terraform modules,
13:10
right? In a way that's like, that I can apply
13:13
them quickly, that reduces the blast radius,
13:15
all of those sorts of things. And so I think
13:17
that was a really interesting thing that, and
13:21
that wasn't Terraform Enterprise specifically,
13:22
that was Terraform writ large. I think Terraform
13:24
Enterprise was a, The first version of Terraform
13:27
Enterprise was the one that I worked on where
13:29
it was for people who wanted the experience of
13:36
having a platform that ran Terraform for them,
13:39
but they needed to have it sort of inside their
13:42
silo, whatever that meant. And so we went through
13:45
a few iterations where basically it was just
13:47
like a – big machine like we scaled it vertically
13:50
inside the actual instance and then you would
13:53
just it would just run Terraform on on instance
13:55
and it was a like so many products when you see
14:01
how the sausage is made underneath you're like
14:03
this is all we're doing you know like it was
14:06
a little bit that in a good way because it basically
14:08
made it made the product easy to manage and and
14:10
simple and stuff like that like we were constantly
14:12
like stripping things out and taking things out
14:14
and like you know at one point the product was
14:17
dependent upon RabbitMQ and we're like, no, we
14:20
need to like tear, like slice this down. Like
14:23
that's too big of a complexity for a thing that's
14:25
just running on a single machine. And so. Is
14:29
that for like worker queues? Yeah, for worker
14:31
queues. Because that was, so at the time, this
14:33
is Terraform Enterprise is a product before
14:37
Terraform Cloud, which is a weird sequencing
14:41
because we had Atlas at the time, which was the
14:44
thing that predated HCP which was like our
14:47
all-in-one platform and uh Terraform Enterprise
14:50
existed as sort of a side thing to atlas and
14:53
then atlas eventually gets broken apart part
14:55
of it became Terraform Cloud we still have
14:57
uh Terraform Enterprise and so that rabbit mq
15:00
part came from the original vision of atlas yeah
15:04
believe it or not i was an early uh customer
15:07
to Terraform Enterprise yeah i we so probably
15:11
much like a lot of the customers we we had a
15:13
pci HIPAA requirement where we needed to have
15:16
state files managed locally. We needed to make
15:19
sure that everything was self -contained and
15:20
then eventually move to Terraform Cloud. Did
15:23
you use it in the... You weren't at desk.com,
15:28
were you? It was not, no. Was it a forms company
15:31
at the time? There's three shapes of Terraform
15:35
Enterprise. The first, the shape number one was
15:38
only ever seen by like two customers. Oh, okay.
15:40
One of which was desk.com when they got bought
15:42
by Salesforce. And then the second one was the
15:46
AWS only Terraform Enterprise where we basically
15:49
shared an AMI and a bunch of Terraform for you
15:51
to run it. That was shape number two. Okay. And
15:55
then shape number three was a rewrite that I
15:58
did, which made it basically Docker -based. And
16:01
then you can install it anywhere. I forget the
16:04
name of the application that it was built on.
16:06
Replicated. Replicated, yes. I remember going
16:10
through and dealing with weird Docker IP address
16:13
allocation issues. All the time, yes. So yeah,
16:17
I guess it was version three. But yeah, it was
16:19
a good product. We eventually moved to Terraform
16:22
Cloud. And then I think they've since, I don't
16:24
know what the company does now. It was a few
16:26
years ago. But it got me excited about IaC because
16:30
at the time, and this is going to cause a lot
16:33
of people to cringe, I was at a company where
16:36
we were using Ansible with no state. We were
16:39
just using Boto3 to manage infrastructure, some
16:44
CDK stuff. So going from that to Terraform was
16:48
a huge jump. More recently, the problem you were
16:52
describing earlier, just about how do you manage
16:55
state files, how do you manage environments,
16:57
how do you manage modules, I feel like Terragrunt
17:01
has really tried to answer that problem. I don't
17:05
know that they've answered it well, to be honest
17:07
with you, but I think they've tried at least
17:09
to answer some of that dry issue. Yeah. I mean,
17:12
they were around right when I was at HashiCorp
17:14
because the Terragrunt has been around for maybe
17:18
not 10 years, but quite a while at this point.
17:20
Yeah. And they had this idea of like, how do
17:22
you slice your Terraform and we'll basically
17:24
like put it in different shapes. And they were
17:28
very successful at basically saying, we're not
17:30
going to let you like Terraform in some ways,
17:32
let you do anything, let you put files wherever
17:35
and let you organize it. However, in Terragrunt
17:37
was like, what if I just put like a corral, like
17:39
I put like a little, like a little. guard around
17:41
this thing you can't do all this stuff it's just
17:44
like it's going to work like this and and that
17:46
is very powerful from a workflows perspective
17:49
because now people are they're having to comport
17:52
themselves maybe to this smaller feature set
17:55
but they're also not having to think as much
17:57
because they're basically being like no it works
17:59
like this like i'm supposed to supposed to put
18:00
it in these boxes we think about that a lot about
18:02
that at Miren because Part of it is what do you
18:06
want someone to have an opinion about? Like Terraform
18:08
made you have opinions about like file structure
18:11
and stuff like that. Right. Terragrunt is all
18:14
about like, OK, well, let's not like people have
18:16
an opinion about all of these things. Let's just
18:18
make those opinions for them and we'll just tell
18:20
them what they are. And then they're not having
18:22
to go around, go about, figure out, like, should
18:24
I have an opinion about this? Like a lot of people,
18:26
when they would start using Terraform, they're
18:28
like, but just can you just tell me what the
18:30
right pattern is? And they're like, well, like,
18:32
did you do that? or does it think this or what you
18:34
know whatever and so uh they would we would sort
18:38
of force them in some ways to have an opinion
18:40
about stuff that they didn't really want to have
18:41
an opinion about that's fair so where do you
18:45
think OpenTofu sits in that opinionated stack
18:50
against Terraform Terragrunt yeah at least whatever
18:54
other alternatives there are well i think open
18:56
tofu is the result of two specific things obviously
19:01
number one was a license change without a question
19:04
no no no question right but i think that the
19:07
second one is part of it which is that the backlog
19:11
of potential features for Terraform that we're
19:15
sitting in PRs or sitting in issues that the
19:18
Terraform team was like no we're not going to
19:19
do that no we're not going to do that and so
19:21
the the uh The license change catalyzed people
19:27
to be like, okay, well, I want a different thing.
19:29
And then it got runway because it was like, well,
19:32
look at all these PRs. Look at all these really
19:34
cool features that Terraform team never wanted
19:36
to do that will just do those things, right?
19:38
And so now it's like it's growing all these different,
19:40
growing in this sort of different direction because
19:43
of that, right? It might still exist even if
19:47
there wasn't all those features, but I don't
19:49
think it would have had the legs that it has
19:52
had. if it didn't have that second part, right?
19:55
And I think that what's interesting is that I
19:58
was at HashiCorp during the creation of OpenTofu.
20:01
So we were talking about it and we were trying
20:03
to figure out what it meant. And what was interesting
20:05
is that OpenTofu was taking on these features
20:07
that Terraform was rejecting. for good reason
20:12
right so there was a lot of interesting features
20:14
there's encryption encrypted state management
20:16
was one of the big things that OpenTofu took
20:18
on that Terraform was like don't do this do not
20:22
like the reason that they didn't do it for the
20:23
longest time was because like we we run the math
20:26
and if you do this what you end up doing is that
20:30
you create like this other big hairy availability
20:34
scenario that is now i want to do stuff and now
20:37
i've lost the keys to do stuff and now i can't
20:40
do anything i'm sort of like really dead in the
20:42
water and so they were like an advocate for like
20:44
do it a different way right they had all these
20:47
different things and so and again like maybe
20:50
that's an example of where Terraform was like
20:52
actually don't let people do everything they
20:53
were trying to sort of rein in the the sort of
20:55
field of opinion and and in some ways OpenTofu
20:57
is like even wider like now you can do all these
21:00
things and maybe some of them are good ideas
21:02
but at least people are trying them out so well
21:05
in secrets probably shouldn't be in state right
21:07
or anything too sensitive or if it is i mean
21:11
lock it down otherwise but but maybe that's the
21:14
answer like i'm sure they were pushing like Vault
21:16
and like using yeah that was really that was
21:19
that part of it was really interesting because
21:21
that was the sort of the evolution of how of
21:24
your state files as definitive reproducible data
21:30
right so like one of the reasons early on in
21:32
Terraform that Terraform would end up having
21:35
secrets inside of it was because they were like
21:37
look like the state file needs to be the thing
21:39
the the the the jewel that you can use to recreate
21:43
whatever you need right and so they they were
21:48
like you should put secrets in your state file
21:50
because you want to do this and the other thing
21:52
and then what they sort of realized over time
21:55
was that like that's a that's a hard position
21:58
to to have defense for when Terraform depends
22:02
on a jillion other APIs. And so it's like, okay,
22:05
well, if it already depends on all these other
22:07
APIs to work anyway, can't you just make secrets
22:10
an API that you have to depend on? And then that
22:12
became a thing that they sort of moved towards.
22:14
In my opinion, they didn't move towards it. quick
22:16
enough like in other words i i always felt like
22:18
hey if you're gonna have secrets just make make
22:21
a secrets part of Terraform Cloud and let secrets
22:23
live there your state files can basically just
22:25
like not have secrets in them no matter where
22:27
you are you can just sort of outsource the secret
22:29
management part to Terraform Cloud um that was
22:31
kind of the idea with variable sets though right
22:33
i mean yeah whether it's exactly right that's
22:36
exactly right yes that's exactly right but it
22:38
could have been at a deeper integration level
22:41
no right so yeah for sure So what did you learn
22:44
about what devs want versus what they think they
22:46
want when it comes to tools like Waypoint? Yeah,
22:50
well, Mitchell and I started Waypoint in 2019,
22:54
which was an interesting time to start a new
22:57
product because right in late 2019. And I think
23:01
that the thing that we were really focused on
23:04
was a control plane. for backend deployment systems.
23:09
So it would be like you would talk to Waypoint,
23:11
and Waypoint would go off and do things like,
23:13
oh, to do that, I need to build a container.
23:15
Let me go build a container. And then, OK, great,
23:18
I will do this, and I will do this other thing,
23:19
and I will do this third thing. And the idea
23:22
was it was just a pure workflow engine for deployment.
23:25
And it farmed out all of its backends to do different
23:29
things. And one of the things that I learned
23:32
working on that was You're asking a lot of someone
23:36
who wants to use this thing to say it doesn't
23:39
do anything until you do all of this homework
23:41
first. Right. It was a lot of homework to get
23:44
to get Waypoint running. And that kind of sucked.
23:48
Right. Like from just from a usability perspective.
23:51
But it also it wasn't great from like we're working
23:54
on a perspective. One of the things that we we
23:57
found out that was like a like a really big hurdle
24:01
was people. people would get to the point where
24:04
they're like, oh, like, oh, I just saw this on
24:06
a system and it can run, it can build Docker
24:07
containers. And that part was easy because we
24:10
would just say, also run Docker on the same machine
24:12
that you want to run Waypoint. Like that was
24:13
sort of, we could easily pave over that. But
24:15
then they'd be like, okay, now I want to deploy
24:17
this to ECS. And we'd be like, okay, well, you
24:19
got to go. You have to go configure, like, container
24:21
registry, and then you've got to push it to the
24:23
configure push to push the container registry,
24:26
and then you've got to tell the container registry
24:27
over here to pull. And it was like just that,
24:29
just getting over the, like, where does this
24:31
image go was a leap. It was a big hurdle for
24:36
people to get over. Sometimes it was too big.
24:39
Like, they wouldn't even get past that point.
24:40
And so we worked on it to try to. to pave over
24:45
it. One of the things that I did was I made it
24:47
so that if you were using, let's say, ECS as
24:50
an example, we would automatically configure
24:52
a repo inside AWS's OCI registry to push those
24:58
images to. So we would kind of just try to pave
25:00
over that as much as we could. But the learning
25:02
experience I took away from it was like, a pure
25:06
workflow engine, it's not enough product. It's
25:11
not enough unless there's this really complicated
25:14
thing that people are doing that you're like,
25:17
I want to come in and I want to just revolutionize
25:18
this one segment with a workflow engine. Great.
25:21
But deployment wasn't that. And what they really
25:25
needed was they needed answers. It didn't need
25:27
to be all the answers, but they needed a lot
25:29
of answers that were already set in stone at
25:31
the beginning. So when I was going to deploy
25:32
stuff, I was like, it just deployed. It just
25:34
did the thing. And part of that comes out of
25:37
the idea of like. again, from HashiCorp that
25:41
we learned was people wanted us to have opinions,
25:45
right? They really wanted HashiCorp's opinions.
25:47
They want to say like, well, okay, well, like
25:48
great, it's Terraform, it's Vault, it's Nomad,
25:50
but how do I set it up? Like, what's the best
25:52
way? All this kind of stuff. They'd constantly
25:53
be asking for our opinions, even though we had
25:55
these sort of generic tools. And so the takeaway
25:58
that I took from it was like, well, let's just
26:00
build a really opinionated tool, right? Let's
26:02
go in and say, hey, we've got opinions. We've
26:05
been working in the field for all this time.
26:07
We've got opinions about how this stuff should
26:09
work and what it should do and how it should
26:10
feel and what the primitive should be. And we
26:13
think that if you agree with those things, you
26:15
will really like our thing because it just does
26:17
all those things, right? It's sort of really
26:19
defining that in -group of people who want something
26:22
that looks and feels in a specific way. And so
26:25
we... When we were working on Miren and from
26:28
Waypoint, we took that opposite approach. We're
26:30
like, let's just build all the pieces. Let's
26:32
build the thing that builds stuff. That builds
26:33
the thing that deploys it. Let's not tell people
26:35
where stuff's being stored, where the images
26:37
are. Let's just make that not a problem at all
26:40
that people have to think about and worry about.
26:42
And that was a big takeaway from Waypoint, right?
26:46
Waypoint, it never really got traction and it
26:51
really got its legs underneath it. Partly, I
26:53
think, because it didn't have enough opinions.
26:56
It didn't have enough opinions about how someone
26:58
should actually use this thing. And then just
27:01
because of things that were happening inside
27:03
HashiCorp at the time, it ends up sort of
27:05
migrating over to HCP and then it migrates away
27:08
from deployment entirely. It migrates basically
27:10
to a Terraform front end. And it wasn't doing
27:13
deployment at all towards the end. It was sort
27:15
of misnamed in my opinion. But yeah, that was
27:20
the big takeaway from those Waypoint-as-deployment
27:24
system days. Very cool. So, OK, jumping back
27:28
to Miren. And I don't want to turn this into
27:32
a product pitch, but I am curious what what I
27:37
do want to understand is what you're trying to
27:39
fix and like what's broken for small teams trying
27:42
to ship. What tradeoffs are you making to keep
27:44
it simple? Yeah, I mean, I think that I'll start
27:48
on the first one, right? Like what is the what
27:50
makes Miren sort of interesting is that. And
27:53
where do people and small teams hit problems,
27:56
right? I think that as I've been looking at deployment
27:59
tools for years, you sort of have stuff that's
28:03
really good for a single developer. Like, oh,
28:05
I've got Docker on a machine somewhere, and I
28:07
just SSH stuff to it or whatever, right? And
28:11
then there's sort of a gulf. And then you have
28:14
like, OK, well, I want like a Kubernetes -based
28:16
thing that's sort of eaten most of the market,
28:19
right? There's not really much in that space
28:23
for like, actually, I'm a team of five people
28:26
that really just wants to like, that are application
28:28
focused. They're just like, I just want to ship
28:30
an application. I want to be able to deploy it
28:32
30 times a day if I need to, right? And there's
28:35
kind of not a lot there. There's a number of
28:39
products that try to take what Kubernetes is
28:42
and build facades and things inside Kubernetes
28:46
in order to get. back to sort of this middle
28:48
ground of like a thing that would feel good for
28:50
a small team. But what small teams always run
28:53
into is now you have not this much stuff, like
28:57
from Kubernetes to where they were, but like
28:59
this much stuff, my arm is way over there because
29:02
now it's like, it's not the, if something goes
29:05
wrong, it's not necessarily in this top layer.
29:07
It's basically somewhere deep inside the Kubernetes
29:09
stack of like your stateful pod had got evicted
29:12
because the memory pressure was too high, whatever
29:14
it is. Right. Yeah. And, we didn't want people
29:18
to have to worry about that because that team
29:19
of five people, they just want to deploy stuff.
29:21
They want to get in the groove of deploying.
29:23
They need something that really is getting out
29:25
of their way that just works the way they need
29:27
it to work. The question was, how do you build
29:31
something that isn't going to have a bunch of
29:33
landmines, right? Like a lot of the deep Kubernetes
29:36
stuff that's really important for big teams are
29:38
landmines for small teams. And the answer was
29:41
like, let's just strip it away. Let's just keep
29:43
stripping stuff away until we get to something
29:45
that feels like it's the right size for a small
29:48
team. And that meant like, yeah, it's not Kubernetes
29:51
-based at all. We were like, we're stripping
29:53
that away entirely. We're like, okay, what's
29:56
the right? what's the right primitive like containers
29:58
are still the right primitive again you feel
29:59
see where Docker is you see where people are
30:01
writ large containers are still that thing but
30:04
then it was like okay great let's say containers
30:07
as a primitive and let's just start building
30:08
on top of them and so there's things that miren
30:10
does that Kubernetes does thousands of times
30:12
better right that's no question right but i think
30:16
that A lot of the things that Miren does, you
30:18
could do with Kubernetes, but Miren does it out
30:20
of the box. And with Kubernetes, you have to
30:22
become like a pretty decent Kubernetes expert
30:24
to do. Like the simplest one that I can give
30:27
you is we do a scale to zero application deployment
30:32
by default. I have a really good reason for doing
30:34
that is because you're going to run this on a
30:35
machine, your own computer somewhere. And if
30:38
it's an app that only is used once every month,
30:41
don't run it. Don't run it. Just let
30:45
it run the one time a month that needs to run.
30:47
Like don't take up resources running it,
30:49
you know, for all 29 other days. And you can
30:52
do that with Kubernetes. There's a bunch of Knative
30:54
stuff. There's a bunch of pieces that you can
30:56
sort of layer on top of Kubernetes in order to
30:58
do that. But now you become an expert in a layer
31:01
on top of Kubernetes instead of. It's just a
31:04
thing that does the thing by default, right?
31:06
And so it becomes a question of those trade -offs,
31:09
right? Like we don't do as much as Kubernetes,
31:12
obviously, but the things that we do do are designed
31:15
specifically to do in that way. And we think
31:20
that those are the right trade -offs to make
31:21
for your teams of like one to 20 is really sort
31:27
of the sweet spot that we target. So what kind
31:29
of applications or workloads or problem statements
31:32
do you see customers bringing to Miren? What
31:36
do they come to Miren for that they wouldn't
31:38
go to Kubernetes for? Yeah, I mean, they come
31:40
to us a lot of times. The ambiguity around the
31:44
Heroku situation is one reason people come to
31:47
us. And they're like, I don't really want to
31:48
take on, I really want a system that just does
31:52
the system the right way. I don't want to become
31:54
an expert in how to layer a deployment system
31:57
on top of Kubernetes. That is like all all the
32:00
helm charts and all the things and keeping up
32:02
on all the versions and all that kind of stuff.
32:03
They're like, I don't want that. I don't have
32:05
the time or the team size to do that. But I want
32:08
a thing that is that just just works. Like what
32:12
we think about it, we think about ourselves as
32:14
being application focused and workflow focused.
32:16
So people come to us and they're like, hey, I
32:18
just want to deploy my stuff. We're like, great.
32:20
That's what we do. We deploy your stuff for you.
32:22
You know, we we support. specifically because
32:26
this is where 90 % of the workload is now. It's
32:28
like we're supporting HTTP applications, you
32:31
know, out of the box as the first class thing,
32:34
because that's where people are, right? We have
32:36
stuff for other, we sort of started to layer
32:38
on things for non -HTTP stuff, but like the bulk
32:41
of our focus has been on how do we run HTTP based
32:44
apps really well and quickly. And so. Where do
32:48
you think deployment and IaC tooling goes next?
32:51
Especially. with AI speeding up code changes?
32:54
Yeah, I mean, I think that people, one of the
32:56
things people ask me is like, is Miren in danger
32:59
of having an AI just code something to use AWS
33:03
that works the same way? And I think the answer
33:05
right now is no. But I think that even in the
33:08
longer term, it's probably also no. And here's
33:11
my take. You can have an AI go through and code
33:15
you up a deployment platform in the same that
33:18
feels similar to the way that Miren feels today
33:20
on top of AWS. It could, you know, those AWS
33:23
graphs, it's like all the different services
33:25
wired together. It could do that. It could wire
33:27
all those services together. Is it on call? Is
33:29
the AI on call when one of those things goes
33:31
wrong? What happens when the AI... is looking
33:35
at it again after a month of it working and is
33:38
now trying to figure out something is wrong and
33:40
it's looking at it again. It's like, this doesn't
33:42
make any sense. Who set this up? I'm actually
33:44
going to delete all these things and rebuild
33:46
it from scratch. You're like, oh my God, don't
33:48
do that, right? And so there's not the discipline
33:51
in order to do that. Now, again, I think people
33:54
will be like, well, maybe the AIs will get that
33:56
discipline. Yeah, I mean, then you're effectively
33:58
hiring an SRE to just build that up. And the
34:02
question is, is that a good use of your time
34:04
and or money? right to to have an SRE spend all
34:07
their time trying to balance AWS services you
34:11
know on one hand uh for some people maybe it
34:14
is uh i would i think our bet is that it's not
34:17
you know like it's just not it it doesn't make
34:20
sense in the the larger frame of things and so
34:23
The other point about AI and IaC is that one
34:28
of the things that we've seen is we've sort of
34:30
pointed AI at our tooling and said, hey, go deploy
34:32
this thing, is that our surface area is very
34:35
small, but very obvious for deployment. And so
34:38
the AI gets it right every single time, right?
34:41
Because it's kind of nothing to get wrong. It's
34:43
like, hey, I see that I can set config variables.
34:45
I see that I can do deployments. I see that I
34:46
can do rollbacks. I see that I can get status.
34:48
Great. So the surface area the AI has to interact
34:51
with is small enough that it just doesn't get
34:54
it wrong. Right. Because it's like there's not
34:55
a bunch of ambiguity about what is actually going
34:58
on. And then, you know, the AI knows that like
35:01
what doesn't know anything. But the API knows
35:04
that behind all of that is the machinery that
35:06
keeps all that stuff working in the shape that
35:08
the API is made to work from. And so, I mean,
35:12
I wonder like. what we're talking about here
35:14
is abstraction levels. I don't think abstraction
35:16
levels are going to disappear as AI gets better.
35:19
It's going to build new abstraction levels. And
35:22
then you're basically like, okay, great. That
35:23
abstraction level is done. We're good. We're
35:25
solid. Now I can go to the next one. And the
35:28
question is, do you, and this was the case for
35:32
Waypoint way back in the day. People build deployment
35:35
systems at their companies very commonly. And
35:39
the thing that we would always hear when we first
35:40
was working at Waypoint was that, I got halfway
35:44
done building this cool ass deployment system.
35:47
And then my boss told me, hey, knock it off.
35:50
Go back to the thing that actually we do as a
35:53
company. And so then they'd have this half built
35:55
deployment system that is just kind of a hunk
35:58
of junk because it never got finished. And then
36:00
they have to just suffer. with it for years.
36:03
And they're like, how come the deployment system
36:05
is so weird that you have to like stand on one
36:06
leg and shake a thing above your head in order
36:08
to make it work? And it's like, well, because
36:10
none of that was supposed to be there. That was
36:12
all supposed to be paved over. But we never got
36:14
enough time to finish it because we're a mortgage
36:17
writing company. Right. So like building a deployment
36:19
system never made sense. And so I think that
36:22
that's still true when people are. applying AI
36:26
to AWS resources. It's just like, you're, you
36:29
just needed a thing to work. You didn't need
36:31
any of those other pieces. We want to be, we
36:33
want to basically as Miren be a person who has
36:38
opinions. You want to basically buy our opinions.
36:40
You want, we want, you want to outsource your
36:42
opinions about how this is supposed to work to
36:44
us. And then we'll just give you a thing that
36:46
works. Oh, that makes sense. Yeah. I, I fell
36:49
into the same trap. I think with Jenkins, like
36:51
most people do. Oh yeah. Me too. You know, you
36:53
didn't have time to. to fix it. And it was a
36:55
house of cards. It just works. So you just left
36:57
it alone. Right. Don't touch it. Yeah. Yeah.
36:58
No better to, to yeah. Go to a company that can
37:02
give you the opinions around the stack. You pay
37:04
for that, pay for that. You pay for that service
37:06
that they're going to maintain. Yeah. I mean,
37:08
I have, I know people who, uh, who pay for AWS
37:11
consultants because, and then they're there,
37:13
that's the same thing. Right. And, and in some
37:15
ways, uh, you know, they're, you're paying for
37:17
that person's opinions. So. Okay, so wrapping
37:20
up, what's one industry belief about DevOps or
37:23
platform engineering that you think is just wrong?
37:26
I think that people fall into the trap of, I
37:32
guess it's not DevOps. I'll say what I was going
37:34
to say, which is, yeah, I was going to say that
37:37
I think that people fall into the trap of feeling
37:39
like, oh, I can't. i shouldn't rewrite that i
37:42
shouldn't do how about my own version of this
37:44
that and the other thing and i think that um
37:46
it's a double -edged sword of basically being
37:49
like oh don't ever always use this one version
37:52
of this thing um even though it does no most
37:56
of it doesn't do what we want but this one little
37:58
slice does right i think that people should be
38:00
more uh willing to say I'm going to do the thing
38:05
that works best for me, and I'm going to do 5
38:08
% of what the other thing does, but it's going
38:10
to be the 5 % that I want and now I can own.
38:12
It's sort of like the industry did a lot of hand
38:16
-wringing around dependencies about 10 years
38:18
ago, about how big your node_modules should be
38:21
and all that kind of stuff, right? I think we're
38:23
running into it again now with supply chain.
38:25
And so I think that having your own versions
38:28
of the things that fit just what you need, it's
38:32
way underrated. So yeah, no, that's fair. I agree
38:36
with that. Evan, where can people find more about
38:39
you? You have sOCIals? Yeah, yeah. You can find
38:42
me on Bluesky. I'm evanphx.dev on Bluesky.
38:47
We have a Miren Discord. If you want to hit
38:49
that up, that's at miren.dev/discord.
38:51
Yeah, that's probably the main two places he
38:53
says. Awesome. Appreciate it. Thank you so much,
38:55
Evan, for coming on. Really appreciate it. Absolutely.
38:57
Thanks so much. All right. That was my conversation
39:00
with Evan Phoenix from Miren. My biggest takeaway
39:03
from this one is that deployment is painful because
39:06
it sits at the intersection of everything else.
39:09
It is not just run the app. It is build the app,
39:13
package it, push it somewhere, wire up the config,
39:16
expose the service, watch the logs, handle rollbacks,
39:21
and make sure the next person can understand
39:24
what happened. And every time we try to simplify
39:27
that, we usually move the complexity somewhere
39:31
else. Sometimes that is fine. Good abstraction
39:34
hides the right things, but bad abstraction hides
39:37
the thing until production is broken. And then
39:41
suddenly, the team that didn't need to know Kubernetes
39:44
has to understand pod eviction, node pressure,
39:48
ingress behavior, registry auth, and whatever controller
39:53
is angry three layers down. I liked Evan's point
39:57
that small teams often do not need the biggest,
40:00
most flexible platform. They need something with
40:03
a smaller surface area that handles the boring
40:06
parts well. That does not mean Kubernetes is
40:09
bad. It does not mean Terraform is bad. It just
40:13
means flexibility has a cost. Every knob is a
40:17
decision. Every decision becomes something to
40:20
support. And eventually, the platform becomes its own pile of work. The AI
40:28
angle makes that even more interesting. AI can
40:31
generate deployment code. It can wire together
40:34
cloud services. It can probably build something
40:37
that works for a while. But is AI on call? Does
40:42
the team understand what it created? Does the
40:45
next person know why those decisions were made?
40:49
That is why good abstractions probably matter
40:53
more, not less. A small, clear deployment surface
40:57
is easier for humans to use and easier for AI
41:01
to interact with safely. So the practical question
41:05
is not always how do we build the most powerful
41:08
platform? Sometimes it is how little platform
41:12
can we get away with while still shipping safely?
41:15
I'll have links to Evan, Miren, the Discord,
41:19
and anything else we mentioned in the show notes.
41:22
If you enjoyed this conversation, follow or subscribe
41:25
to Ship It Weekly wherever you listen to podcasts.
41:28
You can also find the show notes and links over
41:31
on shipitweekly.fm. Thanks for listening, and
41:34
I'll see you later this week.
This conversation with Evan hit on something I think a lot of platform and DevOps teams feel, even if they do not always say it directly: deployment is still weirdly painful.
That sounds almost ridiculous at this point. We have had so many waves of tools that were supposed to make this easier. Cloud platforms. Containers. Kubernetes. Terraform. GitOps. Internal developer platforms. PaaS products. CI/CD systems. Now AI agents that can generate a bunch of code and infrastructure glue before you have even had enough coffee to regret the prompt.
And yet, for a lot of teams, the actual act of taking an application and safely getting it into production still feels harder than it should.
I liked Evan’s framing of deployment as the “final boss” of software delivery, because it really does sit at the intersection of everything else. Deployment is not just “run the app.” It is the app, the runtime, the container image, the registry, the network path, the secrets, the config, the database, the rollback, the logs, the thing that wakes somebody up, and the thing the next person has to understand when something breaks.
That is why deployment tools tend to get complicated so quickly.
Every app is shaped a little differently. One HTTP app is not the same as another HTTP app. A Ruby app is not the same as a Go service. A JavaScript app is not the same as a Java service. A background worker is not the same as a public API. Then you multiply that by every cloud provider, every orchestration model, every security requirement, every team structure, every compliance need, and suddenly the “simple” deployment platform has to make a lot of choices.
And that is really where the conversation got interesting to me.
A lot of engineers say they want flexibility. And honestly, we do. We like having knobs. We like knowing that we can tune something if we need to. We like not being boxed in.
But there is a cost to that.
Every knob is a decision. Every decision is something somebody has to understand. Every exception is something the team has to support. And eventually the platform that was supposed to make delivery easier becomes another product the team has to maintain.
That is not always bad. Sometimes you really do need the big flexible platform. Sometimes Kubernetes is exactly the right answer. Sometimes Terraform gives you the control and consistency you need. Sometimes the team has enough scale, enough complexity, or enough operational maturity that the extra surface area is worth it.
But that is not every team.
Evan kept coming back to small teams. Teams that are application-focused. Teams that just want to ship. Teams that do not have a full platform engineering group sitting around waiting to build and maintain abstractions on top of abstractions. Teams that maybe need deploys, logs, rollbacks, config, HTTP apps, and a path to production that does not require them to become experts in every layer underneath it.
That is the part I think a lot of platform conversations miss.
The goal is not always to build the most powerful platform. Sometimes the better question is: how little platform can we get away with while still shipping safely?
I also liked the Terraform Enterprise and Terragrunt thread because it gets at the same idea from a different angle.
The hard part of Terraform was never only the language. It was the shape around it. How do you organize repos? How do you manage state? How do you split modules? How do you reduce blast radius? How do you avoid one giant infra repo where every change feels like it is poking a thousand resources?
That is why tools like Terragrunt became useful for a lot of teams. Not because everyone loves another layer, but because people wanted an opinion. They wanted guardrails. They wanted someone to say, “put this here, structure it like this, and stop inventing a new pattern for every environment.”
That desire for opinionated tooling is not a weakness. It is usually a sign that the team has better things to do than debate folder structure forever.
The OpenTofu discussion was interesting too. The license change obviously mattered, but Evan’s point was that OpenTofu also had a backlog of features people had wanted for a while. Some of those features were things Terraform had intentionally avoided, and in some cases for valid operational reasons. Encrypted state was the example that stood out. It sounds good, and in many ways it is good, but it also creates another availability and recovery problem if the keys become the thing blocking your infrastructure operations.
That is a very real infrastructure tradeoff. Security, recoverability, simplicity, and operability are always pulling on each other.
The Waypoint section may have been my favorite part of the conversation.
I remember Waypoint being one of those tools that seemed like it should make a ton of sense. A deployment workflow engine from HashiCorp? That sounds like something a lot of teams would want.
But Evan’s takeaway was pretty blunt: a pure workflow engine was not enough product.
That stuck with me.
Because if a tool makes you do a ton of homework before it gives you value, you are going to lose people. If the user has to set up the registry, wire the backend, understand where images go, configure ECS or Kubernetes or whatever else, and then finally maybe get to deploy something, that is a lot to ask.
Especially if the person’s actual job is not “become an expert in deployment plumbing.” Their job is to ship the application.
That also maps directly to the internal platform problem. How many companies have a deployment system someone started building because the team needed something better, then the boss told them to stop working on it because, technically, the company is not in the deployment platform business?
So now everyone is stuck with a half-finished platform that mostly works, but only if you know the weird ritual.
That Jenkins comment in the episode was painfully real. I think a lot of us have had some version of that. The job works, nobody knows why, nobody wants to touch it, and the entire deployment process is held together by fear and one person’s ancient Groovy choices.
The AI angle makes all of this even more important.
AI can absolutely help with infrastructure and deployment. It can generate Terraform. It can wire together services. It can produce config. It can read documentation and maybe map an app to a platform faster than a person could.
But Evan’s question was the right one: is the AI on call?
When the generated infrastructure breaks, who owns it? Who understands why those services were wired together that way? Who knows which parts are safe to change? Who knows whether the AI made a reasonable long-term operational choice or just produced something that worked once?
That is the risk I keep seeing with AI-generated infrastructure. It can make the first draft easier, but the first draft is not the same thing as operational ownership.
And maybe that means good abstractions matter more, not less.
If the platform surface area is small and obvious, AI has fewer ways to get it wrong. Deploy. Roll back. Get logs. Set config. Check status. That is very different from telling an agent to wire together a dozen AWS services and hoping it invents a sane deployment platform your team can operate for the next three years.
So the real future might not be AI replacing deployment platforms.
It might be AI making the difference between good and bad platform abstractions even more obvious.
The practical takeaway for me is this: be careful which opinions you adopt, and be even more careful which opinions you accidentally create.
If you choose Kubernetes, you are adopting opinions. If you choose Terraform, you are adopting opinions. If you choose Terragrunt, OpenTofu, ECS, Heroku, Waypoint, Miren, or an internal deployment system, you are adopting opinions there too.
The question is whether those opinions match your team.
Do they match your application? Do they match your operational maturity? Do they match the amount of infrastructure ownership you actually want? Do they help your team ship safely, or do they just move the pain somewhere else?
That is why I liked this conversation. It was not really a “this tool versus that tool” conversation. It was more about the shape of deployment work, and why smaller, clearer, more opinionated systems may be the right answer for more teams than we usually admit.
Sometimes the best platform is not the most powerful one.
Sometimes it is the one that lets the team ship, understand what happened, roll back when needed, and then get back to building the thing customers actually care about.
Additional Links Mentioned
Miren:
https://miren.dev
Miren Discord:
https://miren.dev/discord
Miren on GitHub:
https://github.com/mirendev
Evan Phoenix:
https://evanphx.dev
Evan on Bluesky:
https://bsky.app/profile/evanphx.dev
Puma:
https://puma.io
Rubinius:
https://github.com/rubinius/rubinius
Terraform:
https://www.terraform.io
Terraform Enterprise:
https://developer.hashicorp.com/terraform/enterprise
Terraform Cloud:
https://developer.hashicorp.com/terraform/cloud-docs
Terragrunt:
https://terragrunt.gruntwork.io
OpenTofu:
https://opentofu.org
HashiCorp Waypoint:
https://github.com/hashicorp/waypoint
Kubernetes:
https://kubernetes.io
Docker:
https://www.docker.com
AWS ECS:
https://aws.amazon.com/ecs/
Heroku:
https://www.heroku.com
Knative:
https://knative.dev
HashiCorp Vault:
https://developer.hashicorp.com/vault