0:00
AI can write a lot of infrastructure code now.
0:03
But if every team organizes Terraform differently,
0:06
names resources differently, handles authentication
0:10
differently, and reinvents the same deployment
0:13
patterns, faster code generation does not necessarily
0:17
mean better infrastructure. That is where conventions
0:21
start to matter. Because the real leverage is
0:24
not just getting AI to write HCL faster. It is
0:28
giving it a framework, a golden path, and enough
0:32
context to know how your organization actually
0:35
works. I'm Brian Teller from Teller's Tech, and
0:38
this is Ship It Weekly.
0:56
Welcome back to Ship
0:57
It Weekly, where I filter the noise and focus
1:00
on what matters when you are the one running
1:03
infrastructure and owning reliability. Most weeks,
1:06
it's a quick DevOps, SRE, platform, cloud, and
1:11
security news recap. In between those, I do conversation
1:15
episodes with people building and operating these
1:18
systems. Today, I'm joined by Erik Osterman,
1:21
founder and CEO of Cloud Posse. Cloud Posse has
1:26
published more than 180 public Terraform modules.
1:31
And Erik also hosts their weekly DevOps office
1:35
hours. We talk about what makes a good Terraform
1:38
module, why infrastructure as code needs stronger
1:42
conventions and frameworks. And where tools like
1:46
Terragrunt, Crossplane, Backstage, and Cloud
1:49
Posse's Atmos fit. We also get into internal
1:53
developer platforms. Why self-service does not
1:57
have to mean another no-code abstraction. And
2:00
how AI changes the equation when your infrastructure
2:04
already has consistent patterns, agent skills,
2:08
and clear deployment workflows. All right, let's
2:12
jump in.
2:17
Erik, thank you for joining me. Awesome.
2:18
Thanks, Brian. Happy to be here. I love to talk
2:21
shop about all these topics. So let's just dive
2:23
right in. Yeah. So before we started recording,
2:26
we were talking a little bit about Cloud Posse
2:28
Terraform modules that you all have. And we were
2:31
talking a little bit about the industry changes
2:33
towards AI development and bug bounty programs
2:36
and that sort of thing. I'm curious to hear your
2:40
perspective on that. And if you've been affected
2:43
at all by... by the AI slop that has been hounding
2:47
open source initiatives. We've been following
2:50
it pretty closely. I mean, seeing what has happened
2:53
with curl and it went from hate to love, kind
2:56
of. There's been a lot more, I would say, security
3:00
researchers coming around and poking and prodding
3:05
at things with different levels of experience.
3:07
I would say the one bad run-in we had, and I'm
3:10
not going to name any names partially because
3:12
I can't remember. But there was some security
3:15
young startup out there that came out and tried
3:18
to say that one of our modules was insecure and
3:22
therefore used by hundreds of other modules that
3:25
were therefore also insecure because one of the
3:28
Go tests was using a package with a CVE. But
3:34
Terraform doesn't even run Go tests. So it was
3:38
just brain dead stupid. That was the worst run-in,
3:41
I would say. Other than that, I would say we
3:44
are pretty AI-friendly, accepting lots of pull
3:46
requests that have co-committers from Claude
3:49
and Cursor and other. And if anything, it just
3:52
makes things easier to review. Have you seen
3:55
an influx over the last year? It's hard to say
4:00
exactly. What we've seen is a steady stream of
4:04
pull requests still coming through to all of
4:06
our repositories over the past year. But I would
4:11
say it's fairly down from, let's say, the heydays
4:15
of 2021 when there wasn't any other really mature
4:20
alternative to using open source. We're not delusional
4:25
in the sense that companies these days aren't
4:28
using AI to write infrastructure as code, as
4:31
they should be, as we are using it to write our
4:33
own modules. I think that doesn't invalidate
4:36
open source. If anything, it just shows the resiliency
4:38
of open source that it takes on different value
4:41
at different phases. So when you're creating
4:45
these modules, how do you decide what becomes
4:47
a module versus what stays a pattern or example?
4:52
Yeah, it's a good question. And an interesting
4:55
thing about that is we, you know, so we started
5:00
the company in 2015. And from 2015 to 2021, I
5:06
would say we were adding like dozens of modules
5:09
every single month. But the rate at which you
5:13
can develop modules is faster than the rate of
5:16
new services being developed. So at this point,
5:21
you know, it's pretty stable. We aren't releasing,
5:24
you know, 50 new modules a year or anything like
5:28
that. So we have a pretty opinionated approach
5:31
to Terraform, take it or leave it. But the idea
5:35
is that all the modules should work well together.
5:37
And the first thing there is that there needs
5:41
to be a consistent naming convention, consistent
5:43
interface to how they work. So interoperability.
5:47
And then they shouldn't be so small that they
5:50
wrap a single resource because then it's just
5:52
unnecessary abstraction. Shouldn't be so large
5:56
that you have a hundred inputs because you can't
5:59
test that many permutations. So there's a happy
6:02
medium or happy balance between that and that's
6:06
what you'll see in a lot of our modules. We like
6:09
to think that they turn a concept that is pretty
6:12
relatable, like what is a VPC, into a declarative
6:17
module resource that you can provision easily.
6:20
But one of the opinionated things that we've
6:22
done differently at Cloud Posse, for example,
6:24
is keep the subnet architecture separate from
6:27
the VPC because there are lots of different strategies
6:30
for how you construct your subnets. Just one
6:33
example of how we've thought about it. One of
6:36
the biggest confusions that we get from our community
6:39
is what is the difference between a root module
6:40
and child module? A good interview question also.
6:45
So yeah, what is the difference? So in Terraform,
6:49
a child module never has state. And that's the
6:52
simplest way. So you have a root module. That
6:54
root module can invoke any number of child modules,
6:58
but you don't embed one root module inside of
7:01
another. So the root is kind of where you run
7:03
Terraform. But it's amazing how often these concepts
7:07
are conflated and not clear, even by practitioners
7:11
who've been using Terraform for a while. If they're
7:13
operating in a mature environment, they maybe
7:15
never started from scratch. Let's say you were
7:18
to start it all over again today. Is there any
7:20
different framing or philosophy that you'd have
7:23
around that? You know, I go back and forth. I
7:27
have a constant inner dialogue about some of
7:32
these things. And, you know, there's a love-hate
7:35
relationship with Terraform. And there's a
7:37
love-hate relationship with IaC. And some of these
7:40
things, I just wish they could be simpler. And
7:44
then when you go look at the alternatives, you
7:45
realize that it... It's hard. And here's what
7:48
I mean. I think what are the main contenders
7:53
today for infrastructure as code, IaC? So you
7:57
have CloudFormation. Honestly, my understanding
8:00
is CloudFormation dwarfs everything else in terms
8:03
of adoption within the side of AWS. Even though
8:06
in my echo chamber, nobody uses CloudFormation.
8:08
They wouldn't be caught dead with it. Then you
8:11
have Terraform, of course, and you have Pulumi.
8:15
Pulumi is interesting because they've had staying
8:18
power. They've survived. There have been a few
8:20
other upstarts that have come around. And we
8:22
also know like CDKTF, that failed as a project.
8:25
I think it's now spun up as a fork. And there
8:28
was another project that came up, Winglang, that
8:31
tried to create a whole language based around
8:33
being interoperable with cloud and local development
8:36
and so forth. Didn't work. So we have these three
8:38
staying powers. Then the FOMO I have is... Should
8:42
we be instead moving over to something more like
8:45
Pulumi and TypeScript or picking the language
8:47
du jour of our choice? And I keep coming back
8:51
to, I think it makes a lot of sense maybe inside
8:55
of a enterprise where they might have more standardization
9:00
on one or so languages. But even then, I don't
9:03
know if the value is there. Let me explain. So
9:06
the problem is with a formal IaC language like
9:09
Pulumi, through TypeScript is that you have unlimited
9:13
possibility and that creates so much variation
9:16
in how the infrastructure is provisioned and
9:20
managed and so forth. And sometimes constraints
9:23
are healthy. you converge on a few necessary
9:27
patterns. And I think that's what's led to HCL's
9:31
dominance in IaC. Because it's limiting, it's
9:35
led to greater interoperability because of the
9:38
module ecosystem. You see so many, there are
9:41
tens of thousands of modules out there, conservatively
9:44
speaking. And it's because of that that it works.
9:47
But you don't really find CloudFormation libraries
9:50
or architectures out there in the same capacity.
9:54
Or if they are, they were meant to be forked.
9:57
They weren't meant to be embedded and reused.
9:59
And I can't speak to Pulumi what the health of
10:03
the ecosystem is of supporting kind of architectures
10:06
around Pulumi. Maybe practitioners would say,
10:09
well, you don't need it. Yeah, I would say the
10:12
majority of use cases I've seen for CloudFormation
10:16
is for vendor-specific setup for IAM or a vendor
10:20
control that needs setup. And I mean, that is
10:24
one of the beautiful things about CloudFormation,
10:27
let's be honest, is that you can represent an
10:29
architecture in a single document, lob it over
10:31
to the API, and it materializes those resources
10:35
and reconciles it, which is nice. I've often
10:38
thought about this, like, what I would love to
10:40
see is an endpoint that speaks like the Kubernetes
10:45
protocol. But the fact that it's Kubernetes or
10:48
anything else, I don't even need to know that.
10:50
It's an implementation detail. I want something
10:53
where we can declaratively describe what our
10:57
architecture is in resources that is as convenient
11:00
as Kubernetes without... Any of the overhead.
11:04
I never want to have to think about updating
11:07
a controller or deploying a new version of an
11:09
operator, spending, you know, one week, a quarter,
11:12
updating 16 clusters a month, a year, basically,
11:16
just upgrading clusters. I don't want that. So
11:18
I want the convenience of CloudFormation with
11:21
the power of a Kubernetes manifest to manifest
11:26
infrastructure. So do you use Crossplane? Well,
11:30
no. So this is an interesting one. For a long
11:35
time, I had FOMO on Crossplane as well. And
11:39
speaking to its strengths, from my perspective,
11:42
where the FOMO is, I mean, it's pretty elegant
11:44
when you have. If you can assume that you have
11:47
a cluster that is managed by other people, so
11:51
other people's problems, OPP, and all I have
11:53
to worry about is I have a Crossplane endpoint
11:56
and then I can manage everything else that way,
11:58
then it sounds kind of nice. I still have concerns
12:03
about understanding what infrastructure is changing
12:06
and what I would manage under a Crossplane.
12:09
And let me explain. This gets down to the internal
12:12
developer platforms topic that we want to talk
12:14
more about. I think one thing that's so glossed
12:17
over is that you have basically layers of infrastructure.
12:21
Crossplane solves, and it's very important.
12:25
but a very small sliver from our perspective
12:28
of what it takes to operate at scale on AWS.
12:31
And what does at scale mean? It's thrown around.
12:33
Let's talk, you know, enterprises that operate
12:36
with hundreds of accounts or at a minimum dozens
12:39
of accounts. You have a complex network architecture.
12:42
You have an IAM architecture. You have a DNS architecture.
12:45
Managing TLS certificates is even just a challenge,
12:49
right? So at the point you have the privilege
12:52
of only worrying, about how you deploy these
12:55
cute little things on top of Crossplane. Yeah,
12:57
that sounds pretty good. How are you solving
12:59
everything else? And you're not doing that with
13:01
Crossplane. No, that's a fair point. Every implementation
13:04
of Crossplane I've seen typically has Terraform
13:08
as the setup of the actual cluster and everything
13:11
beyond that, like at the application level is
13:13
maybe Crossplane. But yeah, it's a chicken egg
13:16
problem. You still have to start somewhere. Yeah.
13:18
Yeah. But before you have the cluster, Well,
13:21
before you have Crossplane, you need the cluster.
13:23
Before you have the cluster, you need a network.
13:25
Before you have the network, you need the account
13:27
and you need the IAM. And then you need, well,
13:30
if you're having networks, you need a network
13:32
architecture. And in order to be able to do anything
13:34
with the cluster to access it, you need a DNS
13:37
architecture. And then how are you delegating
13:39
the zones and where is that zone going to reside?
13:42
And then it just, it spirals so quickly. And
13:46
it's so convenient when somebody else has to
13:48
worry about all of that. I've spent, it's pretty
13:51
nuts in this era of specialization, right? I've
13:54
spent the last 11 years of my life at Cloud Posse
13:59
focused on zero to one. Just how do you go from
14:03
having no AWS infrastructure to getting ready
14:07
to deploy all your stuff on top of Kubernetes?
14:10
And that has brought an interesting perspective
14:13
to things that you can spend a whole life, virtually
14:16
a career on just cold starts on AWS. Yeah. I
14:21
say that with humility. So you had mentioned
14:23
IDPs. Are you using IDPs? Do you find those useful?
14:27
And do you have a platform team that manages
14:29
that? Yeah. And what that means, I think, will
14:32
mean different things to different people. So
14:35
let's qualify this. We're not talking identity
14:38
platforms, but internal developer platforms here.
14:42
And there's been, you know, the OG on the block,
14:46
let's say, would be maybe Backstage. If we're
14:50
talking about internal developer platforms, there's
14:53
some commercial offerings. There's a company
14:55
that spun off doing Backstage service and others
14:58
as well. But this is an area, like if we're talking
15:02
zero to one, this happens after one. And so we've
15:07
been a little bit less focused on like helping
15:09
companies adopt something like a Backstage. But
15:12
here's what I think is often misunderstood.
15:17
It is easy enough to ClickOps up that Kubernetes
15:20
cluster and then, you know, deploy the Helm chart
15:23
and you are up and running. You got Backstage
15:25
and now let's get to business. But I mean, if
15:29
we're building something that's going to have
15:30
lasting value for the company, it is, I don't
15:35
know how to say it. It's like getting the, what
15:39
do you call it? The outside of a car without
15:41
any of the machinery, without the frame and starting
15:44
there and then saying, we will add in. the frame,
15:49
the motors, the wheels later, rather than starting
15:51
from the foundation and building up on top of
15:54
the pan and putting things on. So that's, yeah,
15:58
that's the most important thing, I think, is
16:00
you got to get the framework in place that enables
16:04
you to deploy software of which Backstage is
16:09
just one and an enabler for other things or other
16:12
capabilities that you choose to have. The teams
16:15
that I see that do that, though, Like effectively,
16:18
like they often have teams whose problem is Kubernetes
16:22
and that's what they're managing or their problem
16:25
is managing that IDP because it ends up being
16:29
a big investment. So where it works is in enterprise.
16:33
Yeah, Spotify also says like six to eight developers
16:36
full time. I mean, realistically, you're not,
16:40
yeah, you're not, you don't have the manpower
16:41
to run that. And yeah, we... I mean, we see this
16:46
all the time, right? Like, look, I love Kubernetes
16:49
as much as the next person who loves Kubernetes.
16:52
But I've always taken this standpoint. We never
16:54
recommend Kubernetes to a company that doesn't
16:57
first come to us saying they want Kubernetes
17:00
because you have to kind of opt into the pain
17:02
and understand the trade-offs. So when it comes
17:06
to that, why I bring that up is kind of like,
17:08
well, you got to know what you're signing up
17:11
for when you're trading. You're enabling a lot
17:14
of team members by committing to maintaining
17:16
this one centralized platform. And centralization,
17:19
well, we've learned what that does. So are a
17:23
lot of your customers, do they have like a monolithic
17:26
application? Or do they already have that microservice
17:30
architecture set up? Or is it a split? It is.
17:36
We've seen everything. We've seen everything
17:38
from people running a large monolith on a Kubernetes
17:42
cluster. It's like, and why? To, you know, overly
17:46
architected microservices architectures, which
17:49
is kind of like just asking for pain because
17:52
they're not at the scale where that makes sense.
17:55
And I remember what I was getting to is like,
17:57
we take a lot of our guidance. I mean, us as
18:00
an industry, as practitioners, we look at the
18:03
giants and what they do, right? And that's where.
18:06
Kubernetes came from and that's where network
18:09
defined architectures and that's where Backstage
18:12
and all these things came from. And they made
18:14
a lot of sense when you are at that scale, but
18:17
they don't necessarily make sense at the starting
18:19
point. So we see, I would say the biggest pattern
18:24
that I've seen lately is teams adopting a
18:29
poly-mono repo approach where they having a strict
18:34
poly repo approach where you have one service
18:36
per repo has has its challenges when you start
18:40
having hundreds of repositories and managing
18:43
that at any scale in organization to the extreme
18:47
opposite, the Google-style monorepo where you
18:50
have a thousand services inside of one repo and
18:53
the cognitive load that comes along with that
18:55
I can acknowledge the beautiful things that can
18:58
also happen with a monorepo. But a lot of teams
19:01
also don't have the sophistication to operate
19:03
things like Bazel the way that Google does. Therefore,
19:07
where I'm seeing is a really nice hybrid model
19:10
where you have a number of repositories in an
19:13
organization with a collection of services that
19:16
are loosely related to each other and don't have
19:19
tight couplings across. Because you never want
19:21
to have two services tightly coupled across two
19:24
different repositories or you're in deployment
19:26
hell. Good point. Okay, let's shift gears. So
19:30
one of the things that you had mentioned to me,
19:32
I think, before we scheduled this was you said
19:34
that the majority of work is now generated with
19:36
AI. And it works because you have strong IaC
19:40
frameworks, conventions, and golden paths. What
19:43
changes made it real? Yeah, this is a sore point
19:52
for me. I spent a lot of time thinking about
19:54
it, a lot of time lamenting about it. How teams
20:02
manage infrastructure as code today? It's not
20:06
new. I mean, I've been at this for 11 years and
20:08
even then it wasn't new. 11 years is a long time
20:13
for a language to exist. And let's look at every
20:15
other mainstream language today being used for
20:19
software development that is being vibe-coded
20:22
at what I like to say in vibe years, like moving
20:25
really fast. And they have frameworks. Nobody
20:29
starts a modern project in JavaScript unless
20:33
for like some weird fundamental like philosophical
20:37
reasons, and that project probably isn't going
20:41
to see the light of day. They start with React,
20:44
they start with Next.js, they start with Rails
20:47
or Django, or, you know, the list just goes on and
20:51
on and on, because frameworks give a convention
20:54
to the team to operate. And that includes
20:57
documentation. It includes nomenclature, how you
21:01
refer to things, concepts, strategies, best practices,
21:06
design patterns, and the list goes on and on
21:09
and on. Yet our community within IaC has a purity
21:14
test when it comes to Terraform that we don't
21:19
use wrappers or no, we just got to write vanilla
21:22
Terraform. But it's a lie to themselves and it's
21:25
a lie to the organization because nobody ever
21:27
does. When you only think of Terraform, it doesn't
21:32
actually do anything. It provisions a resource,
21:35
but it doesn't handle how you are going to promote
21:37
that resource, how you're going to reconcile
21:40
it that it was actually deployed, how you are
21:42
going to scale it up across multiple team members,
21:45
how you're going to lay things out on the file
21:47
system, how you're going to manage the state,
21:49
are you going to have lots of different state
21:51
files, how you're going to... And then once you
21:54
realize that you're going to need to solve some
21:56
of these problems, and if you've already committed
21:58
to a path, well, you got to go back to basics
22:01
and re-engineer it. So what we have is all these
22:04
teams working concurrently on the table stakes
22:08
of what it is to do infrastructure and reinventing
22:10
the wheel over and over and over and over and
22:12
over and over and over again, and just writing
22:14
HCL because you don't get anywhere. So where
22:16
I'm going with this is we as a community need
22:20
to gravitate towards frameworks, just like
22:23
you have in real languages. And to say that Terraform
22:27
or HCL is just a configuration language is, I
22:30
think, insincere these days. You have loops and
22:33
iterators and modules and call modules and interpolations,
22:37
and it gets complicated real quick. So we need
22:40
conventions and we need frameworks. And this
22:43
is where we've kind of... gravitated towards
22:46
at Cloud Posse, we started back in 2019-ish working
22:49
on a project called Atmos. Atmos is... an IaC
22:53
framework. It's based on open source. It's what
22:55
we invest in. And it brings those conventions,
22:59
you know, project layout structures, handling
23:04
how you do authentication, handling how you install
23:07
your toolchains, handling how you do versioning,
23:09
handling how you do software promotion, all these
23:12
things that everybody has to figure out. We've
23:14
published and well documented exactly how to
23:17
do that. So this is what I think we as a community
23:20
need to do. Not necessarily our project. Look,
23:22
there needs to be competition. Somebody else,
23:25
please start a competitor. But if you're not
23:28
in the business doing competition, just use Atmos.
23:30
Well, that was kind of the idea with Terragrunt
23:34
to a degree. I mean, right, they were trying
23:36
to solve some of the fracturing of state files
23:38
and being able to automate some of that, keep
23:41
things DRY. But it didn't solve every problem
23:44
either, though. It is. So Terragrunt is an excellent,
23:47
they were the OG way. Once you started doing
23:50
any sort of serious Terraform, you quickly reached
23:54
for Medium and said, somebody must have solved
23:57
this. And then you came across Terragrunt and
23:59
you found that. And I think they've offered a
24:01
tremendous amount of value for a long time and
24:03
provided the community ways to do some of these
24:08
things. What I still challenge is that they
24:12
don't provide idiomatic ways of organizing and
24:16
architecting infrastructure, at least to the
24:19
degree that I think is required. So how do you
24:23
solve, how do you architect? This is why like
24:25
every week, there are like 10 posts on Reddit.
24:28
How do I organize my infrastructure for
24:31
multi-account? Over and over and over and over and
24:33
over and over and over again. And the answers
24:36
are the same. And from our perspective, always,
24:38
always incredibly inadequate. As a company who
24:41
spent 11 years only focused on cold starts. So
24:45
this is what we call design patterns and how
24:48
you organize your projects for AWS is slightly
24:50
different from GCP, is slightly different from
24:54
Azure and so forth. But there are really nice
24:58
golden paths that eliminate the thought process.
25:02
And this is the other thing that I love about
25:03
frameworks, and we'll tie this back to AI here
25:06
in a second, is that when you have a proper framework,
25:09
it reduces decision fatigue because you just
25:12
follow the conventions that are in place. This
25:15
comes down to how you name your resources. Highly
25:18
recommend you have a naming module or a naming
25:21
provider that gives you programmatically consistent
25:24
names. You just got to plug in the fields and
25:26
your resource names are always the same. It's
25:28
not like, is it prod cluster region service or
25:35
service cluster region prod? It just normalizes
25:38
and standardizes that. And then, well, which
25:41
folder do I put this in or where do I put? All
25:44
of that is handled by the framework. Here's the
25:48
beautiful thing. When you adopt a framework,
25:51
especially one like Atmos that has an insane
25:53
amount of documentation and conventions, and
25:56
every AI model now has been trained on it, it's
26:00
pretty amazing because you can literally ask
26:02
it just to build you out an architecture in a
26:06
conventional framework-esque manner. So how
26:10
do you organize your components and folders?
26:12
How do you organize your configurations for those
26:15
components? How do you install your tools? How
26:18
do you handle authentication? And this has been
26:21
one of the realizations we had over the past
26:23
year is that every team needs to solve how you
26:26
even get your tools installed. And that's different
26:28
whether you're running in GitHub Actions, whether
26:30
you're running on Linux, whether you're running
26:31
on Mac or Windows. How do you get Terraform,
26:34
OpenTofu, AWS CLI, all those things? There's
26:37
a different way to do it. And that leads to inconsistency
26:41
and a lack of reproducibility. Frameworks give
26:43
you reproducibility. You'll know you'll have
26:45
that when your pipelines become boring because
26:50
they run one command and everything is handled
26:52
by your framework. Authentication is another.
26:55
Before you can even run Terraform, you need to
26:58
be authenticated. And Terraform, yeah, it can
27:01
do some role assumption based on your ambient
27:03
credentials, but that's pretty weak. As a human,
27:06
I need to integrate with single sign-on and
27:08
log in that way. Maybe I need to break glass
27:10
with an IAM user. In GitHub Actions, I need to
27:13
exchange an OIDC token for my AWS credentials.
27:17
And then I need to rinse and repeat that for
27:19
Windows, Mac, and Linux. And then if we're
27:21
multi-cloud, oh my God, let's start this project all
27:24
over again and do that for Azure and do that
27:26
for GCP. And these are just the table stakes
27:29
to get started with running Terraform. So a framework
27:33
should handle that. And that's what Atmos does
27:35
too. So Atmos normalizes authentication, works
27:38
with all of these things and gives you just the
27:41
credentials you need to run Terraform. So now
27:44
you can have your local skill files in your repos.
27:48
You can use a framework like Atmos in each one
27:50
of your application repos. And when your team
27:52
goes to do anything, they can just talk with
27:56
their, I heard a great term, their ADE. There's
27:59
no longer IDEs, right? It's ADEs, Agent Developer
28:02
Environment. So these days, you know, you have
28:05
your vertical tabs on the left with all your
28:07
sessions and different worktrees, but you're
28:10
just conversing with, you know, Claude Code or
28:12
Codex, et cetera, and manifesting the infrastructure
28:16
you want. This is what's so cool. Now you have
28:19
a framework, you have your skills, all of these
28:21
things you can, and this is what we're doing
28:23
at Cloud Posse every day now, is we're just conversing.
28:26
and having the infrastructure built, running
28:29
it through the pipelines that are in place, and
28:31
it works. It's awesome. So I've attended a couple
28:36
of DevOps office hours and heard the project
28:40
growing over time. It sounds interesting. I haven't
28:45
had a chance to actually use it in production
28:47
because every company I've worked at is kind
28:50
of locked into what they've already implemented.
28:52
Yeah, that is true. Is there a good migration
28:56
path? Yeah, that's a great question. So historically,
29:00
we've had a really bad story on that. And if
29:03
anybody listening has tried Atmos in the past,
29:05
they might have realized that it was hard to
29:08
do in a brownfield environment. One of the things
29:11
that we've really invested in in the early days
29:14
of Atmos was programmatic consistency. And that
29:18
is a double-edged sword. If you are in an environment
29:21
that is not programmatically consistent, from
29:24
naming conventions to tag conventions to how
29:27
things are organized, it used to be incredibly
29:30
painful to use Atmos because it tried to enforce
29:34
conventions where there were none. Well, over
29:38
the past year, and I have a graph somewhere,
29:40
it's just insane. If you look at, you can barely
29:43
see the activity on Atmos compared to the activity
29:46
we've had in the last year. because of the influence
29:49
of AI. And with that has come a tremendous investment
29:53
in developer experience. So now it works really
29:56
well in brownfield environments. We've helped
29:58
companies have existing Terragrunt architectures
30:01
adopted into the Atmos framework. We've had companies
30:04
who had just their hodgepodge scripts and things
30:07
like that, want to bring it into Atmos. Now you
30:10
can do that and benefit from, here's the thing,
30:14
adopting a framework won't magically turn, is
30:19
not a silver bullet. If you have an inefficient
30:22
application and you just load it into Next.js,
30:26
it isn't going to be magically scalable, right?
30:28
You have to build it that way from the scratch.
30:31
However, you can bring it in and build everything
30:35
new that way. So now you have the benefit of
30:38
a consistent way of developing what you had.
30:41
But without rocking the boat too much and doing
30:44
that. And then building out new things in a way
30:47
that makes service-oriented architectures a little
30:51
bit easier. Now, we haven't talked about that,
30:53
but this is really what we're doing when we're
30:56
doing Terraform these days. A lot of companies,
30:58
when they get started, when they do Terraform,
30:59
they have a monolithic infrastructure code base
31:02
with Terraform. One root module with dozens and
31:06
dozens of child modules maybe and instances of
31:08
those and factories of child modules inside of
31:11
a root module. But this is the equivalent in
31:14
application development of having a monolith
31:16
because every change, that whole thing is deployed
31:21
as a unit in a container and shipped out. But
31:24
in service-oriented architectures, notice I'm
31:26
not saying microservices. Microservices is an
31:29
implementation of services-oriented architectures.
31:31
You break that in, break that apart. How you
31:34
break it apart, that depends on the company.
31:37
We like to say the things that typically change
31:40
by themselves are good candidates to break out
31:43
into their own services. And then now if you
31:46
use a framework like Atmos, you have this option
31:48
of starting to peel things out of this monolithic
31:51
Terraform root module that you have into individual
31:54
services that have their own lifecycle. And then
31:58
they can start to share states between the monolithic
32:01
core where you started. We like to call it your
32:04
heritage environment. And then into your individual
32:07
services. Very cool. Okay, so wrapping up, one
32:10
thing you had also mentioned was self-service,
32:12
enabling self-service without adding layers
32:14
of abstraction and no-code platforms. What does
32:18
that look like at a real org? Yeah, yeah. I guess
32:22
there are three things that I really love about
32:25
AI as an engineer. One is I never need to worry
32:29
about merge conflicts anymore. Those are handled
32:32
for me. So in terms of a team environment and
32:34
collaboration, you can move a lot faster, step
32:36
on toes, and just have the AI resolve those merge
32:39
conflicts. Refactoring became a lot easier. So
32:43
the cost of making certain mistakes is less.
32:46
Not zero, but is less. And team enablement. So
32:50
if I had to solve it once... We have as a practice
32:54
internally, every new service that we develop,
32:57
every new application we develop, it comes with
33:01
its own skills. And if someone's not as familiar
33:05
with this, there's an open standard. Go to
33:08
agentskills.sh or .io. I forget which one it is. And this
33:12
is basically what Claude pioneered, but has been
33:15
embraced by every major ADE out there. The ability
33:19
to have these markdown files, basically. And
33:22
it used to be that WikiOps was pejorative, but
33:24
now it is a sign of a healthy project because
33:29
the agent can follow those things. So getting
33:32
back to your point here, how this enables platform
33:34
teams. Now, make sure you have a step one, pick
33:38
a framework. Ours or someone else's. Step two,
33:41
codify your agent skills. Step three, solve how
33:45
you are going to do your. Code reviews, because
33:49
as you open this up for more developers to contribute,
33:52
you're going to be buried under code reviews
33:55
and PR reviews. So you have to solve that and
33:59
then solve your deployment process so those are
34:02
painless and they just go up. And that is the
34:05
replacement for these no-code platforms. I
34:08
think that was the other point I was going to
34:09
say. The saving grace of AI is that I don't need
34:14
to rely on a no-code platform. I want code as
34:17
an artifact. I want it to meet me where I am
34:20
in my agentic editor, and I want the power of
34:24
the MC... Oh, dude, that's the other thing we
34:26
haven't even talked about, but MCPs. I know there's
34:29
some FUD out there that MCPs are dead and that
34:31
you just need CLIs. They are not. They serve
34:35
a specific purpose that will make a lot of sense
34:37
once you start using them and save on your context
34:40
and your tokens. But like MCPs for AWS, MCPs
34:44
for your telemetry, MCPs for your exception tracking
34:48
and your logs, MCPs for any of your internal
34:51
services. And you put those things together with
34:54
agent skills in a service repo that you deliver
34:57
to your users, and they are up and running in
34:59
no time. It is just magical to see these things
35:02
come together. Yeah, no, for sure. I have developed
35:06
a lot of MCPs, as many as I can, just because
35:10
the more context you can give and the more authentication
35:13
paths that you can give to your AI, the better
35:16
information you'll get back. So yeah, totally
35:18
agree. And the enablement it offers, right? It's
35:20
enabling the developers because I mean, most
35:23
developers I know, the reason why they write
35:24
something from scratch is they just didn't want
35:26
to have to ask for help. So if you meet them
35:30
where they are in their editor before they start
35:32
writing from scratch and they give them the skills,
35:34
give them the MCPs, my gosh, they're going to
35:38
move really fast and blow through your token
35:41
budgets. For sure. So where can people find out
35:44
more about you, more about your projects? Yeah,
35:47
we are very visible, very out in the open. Of
35:50
course, there's cloudposse.com, which is our
35:52
commercial site. It talks about some of our commercial
35:54
reference architectures that we do using our
35:56
framework. There is our project called
35:59
atmos.tools, which is where we talk, where we have
36:03
all the thorough documentation on our framework
36:05
for IaC. I didn't mention, but like we handle
36:08
Ansible, Packer, Terraform, Helmfile, and more
36:12
types of components are supporting, are coming
36:16
out for that as well. And on LinkedIn, of course,
36:19
we'd love to connect with anybody on LinkedIn,
36:21
linkedin.com slash in slash Osterman. And you'll
36:25
find me there. Awesome. I'll leave links for
36:27
all those in the show notes as well. Erik, thank
36:29
you so much for coming on. Really appreciate
36:30
it. Thanks, Brian. I had a great time. That was
36:32
my conversation with Erik Osterman, founder and
36:36
CEO of Cloud Posse. The biggest takeaway for
36:39
me is that AI makes conventions more valuable,
36:43
not less. If every team has a different Terraform
36:46
layout, naming scheme, authentication flow, state
36:51
strategy, and deployment process, AI can absolutely
36:55
generate code faster, but it is still generating
36:58
into chaos. Where things get interesting is when
37:02
those decisions have already been made. There
37:06
is a framework. There are known patterns. Authentication
37:09
works the same way locally and in CI. Resource
37:13
naming is predictable. Teams know where components
37:17
belong and how changes move through the pipeline.
37:20
Now, AI has something consistent to work with.
37:24
I also liked Erik's point that Terraform itself
37:27
does not solve all of infrastructure as code.
37:31
It provisions resources, but teams still have
37:34
to figure out everything around it. State, promotion,
37:38
authentication, repository structure, tooling,
37:42
and how different environments fit together.
37:44
Every organization ends up solving those problems
37:47
somehow. The question is, whether you intentionally
37:51
adapt conventions or slowly build your own undocumented
37:55
framework without calling it one. And that connects
37:59
directly to self-service. Maybe the future is
38:03
not another portal hiding infrastructure behind
38:06
more buttons. Maybe self-service is giving engineers
38:09
a well-defined framework, good documentation,
38:13
agent skills, MCPs, and the ability to work through
38:18
the tools they already use. The code still exists.
38:22
The architecture is still reviewable. But the
38:26
amount of context that somebody needs before
38:29
they can contribute gets much smaller. So my
38:33
takeaway is to use AI to eliminate the repetitive
38:36
parts, but give it structure. Standardize the
38:40
decisions that do not need to be made over and
38:44
over again. Build conventions that both humans
38:48
and agents can understand, and make the paved
38:52
road good enough that people actually want to
38:55
use it. You can find Cloud Posse, Atmos, Erik's
38:59
work, and DevOps Office Hours through the links
39:03
in the show notes. Follow or subscribe to Ship
39:05
It Weekly wherever you listen, and find previous
39:09
episodes at shipitweekly.fm. Thanks for listening,
39:13
and I'll see you later this week.
The thing that stuck with me most from this conversation with Erik Osterman is that AI makes conventions more valuable, not less.
That sounds backwards at first.
If AI can write Terraform, why does it matter how your Terraform is organized? Just ask for what you need and let the model figure it out.
But that is exactly the problem.
If every team lays out its repositories differently, names resources differently, handles authentication differently, and promotes changes differently, AI can absolutely generate code faster. It is just generating into chaos.
The model does not know your conventions if you never wrote them down. It does not know which folder a component belongs in, how your state is split, or how a change gets to production. It guesses. And every guess is another variation someone has to review, debug, and maintain.
Faster code generation is not the same thing as better infrastructure.
Erik's point about frameworks is the part I keep coming back to.
Nobody starts a serious web application by writing raw JavaScript against the browser and inventing their own project structure. They start with React, Next.js, Rails, Django. Frameworks give a team shared nomenclature, a project layout, documentation, design patterns, and a set of decisions that are already made.
Infrastructure as code mostly skipped that step.
There is still a kind of purity test around Terraform. No wrappers. Just vanilla HCL.
But as Erik put it, nobody actually does that.
Terraform provisions resources. It does not decide how you lay out your files, how many state files you have, how you authenticate locally versus in CI, how you install the right versions of your tools, how you promote a change from dev to staging to prod, or how twenty engineers work in the same codebase without stepping on each other.
Every team has to solve those problems.
So every team ends up with a framework anyway. It is just usually undocumented, spread across a Makefile, a few bash scripts, a CI template, and the memory of whoever set it up three years ago.
The question is not whether you have a framework.
The question is whether you chose one on purpose.
That idea showed up in the module discussion too.
A good module is not about wrapping as much as possible. If it wraps a single resource, it is unnecessary abstraction. If it has a hundred inputs, nobody can test the permutations. The value is in the middle: a concept people already understand, like a VPC, turned into something consistent and composable.
And consistency is what makes modules work together.
Same naming. Same interface. Same assumptions.
That is also why Erik's take on HCL was interesting. A general-purpose language like TypeScript gives you unlimited possibility, and that is the appeal. But unlimited possibility also means unlimited variation. HCL's constraints are part of why the Terraform module ecosystem got so big. When everyone is working inside the same limits, the pieces fit together.
Sometimes constraints are the feature.
The Crossplane discussion was a good reminder of how much infrastructure exists below the layer most conversations focus on.
Crossplane can be elegant if someone else owns the cluster. But before you have Crossplane, you need a cluster. Before the cluster, you need a network. Before the network, you need accounts and IAM. Then DNS, zone delegation, certificates, and a network architecture that still makes sense at dozens or hundreds of accounts.
Erik has spent eleven years at Cloud Posse on just that zero-to-one problem. That alone says a lot about how big it is.
The same goes for internal developer platforms.
Spinning up Backstage is easy. Building the foundation that makes Backstage worth having is not. Erik's car analogy fits: starting with the portal is like building the body of the car before the frame, the engine, or the wheels. It looks like progress. It does not drive anywhere.
Platforms work when the foundation is already boring.
Which brings it back to AI.
Erik said most of Cloud Posse's infrastructure work is now generated with AI, and the reason it works is not the model. It is everything around the model. A framework with conventions the model already understands. Agent skills checked into the repo. MCPs that give the agent real context. Pipelines that run one command and handle the rest. A code review process that can keep up.
That is the part I think a lot of teams will miss.
They will add AI to an inconsistent environment and wonder why the output is inconsistent.
The model is only as good as the context it has, and conventions are context.
I also liked how Erik reframed self-service.
For years, self-service meant building another layer on top of infrastructure. A portal. A form. A no-code workflow that hides the code from the people using it.
Maybe the future looks different.
Maybe self-service is a well-defined framework, good documentation, agent skills, and MCPs, delivered right inside the editor an engineer already uses. The code still exists. The architecture is still reviewable. But the amount of context someone needs before they can safely contribute gets much smaller.
Erik made a point I think is true of almost every developer: a lot of people build things from scratch because they did not want to ask for help. If the help is already in their editor, they move a lot faster.
That is what a paved road is supposed to be.
Not a gate. Not an approval queue. A path that is easier to use than to avoid.
So my takeaway from this one is simple.
Use AI to eliminate the repetitive parts, but give it structure.
Standardize the decisions that do not need to be made over and over again. Write the conventions down where both humans and agents can read them. And make the paved road good enough that people actually want to use it.
The syntax is getting cheaper.
The decisions behind it are not.
Additional Links
Erik Osterman on LinkedIn:
https://www.linkedin.com/in/osterman
Cloud Posse:
https://cloudposse.com
Atmos:
https://atmos.tools
Cloud Posse Terraform modules on GitHub:
https://github.com/cloudposse
Cloud Posse DevOps Office Hours:
https://cloudposse.com/office-hours
Terraform modules (root vs. child modules):
https://developer.hashicorp.com/terraform/language/modules
OpenTofu:
https://opentofu.org
Terragrunt:
https://terragrunt.gruntwork.io
Crossplane:
https://www.crossplane.io
Backstage:
https://backstage.io
Pulumi:
https://www.pulumi.com
AWS CloudFormation:
https://aws.amazon.com/cloudformation/
Agent Skills:
https://agentskills.io
Model Context Protocol:
https://modelcontextprotocol.io