Back to all episodes
Ship It Conversations: Ned Bellavance of Ned in the Cloud on DevOps Beyond the Buzzwords, Terraform, AI, the Future of Infrastructure as Code, and Why Fundamentals Still Matter
devopsSREplatform engineeringterraformopentofuDevOps learningagentic AIIaCterragruntSwamp
Explore topics
In this episode, Ned Bellavance discusses the essence of DevOps beyond mere ceremonies, emphasizing the importance of principles like communication and feedback.
Ship It Conversations: Ned Bellavance of Ned in the Cloud on DevOps Beyond the Buzzwords, Terraform, AI, the Future of Infrastructure as Code, and Why Fundamentals Still Matter
Ship It Weekly
0:00
36:12
Chapters
Jump to a section in this episode.
Speed & share
Transcript
A lot of teams say they are doing DevOps because
they have stand-ups, sprints, pipelines, and
maybe a platform team. But ceremony is not the
same thing as progress. And now, AI can write
Terraform, scripts, and infrastructure code faster
than ever without necessarily understanding whether
the architecture makes sense. The tools keep
changing. The harder questions do not. What are
we trying to improve? What should we automate?
And does the person operating it understand what
is actually underneath it? I'm Brian Teller from
Teller's Tech, and this is Ship It Weekly. Welcome
back to Ship It Weekly, where I filter the noise
and focus on what matters when you are the one
running infrastructure and owning reliability.
Most weeks, it's a quick DevOps, SRE, platform,
cloud, and security news recap. In between those,
I do conversation episodes with people building,
teaching, and operating these systems. Today,
I'm joined by Ned Bellavance. founder of Ned in
the Cloud and host of the Day 2 DevOps podcast
we talk about why teams often copy the ceremonies
of devops without adopting the principles how
platform teams should decide what actually matters
and why basic security hygiene still beats chasing
every new threat we also get into the future
of infrastructure as code Terraform's scaling
limits, AI-generated code, emerging tools like
Swamp, and why architecture and troubleshooting
skills may become more valuable as writing code
gets cheaper. All right, let's jump in. Today,
I'm joined by Ned Bellavance. He's the founder
of Ned in the Cloud, and he teaches this stuff
for a living. So he's got a really good lens
on where teams actually are. versus where they
think they are. Ned, thank you for joining me.
Thank you. Thank you for having me, Brian. Really
appreciate it. So give me your take. When teams
say that they're doing DevOps or they're doing
platform engineering, what's usually true? What's
usually hype? Jeez, that's a great question.
I think a lot of the hype has to do with process
and ceremony. So a lot of what you'll hear...
when they're talking about practicing DevOps,
is that ceremony that you put around actually
getting work done. So your daily stand-ups and
embracing Agile and doing sprints and all that
kind of stuff, setting up the ceremony and the
look and the feel of what being truly a DevOps
organization is. And a lot of the time they stop
at the ceremony, but they don't actually adopt
the principles that were behind DevOps to begin
with. Because what was DevOps really meant to
solve? It was meant to solve a communication
gap that existed between the operations and the
developers. I mean, it's in the name, right?
We took two different things, we shortened them,
and we smooshed them together, and we got DevOps.
So that was the goal. It was not to have these
strict ceremonies where you had to go through
this arduous process every day and... And that
would make you just by default better and more
efficient as an organization. No, it was these
other things. We needed to break down the walls
that were between the two groups, have a common
language that we could use to communicate needs,
and have a feedback loop. I mean, that's one
of the main icons you see in DevOps is that nice
infinity symbol of deploying it in production
and then having monitoring and feedback go back
to the developers. The developers can improve
and go through the testing. and the qa and then
it automatically gets deployed to production
ad infinitum right so that cycle is less prevalent
than you might be led to believe but people still
tend to adopt the ceremony instead of what actually
will make them more efficient yeah so they're
they're surface level saying hey this is what
we're supposed to do but they're not maybe diving
into what that actually means at the heart of
it right and a lot of them have gotten mired
in buzzwords And that can totally happen because
there are so many buzzwords that are thrown out.
When I first started encountering DevOps, this
had to be 2016, 2000. Yeah, 2016 ish. I was put
on a project where it was literally an application
team that was building something for a financial
institution. And they had sort of some cloud
adjacent folks, but not really. So they brought.
me and some other folks in who did understand
AWS in the cloud to help them build out their
infrastructure as code and their pipeline. And
they had heard all the buzzwords of like agile
and stand up and, you know, OKRs and all that
kind of stuff, but they hadn't really implemented
what would make their life easier, which was
an automated pipeline that actually deployed.
the infrastructure and the code. So that was
what we actually focused on was getting that
pipeline built out and the process in place with
all the necessary checks. And they found that
their development velocity increased because
now when someone pushed new code, it would only
take half a day for it to end up in production
if it passed through all the correct gates. As
opposed to before, it was probably taking a week
for it to end up in production. What does good
look like without needing that giant platform
team? Is it having everything in IaC? What is
it? Where do you start? Well, I think the place
to start for any organization is figuring out
where you are today and being honest about it.
So really, where are we today? I need to know
that. And where are we trying to get to? What
are our goals? What's actually important to the
organization? Because you could build a super
streamlined pipeline that is shipping crap. And
that's not actually going to improve your organization
in any way. So understanding what the organization's
goals are, how those goals affect what you should
be doing as the DevOps team or the platform team
within the organization, and then setting metrics
that actually measure how effective you are at
those goals. That's where you need to start,
just baseline. And then you can come up with
a strategy and an actual plan. to implement and
get better at the things that matter. So, okay,
let's say that we have that initial setup done.
We have CI/CD, we've shifted left to a degree.
What would you say is next? What matters? Is
it authentication? Is it IAM? Is it resilience?
Again, I know it's probably, it depends, right,
for each organization. It depends. It always
depends. I mean, the things that I would look
for and the concerns I would have, a lot of them
center around security. because that's such a
problem right now. And I'm sure you saw Project
Glasswing got talked about pretty recently. Anthropic's
knew, or was it Mythos? I think it's all part
of one larger push. But they basically developed
a new version of their model that is way too
good at hacking things. And so I would be more
concerned with security than probably anything
else once I've gotten my... T's crossed and my
I's dotted. So there are definitely tools that
will help you become more secure, but I think
bringing in either developing in -house or bringing
in some people who really understand the security
stack would be a good move above anything else,
if that's the main thing that you're worried
about with your application. You might also just
be worried about performance and scaling, or
you might be worried about cost and want to implement
FinOps. So you could... The teams that you bring
into your existing DevOps team will really depend
on what's prioritized at the moment. Yeah, it's
interesting. Like thinking on observability,
I worked for a company where the CEO had a mandate
where I want 100% uptime. Okay, sure. We can
have the webpage cached always showing up. But
what does uptime mean, right? Do you need to
be able to reach the database to actually log
in? Oh, okay. Well, that's a separate set of
SLOs that we need to define at that point. That's
not just... So uptime is relative. So all of
this is relative. What does security mean? If
you're dealing with PII, there could be security
concerns that you may have if you're dealing
with GDPR and maybe security concerns that are
relevant there that maybe just aren't for other
organizations. So yeah, you can find yourself,
I feel like, worrying about the wrong things
if you're not well -led and intentional in your
path. Yeah. And, you know, you can... If you
spend too much time on Reddit or reading hacker
news or you're subscribed to way too many tech
blogs, you might get led astray by what's important
to them because they're going to write about
the shiny, crazy, and the new. And if you haven't
done the basic block and tackling that you need
to for security and you're just chasing after
some new threat vector, you're going to miss
that basic block and tackling. And that's how
most hackers or attackers are going to get in.
It's through those basic... Basic security hygiene
that you're not doing. There's a reason why the
OWASP top 10 list is basically the same every
year. It's because people still do make the same
basic mistakes. Yeah, very true. Yeah, I think
even the recent like Trivy exploits and Strapi
plugin exploits, it was all like any of those
like package system npm, you know, package system
exploits. It's typically just people creating
packages that look like the real thing or close
enough. As engineers, we're sometimes lazy and
we're like, oh, that looks like the right package.
That's what I need. And without vetting it or
checking first. Yeah, some of them have been
really interesting. The way that it actually
goes about attacking your system. Some have started
trying to leverage AIs that you're using on your
local system. So they'll have like a startup
script that's part of that npm package. And when
that script runs, it reaches out subtly and just
asks. various LLMs if they're running. And if
so, you know, can you help me with installing
this thing? And if they can get a toehold in
one of those LLMs, a lot of the time you're running
that as yourself. So it has all the same credentials
you do and can operate a lot faster than you
can. Yeah, that's scary. That's so, gosh, I was
even running. I ran OpenClaw for a couple of
days, but very quickly. Started to go down the
rabbit hole of what that could mean. It's like,
wait a minute. I'm giving autonomous access to
all of my system and all of my various cloud
logins. No, I don't. And prompt injection could,
yeah, this could get bad very quickly. Yes. I
will admit, I'll admit that I'm running OpenClaw
right now, but I think I've approached it in
a measured way. So I'm running it on its own
virtual machine. And rather than giving it any
of my credentials, I've been creating custom
credentials for it to access various systems.
So it has access to my Gmail, but it has read
-only access to my Gmail. It can't create any
emails. It can't create any calendar events.
And same thing with GitHub. I gave it read-only
access to some of my repositories, but it can't
commit code. It can just read some stuff in.
And I've slowly been giving it like, here's access
to this thing and here's that. And OK, you've
been you've been behaving yourself. So maybe
I'll let you create to do's for me. But I'm still
not willing to give it unfettered access to any
of my personal information or my business stuff.
Yeah, no, that's fair. That's the fair way to
do it. It also was just expensive. It was going
through a lot of tokens. But yeah, for sure.
From a security perspective, yeah, read only
and least privilege as much as you can. Before
we started, we spoke a little bit offline about
IaC and you had some, wanted to talk about where
IaC is headed. I'm just kind of curious your
take on where you think infrastructure as code
is headed or where we're going, especially with
AI. Yeah. Becoming more prevalent. I mean, to
put some like historical context on it, I've
been working in the IaC space, you know, since
before Terraform was a thing. 2014, 2013, maybe.
And back then, infrastructure as code was CloudFormation.
And it was JSON only. And it was awful. I feel
bad for anybody who lived through that era before
they adopted YAML to make it a little more compact
and readable. And I had to change my mindset
from my scripting background, which was a lot
of PowerShell and Bash, to this idea of a declarative
configuration. Because I'd been running these
long PowerShell scripts to deploy like AWS environments
or Azure environments. And declarative was such
a mind shift for me. Okay, I'm no longer telling
it step by step. This is what I want you to create.
And I need to have it checked to make sure that
thing doesn't already exist before it recreates
it. Now I'm just telling it what I want. But
the language is a little awkward because when
you're declaratively creating multiple instances,
it's not actually a for loop. It's like this
other construct and you just have to get comfortable
with it. And I did, you know, fully embraced
Terraform at a certain point because it was so
much easier to use and it worked across the two
major clouds that I was working and doing consulting.
And that's kind of been the status quo for the
last eight years is just Terraform has... slowly
been improving and adding new features and doing
interesting things. And people have tried to
shake things up by making Pulumi as a good example
of like, we're going to create something that
snaps into your general purpose programming language.
So if you're mostly a developer, maybe you'll
feel right at home with this. But it's still
the same sort of idea of declaring what you want
and making it so and state has to live somewhere.
I don't know where things are going to go from
here. Because Terraform's starting to hit some
limitations on what it's able to do with the
model it started with. And I think, you know,
this is pure speculation on my part. So this
is not information I got from HashiCorp or anything.
But I feel like as they approach a potential
Terraform 2.0 era, they're going to have to
really overhaul some of the internals of how
Terraform does its thing and break some stuff.
And then you can do that in a major revision,
right? But AI kind of flooded onto the scene
and said, hey, maybe there's a faster and quicker
way to do things that's still reproducible. It's
still code that you can check in somewhere. But
now it's not necessarily a human writing any
of that code. And that's where I think we're
quickly approaching. Yeah, I also think about
the divergence with OpenTofu and some of the
changes that they've made kind of influencing,
I guess, what HashiCorp may have on their roadmap.
And as that divergence grows, there's a gap there
that forms that could lead to another product
or a completely different way of deploying that
infrastructure, maybe with AI, like you mentioned.
Yeah, and that split's not healthy for either
ecosystem because it does create that, okay,
well, if I want to migrate from one to the other,
I have to keep an eye on what's supported across
both of them. They've been doing a decent job
of keeping feature parity, but the actual implementation
detail is going to be different. because they
can't copy code from each other, not directly
at least. But I think just that paradigm of declarative
code and then this standard workflow for deploying
and then updating your infrastructure, it's a
good workflow, but it definitely has its size
limitations. So when your state gets too big,
everything slows way, way, way down before you
do any kind of comparison. You have to refresh
the contents of your local state because Terraform
needs a current picture of what your actual architecture,
your deployed architecture looks like. It doesn't
always know. And so it tries to do a full refresh
of everything. Even if you've only changed one
thing in your configuration, I'm changing my
EC2 instance size. That doesn't impact the VPC
or the subnets or the security groups or the
S3 buckets or like the 26 other things that are
in my config. It impacts one resource, but during
that plan, it still refreshes all 27 resources.
That model doesn't scale. And when you want to
take those things and break them apart into separate
states, that is a painful and arduous process.
So I think on the short term, AI is just going
to make writing Terraform and doing these kind
of operational tasks a bit faster. But if we're
looking longer term, I think there's whole new
classes of solutions that are going to come out
that maintain some of the core principles of
what IaC does, but do it in a different way that's
a little more efficient and a little friendlier.
So I did have a company reach out to me where
it was deterministic AI creating infrastructure.
I don't necessarily want to give them a shout
out because I haven't used the product yet, but
I'm curious if that is like this net new possible
way of deploying infrastructure that will be
adopted or at least i'm sure they're trying to
at least make sure that it is but yeah i'd be
curious to see if there's new ways of deploying
infrastructure it's interesting we're talking
about like those calls years ago at a company
I worked for we were 100% in on HashiCorp this
is pre-IBM 100% in on Vault, Terraform, Packer
all of it and we were we had all of our GitHub
repositories in Terraform as well And we quickly
realized that you can hit that API limit very,
very easily. And GitHub's is pretty restrictive
if you're not on like an enterprise, like a higher
level enterprise program. So yeah, we were then
partitioning out repo creation and branch protection
rules based on teams and other arbitrary ways
of our partitioning out the state files. So we
didn't have to hit those resource limits just
by doing a... making sure there was no drift
changes or making sure that we were updating
branch rules. Yeah, that totally makes sense.
Arduous and broken and yeah, just the wrong way
to do it. Yeah, if listeners aren't familiar,
like when you do a whole bunch of reads against
GitHub's API, that counts against your rate limit.
So if you're doing a massive, if you're managing
all these repos with Terraform and you try to
do a single configuration update on one repository
and it's pulling. 100 repos and all the settings
in those, yeah, boom, you hit that limit and
your plan fails. That's hugely frustrating. So
I would like to see a solution that's a little
more intelligent about the graph that exists
and intelligently updating that graph. The other
thing that Terraform is predicated on, and most
of the other ones are as well, is this idea of
CRUD, or CRUDL, really. So the standard operations
for any resource that exists is you can create
it, you can update it, you can read, you can
delete, and you can list out resources. And that's
really well documented in the providers, and
it makes the providers extremely reliable because
they only have these actions, and that's it.
But if you work in operations, you know that's
not the end-all and be-all of infrastructure.
Those are not the only operations that are permitted.
There's operations like... running a sync or
resetting a virtual machine or backing up a database.
And those are actions that don't fit neatly into
one of those categories. Interestingly, HashiCorp
has introduced Terraform actions to sort of try
to help out with those non-CRUD operations.
But I've been playing around with a new tool
that's being developed by System Initiative called
Swamp. And, you know, I'm not going to say that
Swamp is the end all and be all. But its approach
is really interesting in that it builds models
in the same way that you would build resources
for a Terraform provider. But it doesn't have
strictly defined actions that have to exist.
And it's all TypeScript based, at least at the
moment. So when you build out your model, your
model for a resource can have a number of different
actions. And you can call them whatever you want
them to be. You work with an LLM to write the
TypeScript that is going to be behind those actions.
So if I was managing an Azure virtual machine,
I could not only create, update, delete, etc.
I could also restart it. I could kick off an
extension inside of it. I could do a backup of
it. I could do all these other operations. And
I can also have scoped operations. I could have
an operation that is strictly updating metadata
tags. And then I could potentially... hand out
permissions that that's the only thing that a
specific pipeline or operation can do is update
those metadata tags. So I really, I'm enjoying,
I'm experimenting with it now and I'm enjoying
the model and the fact that it's really easy
to extend because you can download somebody else's
set of models. They call it an extension. And
then if the action you're looking for or the
property that you need isn't in it, you can just
work with the LLM and Swamp and and say this
is what i'm looking for add it to the extension
and it will write the TypeScript for you and
test and validate it and if everything works
out you can upload a new version of the extension
for other people to use interesting so okay given
that model with Swamp given where Terraform is
OpenTofu Terragrunt if you want to wrap all
that and make it more dry where we don't know
what the next evolution is necessarily of IaC
that will be adopted by everybody. How do you
recommend preparing for that future? Is it smaller
state files, larger state files? How do we go
about logically building out the infrastructure
with IaC tools in a way that will make that adoption
easier in the future? Any truly viable solution
is going to have an off-ramp from Terraform,
basically. CloudFormation and ARM templates too,
but any real serious solution is going to have
to have a way for you to migrate off your existing
infrastructure as code onto this new platform.
I don't see that happening, especially at enterprises,
within the next five years. I know the speed
at which enterprises tend to move, and they're
not going to throw away all the effort and the
skills that they've put into building up their
Terraform portfolio. for some new solution that's
totally unproven. So if you're looking for like,
should I even bother learning Terraform? Like,
yeah, you probably should. But I think the more
important thing to learn, and this applies whether
you're using Terraform or CloudFormation or Ansible
or Swamp or like 10 other products that are out
there, none of those matter if you don't understand
what you're actually deploying. All of them assume
that you have some level of knowledge of... the
different services and features that exist on
the different cloud platforms and how they interact
and work together. AI can help you with that
a bit. If you describe the shape of the application
you want, it can make some educated guesses on,
well, that sounds a lot like a Lambda. You should
probably implement that as a Lambda. And okay,
you're going to need, you know, an elastic load
balancer to provide availability for these VMs,
stuff like that. It can make some pretty good
educated guesses. But I think... Our job as,
I guess, system administrator turned DevOps engineer
turned AI prompt engineer, our job is still to
understand the fundamental architecture behind
all this stuff and be able to describe it and
explain why we need certain infrastructure set
up in a particular way. That skill is way, way
more valuable than being able to write good code
in any of these different languages because...
Regardless of what tool wins out, AI is going
to be writing most of the code at this point.
I know that's not like a certainty, but it's
like a 90% certainty that writing code now is
extremely cheap. Writing really good architecture
documents is still very expensive. So I would
focus on learning the fundamentals of the different
cloud platforms and learning the fundamentals
of like networking. of Linux, of databases, of
security, and being able to take all of those
different disciplines and turn it into a design
document that can then be interpreted by an LLM.
Okay, so along those lines, if you were talking
to someone that was new in DevOps, cloud, where
would you tell them to focus first? Kind of what
you already mentioned, but I mean, where should...
Where should their focus be? Should it be on
prompt engineering? And should it not be that?
Should it be the fundamentals first and then
bring AI into the mix after the fact so you know
what you're vetting when you look at the output?
Yeah, I feel like my value is the 20 plus years
I've been in IT and just seen some stuff. And
I don't think there's a way to speed run that
experience. And being hands -on, at least for
me, is the best way to get that experience. So
even though... I might never have to make a Cat 5
cable myself. I've done it. I've taken the crimper,
I've separated out all the pairs, I've put it
into the thing and crimped it up and tested it
with a cable tester. I've done that. I know how
that fundamentally works. And even though it's
totally useless because I can buy a Cat 5 cable
for pennies on the dollar, I still think it was
useful to gain that knowledge. And so... If I
could compare that to like writing code versus
having AI write it for me, you might never have
to write another line of Terraform code in like
a year or two. But understanding how to properly
compose a module, understanding how to organize
code effectively, those are the things that are
going to pay dividends when something goes wrong.
Because that's in the true job of the SRE. is
to be there when things go wrong and they always
will. And knowing what wrong looks like and being
able to sort of guess based off of prior experience
what the probable cause is. Lots of different
tools can help you with that. But those like
the key troubleshooting skills is something you
still need to hone on your own. That's fair.
So what training in your experience actually
sticks for people new? Is it labs, projects,
incident drills, internal docs? What do you find
works the best when working with junior engineers
or people fresh? What works the best is to do
it wrong. Okay. That sounds weird, right? What
works the best for me, and so I'll use I statements
because this is what works best for me, is I
only ever read like the first three pages of
the manual and then I try to do it. And it never
works right. It's broken. It's misconfigured.
I'm frustrated. But the process of reading the
docs, figuring out where I went wrong, and then
fixing it and redoing it correctly, I'm going
to remember that whole process because it truly
was a process. Clicking like a let's get started
script and just having it build all the background
for me, that means I'm never going to truly learn
those things. At least, you know, like I said,
for me, there's a reason why. Kelsey Hightower
wrote Kubernetes the Hard Way. It's not because
it's impossible to script the installation of
Kubernetes. Clearly that's not the case, right?
But if you do it the hard way, you're probably
going to screw it up. So you're going to have
to recreate a certificate authority and reissue
certs because you got that wrong. And then you
probably spun up your node wrong or you got the
kubelet misconfigured or your networking is incorrect.
And you're just going to trip over all these
things. And as you trip over those things, you
realize where the gaps are and where things can
go wrong. And so when your production cluster
falls over at 3 a .m., you're going to feel like
you actually have a handle on what's happening
inside that cluster. As opposed to, I've only
ever used this template and deployed automatically
and never had to think about what happens if
the networking gets screwed up within my cluster.
That's fair. I mean, I find even still, I'm learning
after 25 plus years that same way, right? By
breaking things. And I mean, even OpenClaw, if
I wanted to, you know, trying that, it was playing
around with it and letting it break things in
a controlled environment. But yeah, I'm just
playing with it. I, you know, I still find that
I don't know everything about Kubernetes. I don't
know everything about cloud. I can't. It's impossible.
But being open to that, being open to that process
and being open to it. to getting it wrong is
valuable, very valuable. Yeah, I've had three
pillars. Well, I established three pillars for
myself a long time ago. One of them was make
yourself uncomfortable. So just be uncomfortable.
Put yourself in an uncomfortable, difficult situation.
Not too uncomfortable, but like step outside
of your comfort zone, right? Another one was
be kind because everybody else is just trying
to trudge through this thing called life as well.
And the third one was... be prepared to fail
and be okay with it. Because failure is one of
the ways we learn. And I am failing constantly.
I spent last week fighting with this Swamp thing
to set up my home lab. And I had moments where
I was yelling at the monitor and hitting escape
and going outside to take a walk. But after a
week, I used Swamp to rebuild my entire home
lab. It was a big learning experience for me
on how to use LLMs more effectively. To understand
Synology's weird API is a little bit better.
Don't get into that if you don't have to. And
also, I learned some new ways to capture web
traffic and do analysis on it. I didn't even
think that would be a benefit, but it was just
like in the troubleshooting, I learned a whole
bunch of new things. So I think being willing
to experiment and being willing to try something
and fail is the mark. of a lifetime learner and
that's what we all are when it comes to technology
you have to be a lifetime learner because the
technology is not going to stop for you i'd be
very dubious to trust any senior or plus level
engineer that's worked 10 plus years in the industry
and hasn't taken down production or has that
as like a statement like oh i yeah i've never
well if you haven't then you what have you really
been doing like what yeah how deep do you actually
work in the system or do you know the system
um because i think we've all been there right
yeah totally It's an unfortunate reality, but
humbling yourself and working as a team and working
through it rather than just panicking is when
you learn probably the most, too, is in those
scenarios, in those situations, not in the building
of the POC or in the building. I mean, those
can be valuable as well, but or in building of
the SOP, you know, building out your standing
operating procedure that you may run or run book
that you may use someday, but actually doing
it and being in the incident. is where the real
work and value, I think, is built for the most
part. Yeah, and I have a podcast. It's called
Day 2 DevOps. And that was sort of the inspiration
behind the name is because day one or that POC,
when you first deploy everything and it's new
and it's shiny, yeah, it's probably going to
hum along pretty well. But day two, when you
actually have to start operating it and updating
it and real traffic starts hitting it, that's
when you learn how good your architecture actually
was. And that's what we like to focus on. in
the podcast is kind of more realistic implementations
of technology and less the hello world and the
day one of what happens when you deploy tech.
Very cool. So along those lines, where can people
find out more about you find out more about your
podcast? Any other things that you'd like to
plug or talk about? Yeah, I mean, so I have two
things I'd like to plug one is that podcast day
two DevOps. So if that sounds interesting to
you, if you want to hear some interesting conversations.
about DevOps adjacent technologies, Day 2 DevOps,
all spelled out. You can find it there at day2devops.com.
And the other thing that I'm currently
doing is I'm doing a live stream every Thursday
at 1 o'clock Eastern to help people study for
the Terraform Authoring and Operations Professional
exam. So every week I cover a few of the topics
that are part of the exam. And based on my mapping,
that's going to be going on for a couple months
because there's a lot of topics. So if that sounds
interesting to you, if you thought about sitting
the exam or you just have like weird questions
about Terraform, go to my YouTube channel. It's
Ned in the Cloud and check it out. 1 p.m. on
Thursdays. Awesome. I'll put links for all those
in the show notes as well. Ned, thank you so
much for coming on. Really, really appreciate
it. Thanks, Brian. That was my conversation with
Ned Bellavance, founder of Ned in the Cloud. and
host of Day 2 DevOps. The biggest takeaway
for me is that tools and terminology can make
a team look much further along than it really
is. You can have stand-ups, pipelines, infrastructure
as code, and an internal platform. None of that
guarantees you are improving communication, shortening
feedback loops, or shipping something the business
actually needs. The same applies to AI. AI can
make writing Terraform and automation code much
cheaper, but it does not remove the need to understand
what you are deploying, why the architecture
works, what permissions it needs, or what failure
looks like in production. If anything, that judgment
becomes more valuable. I also liked Ned's point
about learning by getting things wrong, not recklessly
in production. but in labs and controlled environments
where you can break something, troubleshoot it,
and understand what was hidden behind the template.
You cannot completely speed run that experience.
So focus less on whether your organization looks
like it is doing DevOps and more on whether the
practices are producing better outcomes. Use
AI to move faster. but understand what it generates
learn the tools but do not confuse the tool with
the architecture and keep building the troubleshooting
experience you will need when the clean demo
becomes a day 2 production system you can find
Ned in the Cloud and the Day 2 DevOps podcast
through the links in the show notes Follow or
subscribe to Ship It Weekly wherever you listen
and find previous episodes at shipitweekly.fm.
Thanks for listening and I'll see you later this
week.
Ship It Conversations: Ned Bellavance of Ned in…
0:00 / 36:12
Listen & Subscribe
Catch This Episode
Listen on your favorite podcast platform
Commentary
Host Commentary
📝 Notes
Show Notes
This is a guest conversation episode of Ship It Weekly, separate from the weekly news recaps.
In this Ship It Conversations episode, I talk with Ned Bellavance of Ned in the Cloud about DevOps beyond the buzzwords, platform engineering, infrastructure as code, AI, and why fundamentals still matter even as the tools change.
Ned is the founder of Ned in the Cloud and host of the Day 2 DevOps podcast, with more than 20 years in IT across systems administration, cloud, architecture, automation, and technical education.
We start with a problem a lot of teams run into: adopting the ceremonies of DevOps without actually adopting the principles. Standups, sprints, pipelines, and tooling can make an organization look mature, but the real goal is better communication, faster feedback loops, and delivery tied to actual outcomes.
We also talk about how teams decide what to prioritize next. Security, reliability, performance, FinOps, and platform work can all matter, but chasing whatever is newest does not help if the basics are still broken.
A big part of the conversation is where infrastructure as code goes from here. We get into Terraform's state and scaling model, API rate limits, the Terraform/OpenTofu split, Terragrunt, and newer approaches like Swamp from System Initiative. AI is making infrastructure code cheaper to produce, but understanding the architecture behind that code is becoming more valuable.
That leads into learning and career development. We talk about why networking, Linux, databases, security, cloud architecture, and troubleshooting still matter, even if an LLM writes most of the syntax. Build things, get them wrong in controlled environments, troubleshoot them, and learn what is happening underneath the abstraction.
The big takeaway: tools will keep changing. Judgment, architecture, troubleshooting, and understanding the systems underneath them are much harder to automate away.
Highlights
• Why DevOps ceremony is not the same as DevOps principles
• How teams should decide what platform work actually matters
• Why basic security hygiene still matters
• Where Terraform's current state model starts to hit limits
• Terraform, OpenTofu, Terragrunt, and the future of infrastructure as code
• How AI changes the value of writing infrastructure code
• Why architecture and troubleshooting skills become more important
• Why breaking things in controlled environments is one of the best ways to learn
Links
Ned in the Cloud: https://nedinthecloud.com
Day 2 DevOps: https://day2devops.com
Ned in the Cloud on YouTube: https://www.youtube.com/c/NedintheCloud/
Ned Bellavance on LinkedIn: https://www.linkedin.com/in/ned-bellavance/
Swamp: https://swamp.club
Terraform: https://developer.hashicorp.com/terraform
OpenTofu: https://opentofu.org
More episodes and show notes: https://shipitweekly.fm
On Call Brief: https://oncallbrief.com
👤 Guest
Ned Bellavance of Ned in the Cloud
Made in Greencastle, Pennsylvania
More from Ship It Weekly
All episodes →
Ship It Conversations: Gareth Kersey on IaCConf 2026, AI, and Corey Quinn’s Terraform Keynote
In this episode, Gareth Kersey discusses IaCConf 2026, focusing on how infrastructure teams adapt to AI-driven changes in software delivery.
EKS Rollbacks, GitHub Actions Supply Chain Attacks, AI Agentjacking, CloudWatch Log Alarms, and Why Safety Nets Don’t Replace Ownership
This episode of Ship It Weekly discusses Amazon EKS's new Kubernetes version rollbacks, GitHub Actions supply chain risks, and AI agentjacking via fake telemetry.
CISA’s GitHub Leak, AI Root Cause Analysis, Copilot Agents, Claude Code in CI/CD, and Kubernetes Seccomp Risk
This episode of Ship It Weekly discusses the implications of the CISA GitHub leak, highlighting the exposure of AWS keys and internal documentation.
AI Agents Target GitHub, Kubernetes 1.37 Deprecations, AWS Transit Gateway Policy Routing, IAM Identity Center Multi-Region, Cloudflare Meerkat, AMOS Mac Malware, and the Risk of Half-Migrated Production Systems
This episode of Ship It Weekly discusses AI agents from Anthropic and OpenAI taking unsanctioned actions during UK cyber tests, including a malicious GitHub pull request.
The thing that stuck with me most from this conversation is how easy it is for a team to look mature without actually being mature.
You can have standups, sprints, CI/CD, Terraform everywhere, a platform team, an internal developer portal, automated security scans, dashboards, SLOs, and a giant pile of tooling. None of that automatically means you are doing DevOps well.
That was one of the first things Ned got into, and I think it is a useful distinction because DevOps has accumulated so much ceremony around it that sometimes the ceremony becomes the goal. Teams start asking whether they have the right meetings, the right tools, the right titles, or the right platform. The better question is whether any of it actually improved how the organization works.
Are developers and operations communicating better? Is feedback getting back to engineers faster? Can you ship a change safely without waiting a week for five different handoffs? Can people understand what happened after something reaches production? Are you fixing the problems that actually matter to the business?
That is much harder to measure than whether everybody attended standup.
It connects to something I see constantly in platform engineering too. It is really easy to start building the platform before deciding what problem the platform is supposed to solve. You can spend six months creating beautiful abstractions, golden paths, self-service workflows, templates, and automation, then discover the thing slowing teams down was an approval process nobody challenged, a flaky test suite, an API rate limit, bad IAM boundaries, or a deployment process that required somebody to manually click three buttons.
The platform is not the outcome. The pipeline is not the outcome. Terraform is not the outcome. They are tools we use to get somewhere else.
I liked Ned's framing here: start by being honest about where you actually are. Then figure out where you are trying to go. Then measure whether the work you are doing is moving you in that direction.
That applies to reliability too. I brought up the example of a company wanting 100 percent uptime because that sounds great as an executive goal. Of course we want 100 percent uptime. But what does uptime mean?
Does the homepage respond? Can the user authenticate? Can the application talk to the database? Can the customer complete the transaction that actually makes the business money?
Those are very different things. You can cache a webpage and proudly show a green uptime dashboard while the application behind it is completely unusable. The number means nothing until you define what the number represents.
Security works the same way. You can say security is the priority, but the actual risks depend on the system. Are you handling PII? Do you have regulatory requirements? Are you exposing public APIs? Are you running AI agents with access to developer credentials?
That is why I liked the part of the conversation where Ned talked about basic security hygiene. There is always a new scary vulnerability, package compromise, supply chain attack, prompt injection technique, or headline about some model doing something terrifying. Those things matter, but there is also a reason the same basic classes of mistakes keep showing up year after year.
People still over-permission credentials. People still install packages without really checking what they are. People still expose things that should not be public. People still reuse secrets. People still grant applications far more access than they actually need.
The shiny new attack gets attention. The boring old mistake gets exploited.
The OpenClaw discussion was a good example of that. Giving an autonomous system access to your email, GitHub, cloud accounts, filesystem, browser sessions, or developer credentials should immediately change how you think about permissions.
The interesting part of Ned's setup was not that he was running an AI agent. It was that he was treating the agent like an untrusted automation system. Separate VM. Separate credentials. Read-only access where possible. Gradually expanding permissions instead of handing it everything on day one.
That is just least privilege.
There is nothing particularly AI-specific about the principle. What AI changes is the speed and autonomy of the thing holding those permissions. If I accidentally give a script too much access, the script can do whatever its code explicitly tells it to do. If I give an agent too much access, I am giving a probabilistic system a collection of tools and asking it to figure out how to accomplish a goal.
That should probably make us more careful about permissions, not less.
Then the conversation moved into infrastructure as code, which I think was probably my favorite part.
Terraform has been incredibly successful for a reason. Declarative infrastructure was a huge improvement over giant procedural scripts that had to manually check whether every resource existed before deciding what to do next. You describe the desired state, Terraform builds a graph, compares what you declared with what exists, and figures out what needs to change.
That model has worked really well. But successful abstractions eventually run into the edges of the assumptions they were built around.
State is one of those edges.
Anyone who has worked with a sufficiently large Terraform estate has eventually had the conversation about how many resources belong in a state file. Too large and plans become painfully slow. Too small and you create dependency and orchestration problems between dozens or hundreds of states.
Then you start introducing wrappers, dependency graphs, CI orchestration, remote state lookups, generated configuration, and conventions about where everything lives. It works, but you can feel the complexity accumulating around the original abstraction.
The GitHub example we talked about is a good one. Terraform might only need to change one branch protection rule, but before it can confidently decide what needs to change, it may need to refresh a huge number of repositories and related resources. Those reads count against API rate limits.
Suddenly the infrastructure problem is not actually creating the resource. The infrastructure problem is figuring out what already exists without exhausting somebody else's API.
That suggests the next generation of infrastructure tooling may not simply be Terraform with nicer syntax. The state and reconciliation model itself may evolve.
Ned talked about Swamp from System Initiative, and I think the interesting part is not whether Swamp specifically becomes the thing everybody uses. Nobody knows that yet. The interesting part is the model.
Terraform providers largely operate around CRUD-style resource lifecycle operations. Create, read, update, delete, maybe list. That maps nicely to provisioning infrastructure.
But operations work is bigger than provisioning.
Restart this virtual machine. Back up this database. Rotate this credential. Update these tags. Run this synchronization. Perform this one operation against this one part of the resource without pretending the resource itself needs to be recreated.
Those are normal operational tasks, but they do not always fit neatly into the original declarative resource lifecycle. That is where I think the next few years of infrastructure tooling get interesting.
AI makes it much easier to create new interfaces around infrastructure because generating the glue code suddenly becomes cheap. And that led into what I think was the most important point Ned made in the entire conversation.
Writing code is getting cheaper.
Understanding what the code should do is not.
That distinction matters a lot.
Terraform syntax used to be a meaningful part of the skill. You had to understand HCL, modules, expressions, loops, dependencies, providers, data sources, state, and all the weird edges of the language. Those skills still matter today, but an LLM can write a pretty decent Terraform module in seconds.
The same thing is happening with Python, TypeScript, Bash, Kubernetes manifests, GitHub Actions, Helm charts, CloudFormation, and Ansible. The cost of producing syntactically plausible infrastructure code has collapsed.
But somebody still needs to know whether that infrastructure makes sense.
Should this workload be a Lambda? Should it be Kubernetes? Should it be a virtual machine? Does it need a load balancer? Should this database be public? What network paths should exist? Where should secrets live? What should happen if the region fails? What data can be lost? How much availability does the business actually need? What happens when traffic increases ten times?
AI can suggest answers, but the person reviewing those answers needs enough context to know whether they are reasonable.
That is why I do not think AI makes fundamentals less valuable. I think it makes them more valuable.
If AI handles more of the syntax, human value moves upward into architecture, constraints, tradeoffs, security boundaries, failure modes, troubleshooting, and understanding how systems interact.
Knowing when the generated answer is technically valid but operationally stupid is going to matter a lot.
Generated infrastructure can look incredibly convincing. Everything can validate. The Terraform plan can look fine. The Kubernetes manifest can apply successfully. The pipeline can turn green.
And the architecture can still be wrong.
This also changes how I think people should learn DevOps and cloud engineering. There is a temptation right now to skip directly to prompting. Why learn Terraform deeply if Claude can write Terraform? Why learn Kubernetes if an agent can create the manifest? Why learn networking if AI can tell you which security group rule you need?
Because eventually something breaks.
And the clean abstraction disappears.
The SRE job is not just creating the thing. It is understanding the thing when the assumptions stop being true.
That is why Ned's example about making a network cable actually made sense to me. You probably do not need to make your own Ethernet cables anymore. That does not mean there was no value in understanding what was inside the cable.
The same thing happens when you build Kubernetes the hard way. Nobody should manually build every production Kubernetes cluster from individual binaries and certificates. That would be absurd. But doing it once teaches you what kubelet is, what certificates are doing, where control plane communication happens, how networking fits together, and which components exist underneath the managed abstraction.
Then when something breaks at 3 a.m., you are not looking at Kubernetes as one giant magic box that stopped working. You have some idea of what could actually be broken inside the box.
That is why the learning discussion near the end of the episode resonated with me. Sometimes the best way to learn something is to do it wrong.
Not intentionally wreck production. Not create chaos just for the sake of chaos. But build something in a place where failure is cheap. Try it. Break it. Misconfigure it. Read the error. Figure out why it did not work. Fix it. Do it again.
That debugging process creates a completely different kind of understanding than following a perfectly scripted tutorial.
Tutorials are great for getting started. They are terrible at teaching you what happens when step seven does not work.
Production is mostly step seven not working.
Experienced engineers sometimes forget how much of their judgment came from those failures. You remember the certificate issue because you spent four hours figuring out why TLS was broken. You remember the networking problem because you accidentally configured a route that black-holed traffic. You remember the IAM permission because the application worked everywhere except production and you eventually realized one role was missing a single action.
Those experiences become intuition.
You start recognizing the shape of failures before you fully understand them. Something feels like DNS. Something feels like permissions. Something feels like stale state. Something feels like a dependency timing issue.
That intuition is difficult to teach and difficult to speed-run.
AI can help you investigate it. It can search logs faster, explain an error message, suggest likely causes, and generate commands. That is useful. But there is still value in the engineer understanding why one hypothesis is much more likely than another.
That is the judgment we should be trying to preserve and teach.
I also liked Ned's three personal principles: make yourself uncomfortable, be kind, and be prepared to fail.
Those are surprisingly good engineering principles.
Make yourself uncomfortable means keep learning things slightly outside what you already know. Be kind matters because incidents and technical disagreements involve humans, and everybody is trying to solve the same problem under some amount of pressure. Being prepared to fail matters because failure is unavoidable if you are actually experimenting and building things.
I joked that I would be suspicious of somebody who had been a senior engineer for ten years and claimed they had never taken down production.
I still mostly believe that.
Not because taking down production is some badge of honor. It absolutely is not. But if you have spent enough years making meaningful changes to complex systems, eventually you will get something wrong.
The more important question is what happens next.
Do you hide it? Do you blame somebody? Do you panic? Or do you work the incident, understand what happened, restore the system, and make the next failure less likely?
That is where the experience comes from.
And that connects perfectly to Ned's Day 2 DevOps framing.
Day one is easy to make look good. The demo works. The POC works. The deployment succeeds. Everybody celebrates.
Day two is when reality arrives.
Users show up. Traffic increases. Certificates expire. Dependencies change. Security requirements change. Costs grow. Someone upgrades a library. A region has problems. The API behaves differently than you expected. The person who built the original system leaves the company.
Now somebody has to operate it.
That is when you find out whether the architecture was actually good.
Honestly, that might be the thread connecting this entire conversation.
DevOps ceremony looks good on day one. Platform demos look good on day one. Terraform looks good when the state is small. AI-generated infrastructure looks good when the code validates. New tools look great in a POC.
The real test is what happens after the novelty wears off.
Can we operate it? Can we troubleshoot it? Can we change it? Can someone other than the original author understand it? Does it actually improve the outcome we cared about? What happens when it breaks?
That is the work.
So my takeaway from this episode is pretty simple. Do not confuse having DevOps tooling with doing DevOps well. Do not confuse infrastructure as code with good architecture. Do not confuse AI-generated code with understanding the system. And do not optimize so hard for avoiding failure that nobody gets the experience required to handle failure when it eventually happens.
Use the new tools. Experiment with AI. Try the new infrastructure models. Let AI write the boring code. That part is useful.
But keep learning networking. Keep learning Linux. Keep learning security. Keep learning databases. Keep understanding how cloud services actually fit together. Keep building things. Keep troubleshooting them. And every once in a while, build something the hard way just so you understand what the easy way is hiding.
Because the syntax is getting cheaper.
The judgment is not.
Additional Links
Ned in the Cloud: https://nedinthecloud.com
Day 2 DevOps: https://day2devops.com
Ned in the Cloud on YouTube: https://www.youtube.com/c/NedintheCloud/
Ned Bellavance on LinkedIn: https://www.linkedin.com/in/ned-bellavance/
Swamp: https://swamp.club
Terraform: https://developer.hashicorp.com/terraform
OpenTofu: https://opentofu.org
Terragrunt: https://terragrunt.gruntwork.io
OWASP Top 10: https://owasp.org/www-project-top-ten/
Kubernetes The Hard Way: https://github.com/kelseyhightower/kubernetes-the-hard-way