A lot of teams say they are doing DevOps because they have stand-ups, sprints, pipelines, and maybe a platform team. But ceremony is not the same thing as progress. And now, AI can write Terraform, scripts, and infrastructure code faster than ever without necessarily understanding whether the architecture makes sense. The tools keep changing. The harder questions do not. What are we trying to improve?
What should we automate? And does the person operating it understand what is actually underneath it? I'm Brian Teller from Teller's Tech, and this is Ship It Weekly. Welcome back to Ship It Weekly, where I filter the noise and focus on what matters when you are the one running infrastructure and owning reliability. Most weeks, it's a quick DevOps, SRE, platform, cloud, and security news recap.
In between those, I do conversation episodes with people building, teaching, and operating these systems. Today, I'm joined by Ned Bellavance.
Founder of Ned in the Cloud and host of the Day 2 DevOps podcast we talk about why teams often copy the ceremonies of devops without adopting the principles how platform teams should decide what actually matters and why basic security hygiene still beats chasing every new threat we also get into the future of infrastructure as code Terraform's scaling limits, AI-generated code, emerging tools like Swamp, and why architecture and troubleshooting skills may become more valuable as writing code gets cheaper.
All right, let's jump in. Today, I'm joined by Ned Bellavance. He's the founder of Ned in the Cloud, and he teaches this stuff for a living. So he's got a really good lens on where teams actually are. Versus where they think they are. Ned, thank you for joining me. Thank you. Thank you for having me, Brian. Really appreciate it. So give me your take.
When teams say that they're doing DevOps or they're doing platform engineering, what's usually true? What's usually hype? Jeez, that's a great question. I think a lot of the hype has to do with process and ceremony. So a lot of what you'll hear... When they're talking about practicing DevOps, is that ceremony that you put around actually getting work done.
So your daily stand-ups and embracing Agile and doing sprints and all that kind of stuff, setting up the ceremony and the look and the feel of what being truly a DevOps organization is. And a lot of the time they stop at the ceremony, but they don't actually adopt the principles that were behind DevOps to begin with. Because what was DevOps really meant to solve?
It was meant to solve a communication gap that existed between the operations and the developers. I mean, it's in the name, right? We took two different things, we shortened them, and we smooshed them together, and we got DevOps. So that was the goal. It was not to have these strict ceremonies where you had to go through this arduous process every day and...
And that would make you just by default better and more efficient as an organization. No, it was these other things. We needed to break down the walls that were between the two groups, have a common language that we could use to communicate needs, and have a feedback loop.
I mean, that's one of the main icons you see in DevOps is that nice infinity symbol of deploying it in production and then having monitoring and feedback go back to the developers. The developers can improve and go through the testing.
And the qa and then it automatically gets deployed to production ad infinitum right so that cycle is less prevalent than you might be led to believe but people still tend to adopt the ceremony instead of what actually will make them more efficient yeah so they're they're surface level saying hey this is what we're supposed to do but they're not maybe diving into what that actually means at the heart of it right and a
Lot of them have gotten mired in buzzwords And that can totally happen because there are so many buzzwords that are thrown out.
When I first started encountering DevOps, this had to be 2016, 2000. Yeah, 2016 ish. I was put on a project where it was literally an application team that was building something for a financial institution. And they had sort of some cloud adjacent folks, but not really. So they brought.
Me and some other folks in who did understand AWS in the cloud to help them build out their infrastructure as code and their pipeline. And they had heard all the buzzwords of like agile and stand up and, you know, OKRs and all that kind of stuff, but they hadn't really implemented what would make their life easier, which was an automated pipeline that actually deployed. The infrastructure and the code.
So that was what we actually focused on was getting that pipeline built out and the process in place with all the necessary checks. And they found that their development velocity increased because now when someone pushed new code, it would only take half a day for it to end up in production if it passed through all the correct gates. As opposed to before, it was probably taking a week for it to end up in production.
What does good look like without needing that giant platform team? Is it having everything in IaC? What is it? Where do you start? Well, I think the place to start for any organization is figuring out where you are today and being honest about it. So really, where are we today? I need to know that. And where are we trying to get to? What are our goals? What's actually important to the organization?
Because you could build a super streamlined pipeline that is shipping crap. And that's not actually going to improve your organization in any way. So understanding what the organization's goals are, how those goals affect what you should be doing as the DevOps team or the platform team within the organization, and then setting metrics that actually measure how effective you are at those goals.
That's where you need to start, just baseline. And then you can come up with a strategy and an actual plan. To implement and get better at the things that matter. So, okay, let's say that we have that initial setup done. We have CI/CD, we've shifted left to a degree. What would you say is next? What matters? Is it authentication? Is it IAM? Is it resilience?
Again, I know it's probably, it depends, right, for each organization. It depends. It always depends. I mean, the things that I would look for and the concerns I would have, a lot of them center around security. Because that's such a problem right now. And I'm sure you saw Project Glasswing got talked about pretty recently. Anthropic's knew, or was it Mythos? I think it's all part of one larger push.
But they basically developed a new version of their model that is way too good at hacking things. And so I would be more concerned with security than probably anything else once I've gotten my... T's crossed and my I's dotted.
So there are definitely tools that will help you become more secure, but I think bringing in either developing in -house or bringing in some people who really understand the security stack would be a good move above anything else, if that's the main thing that you're worried about with your application.
You might also just be worried about performance and scaling, or you might be worried about cost and want to implement FinOps. So you could... The teams that you bring into your existing DevOps team will really depend on what's prioritized at the moment. Yeah, it's interesting. Like thinking on observability, I worked for a company where the CEO had a mandate where I want 100% uptime. Okay, sure.
We can have the webpage cached always showing up. But what does uptime mean, right? Do you need to be able to reach the database to actually log in? Oh, okay. Well, that's a separate set of SLOs that we need to define at that point. That's not just... So uptime is relative. So all of this is relative. What does security mean?
If you're dealing with PII, there could be security concerns that you may have if you're dealing with GDPR and maybe security concerns that are relevant there that maybe just aren't for other organizations. So yeah, you can find yourself, I feel like, worrying about the wrong things if you're not well -led and intentional in your path. Yeah. And, you know, you can...
If you spend too much time on Reddit or reading hacker news or you're subscribed to way too many tech blogs, you might get led astray by what's important to them because they're going to write about the shiny, crazy, and the new. And if you haven't done the basic block and tackling that you need to for security and you're just chasing after some new threat vector, you're going to miss that basic block and tackling.
And that's how most hackers or attackers are going to get in. It's through those basic... Basic security hygiene that you're not doing. There's a reason why the OWASP top 10 list is basically the same every year. It's because people still do make the same basic mistakes. Yeah, very true.
Yeah, I think even the recent like Trivy exploits and Strapi plugin exploits, it was all like any of those like package system npm, you know, package system exploits. It's typically just people creating packages that look like the real thing or close enough. As engineers, we're sometimes lazy and we're like, oh, that looks like the right package. That's what I need. And without vetting it or checking first.
Yeah, some of them have been really interesting. The way that it actually goes about attacking your system. Some have started trying to leverage AIs that you're using on your local system. So they'll have like a startup script that's part of that npm package. And when that script runs, it reaches out subtly and just asks. Various LLMs if they're running.
And if so, you know, can you help me with installing this thing? And if they can get a toehold in one of those LLMs, a lot of the time you're running that as yourself. So it has all the same credentials you do and can operate a lot faster than you can. Yeah, that's scary. That's so, gosh, I was even running. I ran OpenClaw for a couple of days, but very quickly.
Started to go down the rabbit hole of what that could mean. It's like, wait a minute. I'm giving autonomous access to all of my system and all of my various cloud logins. No, I don't. And prompt injection could, yeah, this could get bad very quickly. Yes. I will admit, I'll admit that I'm running OpenClaw right now, but I think I've approached it in a measured way. So I'm running it on its own virtual machine.
And rather than giving it any of my credentials, I've been creating custom credentials for it to access various systems. So it has access to my Gmail, but it has read -only access to my Gmail. It can't create any emails. It can't create any calendar events. And same thing with GitHub. I gave it read-only access to some of my repositories, but it can't commit code. It can just read some stuff in.
And I've slowly been giving it like, here's access to this thing and here's that. And OK, you've been you've been behaving yourself. So maybe I'll let you create to do's for me. But I'm still not willing to give it unfettered access to any of my personal information or my business stuff. Yeah, no, that's fair. That's the fair way to do it. It also was just expensive. It was going through a lot of tokens.
But yeah, for sure. From a security perspective, yeah, read only and least privilege as much as you can. Before we started, we spoke a little bit offline about IaC and you had some, wanted to talk about where IaC is headed. I'm just kind of curious your take on where you think infrastructure as code is headed or where we're going, especially with AI. Yeah. Becoming more prevalent.
I mean, to put some like historical context on it, I've been working in the IaC space, you know, since before Terraform was a thing. 2014, 2013, maybe. And back then, infrastructure as code was CloudFormation. And it was JSON only. And it was awful. I feel bad for anybody who lived through that era before they adopted YAML to make it a little more compact and readable.
And I had to change my mindset from my scripting background, which was a lot of PowerShell and Bash, to this idea of a declarative configuration. Because I'd been running these long PowerShell scripts to deploy like AWS environments or Azure environments. And declarative was such a mind shift for me. Okay, I'm no longer telling it step by step. This is what I want you to create.
And I need to have it checked to make sure that thing doesn't already exist before it recreates it. Now I'm just telling it what I want. But the language is a little awkward because when you're declaratively creating multiple instances, it's not actually a for loop. It's like this other construct and you just have to get comfortable with it.
And I did, you know, fully embraced Terraform at a certain point because it was so much easier to use and it worked across the two major clouds that I was working and doing consulting. And that's kind of been the status quo for the last eight years is just Terraform has... Slowly been improving and adding new features and doing interesting things.
And people have tried to shake things up by making Pulumi as a good example of like, we're going to create something that snaps into your general purpose programming language. So if you're mostly a developer, maybe you'll feel right at home with this. But it's still the same sort of idea of declaring what you want and making it so and state has to live somewhere. I don't know where things are going to go from here.
Because Terraform's starting to hit some limitations on what it's able to do with the model it started with. And I think, you know, this is pure speculation on my part. So this is not information I got from HashiCorp or anything. But I feel like as they approach a potential Terraform 2.0 era, they're going to have to really overhaul some of the internals of how Terraform does its thing and break some stuff.
And then you can do that in a major revision, right? But AI kind of flooded onto the scene and said, hey, maybe there's a faster and quicker way to do things that's still reproducible. It's still code that you can check in somewhere. But now it's not necessarily a human writing any of that code. And that's where I think we're quickly approaching.
Yeah, I also think about the divergence with OpenTofu and some of the changes that they've made kind of influencing, I guess, what HashiCorp may have on their roadmap. And as that divergence grows, there's a gap there that forms that could lead to another product or a completely different way of deploying that infrastructure, maybe with AI, like you mentioned.
Yeah, and that split's not healthy for either ecosystem because it does create that, okay, well, if I want to migrate from one to the other, I have to keep an eye on what's supported across both of them. They've been doing a decent job of keeping feature parity, but the actual implementation detail is going to be different. Because they can't copy code from each other, not directly at least.
But I think just that paradigm of declarative code and then this standard workflow for deploying and then updating your infrastructure, it's a good workflow, but it definitely has its size limitations. So when your state gets too big, everything slows way, way, way down before you do any kind of comparison.
You have to refresh the contents of your local state because Terraform needs a current picture of what your actual architecture, your deployed architecture looks like. It doesn't always know. And so it tries to do a full refresh of everything. Even if you've only changed one thing in your configuration, I'm changing my EC2 instance size.
That doesn't impact the VPC or the subnets or the security groups or the S3 buckets or like the 26 other things that are in my config. It impacts one resource, but during that plan, it still refreshes all 27 resources. That model doesn't scale. And when you want to take those things and break them apart into separate states, that is a painful and arduous process.
So I think on the short term, AI is just going to make writing Terraform and doing these kind of operational tasks a bit faster. But if we're looking longer term, I think there's whole new classes of solutions that are going to come out that maintain some of the core principles of what IaC does, but do it in a different way that's a little more efficient and a little friendlier.
So I did have a company reach out to me where it was deterministic AI creating infrastructure.
I don't necessarily want to give them a shout out because I haven't used the product yet, but I'm curious if that is like this net new possible way of deploying infrastructure that will be adopted or at least I'm sure they're trying to at least make sure that it is but yeah I'd be curious to see if there's new ways of deploying infrastructure it's interesting we're talking about like those calls years ago at a
Company I worked for we were 100% in on HashiCorp this is pre-IBM 100% in on Vault, Terraform, Packer all of it and we were we had all of our GitHub repositories in Terraform as well And we quickly realized that you can hit that API limit very, very easily.
And GitHub's is pretty restrictive if you're not on like an enterprise, like a higher level enterprise program. So yeah, we were then partitioning out repo creation and branch protection rules based on teams and other arbitrary ways of our partitioning out the state files. So we didn't have to hit those resource limits just by doing a...
Making sure there was no drift changes or making sure that we were updating branch rules. Yeah, that totally makes sense. Arduous and broken and yeah, just the wrong way to do it. Yeah, if listeners aren't familiar, like when you do a whole bunch of reads against GitHub's API, that counts against your rate limit.
So if you're doing a massive, if you're managing all these repos with Terraform and you try to do a single configuration update on one repository and it's pulling. 100 repos and all the settings in those, yeah, boom, you hit that limit and your plan fails. That's hugely frustrating. So I would like to see a solution that's a little more intelligent about the graph that exists and intelligently updating that graph.
The other thing that Terraform is predicated on, and most of the other ones are as well, is this idea of CRUD, or CRUDL, really. So the standard operations for any resource that exists is you can create it, you can update it, you can read, you can delete, and you can list out resources.
And that's really well documented in the providers, and it makes the providers extremely reliable because they only have these actions, and that's it. But if you work in operations, you know that's not the end-all and be-all of infrastructure. Those are not the only operations that are permitted. There's operations like... Running a sync or resetting a virtual machine or backing up a database.
And those are actions that don't fit neatly into one of those categories. Interestingly, HashiCorp has introduced Terraform actions to sort of try to help out with those non-CRUD operations. But I've been playing around with a new tool that's being developed by System Initiative called Swamp. And, you know, I'm not going to say that Swamp is the end all and be all.
But its approach is really interesting in that it builds models in the same way that you would build resources for a Terraform provider. But it doesn't have strictly defined actions that have to exist. And it's all TypeScript based, at least at the moment. So when you build out your model, your model for a resource can have a number of different actions. And you can call them whatever you want them to be.
You work with an LLM to write the TypeScript that is going to be behind those actions. So if I was managing an Azure virtual machine, I could not only create, update, delete, etc. I could also restart it. I could kick off an extension inside of it. I could do a backup of it. I could do all these other operations. And I can also have scoped operations. I could have an operation that is strictly updating metadata tags.
And then I could potentially... Hand out permissions that that's the only thing that a specific pipeline or operation can do is update those metadata tags. So I really, I'm enjoying, I'm experimenting with it now and I'm enjoying the model and the fact that it's really easy to extend because you can download somebody else's set of models. They call it an extension.
And then if the action you're looking for or the property that you need isn't in it, you can just work with the LLM and Swamp and and say this is what I'm looking for add it to the extension and it will write the TypeScript for you and test and validate it and if everything works out you can upload a new version of the extension for other people to use interesting so okay given that model with Swamp given where
Terraform is OpenTofu Terragrunt if you want to wrap all that and make it more dry where we don't know what the next evolution is necessarily of IaC that will be adopted by everybody.
How do you recommend preparing for that future? Is it smaller state files, larger state files? How do we go about logically building out the infrastructure with IaC tools in a way that will make that adoption easier in the future? Any truly viable solution is going to have an off-ramp from Terraform, basically.
CloudFormation and ARM templates too, but any real serious solution is going to have to have a way for you to migrate off your existing infrastructure as code onto this new platform. I don't see that happening, especially at enterprises, within the next five years.
I know the speed at which enterprises tend to move, and they're not going to throw away all the effort and the skills that they've put into building up their Terraform portfolio. For some new solution that's totally unproven. So if you're looking for like, should I even bother learning Terraform? Like, yeah, you probably should.
But I think the more important thing to learn, and this applies whether you're using Terraform or CloudFormation or Ansible or Swamp or like 10 other products that are out there, none of those matter if you don't understand what you're actually deploying. All of them assume that you have some level of knowledge of...
The different services and features that exist on the different cloud platforms and how they interact and work together. AI can help you with that a bit. If you describe the shape of the application you want, it can make some educated guesses on, well, that sounds a lot like a Lambda. You should probably implement that as a Lambda.
And okay, you're going to need, you know, an elastic load balancer to provide availability for these VMs, stuff like that. It can make some pretty good educated guesses. But I think...
Our job as, I guess, system administrator turned DevOps engineer turned AI prompt engineer, our job is still to understand the fundamental architecture behind all this stuff and be able to describe it and explain why we need certain infrastructure set up in a particular way. That skill is way, way more valuable than being able to write good code in any of these different languages because...
Regardless of what tool wins out, AI is going to be writing most of the code at this point. I know that's not like a certainty, but it's like a 90% certainty that writing code now is extremely cheap. Writing really good architecture documents is still very expensive. So I would focus on learning the fundamentals of the different cloud platforms and learning the fundamentals of like networking.
Of Linux, of databases, of security, and being able to take all of those different disciplines and turn it into a design document that can then be interpreted by an LLM. Okay, so along those lines, if you were talking to someone that was new in DevOps, cloud, where would you tell them to focus first? Kind of what you already mentioned, but I mean, where should... Where should their focus be?
Should it be on prompt engineering? And should it not be that? Should it be the fundamentals first and then bring AI into the mix after the fact so you know what you're vetting when you look at the output? Yeah, I feel like my value is the 20 plus years I've been in IT and just seen some stuff. And I don't think there's a way to speed run that experience.
And being hands -on, at least for me, is the best way to get that experience. So even though... I might never have to make a Cat 5 cable myself. I've done it. I've taken the crimper, I've separated out all the pairs, I've put it into the thing and crimped it up and tested it with a cable tester. I've done that. I know how that fundamentally works.
And even though it's totally useless because I can buy a Cat 5 cable for pennies on the dollar, I still think it was useful to gain that knowledge. And so... If I could compare that to like writing code versus having AI write it for me, you might never have to write another line of Terraform code in like a year or two.
But understanding how to properly compose a module, understanding how to organize code effectively, those are the things that are going to pay dividends when something goes wrong. Because that's in the true job of the SRE. Is to be there when things go wrong and they always will. And knowing what wrong looks like and being able to sort of guess based off of prior experience what the probable cause is.
Lots of different tools can help you with that. But those like the key troubleshooting skills is something you still need to hone on your own. That's fair. So what training in your experience actually sticks for people new? Is it labs, projects, incident drills, internal docs? What do you find works the best when working with junior engineers or people fresh? What works the best is to do it wrong. Okay.
That sounds weird, right? What works the best for me, and so I'll use I statements because this is what works best for me, is I only ever read like the first three pages of the manual and then I try to do it. And it never works right. It's broken. It's misconfigured. I'm frustrated.
But the process of reading the docs, figuring out where I went wrong, and then fixing it and redoing it correctly, I'm going to remember that whole process because it truly was a process. Clicking like a let's get started script and just having it build all the background for me, that means I'm never going to truly learn those things. At least, you know, like I said, for me, there's a reason why.
Kelsey Hightower wrote Kubernetes the Hard Way. It's not because it's impossible to script the installation of Kubernetes. Clearly that's not the case, right? But if you do it the hard way, you're probably going to screw it up. So you're going to have to recreate a certificate authority and reissue certs because you got that wrong.
And then you probably spun up your node wrong or you got the kubelet misconfigured or your networking is incorrect. And you're just going to trip over all these things. And as you trip over those things, you realize where the gaps are and where things can go wrong. And so when your production cluster falls over at 3 a .m., you're going to feel like you actually have a handle on what's happening inside that cluster.
As opposed to, I've only ever used this template and deployed automatically and never had to think about what happens if the networking gets screwed up within my cluster. That's fair. I mean, I find even still, I'm learning after 25 plus years that same way, right? By breaking things.
And I mean, even OpenClaw, if I wanted to, you know, trying that, it was playing around with it and letting it break things in a controlled environment. But yeah, I'm just playing with it. I, you know, I still find that I don't know everything about Kubernetes. I don't know everything about cloud. I can't. It's impossible. But being open to that, being open to that process and being open to it.
To getting it wrong is valuable, very valuable. Yeah, I've had three pillars. Well, I established three pillars for myself a long time ago. One of them was make yourself uncomfortable. So just be uncomfortable. Put yourself in an uncomfortable, difficult situation. Not too uncomfortable, but like step outside of your comfort zone, right?
Another one was be kind because everybody else is just trying to trudge through this thing called life as well. And the third one was... Be prepared to fail and be okay with it. Because failure is one of the ways we learn. And I am failing constantly. I spent last week fighting with this Swamp thing to set up my home lab.
And I had moments where I was yelling at the monitor and hitting escape and going outside to take a walk. But after a week, I used Swamp to rebuild my entire home lab. It was a big learning experience for me on how to use LLMs more effectively. To understand Synology's weird API is a little bit better. Don't get into that if you don't have to.
And also, I learned some new ways to capture web traffic and do analysis on it. I didn't even think that would be a benefit, but it was just like in the troubleshooting, I learned a whole bunch of new things. So I think being willing to experiment and being willing to try something and fail is the mark.
Of a lifetime learner and that's what we all are when it comes to technology you have to be a lifetime learner because the technology is not going to stop for you I'd be very dubious to trust any senior or plus level engineer that's worked 10 plus years in the industry and hasn't taken down production or has that as like a statement like oh I yeah I've never well if you haven't then you what have you really been
Doing like what yeah how deep do you actually work in the system or do you know the system um because I think we've all been there right yeah totally It's an unfortunate reality, but humbling yourself and working as a team and working through it rather than just panicking is when you learn probably the most, too, is in those scenarios, in those situations, not in the building of the POC or in the building.
I mean, those can be valuable as well, but or in building of the SOP, you know, building out your standing operating procedure that you may run or run book that you may use someday, but actually doing it and being in the incident. Is where the real work and value, I think, is built for the most part. Yeah, and I have a podcast. It's called Day 2 DevOps.
And that was sort of the inspiration behind the name is because day one or that POC, when you first deploy everything and it's new and it's shiny, yeah, it's probably going to hum along pretty well. But day two, when you actually have to start operating it and updating it and real traffic starts hitting it, that's when you learn how good your architecture actually was. And that's what we like to focus on.
In the podcast is kind of more realistic implementations of technology and less the hello world and the day one of what happens when you deploy tech. Very cool. So along those lines, where can people find out more about you find out more about your podcast? Any other things that you'd like to plug or talk about? Yeah, I mean, so I have two things I'd like to plug one is that podcast day two DevOps.
So if that sounds interesting to you, if you want to hear some interesting conversations. About DevOps adjacent technologies, Day 2 DevOps, all spelled out. You can find it there at day2devops.com. And the other thing that I'm currently doing is I'm doing a live stream every Thursday at 1 o'clock Eastern to help people study for the Terraform Authoring and Operations Professional exam.
So every week I cover a few of the topics that are part of the exam. And based on my mapping, that's going to be going on for a couple months because there's a lot of topics. So if that sounds interesting to you, if you thought about sitting the exam or you just have like weird questions about Terraform, go to my YouTube channel. It's Ned in the Cloud and check it out. 1 p.m. On Thursdays. Awesome.
I'll put links for all those in the show notes as well. Ned, thank you so much for coming on. Really, really appreciate it. Thanks, Brian. That was my conversation with Ned Bellavance, founder of Ned in the Cloud. And host of Day 2 DevOps. The biggest takeaway for me is that tools and terminology can make a team look much further along than it really is.
You can have stand-ups, pipelines, infrastructure as code, and an internal platform. None of that guarantees you are improving communication, shortening feedback loops, or shipping something the business actually needs. The same applies to AI.
AI can make writing Terraform and automation code much cheaper, but it does not remove the need to understand what you are deploying, why the architecture works, what permissions it needs, or what failure looks like in production. If anything, that judgment becomes more valuable. I also liked Ned's point about learning by getting things wrong, not recklessly in production.
But in labs and controlled environments where you can break something, troubleshoot it, and understand what was hidden behind the template. You cannot completely speed run that experience. So focus less on whether your organization looks like it is doing DevOps and more on whether the practices are producing better outcomes. Use AI to move faster.
But understand what it generates learn the tools but do not confuse the tool with the architecture and keep building the troubleshooting experience you will need when the clean demo becomes a day 2 production system you can find Ned in the Cloud and the Day 2 DevOps podcast through the links in the show notes Follow or subscribe to Ship It Weekly wherever you listen and find previous episodes at shipitweekly.fm. Thanks for listening and I'll see you later this week.