Ship It Weekly Transcripts

This week is a reminder that tooling is not neutral. The stuff we call tooling ends up holding secrets, holding budgets, and holding production together. And when that tooling breaks or gets repriced or gets abandoned, you feel it immediately, usually at 2 a .m. So today, a nasty N8n security issue that hits the workflow automation crowd, quietly raising prices on scheduled GPU capacity.

And a Netflix pattern that basically says stop running critical ops as fragile scripts. Hey. I'm Brian Teller. I work in DevOps and SRE, and I run Teller's Tech. This is Ship It Weekly, where I filter the noise and pull out what actually matters when you're the one running infrastructure and owning reliability. Quick housekeeping, shipitweekly .fm has all of the links and show notes.

And if you want to come on for a guest episode and talk through real DevOps, SRE, platform engineering, migrations, governance. Incident scars, whatever, there's a contact form on the site. And if the show's been useful, follow it wherever you listen. A rating or review helps way more than it should.

All right three main stories then the lightning round then a closer story one n8n which a lot of teams use as a workflow automation platform shipped a security fix for a critical issue that can turn into local file access and potentially worse depending on how you run it this is another your internal tool is part of your attack surface moment story two AWS is raising prices around EC2 capacity blocks for ML.

Basically, reserved scheduled GPU capacity. If you do any GPU work, cost assumptions just shifted. If you don't, it still matters because it's a signal. Constrained compute is not... Getting cheaper. Story 3. Netflix wrote about using Temporal to make cloud operations and deployments more reliable, claiming a huge reduction in transient failures. Even if you ignore the exact number, the pattern is gold.

Let's get into it. Story 1 is N8n. If you haven't run into it, N8n is a workflow automation platform. The Glue Tool. The thing that ties together Slack, Jira, GitHub, webhooks, cloud APIs, whatever. And that right there is why I care. A critical vuln dropped, CVE -2026 -21858, and multiple write -ups describe it as a worst -case type issue.

Unauthenticated attackers, remote code execution, full compromise potential, depending on how you run it. You'll also see it framed as unauthenticated file access via certain form -based workflows, which can lead to exposure of sensitive info and then escalation from there. Either way, this is not a patch -it -next -sprint situation. Now, the reason this matters isn't just because N8n is popular.

It's because the thing we keep doing in modern ops is we build these automation islands and then we forget they exist. Somebody sets up N8n or Zapier or a self -hosted workflow runner. It starts as just notifications. Then it becomes kick off a deploy. Then it becomes call this internal endpoint. Then it becomes rotate this secret. Then it becomes close this incident.

And at some point, that tool has enough power to wreck your week. This is the same story as CI controllers. Jenkins controllers, GitHub runners, internal bot accounts, chat ops systems. They're not just tooling. They're privileged systems. So here's the practical takeaway. And I'm going to say it like I'd say it to my own team. If a tool can trigger actions in production, it is production -adjacent infrastructure.

Treat it like that. That means three things. First, exposure. If you have this thing reachable from the public internet, you need to be able to explain why. Because it was easier is not a reason. Put it behind SSO. Restrict inbound, and don't let webhooks turn into open doors. Second, credentials. This is where people get killed. Workflow tools tend to accumulate credentials, like lint accumulates in your dryer.

A token here, a key there, just make it work. And then you've got one compromise system that can call everything. So the move is per workflow credentials and least privilege. If workflow A posts to Slack, it should not have the token that also opens PRs and it definitely shouldn't have something that can touch cloud IAM. Third, ownership and patching. Who owns it? What's the patch cadence?

What's the backup and restore story? Is it monitored? Are logs shipped? Does it have an incident plan? If the answer is, uh, I don't know, you've got an outage waiting for a calendar date. And one more thing that matters in the real world. After a vulnerability like this, I assume credentials might be exposed until proven otherwise. Rotate the meaningful ones. GitHub tokens.

Cloud keys, any API tokens used by workflows. You don't need to rotate your entire company, but rotate the keys that would ruin your month if they leaked. The shortest version of story one is automation is a control plane and control planes need breaks. Story two, AWS is raising prices on EC2 capacity blocks for ML.

Capacity blocks are basically AWS saying, if you need GPU capacity at a specific time, you can book it in advance so you actually get it. It's aimed at ML training and other accelerator -heavy jobs where best effort doesn't cut it. The reporting this week calls out roughly a 15 % increase for certain P5e .48 Extra Large and P5en .48 Extra Large capacity block pricing. There's also mention...

That one region had an even bigger jump, and AWS's stance is basically, capacity blocks pricing is dynamic based on supply and demand. It's reviewed quarterly, and this is what they expect this quarter. And here's why I care even if you don't touch GPUs. Because this is a clean example of something ops people learn the hard way. Cost is part of reliability.

If your ML platform team has runs scheduled and suddenly the unit price shifts, the impacts aren't just the bill is bigger. The impacts are priorities get changed, capacity gets reduced, experiments get delayed, and now you're dealing with delivery pressure and business expectations. Also, it creates internal trust issues. Finance thinks you blew the budget. Engineering thinks finance doesn't get it.

Leadership asks why it wasn't predicted. And the answer is suppliers can change pricing. And if you don't have visibility and tripwires, you only learn when it's too late. So what should a normal platform team do? Start with awareness. Inventory, who uses capacity blocks? What instance families? Which regions? And how much of your spend is sensitive to a rate change? Not your entire AWS bill.

Specifically, the things that could jump and matter. Then, communicate early. If this price increase starts January 2026, you want that message in front of anyone who forecasts spend. Heads up, same workload may cost more starting January. We're adjusting forecasts. Then, set a tripwire. Something simple.

Alert when the effective rate for a known SKU changes above a threshold, or when spend jumps without a matching usage change. The point is to learn about it this week, not at the end of the month. And the bigger philosophical takeaway is this. The more you build on scarce resources, the more you're building on market behavior. GPUs, specialized networking, reserved capacity, all of it.

This is why mature platform teams build two lanes, the premium guaranteed lane for workloads that truly need it and the good enough flexible lane for everything else. Because you want options when the market shifts. Story two is basically the cloud is still a supplier and suppliers behave like suppliers. All right. Story three. This one is my favorite because it's a real platform move.

Netflix published How Temporal Powers Reliable Cloud Operations at Netflix. And the key line that grabbed everyone is that Temporal helped them reduce transient deployment failures from 4 % to 0 .0001%. The context is operating Spinnaker and all the long -running workflows involved in cloud operations and deployments. Now, I'm not here to fact -check the exact decimal.

I don't really care, because I've lived the problem they're solving, and you probably have too. A ton of operational work is basically a workflow. It has steps, it has dependencies, it has waiting. It has retries. It has do this, verify that, and if it fails, back up. And most teams implement these workflows as the flimsiest possible thing. A bash script, a Jenkins job, a GitHub action with a bunch of steps.

Or worse, a runbook that assumes the human will do all of the careful stuff correctly while the world is on fire. Durable execution is basically the adult version of that. It's saying a workflow engine should remember state. Handle retries correctly, persist progress, and be able to resume without you guessing which steps happen. Here's the steal this now version. Pick one operational process that you hate.

One that flakes or takes forever or depends on a hero. Something like rotating certs across services, rolling a new AMI, doing a blue -green cutover, draining and replacing nodes, promoting a release across regions, migrating a database with guardrails. Now ask, if this process fails halfway through, can we resume cleanly? Or... Do we restart and hope we don't double apply something? Do we know which step we're on?

Or do we just see job failed and now everyone is scrolling logs? Are the steps item potent? Or are we creating duplicate resources and then cleaning up later? If you answered no to any of that, you're already in the category Netflix is talking about, just at a different scale. And you don't need Temporal specifically to adopt the mindset. Temporal is one implementation.

The mindset is treat critical ops like software, not like rituals. Make the workflow explicit. Make it resumable. Make it observable. Make it safe to retry. And then... Build it so the average on -call engineer can run it without needing the one person who wrote the original script three years ago. The reason this is such a big deal in 2026 is because automation is accelerating.

Agents are going to write code and open PRs and trigger pipelines faster than humans can reason in real time. So if your underlying workflows are fragile, You are basically giving a faster engine to a car with bad brakes. That's not progress, that's just more dramatic failures. This is why Story 3 matters. It's about making ops workflows reliable enough that speed doesn't kill you. Alright, lightning round.

First, Kubernetes Dashboard. The Kubernetes Dashboard project is archived. And no longer maintained. And the SIG UI note basically says there aren't active contributors and maintainers to keep it going. They recommend Headlamp as an alternative. And the repo itself is now marked archived. This is one of those reminders that ops dependencies die.

If your org still relies on Kubernetes dashboard as part of your standard workflow, you should migrate now while you're calm and not later when your security team shows up with a clipboard. Next. Docker Hardened Images. Docker's pushing a story that hardened images are now free and open source under Apache 2 .0.

Positioned as a secure, minimal, production -ready set of images with stuff like SBOMs, Providence, and SVE transparency baked in. You don't have to buy the entire pitch, but the direction is right. Containers are supply chain. Base images are supply chain. If you're still building on random community images and hoping for the best, you are basically taking on risk you don't need. And finally, PipeDash.

It's a desktop app that aggregates CICD pipelines across providers so you're not bouncing between GitHub Actions, GitLab CI, Jenkins, etc. This is a small quality of life thing, but I love anything that reduces where do I even look during incident weeks. For the closer, here's the human thread across all of this. We are entering a phase where speed is cheap, automation is cheap.

Agents will make changes faster than humans can keep up. The hard part is designing systems where speed doesn't turn into security incidents, budget blowups, and we don't know what just happened outages. N8n is the story of speed without perimeter thinking. AWS Capacity Blocks is the story of speed without cost reality. Netflix with Temporal is the story of speed with the boring engineering to make it survivable.

And that's the mindset I'd take into 2026. Don't just automate. Make automation resumable. Make it observable. Make it least privileged. Make it safe to stop. Because if you build a faster machine without brakes, it's not going to fail gracefully. It's going to fail loudly. Alright, that's it for this episode of Ship It Weekly.

We covered the critical N8N vulnerability and why workflow automation tools are basically control planes now. We hit AWS raising capacity block pricing for GPU workflows and why cost is part of reliability. And we talked about Netflix using Temporal. And the bigger pattern of durability execution for cloud operations.

The lightning round was Kubernetes dashboard getting sunset, Docker hardened images going free and open source, and a pipeline dashboard aggregator. If you got something out of this, hit follow or subscribe wherever you are listening. Links and show notes are on shipitweekly .fm. And if you want to come on for a guest episode, hit the contact form on the site. I'm Brian, and I'll see you next week. Thank you.

Thank you. Hey, I'm Brian Teller. I work in DevOps and SRE and I run Teller's Tech. Ship It Weekly is where I filter the noise and pull out what actually matters when you're the one running infrastructure and owning reliability. If something's just hype, we'll call it hype. If it changes how you operate, we'll talk about it. Most weeks, it's a quick news recap.

In between those, I drop interview episodes with folks across DevOps, SRE, and platform engineering. Today is one of those interviews. Today, I'm joined by Danny Teller. He's a DevOps architect at Tipalti, and we're going to talk about internal developer platforms and why he thinks we're headed towards more internal IDPs instead of just backstage everywhere. Danny, can you introduce yourself? Yeah, definitely.

Thank you very much, Brian. Well, thank you for having me, first of all. As I stated, I'm the DevOps architect for Tipalti. Tipalti is a... Rather a medium -sized business company that is in the fintech world. And most of the things that I handle is working with our developer teams and our architects to kind of steer the direction of the company from not a software perspective, but a DevOps perspective.

And that also includes platform engineering because we can't have just one of them anymore. So you got to go with both. Yep, for sure. So one of the things that made me want to reach out to you was about this talk about IDPs. And I'm curious your thoughts on...

Idps like what what are your thoughts on where do you think idps are headed for everyone else internal developer platforms I know idp is an acronym for for many different things uh in the devops identity it could be identity it could be yeah yeah what's your thoughts well it's obvious that in recent years there's been a rise of platforms in any company you can take backstage for example which was open sourced and

Everybody can take it and use it as they see fit And then in recent years, we had rising companies like, what are they called?

Northfolk, I think, or Northflank. And you have Port .io, which are offering a huge integration suite. And so on. And many companies that started to buy into platform engineering. There were already platform engineering teams across different companies, but they were doing something internal. Some of them transitioned from DevOps into DevX, which are more...

Program oriented rather than ops oriented and they started picking up things like backstage and introducing it to the organization to you know simplify the way developers work because one of the core things you have in platform engineering is reducing everything from developer perspective from operations perspective and developers lifetime to literally nothing so they have nothing to worry about but the golden

Standard can be like he types in something and into chat and he gets everything done behind the scenes that's kind of like that golden mantra, if you will.

So we have all that rising every day. And that's why things like Backstage are highly adopted and have a great community. That's why we have great products like Port .io and other ones and so forth. What do you think changed that makes internal tools more realistic now? I think with all the changes recently in the AI world, in Gen .AI. It kind of simplified how we view IDPs.

If you're talking about last two years, then we've been constrained to companies and open source solutions. So if you take Backstage, which is a great product when it works, which is a great sentence to say. So when it works, it's great. But when something breaks because it's so tightly knit together with Node .js and TypeScript and all the... Stuff it's written in, it makes it very difficult to maintain.

So unless you have developers that specialize in that and also understand infrastructure, then you can't actually maintain it. Every little thing breaks it. You always have to rebuild the image and it makes it very difficult to manage. Even if you look at Backstage, Portal Backstage, that recently launched from Spotify, the SaaS offering, this is what they do behind the scenes.

They explicitly said that whenever you make a change to Backstage, we rebuild your image. So they took what you're doing already and just call it a SaaS platform. And it's written just in their docs. It's a little funny.

I don't know if the docs are still written this way, but I do remember Spotify saying too that they recommend at least like six full -time developers managing backstage, which is just cost prohibitive for most companies. I mean, they just can't afford. But that level of staff to run an internal platform. Yeah, I mean, the DevEx team should not just be doing backstage.

They have to also develop more internal tools for the company. I mean, whatever DevEx engineers do is usually with creating, you know, like the pre -commit hooks, for example, like they were created. I mean, it's kind of a thing for platform engineering to have or DevEx developer experience to have. So this is what they do day in, day out. And then came stuff like IDPs.

And they made things a lot easier because you have a central control plane that can be integrated into virtually anything. And you can see a developer just working with one single pane of view instead of tens, which include metrics, traces, monitoring, deployments, pipelines, so on and so forth. So it's great. The idea is great. But then you're encountering a bigger problem. And that's when you start to manage it.

And because these teams are very... Oriented in writing it, that's great. But now that you have Gen AI, you can kind of cut that in half. Even your platform engineers, even your classic DevOps engineers, they can do it on their own because they are familiar with the tools like Backstage. They are familiar with tools like kubectl. They are familiar with Terraform, so on and so forth.

So even DevOps that are not trained as software engineers do know how these things work. So specking it out with tools like SpecKit or creating plans with Cursor or Cloud Code and Skills becomes very easy. Instead of, so you can just ask it, okay, go ahead and study how Backstage works. Let's create a Pythonic Backstage instead. Makes it also just as easy to do. Yeah, that makes sense.

So what do you think Backstage got right? I think Backstage as a concept, it's great. I think it's really, it's one place where you just throw many integrations. And it just works. And people have one centralized location for that. I mean, they got that right. The idea is great. Condensing also tech docs in there, which is my favorite feature in backstage. I love that feature.

And the tech radar that they have there, it's just great. It's a great user experience. When it works and you see it live and everything, you just want to buy into that. But when you start managing, that's when you lose it. So can you give me an example? Like, what did you build and what problem exactly were you solving with that build out? Yeah, so, well, we at first tried working with Backstage. We did.

We ran for a whole year. But given that we've had many projects in the same time, so managing Backstage kind of fell apart. And we had a couple of our three engineers trying to manage the backstage as a whole and doing their own thing in developer experience as well. And it just fell short. So you can maintain that for 350 developers. It's insane.

And that's not including the DevOps team, which was also trying to help at the time. So even though we integrated stuff into that, we tried to work with it, it just proved to be a maintenance nightmare. So it dropped off. And then you start looking into other products. You see like Port, like I mentioned, and other ones like Northflank and so forth. And great. So they can sit in your system.

They can integrate their SaaS platform. But when you start seeing how they work, when you start seeing if you can do it on your own, because the plugins are already there. They're already online anywhere. And even if you point an LLM to certain plugins and tell it to learn it and then recreate it, you can just do it with zero maintenance. It becomes super easy.

Barely an inconvenience yeah I so at my last company we actually ran roadie for a while which is another sas backstage like offering and I found that it was good at helping us get set up quickly but beyond that um and we were very immature to the whole idp process at the time this was a new experience for us we were just moving from a devops team into a platform like devex experience team but I did find that like I I

Don't really know what it really bought us at the end of the day and it may be a better product now this was a few years ago but I don't know what it really bought us above what backstage could offer or or any other like you mentioned another sass um in the adoption yeah definitely I mean you don't know you can't tell until you actually try it but for us anyway all these products they didn't they just fell short so

We decided to try something on our own because well At the end of the day, through IDPs, I'm supposed to be able to provision whole environments per se.

I'm supposed to be able to see every other environment that I have as a developer, even up to production and so on and so forth. So in the initial stage, we decided to drop all the other ones and focus on the real pain for our developers. And that was actually developer environments. It's something that's been brewing in our company for a very long time. We've been trying to do it for a very, very long time.

And it proved to be very difficult. Because of the nature of FinTech and all the stuff that you need to work with and maintain all the time, it just becomes a small developer environment becomes very, very large. So we tried to minify that as much as we could. And eventually we got to something that you can work with. It wasn't 100 % of the entire product, but 85 % is already good enough.

I mean, missing a couple of features, but it's already something a developer can use to work with. So we went with the classics. We decided to go with Argo CD, cross -plane, and all the infrastructure is going to manage it in the first stage. And that's going to start with that. You just commit a normal YAML and you get the entire infrastructure in your backend. Everything.

You can gain all the databases, all the stuff that you need to run the application smoothly. You get it out of the box. Everything is prepared using best practices from Platform Engineer. Golden images, golden charts, if you will. Networking, all clusters, standalone clusters, not namespaces. Even monolithic machines if we have to. So everything is bundled together.

So in about 30 minutes, you can spin up a whole semi -product environment. And the next stage after that is, well, you know, connecting an AI to that. Because if today I'm spinning an entire flow, everything the system has, I'm just saying, go ahead and use it.

So we're trying to play with various ways to see how we can ask an LLM like Claude or ChadGBT through Slag and say, listen I need this and this xyz flow go ahead and create that for me so instead of creating the entire flow which is say five to six machines including any uh kubernetes cluster and some old stuff legacy machines we can dumb it down to something really really small and the way to do that is playing with

Llm you have to make sure your model can actually access all these flows and to see them and compile exactly what it needs and then request that from from crossplane in our ground So this is something we're toying around with to see how we can do that.

But we have kind of success in that area because we've developed quite an idea for that. And it proves to be very useful for now. So it goes great, really. I don't mind sharing it. It's fairly straightforward. Is it just an MCP server that can reach out to Argo and kubectl? So pretty much behind the scenes, you have the Slack bot, which connects to Cloud or Bedrock or any other one. In turn, it does use an MCP.

So that MCP actually is a tool that calls to GitHub, and it kind of rags all the repositories back to itself using a small depth. Or actually, it just takes a specific file. That specific file is kind of an XML structured file that says, this is the repository. This is what it contains. This is the flow. This is the database needed. These are their names.

This is kind of what this repository represents and how it works. So when we use requests, I need payments flow, for example. Then it's going to pull all these repositories. They're actually relevant repositories because we have a RAG map of all of these repositories. It just pulls all these relevant files, piles that into a list of flows, repositories and flows.

And after that, it just communicates with using another MCP with Crossplane and Argo, submits a YAML that it needs, and then provisions it for the developer. Very cool and interesting. So going back to these developer environments that you set up, I'm assuming they're ephemeral in nature, like because you're using Crossplane to provision everything. Yes, we're using also a TTL controller as the annotation.

Basically, you get an environment from 8 to 5. That's it. And if you need to extend it, you can. Interesting. Are you using Terraform to provision the initial infrastructure or is everything through Crossplane? Everything is Crossplane. Okay. Interesting. And why did you choose that approach? Is it just to be closer to the actual application in Argo?

Yes, I wanted to use Argo as much as I could because it just makes sense. You already have the provisioners within Crossplane. So I don't need to toy with anything else. And eventually, if the developer in the first offering just submits a YAML file, which is, what, five lines long, then already they have something reduced instead of just provisioning through a pipeline and everything.

Just git commit and you get an environment. It's far simpler, far better, and they always get the same thing. That's very cool. So, okay, backing up again. Sorry, going back to the developer platforms and this new approach that you, you know, you went the SaaS way, it didn't work. You decided to do this internal thing. How did you get adoption from the overall engineering team? It's quite easy. Okay.

It's quite easy because it's a pain that we had to solve. So this is a major, sorry, even pain in the ass of developers since they couldn't have normal developer environments. So once you sell something that gives them exactly what they need. Which is environments, so it's kind of easy to roll them in. So they have one place. They're not even looking. All they get is this environment that they're connected to.

They have their own Argo CD set up in that environment. They can provision whatever they want. And the second phase, which will connect the LLM, will be even simpler for them. So every developer today wants to work with as less screens as possible. They just want to stick to their VS Code, their Visual Studio, JetBrains, whatever, and just stick their nose in there all day long.

They don't want to look at metrics and stuff. Just query this. I want this from QL to tell me if there's a problem in my app at 2 p .m. To 1 a .m. That's what they want.

So I think that it's easier to maintain that locally using a DevOps team, platform team, or even if you want to go even further in AI ops team, because I do believe that using things like agent core and bedrock is going to be the future of these things. You're not going to see more. More of graphs lying around or backstage lying around.

Because if you can get it through text, and that text can also generate an image for you, which is pretty damn good, by the way, for now, from nano banana, so on so forth. So you're kind of losing the entire idea of having platforms like backstage or other ones, all you need to worry about is integrations. That's interesting. That's an interesting idea. I'm curious, what was the biggest surprise with this approach?

Was it the tech? Was it the ownership model? Was it like change management? Was there anything that was a surprise, I guess? No, actually, there were no surprises. We planned it thoroughly because we knew what we wanted to achieve, how and when. And the biggest thing is what we're struggling with the most is what most people are struggling with. And that's fine -tuning the models to get exactly what you want.

So we're constantly working also with AWS engineers because we're also working with Bedrock all the time and we're leveraging that. We're kind of back and forth with them all the time for the support. And it is working. We are getting better and better at achieving what we want.

So, okay, let's say a listener was listening to this and they thought the internal integration aspect is really cool, but they've never set up an IDP before. They're deciding between...

That approach or or backstage what what signals would push them one way or another in your opinion I think it's maturity definitely maturity because I can also vouch for my for what he did and myself as well that we started off like everybody you go with things that are well known and instead of creating something on your own you want to test these waters first because maybe you'd be surprised that you're using

Backstage in such a small manner that maintaining it does give you the benefit of everything you need and you don't change it as much you maybe update a version here and there but you don't need all everything it can do or if you go and you find that you work with companies that give you the idp experience and that's also good enough for you and you can you can stick with that and it's great I mean if it works for

You then stay there but when you get to these points where everything they give you is just not enough that's when you start working by yourself That makes sense.

So what's one piece of advice that you would give like platform or DevEx teams that are wanting to start an IDP come 2026? I would actually leverage as much as AI as I can with that and also work with AI as best as I can. I mean, even the recent features of Cursor that they support skills from Claude or using Gemini CLI or knowing which model.

Is best for what kind of work after I get this through this hurdle and I understand what's going on and how to use it I will then go after idps I unders I will have the models understand and map out how they work and then start fixing my own ideas around that very cool all right let's shift gears a bit I wanted to talk about talk about two other things but uh that were non -idp related uh one was your mongo atlas I

Think it's called m atlas yeah yeah the mat list kind of curious what caused you to build this and was it because like terraform operator limits weren't enough or like what what was the reasoning like what were what pain point were you solving frustration okay yeah so the story starts with uh mongo when we started working mongo we had our terraform modules everything was up and running great but at some point the

Drift was just too large to maintain from um from Terraform perspective, from Terraform, the Mongo modules that we built ourselves, and what was actually within our Mongo Atlas cloud.

So that was a little difficult to manage. So we said, okay, we have all these resources, and doing Terraform import is not a nice way to do it. It's error prone. So we decided to let that go and see if we can do it in a different way. We tried exploring the Atlas CLI, but it gave us only a management perspective creation. Creation experience. It was very difficult to work with.

And then we decided to try the Kubernetes operator. And then we found two big issues. You can't own existing resources like you do in Terraform import. So if you create a similar YAML with a similar name resource and everything, it just says, no, it exists. You can't do anything with it. And the other part is it was semi -limited to what we needed to do.

So given all these issues and the lack of abilities to work back from existing infrastructure, it kind of led to the creation of a MATLAS. MATLAS kind of tries to combine the user experience from kubectl and also the abilities of Terraform. So in a nutshell, you can basically say, okay, I've got this organization in Mongo Atlas. I want to map all the infrastructure. All I have to do is hit a button.

And well, I'll hit a feature. It's called Discover. And it just pushes everything into a YAML file. And moreover, you can transform that YAML file using MathMath as well into a workable YAML. Ah, interesting. That's cool. Yeah. And since I don't rely on states at all, because I thought that it would be counterintuitive.

So if you change something because a lot of people may go on to atlas because of a problem or whatever and start changing settings because it does happen it's a database you don't know what's going to happen the next day so I left that part intentionally out so if you change something online you can just update that yaml again with the discover feature and have the same thing already as infrastructure as code if you

Apply something and then you can also do it in a surgical manner like a partial reconciliation so I don't break everything that's already existing in mongo atlas so people can find matless on github and they just search up okay awesome yeah definitely I mean it's kind of I've recently uh read from platform engineering labs that they have this tool called form a or form I and it kind of does it it's it's a tool

Written in pkl in app in apple language and the idea behind it is great it's kind of just trying to remove what terraform does and also do the partial reconciliation and the discovery and everything so when I read about that it reminded me of like I'm pretty much doing the same thing just in go and not in pkl interesting I have not heard of that tool interesting very cool it's fairly new it's like six months old I

Believe and then uh so lastly I wanted to just touch on this a little bit you are a a golden jacket uh holder I'm curious what that process was like going through all the certifications well it was fun and it was tedious at the same time But there was one aspect of it which really frustrated me a lot.

And I do hope that AWS will listen to that and do change it in the future because they do have the capacity to do that. One thing that really pissed me off about the exams is they're really outdated. That's the one thing that really hits me every time. For example, you have the DevOps engineer exam, which I did that about earlier this year.

When I already did that exam, the code commit service was already taken offline. Yet the exam had five to six questions around that. So why are you asking me questions about a service that no longer exists? Oh, that's frustrating. Yeah. Yeah. So, and the thing that also bugs me on that is they constantly say, yeah, go to skill builder and read the docs all the time. Read the docs. Okay.

The docs said the service is not existing anymore. So why should I learn about it? Yeah. Well, then you have situations like CodeStar going away and then changing. There's a lot of inconsistency. I agree with that. Yeah, and that's really the real frustrating part. But apart from that, it's fun to do.

I mean, it does give you a lot of perspective on AWS services and how it works, especially when you read the docs and try to understand further than what's actually asked about in the exams. And it's great. I mean, it's a great feeling when you finish all of that and, you know, kind of just putting it behind you. Well, congratulations. I mean, that's an awesome accomplishment.

What recommendations would you have for listeners that maybe want to head down that road of trying to pick up all the certifications? I would start from the very lowest ones. I mean, it's true what they say about the discounts. So start from the very first ones and get that 50 % discount for the next one. And it does train you for the other exams because you get to understand how these exams are structured.

Yeah, of course, you can go online and find these dumps or whatever people really have out there. You can do that. But once you really do these exams and even fail once or twice these exams, you understand the structure and it's quite easily understandable for the next time you go ahead and do that. Yeah, very true. Very true. All right. Anything you'd like to leave our listeners with? Any recommendations?

Yeah, actually, I would. Yeah. I mean. There's one thing that's been on my mind quite lately, and it's regarding for all platform engineers, developers, DevOps teams, literally quite everyone. And it feels that with all the rise and changes in Gen AI, a lot of people think that they're staying far too behind.

And I believe that because of these changes, we should strive to really get our knowledge up and running in our heads far faster than we usually should. And yeah, subscribe to these magazines, TLDR, so on and so forth, and read more articles and so on. So because it does give you the benefit of staying ahead of others. And you need to do that not because of job security, because it's just that interesting. Yeah.

Yeah, I think anyone that's in SRE, DevOps, platform engineering. You have to have a thirst for knowledge because you're always inundated with different tools, CI, CD processes, different ways of doing things. And if you're not staying on top of that, then you really are going to be behind. And the thing I always come back to is, you know, I do a big process or I'm rolling out a new AWS account, a new way of.

Provisioning basically a new poc that I'm doing what I did five years ago for that poc is not what I would do today I would architect it completely different than I would have just five years ago maybe even two years ago so yeah you you kind of need to always have that that thirst and hunger for learning and just staying on top of new things. You can't stay on top of everything, obviously.

I mean, that's also, you know, the reality is imposter syndrome does exist. You can't know everything and that's okay. That's why they're teams. That's why we're on teams and we're collaborators, you know, but just trying to be present, I think is really important. Yeah, most definitely. Awesome. Well, where can people find you online? Mostly I'm on LinkedIn. So that's the best way to reach out to me.

Some people already have, so. Awesome. So Danny Teller on LinkedIn, if you want to reach out. I'll also leave your MATLESS GitHub link in the show notes. And thanks for coming on, Danny. Really appreciate it. Thank you for having me. I had a great time here. Thanks, Brian. Thanks. All right. That's the interview with Danny. Quick reminder on the format.

Ship It Weekly is still the weekly news recap, and I'm dropping these guest convos in between. If you want to catch both, hit follow or subscribe wherever you are listening. And if this episode was useful, share it with a platform or DevEx friend and leave a quick rating or review. It's annoying how much that actually helps the show. We'll be back later this week with a regular news episode. We'll see you then. Thanks.

You can automate more than ever right now. Agents, pipelines, self -healing systems, all of it. But if you don't build brakes and blast radius controls at the same time, you're just speeding up the failure too. That's the theme for today. Hey, I'm Brian Teller. I work in DevOps and SRE and I run Teller's Tech.

This is Ship It Weekly, where I filter the noise and pull out what actually matters when you're the one running infrastructure and owning reliability. If something's hype, I'll call it hype. If it changes how you operate, we'll talk about it. Happy New Year! It's January 2nd, so if you're listening on a walk, cleaning up a lab, or just avoiding thinking about Monday yet, welcome.

Quick housekeeping, the site is shipitweekly .fm. That's where the links and show notes live, plus the contact email. Also, I'm looking for interview guests. If you're building real infra, platform, SRE, DevEx, governance, migrations, any of it, and you want to come on for a chill conversation episode, hit the contact form. On shipitweekly .fm.

And one quick ask, if the show has been useful, follow or subscribe wherever you are listening. And if you've got 10 seconds, a rating or review really helps way more than it should. All right, three main stories for today. Cloudflare's Code Orange resilience plan and their whole fail small approach. It's basically the internet's biggest edge provider saying we're changing how we ship and how we contain blast radius.

Second, Pulumi making a very direct move at Terraform and HCL. This is about the IAC control plane. Governance, and what the next year looks like for teams trying to standardize without rewriting the world. Third, Meta's DRP, a root cause analysis platform at scale, not we bought a tool. More like we turned incident investigation into software. Then we'll do a lightning round with a few practical updates and tools.

And we'll close with a human story about the ironies of automation and why faster systems can actually increase the burden on humans if you don't design the oversight part. Let's get into it. So Cloudflare published a post about Code Orange and Fail Small. And the vibe is basically, we've had enough of big correlated failures. And I love that they're saying the quiet part out loud.

When you run global infrastructure, a lot of the stuff that hurts you isn't a single server dying. It's one config change. One policy tweak, one rollout mistake, and suddenly you have the same failure everywhere at once. That's the nightmare. Correlated failure. So fail small is the strategy to force problems to stay local. If you ship a bad change, it should not be able to take down everything.

It should hit one slice, one region, one cell, one shard, one whatever, and then you learn and stop it. That's the ideal. Now, the reason this matters for normal humans like us is because this is the same exact pattern inside companies, just scaled. Down. One shared Terraform module change breaks 50 repos. One CI permission tweak breaks every pipeline. One harmless IAM policy update knocks out a production deploy.

One bad external DNS change takes out half your ingress. You know the vibe. Cloudflare's post is basically a reminder that blast radius control is a feature you have to design and defend. It doesn't appear automatically. And it's also a reminder that reliability work often starts with how do we ship changes safely, not how do we add more dashboards. If you want something...

Actionable from this, here's the mental model I keep in my head. If you cannot confidently say this change will only affect a small part of traffic, then you need better segmentation or better rollout controls. That might mean canaries. It might mean progressive delivery. It might mean feature flags that can be disabled quickly.

It might mean splitting a giant shared module into smaller modules so one change doesn't cascade. But either way, the goal is the same. Make failure smaller than your ability to recover. And going into a new year, this is a good story to start with because everyone comes back from the holidays with energy and plans and big changes. And the fastest way to ruin January is to ship a big change with a big blast radius.

So that's story one. Cloudflare is doing the grown -up version of build guardrails for change. It's not glamorous, it's mandatory. All right, story two is a little more control plane politics. Pulumi dropped a blog post that's basically a direct play at Terraform and HCL shops. If you zoom out, this isn't just here's a feature.

It's them saying you can manage your Terraform and your HCL inside Pulumi's ecosystem, and you can migrate without rewriting everything, which is smart. Because the big friction for most platform teams is not that Terraform is bad. It's that standardization is hard and change is expensive. A lot of teams want stronger guardrails, better policy enforcement, better visibility. Better workflows.

They also do not want a two -year rewrite project where you rebuild everything in a new language. So Pulumi is trying to reduce the rewrite tax. They're basically saying, keep your existing Terraform, but move the management layer. Use Pulumi Cloud as the place you orchestrate runs, policy, approvals, drift. Governance.

And they also mention native HCL support inside Pulumi's IAC, so HCL isn't treated like a second -class citizen. This is why this story matters for your audience. It's not about who wins IAC. It's about control planes. In 2026, the platform conversation is increasingly about who owns the workflow and guardrails around infrastructure changes, not which syntax you prefer.

And it ties directly to what we talked about in episode eight with Cloudflare's IAC governance and shift left patterns. Same idea. You want infrastructure changes to be reviewed, tested, policy checked, and auditable before it touches production. Now, if you're a Terraform shop, I'm not telling you to run to Pulumi. But I am saying this is worth watching because it's a sign of where the market is going.

Teams want guardrails. They want policy. They want better workflows. They want less state pain. They want less secrets leakage. They want less, oops, I applied the wrong workspace. And the vendors are fighting for that control plane. Practical takeaway here.

If your IAC workflow is still people run applies locally, or we have a fragile CI pipeline that's half tribal knowledge, then you're going to feel this pressure all year. Whether you solve it with Terraform Cloud, Spacelift, Atlantis, GitHub Actions, Pulumi, or something homegrown, the point is the same. The workflow is the product. Alright, Story 3 is my favorite kind of SRE story.

We built a system to do the thing humans keep doing manually. Meta wrote about DRP, their Root Cause Analysis Platform. And the idea is actually pretty simple. But the execution is what makes it interesting. When an incident happens, teams do the same basic investigation patterns over and over. You check recent deploys. You check configuration changes. You look for correlated alerts. You compare time series.

You scan logs for known signatures. You look for dependency failures. You try to answer the same questions every time. And that's slow. It's manual. It depends on who's on call. It depends on tribal knowledge. So Meta built DRP to codify investigations into analyzers. Think of an analyzer as an automated investigator that knows how to pull signals, correlate timelines, and produce a hypothesis.

They describe it as a platform that runs these analysis at scale. A lot of teams, a lot of analysis, all the time. Even if you ignore the big headline numbers, the approach is the real story. SRE, maturity, isn't just more runbooks. It's turning repeated operational work into software. This is the same idea behind good incident tooling in general. You don't want hero debugging. You want systematic debugging.

You want the first 15 minutes of an incident to be consistent no matter who's on call. DRP is basically that idea productized internally. So what's the takeaway for normal shops? You don't need a meta -level RCA platform, but you can steal the pattern. Take the top five recurring incident types you see. The stuff that keeps happening. Then ask, what are the first three questions we always ask during those incidents?

Then automate those questions. That could be as simple as a Slack bot that pulls recent deploys for this service, last config changes, related incidents, top error rate deltas, and posts it automatically in the incident channel. Or a runbook that starts with real queries and links, not vague instructions like check the logs. Or a...

A simple script that generates a timeline, pulls the right dashboards, and collects the evidence. You don't need to solve everything. You just need to stop doing the same manual steps forever. This is a great new year story because it's a reminder that SRE work is supposed to reduce human toil, not just respond faster to pain. Alright, lightning round. First up, AWS ECR can now create repositories on a push.

This is a small feature that can save a lot of time in orgs that spin up a lot of services, environments, or ephemeral projects. Instead of manually creating the repo first, you can push and ECR can create it. But, as always, convenience is a governance problem in disguise. If you let anyone auto -create repos with any naming pattern, you'll end up with a junk drawer.

So you probably want guardrails around who can do it, what names are allowed, and what defaults get applied, like scanning, lifecycle rules, encryption, that sort of thing. Next, GitHub put out a Let's Talk About GitHub Actions post. I like these when they're less here's marketing and more here's how the platform is evolving.

It's worth skimming if you live in actions and you want to understand where they're investing. Especially if actions is your company's CI backbone. Because small improvements in reliability and performance turn into real saved time during incident weeks. Quick callback. Episode 6 covered the runner pricing pause. Same takeaway still holds. Actions isn't just CI. It's a control plane.

Next, Lauren Hochstein had a great SRE -ish post about incident data and statistical process control. The short version is, MTTR is usually not under statistical control, which means the average can be misleading. If you've ever had one weird, horrible incident blow up your average, you already know this. The point is not stop measuring.

The point is don't drive the car by staring at one metric that gets yanked around by outliers. Look at distributions, look at trends, and focus on repeatable improvements. Next, Drift Hound. This is a tool aimed at drift detection and visibility for terraform.

The reason it's worth mentioning is that Drift is one of those problems that gets worse as your infra footprint grows, and it quietly undermines IAC as a source of truth. Even if you don't use this tool specifically, it's a good reminder to revisit how you detect Drift and how you respond to it, because we'll notice eventually is not a strategy. And last, Superset.

This one is interesting because it's about running multiple coding agents locally in parallel, like a manager for AI agents concept. Useful, but also a good reminder that agentic tools amplify both speed and risk, which is a perfect segue into the human story. For the human closer, I want to pull from Uwe Fredersen's AI and the ironies of automation.

The core point is something we keep learning the hard way in operations. Automation doesn't remove responsibility. It moves responsibility. If a system gets faster, the human oversight problem gets harder, not easier. Because now the system can do more things faster, including the wrong things. And the human has less time to notice.

Less time to understand and less time to stop it if you've ever been on call during an automated rollout that goes sideways you know this feeling you are watching the machine do damage at a speed you can't match so the real job isn't just add automation it's design the control loop how do you detect that automation is going wrong how do you slow it down How do you contain the blast radius?

How do you make it observable so humans can understand what it's doing? And how can you make rollback as easy as rollout? This ties back to everything in this episode. Cloudflare's fail small is literally about making automation safer by limiting correlated impact. Pulumi's control plane move is about controlling the workflow.

Policies and safety rails around changes meta's drp is about automating investigation but in a way that makes humans faster and more consistent not replaced and blind and the agent tools in the lightning round are about the new frontier of this if you give an agent the ability to write code open prs run ci or touch infra you need to treat it like a junior engineer with a caffeine problem fast, helpful, and absolutely capable of doing something dumb if you give it too much freedom.

So going into the new year, here's the mindset I'd keep. Automate? Yes. But build the brakes at the same time. Alright, that's it for this episode of Ship It Weekly. We covered Cloudflare's Fail small push. Pulumi's move to manage all of your IAC, including Terraform and HCL. And Meta's DRP platform for automated incident investigations. The lightning round was AWS ECR auto -creating repos on push.

GitHub Actions platform direction. A better way to think about incident metrics, drift detection, and a quick look at multi -agent tooling. If you got something out of this, hit follow or subscribe wherever you are listening. And if you can, leave a quick rating or review. It helps a ton. Links and show notes are on shipitweekly .fm. And last reminder, I'm looking for interview guests.

If you want to come on and talk through real DevOps, SRE, platform engineering, cloud governance, migrations, incident scars, whatever, hit the contact form on the site. I'm Brian. Happy New Year, and I'll see you next week.

Hey, I'm Brian Teller. I work in DevOps and SRE and I run Teller's Tech. Ship It Weekly is where I filter the noise and pull out what actually matters when you're the one running infrastructure and owning reliability. If something's just hype, we'll call it hype. But if it changes how you operate, we'll talk about it. Most weeks, it's a quick news recap.

In between those, I drop interview episodes with folks across the DevOps world. This is one of those interviews. Today, I'm joined by Eric Pate. He's a cloud and DevOps engineer. He has a background in building and running enterprise web and ERP systems. And he has a really grounded approach to leveling up through hands -on projects. Eric? First off, Brian, thanks for having me. I appreciate it. Yeah, no worries.

I'm excited about this conversation because my DevOps journey. Definitely hasn't been a straight line. I come from a software engineering background. I have spent years building web applications. And I must say that at some point in my career, I realized I was fascinated about cloud technology and how things run in the cloud beyond just shipping code, you know. I'm looking forward to sharing what has worked for me.

What hasn't and hopefully a few lessons that can help someone else who is on a similar path as me. I appreciate it. That sounds awesome. It's interesting because I started as a developer, I guess, too. I worked...

Tech support but then I did some like early web development and kind of jumped into devops kind of a lot of the same way a different industry but um a lot of the same way you did so that's very exciting exactly brian so can you just tell us who you are and what you do day to day well like I said I'm a cloud and devops engineer with expected in AWS, Linux, CI, CD automation, containerization, infrastructure as code.

Basically coming from a software engineering background. So I principally develop web applications and then ship through cloud technology. And yes. Oh, that's very cool. And so what system, I guess most of the systems, are they AWS based like most of the cloud systems or? Yes, they are. So coming from my previous background. All right, so we develop PHP -based applications.

And previously, we host the applications on web servers, not cloud platforms, so to speak. But then with the advent of cloud technology and the thing in Android cloud, there has been the need to migrate from traditional deployment methods to cloud technology.

What made you realize when you were doing this full stack web development, what made you realize that writing code is only part of the picture and you wanted to dive into DevOps as well? Right. So let me say this, that I found myself more curious about modern methodologies and technologies around code deployment and how applications scale and remain reliable in production environments. Yes. Very cool.

Being a developer to touching DevOps stuff? Was it like networking, CICD, just AWS, Linux? Like what was the part that you found to be the hardest? Well, Brian, that's a great question, Brian. How does LevelUp for me, I would say, was familiarizing myself with key DevOps concepts and with the stack of technologies that come with it, I .e.

Source code management using Git, package management using Docker, building CICD pipelines.

Using tools such as Jenkins, GitHub Actions, container orchestration using Kubernetes, you know, mastering cloud services on platforms like AWS, Google Cloud, and also, you know, getting a grip on infrastructure as code, you know, using Terraform Ansible, and of course, implementing continuous monitoring and observability with tools, you know, such as Prometheus, Grafana, and things of the sort.

I must say that, you know, beyond that, tools also another hardest level level up for me was the shift shift in mindset all right the mindset shift I .e you know moving from let me say I know how this process works on my local host to how how that same process behaves in production at scale Now, that's a big jump.

You know, that's a big jump, which requires grit and a lot of hands -on experience, hands -on practice, and also a grip on the technology that you're working with. Yes, Brian. Yeah, those are two really good points. A, there's just a lot to know. When you talk DevOps, that is just such a wide breadth of tools and technologies. And then the other side of it, yeah, testing locally, great, it works.

Then you scale that up to 10 ,000 users, 100 ,000 users, a million, completely different complexities than you're going to have running that locally. And things are going to show up that you didn't even think of when you were down in the 10, 100 early user stage. Exactly, Brian. Yeah. So was there anything that clicked first for you or were there things that took longer than expected as far as like learning like AWS?

Because like speaking of like DevOps being wide, AWS has a ton of different technologies and a ton of different.

Ways of doing the same thing so is there anything like in particular that maybe clicked earlier or or was harder to learn over time that's a great question again brian I would say this that um what really clicked for me was realizing that um devops sits at what what what I will call the intersection of you know software engineering and systems thinking and business impact what you're doing ultimately and largely impacts on business decisions.

So I enjoyed solving problems like, why is this deployment failing? Or why does this service degrade and that's a load? And how do you, or how do we automate this so it never breaks again? You know, that shift in mindset, just from writing features to owning the process, owning the system, owning reliability and delivery.

You know actually pulled me in yes so so that's it for me now brian what let me add this that what didn't click for me like you said earlier all right was the sheer breadth and depth of new tools and technologies I had to learn you know you you you are not dated with lots of of of tools and technologies to learn technologies in respect of cloud in this respect of networking You need to know the ins and outs of Linux, the ICD pipelines, observability, you know, it can feel overwhelming, really overwhelming.

But early on, I tried to try to learn everything at once. And that I would say was my mistake. All right. You need to progress slowly. You need to trust the process, follow along slowly. So progress really accelerated when I focused on the fundamentals and learned. By building, you know, and breaking things on my home lab. Exactly. Yeah, that's a good point.

So yeah, just focusing on one area and then building on that. Exactly, right. So looking back at your background, you've worked on enterprise grade systems for big organizations. What changes when it's not a side project, kind of like we talked about earlier, running locally in dev, what changes between that and like mission critical, like enterprise grade systems? Like what are some things maybe you have to worry about?

Um that you wouldn't in a local development environment all right so brian my my first instance in deploying a mission critical application was a voting was an online voting system okay for for a school cool and yes and you can imagine that the system works fine on your machine And then the moment you have to go live or go or deploy on production, all right, the results you are getting is entirely different.

Users are trying to access the system and they cannot. And you know, elections are contentious matters and that you cannot afford to let your users lose trust in your system. And so for me, that was a great learning curve for me. You really have to be prepared for such instances that you need to have some virtual environments where you can literally test what you have built and you want to deploy.

So for some mission -critical systems, I have been privileged to work with teams that have deployed online voting systems. We have deployed some enterprise resource planning platform for institutions like the food. An agri -organization of the United Nations, the Ghana office to be specific. So that platform encapsulates everything to make their business processes work efficiently.

So there are modules on procurement, modules on human resource management, modules on financial management and things like that. These are mission -critical systems which you can't afford to let them fail. Yeah. So for me, they have been a great learning experience for me practicing on or working on these systems. Yes, Brian. So what did that era teach you that actually helped you in DevOps now?

Like reliability mindset or like working with stakeholders or like change management? What was the most valuable, I guess, part of that? Brian, you literally mentioned everything that was a learning experience for me, right? From system reliability to change management.

It was a great learning experience for me that for users, once your system is unreliable, it's not available, it's not giving them the expected results, they begin to lose trust in the system. And then also the change management process is very critical and very important. There has to be user acceptance. Be user accepted.

You need to tag them slowly and make sure that they learn or they get comfortable with the system. And so lots of sensitization, stakeholder engagements, lots of consultations. All right, you run campaigns, you run some communication campaigns here and there to ensure that there is some... There is user acceptance, all right?

There's user acceptance because no matter how good your system is, if users do not accept the system, the system is 100 % bound to fail. Yeah, that's very true. Very true. Okay, so switching gears, you posted about your home lab on the go, which I believe was a Mac mini. I'm just curious, why that approach? Why a Mac mini? Mac mini, all right.

So let me just say this, that I purchased, This Mac Mini just some time ago. And I said to myself, well, why don't I just turn this into my home lab? Dedicated specifically for practicing, you know, and for honing my tech skills. So I own a laptop and I also have this home lab dedicated specifically to... Practicing my, my DevOps skills. And so with this, this platform, I am not afraid to break things down.

All right. For sure. Yeah. To feel in the, make all the mistakes. All right. To, to, um, what should I say? To make all the mistakes that, that you need to make while, while, while learning as well. Yeah. So this home lab is, is dedicated. To help me practice my DevOps skills and also to hone my tech skills as well. Very cool. So are you doing like Docker, pipelines, monitoring, like break -fix? Exactly.

So just last night, so I decided to containerize a static version of my website. All right. So I Dockerized the website, all right, built the image on that platform. All right. I pushed that Docker image to Docker Hub, right? And then I went on also to run the Docker run command, right? So which literally converted the image to a container.

And then I ran the application from the web browser, you know, to local host port 8080. And that gave... So I used that home lab, all right, to practice writing of Docker files, all right, CICD pipelines, constructing my Docker Compose files, you know, building my Jenkins files.

And so you would be seeing a post for me very shortly detailing the process that went through and containerizing the static version of my website. Very cool. So basically that home lab helps me practice without guardrails. All right. Make the mistakes. All right. Do my debugging in there. Build Docker images. Run Docker images. Run Jenkins files. You know, build my Jenkins files. Run CICD pipelines.

Push my image to Docker Hub. Pull from Docker Hub. Things like that. Very cool. Well, I'll look forward to that post. And I think, I think that just goes to show you that you don't need fancy hardware either for a home lab.

Just take what you take, whatever you have or, or what you've used previously, or, you know, if you have an old Mac mini or whatever, you don't need, you don't need like a big rack and stack server set up. Like you don't need a bunch of fancy machines. Exactly, Brian. Exactly, Brian. So shifting gears a little bit from the home lab, and we talked about this at the top too a little bit, but I'm just curious.

So we talked about how the tools are endless, right? You have Terraform, Kubernetes, Jenkins, Prometheus, Grafana, Ansible. How do you choose, how do you personally go by choosing what to learn next? All right. So this has been my structure. All right. So I begin with source code management. All right. So the basics for me is very critical. You need to be familiar with the basics.

You need to have a stronghold on the basics, which is Linux. All right. So things to do with Linux commands, mastering bash scripting are very important and very, very, very, very critical because at the core of DevOps is Linux. You cannot be a DevOps engineer. And not no Linux. Linux is very critical. So I focus on the fundamentals, which is Linux.

And then I move on to source code management tools, you know, like Git. And then I progress into package management using Docker, right? And then I progress to CICD pipelines using Jenkins. So the idea is how do I use automation, all right, to build? The Docker image and then push it to a repository. All right. Okay.

And then from the Docker, from the CICD pipelines and practice, all right, so I move on to the cloud platform. All right. So that now after pushing to the Docker repository, all right, it has to be pulled onto your cloud platform. Now we have cloud services also that can serve us as your... Your Docker repository. All right. So the various cloud platforms have services that can serve as, you know, Docker repository.

And then I move on to learning infrastructure as code. All right. But in between that one, so from Docker to the cloud platform, my process would be to learn Kubernetes next. And then after learning Kubernetes, you learn Terraform. Okay.

Or infrastructure as code so basically terraform ansible and then you move on to to to implementing continuous monitoring and observability with you know tools like prometheus and grafana gotcha so so so for me that has been my structure well some people have their structure.

But for me, at the heart of DevOps has been to learn and show that you get the fundamentals right and show that you have a good grip on Linux and then you progress from there. And then learn the entire, through the process, learn the entire pipeline and follow it through. It sounds like, right? Exactly. Exactly, Brian. Exactly, Brian. So what are your thoughts on certifications versus like...

Doing your own projects what do you think is more valuable well for certifications let I would say this that's a very dicey dicey matter yeah it is that's a dicey matter right well here is it hands down I'll see real projects would always make the big difference for anyone right certifications and courses will help build structure but running real you know, setting up CICD pipelines, deploying to cloud infrastructure, breaking things, monitoring them, fixing things, you know, that's where everything clicks.

That's where everything clicks, yes. So certifications help, all right? Like I said earlier, certifications and courses would help you build structure. But, you know... Hands down, I would say real projects would always make the big difference. Real projects and hands -on projects will make the real difference. Because you have to fail, right? You have to learn to fail and be okay with that.

You have to learn to fail. You have to learn to fail. You have to learn to fail. For me, there have been several times where I have just tried to organize a Docker Compose file. As basic as getting it to run successfully. Brian, and it's been a challenge. And so I put that file down, go to sleep, come back the following morning, and it starts to end. And surprisingly, things begin to click for me. All right.

And then when you realize that you made mistakes, oh, I made the mistake here, I made the mistake there, I made this mistake. Just two weeks back, all right, one Docker Compose file that I wrote just wasn't working. Only for me to realize there was a dash, a dash I had introduced somewhere that shouldn't have been there. And it's been a great learning experience for me.

So I would say that hands down, real practice and real projects would always make a difference. Yeah, I agree with that. I think, yeah, I think certifications, boot camps have their place, but as a foundational, a structure school, same thing, but you need that hands -on learning. At some point you need to actually dive in. Be comfortable with failing because that's okay. Like, you know, it's okay.

You're going to stumble a little bit and that's okay. So wrapping up a bit, what's one piece of advice you would give someone trying to break into DevOps in 2026? Where should they start? What should they do? All right. So what I would say is this, that let me say this, that I would just say, focus on depth and not hype. Focus on depth and not hype. So I...

I have found out that employers, you know, care more about how you think and solve problems, right? How you think and solve problems than knowing every tool there is, all right? How you think and how you solve problems, all right? Because at the heart of that all, that is what DevOps really is. Simplifying things, automating processes, and then shipping code faster and efficiently.

All right and so how you think now my my my other advice would be this that consistency would always beat intensity all right that consistency would always beat intensity you don't need to learn everything in three months you need to show up steadily all right you you need to show up you need to have a routine if you tell yourself that you would dedicate two hours each day to learning devops do that You need to show steady progress.

You need to build. You need to document. You need to reflect. You need to repeat. And it's okay to fail. It's okay to fail. Do this repeatedly. And I would say also that slow down and be consistent. You don't need to master everything overnight. Just show up. Build something small and keep going. And also one thing I would say also would be this. Learn to share your journey, all right?

Learn to write, learn to post, learn to talk about what you are learning because it's only through that process that it reinforces your knowledge. It creates visibility as well. And then opportunities often come from when people see the way you think and not just what's on your resume. The way you think, the way you solve problems, the way you handle challenges. That would be for me, Brian.

Yeah, I agree with that a lot. All right, so where should people follow you and what should they check out? All right, Brian, so people can follow me on LinkedIn and then also my personal website, which is ericpatty .com, ericpatty .com. I'm also on Facebook and also on Instagram. And so I'm primarily on LinkedIn and I have a website, ericpatty .com. Yes, Brian. Awesome. Appreciate it.

Well, thank you, Eric Pate, for coming on and being on Ship It Weekly. I really appreciate it. Thank you, Brian. It's been great. Thank you. All right. That's the interview with Eric. Quick reminder on the format. Ship It Weekly is still the weekly news recap. And I'm dropping these guest convos in between. If you want to catch both, hit follow or subscribe wherever you are listening.

And if this episode was useful, share it with a coworker or your on -call buddy and leave a quick rating or review. It's annoying how much that stuff actually helps the show. We'll be back next week with the regular news episode. We'll see you then. Thanks for listening.

Thank you. Hey, I'm Brian Teller. I work in DevOps and SRE and I run Teller's Tech. Ship It Weekly is where I filter the noise and pull out what actually matters when you're the one running infrastructure and owning reliability. If something's hype, I'll call it hype. If it changes how you operate, I'll break it down in plain English. Most weeks, this is a quick news recap.

In between those, I drop interview episodes with folks across the DevOps world. Happy holidays. Merry Christmas, all that good stuff. It's the day after Christmas, so if you're listening while hiding from your family, checking one thing real quick, I respect it. Quick piece of housekeeping, the new site is live at shipitweekly .fm. That's where I'm putting links and show notes. Also, I'm looking for interview guests.

If you're building real infra or platform stuff and you want to come on for a chill conversation episode, hit the email on shipitweekly .fm. If you've got war stories, even better. And one quick ask before we start. If the show has been useful, hit follow or subscribe wherever you are listening. And if you've got 10 seconds, a rating or review really helps way more than it should.

All right, three main stories for today. First, Cloudflare wrote up how they built an internal maintenance scheduler on workers. This is real platform engineering. Memory limits. Data modeling, query optimization, caching, Parquet for historical analysis. Second, AWS databases are now available directly in Vercel marketplace. It's a quiet shift, but it's a big one.

Devs can click button real AWS databases from inside Vercel, but you still have to own governance, billing, and the blast radius. Third, there's an open source project from AWS called Team. Temporary Elevated Access Management, and it's built around IAM Identity Center. It's approval -based, time -bound access. This is one of those everybody wants it, few implement it cleanly problems.

Then we'll do a lightning round, and we'll close with Mark Brooker's What Now? Handling Errors in Large Systems. Let's get into it. Cloudflare has a really good post on how they built an internal maintenance brain on workers. The core problem is kind of obvious once you hear it.

When you run infra at their scale you cannot rely on humans to remember every dependency and every weird routing rule and every if these two things go down at the same time a customer special setup gets wrecked scenario so they built a centralized scheduler that treats maintenance like a set of constraints like we must always have at least one of these routers active or this customer pins traffic through these data centers, so don't take all of them out at once.

The fun part is how they got it working within worker's limits. Their first naive approach was basically load everything into one worker. All the relationships, all the product config, all the metrics, then compute constraints. And even in proof of concept, they hit out of memory errors. So they took a step back and said, okay. Workers have limits. We can't treat this like a giant in -memory analytics job.

We need to only load the data that matters for the specific maintenance request. If you get a maintenance request for your router in Frankfurt, you probably do not need to load Australia. You need the dependency neighborhood around that router.

That pushed them into graph modeling they describe constraints as objects and associations basically vertices and edges routers are objects pulls are objects and the dependencies are associations and then they built a typed association interface so the constraint logic stays simple but the backing implementation can get smarter over time then they flip their data fetching style instead of pulling down huge responses and filtering locally They started doing targeted requests through the graph interface.

They claim response sizes dropped by 100 times in one spot. That's huge. Also, it's the exact kind of win you get when you stop shipping your entire dataset into your app layer just to throw most of it away. Of course, that created the next problem. Sub -request limits. They traded a few massive requests for a ton of tiny requests. And then they started breaching sub -request limits.

So they built a fetch pipeline with request deduping, a small LRU cache, edge caching via caches .default, and sane retry and back off. After tuning, they were seeing about a 99 % cache hit rate on those fetches. That's wild. And it's It's also how you make something like this survive at scale without tuning your internal APIs into a creator. Then there's a super relatable metric story.

They use Thanos for Prometheus queries, and they call out what a lot of teams do by accident. Ask for everything. Get megabytes back, parse JSON into single -threaded runtime, then filter most of it out. That's basically self -inflicted pain.

Instead they use the graph to find the specific relationships first then issue much more targeted thanos queries they say average response size went from multiple megabytes to about one kilobyte in one case so the theme of this story is stop dragging huge blobs of data into your application just so you can toss 99 of it and then they bring it home with historical analysis real time is one thing historical is a different because now you're scanning months of data to see if your logic is actually safe and accurate.

They talk about how Prometheus TSDB blocks are not really designed for object storage access patterns and how that turns into a lot of random reads. So they adopt a parquet conversation layer for historical data. Columnar format. Better stats, and you can fetch what you need without slamming object storage with random IO. Takeaways you can steal even if you're not Cloudflare.

If you're building platform brains like schedulers, deploy orchestrators, policy evaluators, you will hit limits. Memory, CPU, API quotas, request fanout. You win by changing the shape of the problem, not brute forcing it. Graph interfaces are a cheat code for dependency -heavy domains. And targeted queries plus caching plus backoff is still undefeated.

All right, let's move from Cloudflare did adult engineering to developers can click -button databases. AWS announced that AWS databases are now available on the Vercel marketplace. The headline is simple. From Vercel, you can provision and connect to Aurora Postgres, Aurora DSQL, and DynamoDB in seconds. And here's the part platform folks should not ignore.

The onboarding path is basically create a new AWS account from Vercel with some starter credits. So the dev experience is you're already in Vercel. You click a thing, and now you have a database, and the app is wired up. That is great for Velocity. It is also the kind of thing that bypasses governance if you don't get in front of it. Even the AWS side is legit.

You still have to deal with who owns that AWS account long term. Is it inside your AWS organization or is it an orphan account that exists because Vercel made it easy? How do SCPs apply? How do guardrails apply? How do you do tagging and cost allocation so finance doesn't show up later asking why there are mystery accounts? What does networking look like? Are public endpoints acceptable?

Do you need private connectivity? Do you have a VPC strategy that fits Vercel -first teams? What's your audit baseline? Cloud trail? Config? Detective controls? All the boring stuff. Also, Region selection matters for data, residency, and latency. It's not just pick whatever. Vercel also hinted this is evolving.

They're talking about coming soon support for provisioning into an existing AWS account, not just a new one. If that lands cleanly, that's the version I'd actually want as a platform team because now you can meet developers where they are without losing governance. So the takeaway, if your dev platform can create AWS resources, your governance has to meet it there. The database is easy.

The ownership model is the hard part. All right. Story 3 is about access, which matters even more once you have more accounts and more surfaces. TEAM stands for Temporary Elevated Access Management. It's an open -source solution built around AWS IAM Identity Center. The pitch is basically approval -based, time -bound elevated access to AWS accounts.

Users request elevated access for a specific period of time, with a reason. Approvers approve or deny it. If approved, access is granted. When time expires, it is automatically removed. That automatic removal is the whole point. Because most orgs fail here. Someone gets admin just for this incident, and then it stays for months. Privilege creep becomes the default. Team also leans into auditing and visibility.

Who requested what, who approved it, when it expired, plus session logging. How I'd frame it. This is not your break glass story. Break glass is the world is on fire and we need access right now. It should be rare, noisy, and heavily monitored. Team is the daily, I need admin for 45 minutes to do this legit change workflow.

If you want this to actually work culturally, approvals need to be fast enough that people don't route around it. And the default permission sets need to be sane so elevation is actually meaningful, not just ceremony. If you are already on IAM Identity Center and you've been hand -waving we should do JIT access, team is at least worth a look.

Even if you don't adopt it, it's a good reference for what time -bound elevation can look like without building everything from scratch. Alright, time for the lightning round. GitHub Actions improved performance on the workflows page. Small change, but if you live in Actions, you'll feel it.

Big workflows render better now, lazy loading, and you can filter jobs by status so you can just see failures or in -progress stuff. During an incident, this is a real quality of life upgrade. Next, the weird one, Lambda managed instances. This is basically run Lambda functions on EC2, but AWS manages the lifecycle of those instances for you. It's meant for steady state workloads and specialized compute needs.

It also changes concurrency and execution assumptions a bit. So you actually need to care about thread safety and shared state in ways you might not with regular Lambda. Interesting, slightly cursed. But I get why it exists. Atmos quick hit. There's a Cloud Posse Atmos issue where vendoring a component with an invalid URL triggers a weird GitHub username prompt.

I'm mentioning it less because of the bug and more because Atmos is clearly turning into a bigger workflow ecosystem now. CLI, dev containers, IDE integrations, the whole thing. And last, k8sdiagram .fun. It's a free Kubernetes diagram builder that can also generate YAML for common resources.

I would not blindly apply auto -generated YAML to prod, but for teaching, prototyping, or explaining architecture to humans, it's actually super handy. All right, let's close with the human story, because this ties into everything we just talked about. Mark Brooker wrote a post called, What Now? Handling errors in large systems. It's basically an interactive error handling game.

You decide whether a system should crash or keep going when something goes wrong. And then he explains his take. The key idea is simple. And it's something we forget all the time. Error handling isn't a local decision. It's a global property of the system. We love to argue about one line of code. Should this crash? Should this retry? Should this be best effort?

Mark's point is, that decision only makes sense if you understand the architecture around it. He asks questions like, are failures correlated? If the same bad input can hit every node, crashing can amplify the blast radius. Can a higher layer handle the air? Some architectures are designed to tolerate a few crashes. None are designed to tolerate a ton of crashes continuously. Is it actually safe to keep crashing?

Is it actually safe to keep running? Sometimes continuing means silent corruption, which is worse than a crash. Then he ties it to blast radius reduction. Cell -based architectures, independent regions, isolating failures so you don't have a single mistake become a global outage. That connects to today's stories perfectly.

Cloudflare's scheduler exists because humans will guess wrong sometimes, and the systems need to prevent correlated failure. The Vercel Marketplace DB story is a new surface area. The failure modes aren't just technical, they're governance and ownership failures. And team is literally about reducing the risk of one person with standing admin turning a mistake into a disaster. So yeah.

If you want a good mindset going into next year, stop treating air handling like a code style preference. Treat it like architecture. All right, that's it for this episode of Ship It Weekly. We covered Cloudflare's maintenance scheduler on workers and the platform limits force better design lessons. AWS databases inside the Vercel marketplace and what that means for governance and blast radius.

And team as a practical path to time -bound elevated access with IAM Identity Center. If you got something out of this, hit follow or subscribe wherever you are listening. And if you can, leave a quick rating or review. It's annoying how much that helps the show. Links and show notes are on shipitweekly .fm. And one last reminder, I'm looking for interview guests.

If you want to come on and talk through real DevOps or platform work you're doing, hit the email on the site. I'm Brian. Thanks for listening. Happy holidays. And I'll see you next week, which is technically next year.