Ship It Weekly Host Commentaries

This episode is about local optimization.

CloudFormation reports success faster. Spark absorbs more AI-oriented workloads. Coding agents generate more code. Platform teams automate more controls. Each of those can be a real improvement, but improving one stage does not automatically improve the system around it.

Sometimes the bottleneck disappears. Sometimes it just moves.

CloudFormation Express mode is a good example. AWS can now return deployment completion before every resource has finished stabilizing. That could make development and agent-driven infrastructure loops much faster, especially when you are repeatedly changing a template and waiting several minutes to discover one bad property.

But the speed comes from changing what “complete” means.

CloudFormation may be done while the resource is still becoming operational. That is not necessarily a problem, but pipelines have to understand the distinction. A green deployment should not automatically trigger integration tests, traffic shifts, or dependent work unless something else confirms that the resource is actually ready.

The uncertainty did not disappear. CloudFormation just stopped waiting for it.

Spark 4.2 raises a different version of the same question. Native vector operations and nearest-neighbor joins mean some teams can keep recommendation, matching, retrieval, and embedding workloads inside Spark rather than introducing another specialized platform.

That does not make every vector database unnecessary. Low-latency application serving is different from large-scale batch analysis. But it should push teams to start with the workload instead of the product category.

Do we need a dedicated serving layer? Or is the data already in Spark, with the work happening offline?

Every new platform comes with credentials, networking, scaling, monitoring, patching, backups, cost, and another operational boundary. Sometimes the specialized system is absolutely worth it. Other times, the best platform decision is not adding one.

The GitLab research gets closer to the organizational side of this. Developers say AI is helping them produce code faster, but review and validation are not accelerating at the same pace.

That feels predictable once you stop treating software delivery as typing.

Software delivery is a chain of queues. Implementation feeds review. Review feeds testing. Testing feeds deployment. Deployment feeds operations. If one stage suddenly produces twice as much work, the next stage does not magically gain twice the capacity.

The pull request gets created faster, but someone still has to understand it. Terraform gets generated in minutes, but someone still has to evaluate the blast radius. More changes reach production, but the on-call team still owns what happens afterward.

A local productivity gain can become a system-level slowdown when it increases work in progress faster than the organization can validate it.

The traceability issue may be even more important. When something fails, can the team determine where the change came from, what it was intended to do, what context the agent had, what validation ran, and who accepted responsibility for it?

Those are not just AI governance questions. They are basic incident-response questions.

Generating code is becoming cheaper. Understanding and operating it is not getting cheaper at the same rate.

The Sevdesk platform-governance story shows what happens when teams forget that internal platforms are still products.

The technically simple approach is to publish a long list of requirements, set a deadline, and start blocking deployments. The controls may be correct, but the rollout can still fail because developers receive noise instead of useful guidance.

Sevdesk took a more gradual approach: build visibility, identify ownership, group findings, provide feedback, soft-enforce, and then enforce. They called it minimum viable governance.

I like that framing because it does not mean minimum standards. It means starting with the smallest set of controls that meaningfully reduces the company’s risk and introducing them in a way the organization can absorb.

If every finding is urgent, none of them are.

If every policy failure becomes a ticket, the platform becomes another queue.

And if the only thing developers experience is a failed deployment, the platform team may win the policy argument while losing adoption entirely.

The platform team’s product is not the policy. The product is changed behavior.

The lightning-round stories reinforced the same theme in smaller ways.

OpenShift being supported does not mean it is current. GitHub’s autoscaling configuration was valid, but it was based on thresholds that no longer matched the live system. Cloudflare showed why a successful GET request does not prove that POST requests, logins, purchases, or webhooks are working. GitHub Code Quality showed how a preview feature can quietly become both a production dependency and a production bill.

In each case, the surface signal was incomplete.

Supported is not patched.

Valid is not correct.

Reachable is not functional.

Enabled is not free.

That leads naturally into Reid Savage’s article about their first year managing an SRE team.

“Could versus should” sounds like a question about ownership, and it is. A capable SRE team could take the ticket, operate the service, debug the deployment, or absorb another responsibility that does not have a clear home.

But Reid’s larger point is about judgment.

Sometimes a manager should make the call. Sometimes they should recommend a direction. Sometimes they should add context. Sometimes they should stay quiet and let the team own the decision.

The challenge is not knowing what you are capable of doing. It is understanding which kind of participation improves the outcome.

That applies to technology decisions too.

CloudFormation can return faster. Should this production pipeline use that mode?

Spark can perform the vector operation. Should the workload live there?

AI can generate more code. Should all of it immediately enter the review queue?

The platform can block every violation. Should blocking be the first way developers learn that a policy exists?

Capability expands the decision space. It does not make the decision for us.

Good engineering increases what is possible.

Good judgment decides what is useful.

Faster is not the same as finished.

And could is not the same as should.

The thing that stuck with me most from this conversation is that production AI has to grow up really fast.

It is fun when it is a prototype. It is fun when someone wires up a model, gives it a few tools, and suddenly it can answer questions that used to require bouncing between dashboards, logs, traces, docs, deploy history, and half a dozen Slack threads.

That part is genuinely useful.

But the second people start depending on it, the bar changes.

Now it is not just “does this look cool in a demo?” It is “can we operate this?” Can we measure it? Can we tell if it helped? Can we tell if it made things worse? Can we understand the path it took, the tools it used, the context it kept, the context it ignored, and whether the answer it gave was actually useful?

That is where I think this conversation with Mat lands in a really practical place.

AI observability is not just normal service monitoring with a new label slapped on it.

Latency still matters. Errors still matter. Cost still matters. A 200 response still tells you something. But with agents, that is only the outer shell of the problem.

The more interesting question is behavior.

Did the agent answer the user’s actual request? Did it pick the right tool? Did the user have to correct it? Did a prompt change improve one workflow while quietly making another one worse? Did a model upgrade change the personality or strategy of the system in a way nobody noticed until users did?

Those are messy questions.

And they are very production-shaped questions.

That is why I liked the part where Mat talked about evals and LLM-as-judge workflows. I know there is a lot of debate around using an LLM to judge another LLM, and I get some of the skepticism. It is not magic. It is not perfect. But it gives teams a way to start measuring behavior instead of just arguing about vibes.

And honestly, a lot of teams are still in the vibe-testing phase.

That is not always wrong at the beginning. Sometimes you do need to build the thing, play with it, see if it works, and figure out whether there is anything there. That is how a lot of useful software starts.

But vibe testing cannot be the long-term operating model.

At some point, the system needs feedback loops. It needs evals. It needs instrumentation. It needs versioning. It needs cost visibility. It needs guardrails. It needs some way to tell whether it is getting better or just getting more confident.

That part feels very familiar to me.

We have seen this pattern with every other generation of production systems. The demo is one thing. The operating model is another.

A dashboard demo is easy. An observability practice is hard.

A CI pipeline is easy. A safe delivery system is hard.

A Kubernetes cluster is easy. A platform people can safely depend on is hard.

An AI assistant is the same way.

You can get something impressive on screen pretty quickly. But the production version has to survive real users, real data, real workflows, real permissions, real costs, and real failure modes.

The other part that stood out to me was the UX discussion.

I think this gets overlooked in a lot of AI conversations. People focus on the model, the prompt, the tools, the architecture, and the eval strategy. All of that matters. But for operators, trust often comes down to whether the system can show its work in a way that is actually useful.

A wall of generated text is not enough.

If an AI assistant tells you “latency looks elevated because service X is timing out against service Y,” that might be helpful. But if it can show you the graph, deep link you into the right Grafana view, apply the filters, and let you inspect the data yourself, that is a different experience.

Because operators do not just want an answer.

They want evidence.

They want to see the shape of the problem. They want to know what changed. They want to know if the model is summarizing reality or just saying something that sounds plausible.

That is why I liked Mat’s point that dashboards are not going away in the AI world. I agree with that. Maybe the way we get to the dashboard changes. Maybe AI helps us generate the query, narrow the scope, summarize the pattern, or find the right slice of telemetry faster.

But the underlying need does not go away.

We still want to know what is true.

And in production, “sounds right” is not the same thing as “is right.”

The guardrails conversation matters for the same reason. Once an agent has tools, the question changes. It is no longer just “what can it say?” It becomes “what can it do?”

Can it read logs? Can it inspect customer data? Can it open an issue? Can it open a pull request? Can it roll back a deploy? Can it restart a service? Can it change config? Can it leak something by accident because it was trying very hard to be helpful?

That is the uncomfortable part.

LLMs are eager. They want to complete the task. If you give them a pile of tools, they may use those tools in ways you did not expect. Sometimes that is impressive. Sometimes that is exactly the problem.

So the control cannot just be “we told it not to do bad things.”

That is not enough.

The control is permissions. Boundaries. Scoping. Testing. Audit trails. Human approval where the blast radius is real. Clear separation between read-only investigation and write-capable remediation. And a healthy amount of skepticism any time a system can move from observing to acting.

That does not mean “do not use agents.”

It means treat them like production automation.

Because that is what they become the second they touch production systems.

I also liked Mat’s point that imperfect does not mean useless.

This is one of those AI debates where people tend to jump to extremes. Either AI is magic and will solve everything, or it gets something wrong and therefore it is garbage.

Neither version is useful.

Most of the systems we operate are imperfect. Networks fail. APIs timeout. Humans miss things. Dashboards hide important context. Alerts fire too late or too often. Runbooks rot. Deploy scripts have weird edge cases. On-call engineers get tired.

We do not throw all of that away because it is imperfect.

We build systems around the imperfections.

Retries. Timeouts. Circuit breakers. Rollbacks. Peer review. Monitoring. Tests. Runbooks. Incident review. Guardrails.

AI needs the same kind of thinking.

Do not trust it blindly. Do not dismiss it completely. Figure out where it helps, where it fails, what it is allowed to touch, how you measure it, and where a human still needs to make the call.

That is probably the most practical frame for platform and SRE teams right now.

Start small.

Let AI help with query generation. Let it help with first-pass investigations. Let it summarize telemetry. Let it connect dots across logs, metrics, traces, deploys, and docs. Let it take some of the annoying search and translation work out of operations.

But do not skip the operating model.

If the agent becomes part of the workflow, it needs ownership. It needs evals. It needs observability. It needs cost controls. It needs guardrails. It needs a clear answer to “what happens when it is wrong?”

That last question is the one I keep coming back to.

What happens when it is wrong?

If the answer is “a human reads the suggestion and ignores it,” that is one kind of risk.

If the answer is “it opens a pull request,” that is another.

If the answer is “it rolls back production,” that is another.

If the answer is “it reads sensitive logs and posts a GitHub issue,” that is a very different conversation.

The authority matters.

The blast radius matters.

The audit trail matters.

And the boring production discipline matters.

So my takeaway from this episode is not that every team needs to rush out and build AI agents into their operations stack tomorrow.

It is more that AI is becoming another layer of production software. And if it is going to sit near observability, incident response, deployment workflows, or infrastructure automation, then it needs to be treated like part of the system.

Not like a toy.

Not like magic.

Not like a chatbot off to the side.

Like production software with weird failure modes, real upside, real cost, and real operational responsibility.

That is the work.

Ship the small thing.

Measure it.

Learn from it.

Put guardrails around it.

Then ship again.

This episode is about exposure.

Not the loud kind. Not always a breach. Not always an exploit. Not always some dramatic chain of events where one vulnerability turns into full compromise in five minutes.

Sometimes exposure is quieter than that.

A GitHub API response. A public repo name. An old token. A CLI tool reading more files than you expected. A cloud resource that is reachable even though the diagram says it should not be. A patch backlog that looks fine until the discovery rate changes.

That is the uncomfortable part. Exposure often starts as visibility. And visibility can feel harmless.

The repo is public anyway. The API only returned metadata. The tool only read files on my laptop. The port was only open for testing. The patch is on the list. The postmortem is documented.

All of that sounds reasonable in isolation. But attackers do not need your systems to be obviously broken. They need enough of a map to make better decisions than you think they can make.

That is why the Datadog GitHub API enumeration story matters.

It is not a classic zero-day story. It is not someone popping shells through a wild exploit chain. It is API traffic. Questions asked through interfaces that are supposed to answer questions.

Who belongs to this org? What repos are visible? What naming patterns exist? What project names leak through public references? Which users are active? Which private repo paths might exist? What OAuth tokens or personal access tokens are worth abusing?

Individually, a lot of that can look boring. At scale, it becomes reconnaissance. And recon is not harmless just because the first layer of data is public.

Public metadata reduces uncertainty. It tells an attacker where to look next. It helps them write better phishing messages. It helps them identify teams, repos, naming conventions, internal systems, release patterns, and likely owners. It turns “we know nothing” into “we know enough to keep going.”

The ghost account part makes that even more interesting.

A brand-new account firing API requests at a bunch of organizations is suspicious. An older account that has existed for years feels different. It has age. It has history. It blends into the noise better. And if the traffic uses normal-looking user agents or analytics-style names, the whole thing can look less like an attack and more like background internet weather.

That is the hard operational problem. Not every bad thing looks bad at the point where you could still stop it easily. By the time enumeration turns into private repo access, token abuse, or cloning, the quiet phase already did its job.

So the practical takeaway is pretty simple. GitHub is not just source control. It is a production surface.

It holds code. It holds workflow logic. It holds release paths. It holds references to infrastructure. It holds build instructions. It holds internal names. It often holds too much history. And it sits close to secrets, cloud roles, registries, package publishing, deploy automation, and developer identities.

That means GitHub API activity matters. Token scopes matter. OAuth apps matter. Old PATs matter. Outside collaborators matter. Dormant users matter. Audit logs matter. GraphQL usage patterns matter. Private repo path probes matter.

Because the repo platform is part of the system now. Not adjacent to it. Part of it.

The Grok Build CLI story comes at exposure from a different direction.

This one is about AI coding tools and the data boundary around them. And honestly, this is where a lot of teams are behind. Because many organizations still think of coding assistants like editor features. Like autocomplete got smarter. Like a plugin got more helpful.

But some of these tools are not just suggesting text. They read files. They inspect repos. They look at environment files. They summarize docs. They understand build systems. They run commands. They install packages. They connect to cloud services. They send context somewhere else for processing.

That is not just an editor feature. That is access.

If a tool reads your .env file, that is access. If it reads Terraform, Helm charts, Kubernetes manifests, GitHub Actions workflows, runbooks, incident notes, or internal docs, that is access. If it sends that context to a hosted service, that is data movement.

And once data moves, the questions change.

Where did it go? Was it retained? Was it used for training? Can admins control it? Can the org audit it? Does it respect ignored files? Does it read Git history? Does it upload the whole repo or selected context? Can engineers use it in production repos? Can they use it in customer code? Can they use it in security repos? Can they use it in infra repos?

Those questions can sound like procurement paperwork. They are not. They are production security questions.

Repos are not just code anymore. They are architecture. They are service names. They are IAM assumptions. They are deployment rules. They are CI/CD logic. They are incident history. They are internal conventions. Sometimes they are secrets. And increasingly, they are instructions for agents.

That makes repo context sensitive even when the source code itself is not some secret algorithm.

This is why “it was just on my laptop” does not work as a boundary. A laptop can be the place where source code, credentials, cloud sessions, dev tooling, observability access, and AI tools all meet. That is a lot of trust in one place.

So the point is not “never use AI coding tools.” That is not realistic. The point is to stop pretending they are harmless by default.

They need policy. They need approved usage. They need repo boundaries. They need secret handling. They need enterprise controls. They need auditability. And engineers need to know what is okay and what is not.

Because if an AI tool can read part of your system, it is now part of your system.

That same idea shows up in the AWS Security Hub story.

Security Hub Network Scanning and exposure impact analysis are interesting because they move closer to reality. Not just “based on config, this might be exposed.” More like “we checked from the outside, this is reachable.”

That matters because cloud environments drift. Everyone who has worked around AWS long enough has seen some version of this.

A public IP attached during troubleshooting. A port opened temporarily and never closed. A test load balancer that outlived the test. A Kubernetes service copied from dev and accidentally made public. A security group that looks fine in one view but behaves differently when all the networking pieces come together. A resource no one remembers owning.

Config analysis is useful. But the internet does not care what your diagram intended. The internet cares whether it can reach the thing.

That is why active scanning matters. Can I reach it? What port is open? What service is there? What evidence proves it? That is closer to how an attacker thinks.

But reachability is still only half the story. The blast radius matters too.

An exposed test machine with no useful permissions is one kind of problem. An exposed instance with an IAM role that can read production secrets, write to S3, reach a database, or assume another role is a very different problem. Same open port. Very different incident.

That is where impact analysis gets useful. The exposed resource is the front door. The permissions behind it decide how bad the house fire can get.

So prioritization should not only be severity labels. It should be reachability plus blast radius.

Public and reachable matters. Public and reachable with dangerous IAM matters more. Public and reachable with a path to sensitive downstream systems matters a lot.

And when something is exposed, closing the port is not the whole fix. It is the first fix.

The better questions come after. Who owns this? Why did it exist? How long was it exposed? What could it access? Did anything touch it? What logs prove that? What guardrail prevents it from coming back?

That is the difference between closing a finding and improving the system. One clears the dashboard. The other reduces future risk.

The Microsoft story takes the same exposure theme and applies it to time.

AI-powered vulnerability discovery changes the tempo. That is the part operators need to care about. Not because everyone suddenly needs to become a Windows expert. Because the larger pattern applies everywhere.

AI helps write code faster. It helps analyze code faster. It helps defenders find bugs faster. And eventually, attackers get better at finding useful paths faster too.

Microsoft is basically saying the discovery side is accelerating. More issues found. More validation. More high-confidence findings reaching engineers. More security updates.

That creates pressure on the people who have to patch real systems.

And patching is not free. Patches break things. Drivers break things. Agents break things. Middleware breaks things. Security fixes have side effects. Anyone who has owned production systems knows this.

So the answer cannot be “patch everything instantly and hope.” But the answer also cannot be slow spreadsheet theater forever.

A lot of patch management still depends on manual coordination. Someone exports a report. Someone ranks CVEs. Someone emails owners. Someone asks for exceptions. Someone waits for a maintenance window. Someone says the app team is not comfortable yet. Someone updates the spreadsheet. Then everyone hopes the numbers are real.

That process already struggles. If the volume and speed of vulnerability discovery keeps increasing, that process gets worse.

So patching has to look more like production operations. Know what you own. Roll out in stages. Measure failures. Have rollback paths. Know coverage. Know exceptions. Have an emergency lane. And be able to explain the difference between “we patched” and “we think we patched.”

That is the platform angle. Patch management is not just a security team problem. It is operational infrastructure. It needs the same kind of thinking we apply to deploys, rollbacks, SLOs, automation, and incident response.

Security updates are becoming production operations. So the process around them needs to grow up.

The lightning round keeps circling the same theme.

HalluSquatting is exposure through generated assumptions. If an AI agent invents a package name and an attacker registers that name, the model’s hallucination becomes a dependency risk. That is wild, but it is also kind of obvious once you see it. Do not let agents install packages just because the name sounds real.

Cloudflare dashboard and API incidents are exposure through control-plane dependency. The edge can still be serving traffic, but if the dashboard or API is unavailable, your ability to respond may be limited. Data plane and control plane are different. Both need to be part of your reliability model.

node-tar CVEs are exposure through boring input handling. Archive parsing sits in package managers, build systems, CI jobs, artifact pipelines, upload services, backups, and scanners. If you process untrusted archives, tar parsing bugs matter.

ServiceNow authentication risk is exposure through enterprise workflow platforms. A single unauthenticated endpoint in a system full of tickets, assets, incidents, employee data, workflows, and operational context is not a small thing. It is an enterprise visibility problem.

France pushing quantum risk into procurement language is exposure through time. Data that needs to stay secret for years has different risk. Post-quantum migration becomes real through procurement, compliance, vendor questionnaires, and data retention expectations before it becomes real in the backlog.

And Lambda Managed Instances for Java cold starts are a reminder that abstractions still have mechanics underneath. Serverless can hide servers. It cannot hide physics. Cold starts, warm capacity, runtime behavior, and cost still matter.

That is why the human closer lands on maps.

This episode is full of maps. GitHub maps. Repo data maps. Internet exposure maps. IAM blast-radius maps. Patch coverage maps. Incident timelines. Postmortem documents. Approval flows.

Maps are useful. I like maps. I want better maps.

But a map is not a fix.

An exposure finding does not close the port. An API log does not revoke the token. A patch bulletin does not patch the fleet. A postmortem timeline does not create learning. A human approval step does not guarantee judgment if the human has no context.

That last one is especially important right now.

A lot of systems add a human-in-the-loop step because it sounds safe. And sometimes it is. But not automatically.

If the human is pulled in at the worst moment, with the least context, under the most pressure, the approval step can become theater. They are technically in the loop. But they do not really have the time, context, or authority to make the system safer.

Post-incident reviews can fall into the same trap.

A meeting is not learning. A timeline is not learning. A document in the wiki is not learning. Those things can help. But learning happens when the system changes.

Ownership becomes clearer. The runbook improves. Alert noise goes down. A risky default gets removed. A manual step gets automated. A guardrail gets added. The next operator is less dependent on luck.

So the useful question is not just, “did we review the incident?” The useful question is, “what changed because of it?”

That applies to every story this week. If GitHub API traffic can map your org, what changed? If an AI tool can read your repo, what changed? If Security Hub finds something exposed, what changed? If patch pressure is going up, what changed? If a human approval step is mostly theater, what changed?

Visibility is good. Maps are good. Scanners are good. Dashboards are good. Postmortems are good. But none of them are ownership by themselves.

Exposure only becomes safer when someone uses the map to change the system.

That is the work.

Visibility is not ownership.

A map is not a fix.

---

Extra links worth including:

Datadog: Coordinated GitHub API enumeration
https://securitylabs.datadoghq.com/articles/coordinated-github-api-enumeration/

The Verge: Grok Build CLI repository upload report
https://www.theverge.com/ai-artificial-intelligence/965600/spacexai-grok-build-repository-upload

AWS Security Hub Network Scanning
https://aws.amazon.com/about-aws/whats-new/2026/07/aws-security-hub-network-scanning/

AWS Security Hub impact analysis for exposure findings
https://aws.amazon.com/about-aws/whats-new/2026/07/impact-analysis-aws-security-hub/

Microsoft: Windows vulnerability management and AI-powered discovery
https://blogs.windows.com/windowsexperience/2026/07/09/evolving-windows-vulnerability-management-to-meet-the-speed-of-ai-powered-discovery/

SRE Weekly Issue 525
https://sreweekly.com/sre-weekly-issue-525/

HalluSquatting / hallucinated package risk
https://www.endorlabs.com/learn/slopsquatting-when-ai-agents-hallucinate-malicious-packages

AWS Lambda Managed Instances for Java cold starts
https://aws.amazon.com/blogs/compute/eliminating-java-cold-starts-with-aws-lambda-managed-instances/

This week’s On Call Brief
https://www.tellerstech.com/on-call-brief-news/2026-W29/

Ship It Weekly shop
https://shop.tellerstech.com/

More Ship It Weekly episodes
https://shipitweekly.fm/

This episode is about safety nets.

That sounds comforting, and honestly, it should. Rollback buttons are good. Better alarms are good. Zone-aware routing is good. Secret scanning with more context is good. Internal advisory workflows are good. Better agent guardrails are good. I do not want fewer safety nets in production systems. I want more of them.

But safety nets create a weird kind of risk too. They can make teams feel like the hard part is solved. And usually, it is not.

A rollback feature does not mean the rollback is safe. An alarm does not mean the signal is worth waking someone up. A scanner finding a secret does not mean rotation is easy. An AI coding agent finding a fix does not mean it should run commands. And zone-aware routing does not mean you understand your failure domains.

That is really the thread through this episode. The tools are getting better. The recovery paths are getting better. The visibility is getting better. But the systems underneath are still complicated, still connected, and still full of places where one bad assumption can turn a small problem into a production incident.

Start with EKS version rollbacks.

This is a genuinely useful feature. For a long time, Kubernetes upgrades have felt like a one-way door. You check your cluster version. You check release notes. You check deprecated APIs. You check add-ons. You check admission controllers. You check node versions. You check whatever chart everyone is scared to touch. Then eventually, you upgrade.

And if something breaks, the path back is not always clean.

So EKS adding version rollback is a big deal. It gives platform teams a recovery option when a Kubernetes minor version upgrade exposes some compatibility issue they did not catch ahead of time. That helps, especially if you run a lot of clusters. Especially if you have strict change windows. Especially if every Kubernetes upgrade already involves a calendar invite, a Jira ticket, three approvals, and someone saying “should be fine” in Slack.

But rollback does not erase the work. It changes the risk model. That is the important distinction.

The existence of rollback does not mean workloads are compatible. It does not mean add-ons are safe. It does not mean admission webhooks behave correctly. It does not mean node groups are ready. It does not mean disruption budgets are sane. It does not mean the rollback itself will be instant. And it definitely does not mean the cluster goes back in time like nothing happened.

Rollback is an operation. It has requirements. It has timing. It has dependencies. It can fail in its own special way.

That is why this feature belongs in the upgrade runbook, not in the “we’ll figure it out if it breaks” bucket. The better question is not “can we rollback?” The better question is “have we tested the rollback path?”

What happens to nodes? What happens to add-ons? What happens to workloads? What happens with EKS Auto Mode? What happens if disruption budgets slow the rollback down? What happens if the thing that broke during upgrade also affects the path back?

A rollback button is great. But if nobody owns the upgrade path, the rollback button can become false confidence. That is the thing to watch.

Cordyceps hits the same theme from a different angle.

This one is about GitHub Actions and CI/CD supply chain risk. And the reason it matters is not just “workflows can be vulnerable.” We already know that. The reason it matters is that a lot of teams still treat workflow YAML like it is not real code.

It is in .github/workflows, so it feels like config. It gets copied from a blog post. It gets borrowed from another repo. It gets tweaked until the build is green. And then it sits there for years with production authority.

That is the problem.

GitHub Actions workflows can run commands. They can read secrets. They can publish packages. They can assume cloud roles. They can sign artifacts. They can push containers. They can cut releases.

That is not harmless automation. That is privileged execution.

And the nasty part with CI/CD attacks is that the dangerous path is not always obvious. It might not be one line that screams “this is bad.” It might be a low-trust workflow producing an artifact, then a higher-trust workflow consuming it later. It might be a branch name landing in shell. It might be a pull request title becoming command input. It might be a cache crossing a privilege boundary. It might be a comment, issue body, label, tag, or generated file moving from untrusted context into trusted automation.

Each step looks normal enough on its own. Together, you accidentally built an attack chain.

That is why pipeline security is hard. The boundary is not always one file. It is the relationship between triggers, permissions, artifacts, caches, tokens, branches, environments, and who is allowed to influence each of those pieces.

And now AI makes this more interesting.

Because if AI coding agents are generating CI/CD configuration, they can reproduce bad patterns quickly. Not because the agent is evil. Because insecure workflow patterns are everywhere. If the training data, examples, old repos, or copied snippets include bad assumptions, the agent may hand you a pipeline that works beautifully and trusts the wrong thing.

That is a rough combination.

Fast generation. High privilege. Weak review. Production authority.

So the fix is not “stop using GitHub Actions.” That is not realistic, and it is not the point. The fix is to treat workflow logic like production code.

Review it. Threat model it. Limit permissions. Separate trusted and untrusted workflows. Be careful with pull_request_target. Watch shell interpolation. Be suspicious of artifacts and caches crossing privilege boundaries. Assume your CI/CD system is part of your supply chain, because it is.

And attackers already treat it that way.

Then agentjacking takes the trust boundary even closer to the day-to-day work.

This story is about fake telemetry influencing AI coding agents. And I think this one matters because it changes how we think about logs, errors, and incident context.

For years, telemetry has mostly been evidence. An error happens. A log line appears. A stack trace gets captured. A Sentry issue gets opened. A human reads it and decides what to do.

The text might be noisy. It might be wrong. It might include user input. But a human is still sitting between the evidence and the action.

AI agents change that.

Now the same text may be read by something that can act. It might inspect a repo. It might suggest a patch. It might run tests. It might install packages. It might execute commands. It might open a pull request. It might touch credentials sitting in the local environment.

That changes the boundary.

The question is no longer just “can an attacker compromise the server?” The question becomes “can an attacker put text somewhere the agent will trust?”

That is a much lower bar.

And it is uncomfortable because our systems were not always designed with that in mind. We sanitize logs for display. We redact secrets. We limit access to observability tools. But we have not always treated error text as something that could steer an automated actor.

Now we have to.

If an agent can read hostile context and then take action, that context is part of the security model. Logs are not just logs. Issue text is not just issue text. Stack traces are not just stack traces. Runbooks are not just documentation. They can become instruction material.

So the control cannot just be “the agent is smart.” That is not a control.

The control is authority.

What can the agent read? What can it run? What can it write? Can it inspect without executing? Can it propose without applying? Can it separate trusted remediation guidance from user-controlled error content? Can you audit what it saw? Can you audit what it did? Are there live credentials nearby when it makes a bad decision?

That is the platform question.

Not “does this make developers faster?” That is a product question.

The platform question is “what happens when it is wrong, and what can it touch while being wrong?”

That is where agentjacking lands for me. It is not just an AI story. It is a trust boundary story. The text around your systems can now cause action. So the trust model has to include the text.

CloudWatch log query alarms are a much more normal operations story, but they fit the theme too.

This is a useful feature. A lot of real incidents show up in logs first. A specific exception. A repeated auth failure. A dependency timeout. A database connection error. A queue consumer screaming before the metric catches up.

For years, turning that into an alarm meant extra glue. Metric filters. Custom metrics. Another vendor. A Lambda path. A weird pipeline where a log pattern becomes a metric and the metric becomes an alarm.

Now CloudWatch can create alarms from log queries directly. That is good. It shortens the path from signal to action.

But it also shortens the path from random scary log line to pager noise.

And that is where teams can hurt themselves. Because the hard part of alerting is not creating the alert. The hard part is deciding whether the alert should exist.

Who owns it? What does it mean? How often does it happen? Is it user-impacting? What threshold makes sense? What does the on-call do when it fires? Does it flap? Does the log format change every deploy? Is it a symptom, or just a noisy internal detail?

If nobody can answer those questions, you did not create an alert. You created a search shortcut with a pager attached.

And every on-call team has lived through some version of that.

Someone sees an error. Someone says “we should alert on this.” Nobody wants to be the person arguing against “better monitoring.” So the alert ships. Then three weeks later, it is waking people up for a condition nobody understands and nobody owns.

That is how observability becomes punishment.

So yes, use log query alarms. They are useful. But use them for known failure modes. Clear owner. Clear threshold. Clear action. Clear reason to interrupt a human.

That is the difference between faster signal and faster noise.

The lightning round keeps adding to the same shape.

ECS Service Connect zone-aware routing is a good example. On the surface, it sounds like a networking improvement. Prefer same-AZ traffic when possible. Reduce latency. Reduce cross-AZ cost. Improve behavior during partial failures.

That all sounds great.

But it also reminds us that service-to-service routing is never just “can service A reach service B?” It is where traffic goes. How much it costs. What happens when an Availability Zone gets weird. What fails open. What fails closed. What happens when an old service needs a redeploy to pick up the new behavior.

Networking defaults matter because defaults become architecture.

etcd 3.7 is another quiet but important one.

Most people do not wake up excited to talk about etcd. Fair. But Kubernetes reliability still depends on the boring distributed database underneath it.

RangeStream, performance improvements, cleanup of older v2 store remnants, protobuf updates, dependency updates. None of that is flashy. But control plane reliability is built out of boring things working correctly under pressure.

The API server, scheduler, controller manager, kubelet, CNI, CSI, admission webhooks, and etcd all have to hold together. So when etcd gets better, Kubernetes gets a little more room to breathe.

GitHub innersource advisories are also easy to underplay.

Public CVEs get attention. Internal dependencies often do not. But a lot of companies run on private packages, internal platform libraries, shared GitHub Actions, private modules, and internal service templates.

Those things need security workflows too. Disclosure. Affected versions. Dependabot notifications. Remediation paths.

A vulnerable internal package can still be a real supply chain problem, even if it never touches the public internet.

Secret scanning metadata is similar.

Finding the secret is not the whole incident. It is just the beginning.

Is it real? Is it live? Who owns it? What system uses it? Is it expired? Can it be revoked safely? Will rotating it break production? Does it need a workspace URL, endpoint, project ID, or other context to validate?

Better metadata matters because incident response is full of these small, painful questions.

The scanner saying “I found something” is useful. The scanner helping you figure out what it is, where it belongs, and whether it is still active is much more useful.

And CloudWatch Application Signals service events hit the same operational theme.

When something is slow, you do not only need to know that it is slow. You need to know what changed.

Was there a deploy? Was there an exception spike? Was there a latency event? Did the problem start after a rollout? Did the shape of traffic change?

Observability gets better when the signal and the context live closer together. But again, only if someone owns the events and knows how to use them.

Otherwise, it is just another wall of things that happened.

That is why the closer lands on ownership.

The episode is not anti-tool. It is the opposite. Use the tools. Use the rollback button. Use log query alarms. Use zone-aware routing. Use better scanner metadata. Use internal advisories. Use AI agents where they help.

But do not confuse better tooling with transferred responsibility.

The system is still yours. The pager is still yours. The blast radius is still yours. The customer impact is still yours. And the judgment is still yours.

So the practical questions this week are pretty simple.

Where are the safety nets? Who owns them? Have they been tested? What do they not cover? What do they make easier? What do they make riskier? What authority do they inherit? What happens when they are wrong? And who is responsible when everyone assumed the tool meant the thinking was done?

That is the work.

Safety nets are good.

But they do not replace ownership.

---

Extra links worth including:

Amazon EKS Kubernetes version rollbacks
https://aws.amazon.com/blogs/aws/upgrade-amazon-eks-clusters-with-confidence-using-kubernetes-version-rollbacks/

Novee Security: Cordyceps supply chain research
https://novee.security/blog/cordyceps/

Tenet Security: Agentjacking through fake Sentry errors
https://tenetsecurity.ai/blog/agentjacking-coding-agents-with-fake-sentry-errors/

Amazon CloudWatch log query alarms
https://aws.amazon.com/about-aws/whats-new/2026/07/amazon-cloudwatch-log-alarms/

ECS Service Connect zone-aware routing
https://aws.amazon.com/about-aws/whats-new/2026/07/ecs-service-connect-zone-aware/

etcd 3.7 announcement
https://etcd.io/blog/2026/announcing-etcd-3.7/

GitHub innersource security advisories
https://github.blog/changelog/2026-07-08-innersource-security-advisories-are-generally-available/

GitHub secret scanning extended metadata and multipart validation
https://github.blog/changelog/2026-07-07-secret-scanning-extended-metadata-and-multipart-validation/

CloudWatch Application Signals service events
https://aws.amazon.com/about-aws/whats-new/2026/06/cloudwatch-service-events/

This week’s On Call Brief
https://www.tellerstech.com/on-call-brief-news/2026-W28/

More Ship It Weekly episodes
https://shipitweekly.fm/

This conversation with Evan hit on something I think a lot of platform and DevOps teams feel, even if they do not always say it directly: deployment is still weirdly painful.

That sounds almost ridiculous at this point. We have had so many waves of tools that were supposed to make this easier. Cloud platforms. Containers. Kubernetes. Terraform. GitOps. Internal developer platforms. PaaS products. CI/CD systems. Now AI agents that can generate a bunch of code and infrastructure glue before you have even had enough coffee to regret the prompt.

And yet, for a lot of teams, the actual act of taking an application and safely getting it into production still feels harder than it should.

I liked Evan’s framing of deployment as the “final boss” of software delivery, because it really does sit at the intersection of everything else. Deployment is not just “run the app.” It is the app, the runtime, the container image, the registry, the network path, the secrets, the config, the database, the rollback, the logs, the thing that wakes somebody up, and the thing the next person has to understand when something breaks.

That is why deployment tools tend to get complicated so quickly.

Every app is shaped a little differently. One HTTP app is not the same as another HTTP app. A Ruby app is not the same as a Go service. A JavaScript app is not the same as a Java service. A background worker is not the same as a public API. Then you multiply that by every cloud provider, every orchestration model, every security requirement, every team structure, every compliance need, and suddenly the “simple” deployment platform has to make a lot of choices.

And that is really where the conversation got interesting to me.

A lot of engineers say they want flexibility. And honestly, we do. We like having knobs. We like knowing that we can tune something if we need to. We like not being boxed in.

But there is a cost to that.

Every knob is a decision. Every decision is something somebody has to understand. Every exception is something the team has to support. And eventually the platform that was supposed to make delivery easier becomes another product the team has to maintain.

That is not always bad. Sometimes you really do need the big flexible platform. Sometimes Kubernetes is exactly the right answer. Sometimes Terraform gives you the control and consistency you need. Sometimes the team has enough scale, enough complexity, or enough operational maturity that the extra surface area is worth it.

But that is not every team.

Evan kept coming back to small teams. Teams that are application-focused. Teams that just want to ship. Teams that do not have a full platform engineering group sitting around waiting to build and maintain abstractions on top of abstractions. Teams that maybe need deploys, logs, rollbacks, config, HTTP apps, and a path to production that does not require them to become experts in every layer underneath it.

That is the part I think a lot of platform conversations miss.

The goal is not always to build the most powerful platform. Sometimes the better question is: how little platform can we get away with while still shipping safely?

I also liked the Terraform Enterprise and Terragrunt thread because it gets at the same idea from a different angle.

The hard part of Terraform was never only the language. It was the shape around it. How do you organize repos? How do you manage state? How do you split modules? How do you reduce blast radius? How do you avoid one giant infra repo where every change feels like it is poking a thousand resources?

That is why tools like Terragrunt became useful for a lot of teams. Not because everyone loves another layer, but because people wanted an opinion. They wanted guardrails. They wanted someone to say, “put this here, structure it like this, and stop inventing a new pattern for every environment.”

That desire for opinionated tooling is not a weakness. It is usually a sign that the team has better things to do than debate folder structure forever.

The OpenTofu discussion was interesting too. The license change obviously mattered, but Evan’s point was that OpenTofu also had a backlog of features people had wanted for a while. Some of those features were things Terraform had intentionally avoided, and in some cases for valid operational reasons. Encrypted state was the example that stood out. It sounds good, and in many ways it is good, but it also creates another availability and recovery problem if the keys become the thing blocking your infrastructure operations.

That is a very real infrastructure tradeoff. Security, recoverability, simplicity, and operability are always pulling on each other.

The Waypoint section may have been my favorite part of the conversation.

I remember Waypoint being one of those tools that seemed like it should make a ton of sense. A deployment workflow engine from HashiCorp? That sounds like something a lot of teams would want.

But Evan’s takeaway was pretty blunt: a pure workflow engine was not enough product.

That stuck with me.

Because if a tool makes you do a ton of homework before it gives you value, you are going to lose people. If the user has to set up the registry, wire the backend, understand where images go, configure ECS or Kubernetes or whatever else, and then finally maybe get to deploy something, that is a lot to ask.

Especially if the person’s actual job is not “become an expert in deployment plumbing.” Their job is to ship the application.

That also maps directly to the internal platform problem. How many companies have a deployment system someone started building because the team needed something better, then the boss told them to stop working on it because, technically, the company is not in the deployment platform business?

So now everyone is stuck with a half-finished platform that mostly works, but only if you know the weird ritual.

That Jenkins comment in the episode was painfully real. I think a lot of us have had some version of that. The job works, nobody knows why, nobody wants to touch it, and the entire deployment process is held together by fear and one person’s ancient Groovy choices.

The AI angle makes all of this even more important.

AI can absolutely help with infrastructure and deployment. It can generate Terraform. It can wire together services. It can produce config. It can read documentation and maybe map an app to a platform faster than a person could.

But Evan’s question was the right one: is the AI on call?

When the generated infrastructure breaks, who owns it? Who understands why those services were wired together that way? Who knows which parts are safe to change? Who knows whether the AI made a reasonable long-term operational choice or just produced something that worked once?

That is the risk I keep seeing with AI-generated infrastructure. It can make the first draft easier, but the first draft is not the same thing as operational ownership.

And maybe that means good abstractions matter more, not less.

If the platform surface area is small and obvious, AI has fewer ways to get it wrong. Deploy. Roll back. Get logs. Set config. Check status. That is very different from telling an agent to wire together a dozen AWS services and hoping it invents a sane deployment platform your team can operate for the next three years.

So the real future might not be AI replacing deployment platforms.

It might be AI making the difference between good and bad platform abstractions even more obvious.

The practical takeaway for me is this: be careful which opinions you adopt, and be even more careful which opinions you accidentally create.

If you choose Kubernetes, you are adopting opinions. If you choose Terraform, you are adopting opinions. If you choose Terragrunt, OpenTofu, ECS, Heroku, Waypoint, Miren, or an internal deployment system, you are adopting opinions there too.

The question is whether those opinions match your team.

Do they match your application? Do they match your operational maturity? Do they match the amount of infrastructure ownership you actually want? Do they help your team ship safely, or do they just move the pain somewhere else?

That is why I liked this conversation. It was not really a “this tool versus that tool” conversation. It was more about the shape of deployment work, and why smaller, clearer, more opinionated systems may be the right answer for more teams than we usually admit.

Sometimes the best platform is not the most powerful one.

Sometimes it is the one that lets the team ship, understand what happened, roll back when needed, and then get back to building the thing customers actually care about.

---

Additional Links Mentioned

Miren:
https://miren.dev

Miren Discord:
https://miren.dev/discord

Miren on GitHub:
https://github.com/mirendev

Evan Phoenix:
https://evanphx.dev

Evan on Bluesky:
https://bsky.app/profile/evanphx.dev

Puma:
https://puma.io

Rubinius:
https://github.com/rubinius/rubinius

Terraform:
https://www.terraform.io

Terraform Enterprise:
https://developer.hashicorp.com/terraform/enterprise

Terraform Cloud:
https://developer.hashicorp.com/terraform/cloud-docs

Terragrunt:
https://terragrunt.gruntwork.io

OpenTofu:
https://opentofu.org

HashiCorp Waypoint:
https://github.com/hashicorp/waypoint

Kubernetes:
https://kubernetes.io

Docker:
https://www.docker.com

AWS ECS:
https://aws.amazon.com/ecs/

Heroku:
https://www.heroku.com

Knative:
https://knative.dev

HashiCorp Vault:
https://developer.hashicorp.com/vault