Try reloading the player or open this episode directly on YouTube.
This episode of Ship It Weekly discusses the recent GitHub outage affecting critical developer workflows and highlights the PleaseFix vulnerability in agentic browsers.
Now Playing
GitHub Outage, PleaseFix Agentic Browser Vulnerability, AWS Certificate Manager Drops Email Validation, Cloudflare TypeScript CI Workflows, AI Observability Consolidation, and the Hidden Cost of “Simple” Platform Changes
Ship It Weekly
0:0017:39
Chapters
Jump to a section in this episode.
Speed & share
Transcript
GitHub had another widespread outage this week.
Agentic browsers have a vulnerability that can
turn a webpage into an execution path. And AWS
is finally pushing Certificate Manager users
away from email validation. This week is mostly
about dependencies we treat as boring until they
become the incident. I'm Brian Teller from Teller's
Tech, and this is Ship It Weekly. Welcome back
to Ship It Weekly, the show about the DevOps,
SRE, cloud, platform, and security stories that
matter when you are the person keeping the thing
running at three in the morning. For the weekly
story list and source links, check out OnCallBrief.com. For past episodes and show notes, head
over to ShipItWeekly.fm. This week, GitHub had
a major outage. We have a new agentic-browser
vulnerability called PleaseFix. AWS Certificate
Manager is ending email validation. And Cloudflare
is experimenting with TypeScript-native CI workflows.
Then we have a quick lightning round and a human
closer about what happens when policy changes
far away from engineering still land directly
on the people running production. Let's get into
it. First up, GitHub had another widespread outage.
The incident affected the website, API, Actions,
pull requests, issues, webhooks, authentication,
and Copilot. At points, web and API error rates
were around 20%, with some services reportedly
much worse. The obvious takeaway is that GitHub
was down. The more useful one is that GitHub
is not really just a developer tool anymore.
For a lot of companies, GitHub sits directly
in the production path. It runs CI/CD. It holds
source code. It manages pull requests and approvals.
It may be part of authentication. It triggers
deployments. And increasingly, AI tooling depends
on it too. So when GitHub has an outage, the
impact is not just that engineers cannot push
code. You can lose deployment capability, change
management, incident automation, and sometimes
even access to the thing you need in order to
fix something else. That means GitHub belongs
in dependency planning the same way any other
critical SaaS provider does. Can you deploy if
it is unavailable? Can you roll back? Can you access
the last known-good artifact? Can you authenticate
to the systems you need? And do your incident
procedures still work if the place where your
runbook lives is also having an outage? There
is also a difference between being unable to
ship new code and being unable to recover existing
code Those are not the same risk. Maybe it is
completely acceptable for deployments to stop
when GitHub is unavailable. In fact, that may
be safer. But rollback is different. If your
rollback procedure starts by checking out a repository,
running a GitHub action, or waiting for an approval
inside GitHub, then your recovery path shares
the same dependency as your deployment path.
That is worth testing before the outage. You
do not necessarily need a second source-control
platform sitting around. But keeping signed artifacts
somewhere independent, documenting emergency
deployment procedures, and knowing exactly which
parts of your recovery path depend on GitHub
can make a huge difference. Developer infrastructure
is production infrastructure now. Treating it
otherwise is mostly wishful thinking. Next,
there is a vulnerability called PleaseFix affecting
agentic browsers. The core issue is that the
browser is no longer just displaying content.
An agentic browser can interpret content, make
decisions, use tools, and take actions. That
changes the threat model. A malicious webpage
is not only trying to trick a person anymore.
It may be trying to influence an autonomous system
that has access to credentials, local files,
cloud services, or external APIs. That is a very
different boundary. The interesting part here
is not whether the model is smart enough to recognize
a bad instruction. The interesting part is whether
the surrounding system allows that instruction
to become an action. If a webpage can indirectly
cause the agent to run a command, submit data,
or interact with another service, then the browser
has effectively become part of your execution
environment. And that means the same controls
we use everywhere else still matter. Scoped credentials.
Tool-level permissions. network restrictions,
human approval for sensitive actions, and a clean
separation between untrusted content and privileged
operations. I also think this is where the phrase
human-in-the-loop gets oversold. A confirmation
dialog is useful, but if the agent controls
the text surrounding that confirmation, summarizes
the action for the user, or performs enough harmless-looking steps before asking for approval, then
clicking yes can become almost automatic. The
better control is to make high-risk actions
technically distinct. Reading a webpage should
not implicitly grant the ability to upload a
local file. Summarizing an issue should not grant
the ability to merge a pull request. Browsing
documentation should not grant shell access.
Those boundaries should exist outside the model.
We keep talking about prompt injection like it
is a weird AI-specific problem. A lot of the
time, it is really an authorization problem wearing
new clothes. The input is untrusted, the action
is privileged, and there is not enough enforcement
between the two. Third, AWS Certificate Manager
is ending support for email validation for new
public certificates. DNS validation is becoming
the path forward. That sounds boring. It is also
exactly the kind of boring change that breaks
things months later because nobody remembers
which certificate still depends on an old workflow.
Email validation has always been awkward operationally.
Someone has to receive the validation message.
The right mailbox has to exist. The right person
has to notice it. And renewals can depend on
that process continuing to work. DNS validation
is much easier to automate and much easier to
make durable. This also lines up with another
recurring theme from this week. Certificate
expiry is still causing real outages, which is
kind of amazing in 2026. Certificates are predictable.
They have expiration dates. They are machine-readable. And yet, teams still get surprised
by them. Usually, the problem is not the certificate
itself. It is ownership. Nobody knows who owns
the domain. The renewal process depends on an
old mailbox. A certificate was created manually
years ago. Or the monitoring only checks the
application after the certificate has already
expired. Another trap is monitoring only the
certificate you think production is serving.
There may be a load balancer with one certificate,
a CDN with another, an internal service mesh
issuing its own certificates, and some forgotten
endpoint using something completely different.
The useful inventory is not just what certificates
exist. It is which endpoint presents which certificate,
who renews it, and what happens if renewal fails.
That is a much more operational question. The
fix is not complicated. Inventory the certificates.
Know who owns them. Automate renewal where possible.
Alert well before expiration, and test the renewal
path before the deadline. Certificate management
is one of those areas where boring automation
is dramatically better than heroic incident response.
Fourth, Cloudflare is experimenting with CI workflows
written in TypeScript. The interesting idea is
that the workflow itself becomes normal application
code. You can define steps, retries, concurrency,
and caching behavior in TypeScript instead of
expressing everything through a large YAML configuration.
There are some obvious advantages. You get normal
language constructs. And you can test parts of
the workflow more like software. But there is
also a tradeoff. The more programmable CI becomes,
the easier it is for the pipeline to turn into
another application nobody really owns. We have
all seen YAML pipelines become impossible to
reason about. Code can absolutely become impossible
to reason about too. There is also a governance
question here. One advantage of declarative CI
is that the set of things a pipeline can express
is intentionally constrained. Once workflows
become arbitrary code, you gain flexibility.
But you also need stronger conventions around
libraries, reviews, dependency management, and
security. Otherwise, every team eventually invents
its own tiny workflow framework. So, I do not
think that the lesson is that YAML is bad and
TypeScript is good. The lesson is that CI
systems are software systems. They need structure.
They need tests. They need clear ownership. They
need versioning. And they need enough observability
that someone can understand why step seven retried
four times and then deployed anyway. If CI is
part of your production delivery path, treating
the workflow definition like real software makes
a lot of sense. Just remember that real software
also comes with maintenance. Quick lightning
round. First, Dynatrace is acquiring Arize for
about $915 million. That is another sign that
AI observability is becoming part of the mainstream
observability platform instead of living in a
separate niche. Second, AWS open-sourced Dogwood
for governing agent tool calls. The interesting
part is policy around what an agent can do, when
it can do it, and under what conditions. That
fits directly with the PleaseFix story. Third,
Pulumi 3.258 adds opt-in credential encryption.
Small change, but a useful reminder that IaC
state and configuration often contain more sensitive
material than teams realize. And fourth, one
AWS credential breach was reportedly noticed
because the bill changed. Unexpected egress charges
became the security signal. FinOps data is not
just about cost anymore. Sometimes it is telemetry
for compromise. The human closer this week comes
from a story titled Mario saved the EU but broke
my system. The details are pretty fun, but the
part I liked is the familiar engineering experience
underneath it. A policy or platform decision
gets made somewhere far away from the team operating
the system. The change may be reasonable. It
may even be the right thing to do. But the operational
consequences still land on somebody. That somebody
is usually an engineer who did not write the
policy, did not choose the deadline, and still
has to make production work afterward. That happens
with regulations. Security requirements, browser
changes, cloud provider defaults, certificate
rules, and internal compliance projects. From
the outside the change can look simple. From
the inside it can touch identity, networking,
billing, deployment, observability, and a pile
of assumptions nobody documented. I think that
is one of the underrated parts of infrastructure
work. You spend a lot of time translating decisions
made at one layer into consequences at another.
And sometimes the hardest part is not technical
at all. It is explaining that a change described
as one checkbox in a compliance document might
actually require three teams, a migration plan,
downtime coordination, and six weeks of testing.
That is not engineers being difficult. That is
the difference between describing an outcome
and implementing it safely. The best platform
and infrastructure teams get good at making that
translation visible early, not after the deadline,
not after the outage. Early enough that everyone
understands what the change actually costs. And
the best engineers are usually the ones who can
do that without turning every change into a
fight. Understand why the change exists. Figure
out where the blast radius actually is. Communicate
the tradeoffs clearly. And make the transition
boring. That is usually the real job. Not preventing
change. Making change survivable. That's it for
this week of Ship It Weekly. We covered GitHub's
outage, the PleaseFix agentic-browser vulnerability,
AWS Certificate Manager moving away from email
validation, and Cloudflare's TypeScript CI workflows,
plus Dynatrace and Arize, AWS Dogwood, Pulumi
credential encryption, and an AWS breach detected
through unexpected egress costs. Follow or subscribe
wherever you are watching or listening. You can
find the weekly story list and source links at
OnCallBrief.com and past episodes and full show
notes at ShipItWeekly.fm. I'm Brian Teller from
Teller's Tech. Thanks for listening. And remember,
The theme I kept coming back to this week is how many systems quietly become critical long before we start treating them that way.
GitHub is probably the clearest example. For years it was easy to think of source control as something developers used to write and review code. That is not really what GitHub is anymore. It sits in deployment paths, approval workflows, automation, identity, incident response, release management, and increasingly AI tooling. When it goes down, the impact is not just “developers cannot push.” In some environments, you lose part of your ability to operate production.
The question that matters to me is not whether you can keep shipping during a GitHub outage. Stopping deployments may actually be exactly what you want. The harder question is whether you can recover. If rollback depends on checking out a repository, starting a GitHub Action, or getting an approval through the same service that is unavailable, then your emergency path has the same dependency as your normal path. That is the kind of thing that looks completely reasonable until the day you actually need it.
I do not think the answer is necessarily maintaining a second Git platform and duplicating everything. That can create more complexity than it solves. But keeping known-good artifacts somewhere independent, understanding exactly which operational procedures depend on GitHub, and testing what happens when those dependencies disappear is pretty reasonable. The important part is knowing where the dependency exists instead of discovering it during the incident.
The PleaseFix story is a different kind of dependency problem, but it gets at something I think we are still learning with agentic systems. We keep focusing on whether the model itself can recognize malicious instructions. That matters, but it is not the security boundary I would want to bet everything on.
If an agent can read an untrusted webpage and also has access to privileged tools, credentials, files, or external APIs, then the important question is what sits between those two capabilities. A webpage should be allowed to influence what the agent thinks about. It should not automatically be allowed to influence what the agent is authorized to do.
That sounds like a subtle distinction, but it is really just an old security principle showing up in a new place. Untrusted input should not directly control privileged execution. We already know how to think about that in shells, web applications, CI systems, and APIs. Agentic browsers just make the path less obvious because there is a model in the middle translating one into the other.
I also think “human in the loop” gets treated as more protection than it sometimes provides. If the same agent summarizes what it wants to do, provides the explanation, and then asks you to approve it, the human is not necessarily making an independent decision. The safer architecture is one where dangerous actions are structurally different. Reading a webpage and uploading a file should require different authority. Looking at an issue and merging code should require different authority. The model should not be the component defining where that line sits.
The AWS Certificate Manager change is much less exciting, but it may be the most operationally familiar story in the episode. Certificates are one of those things that feel solved until one expires. The expiration date was always known. The certificate was always discoverable. The outage still happens because ownership, renewal, monitoring, or automation was not as clear as everyone assumed.
Moving away from email validation is a good default because email introduces a human process into something that is much better handled as infrastructure. Mailboxes disappear. People change roles. Distribution lists get forgotten. DNS validation is not magically perfect, but it gives teams a much more durable automation path.
The bigger lesson is that certificate inventory needs to be tied to endpoints and ownership. Knowing that a certificate exists is not enough. You need to know where it is actually being served, who is responsible for renewing it, what system performs that renewal, and how you know when that process stops working. The certificate itself usually is not the surprising part. The surprising part is discovering the one forgotten endpoint that uses a completely different renewal path.
Cloudflare’s TypeScript CI work is interesting because it pushes in the opposite direction. Instead of taking something complicated and making it more constrained, it takes CI and makes it more programmable.
I can see the appeal immediately. Types, functions, libraries, tests, reuse, normal programming constructs. Anyone who has maintained a giant YAML pipeline has probably had the thought that this would be easier if it were just code.
But code is not automatically simpler. We have spent years proving that.
Once pipelines become arbitrary software, they inherit software problems. Dependency management, abstraction layers, shared libraries, version compatibility, testing, security review, and eventually some internal framework that only two people completely understand. The interesting question is not YAML versus TypeScript. It is whether we are finally willing to acknowledge that CI/CD has become application software and operate it accordingly.
That means ownership. Tests. Observability. Release discipline. Documentation. And probably a willingness to delete clever abstractions when they start making the system harder to understand than the problem they were supposed to solve.
Even the lightning stories fit this broader pattern. Dynatrace buying Arize shows AI observability getting absorbed into the normal observability stack. Dogwood is another attempt to put policy between an agent and the tools it can use. Pulumi is adding stronger credential protection around infrastructure configuration. And the AWS compromise detected through egress costs is a reminder that useful operational signals do not always come from the security product.
Sometimes the first indication that something is wrong is the bill.
That is why I like having FinOps, security, SRE, and platform engineering increasingly overlap. They are all looking at different symptoms of the same systems. A cost anomaly might be a deployment mistake. It might be a runaway workload. It might be credential abuse. The more those teams can share signals instead of treating them as separate domains, the faster somebody is likely to notice that the system is behaving differently than expected.
If I had to boil this episode down to one thing, it would be that boundaries and dependencies both need to be explicit.
Know which systems your recovery path depends on.
Know what authority an agent actually has.
Know who owns the certificate.
Know whether your CI pipeline is configuration or software.
And know which signals might tell you something is wrong before the obvious alarm fires.
The things that cause the biggest incidents are often not mysterious.
They are usually the dependencies everybody knew existed, but nobody realized had become critical.
📝 Notes
Show Notes
This week on Ship It Weekly: GitHub suffers another widespread outage affecting the web interface, APIs, Actions, authentication, Copilot, and other critical developer workflows. Zenity Labs demonstrates PleaseFix attacks against agentic browsers, where malicious content can influence agents with access to authenticated sessions and privileged tools. AWS Certificate Manager is moving away from email validation, and Cloudflare is experimenting with CI pipelines defined as TypeScript instead of YAML.
The bigger theme this week is dependencies and boundaries we tend to ignore until something breaks. GitHub is no longer just where the code lives. Agentic browsers are no longer just displaying webpages. Certificate renewal is not something you want depending on someone checking an inbox. And CI pipelines have become software systems of their own.
The theme I kept coming back to this week is how many systems quietly become critical long before we start treating them that way.
GitHub is probably the clearest example. For years it was easy to think of source control as something developers used to write and review code. That is not really what GitHub is anymore. It sits in deployment paths, approval workflows, automation, identity, incident response, release management, and increasingly AI tooling. When it goes down, the impact is not just “developers cannot push.” In some environments, you lose part of your ability to operate production.
The question that matters to me is not whether you can keep shipping during a GitHub outage. Stopping deployments may actually be exactly what you want. The harder question is whether you can recover. If rollback depends on checking out a repository, starting a GitHub Action, or getting an approval through the same service that is unavailable, then your emergency path has the same dependency as your normal path. That is the kind of thing that looks completely reasonable until the day you actually need it.
I do not think the answer is necessarily maintaining a second Git platform and duplicating everything. That can create more complexity than it solves. But keeping known-good artifacts somewhere independent, understanding exactly which operational procedures depend on GitHub, and testing what happens when those dependencies disappear is pretty reasonable. The important part is knowing where the dependency exists instead of discovering it during the incident.
The PleaseFix story is a different kind of dependency problem, but it gets at something I think we are still learning with agentic systems. We keep focusing on whether the model itself can recognize malicious instructions. That matters, but it is not the security boundary I would want to bet everything on.
If an agent can read an untrusted webpage and also has access to privileged tools, credentials, files, or external APIs, then the important question is what sits between those two capabilities. A webpage should be allowed to influence what the agent thinks about. It should not automatically be allowed to influence what the agent is authorized to do.
That sounds like a subtle distinction, but it is really just an old security principle showing up in a new place. Untrusted input should not directly control privileged execution. We already know how to think about that in shells, web applications, CI systems, and APIs. Agentic browsers just make the path less obvious because there is a model in the middle translating one into the other.
I also think “human in the loop” gets treated as more protection than it sometimes provides. If the same agent summarizes what it wants to do, provides the explanation, and then asks you to approve it, the human is not necessarily making an independent decision. The safer architecture is one where dangerous actions are structurally different. Reading a webpage and uploading a file should require different authority. Looking at an issue and merging code should require different authority. The model should not be the component defining where that line sits.
The AWS Certificate Manager change is much less exciting, but it may be the most operationally familiar story in the episode. Certificates are one of those things that feel solved until one expires. The expiration date was always known. The certificate was always discoverable. The outage still happens because ownership, renewal, monitoring, or automation was not as clear as everyone assumed.
Moving away from email validation is a good default because email introduces a human process into something that is much better handled as infrastructure. Mailboxes disappear. People change roles. Distribution lists get forgotten. DNS validation is not magically perfect, but it gives teams a much more durable automation path.
The bigger lesson is that certificate inventory needs to be tied to endpoints and ownership. Knowing that a certificate exists is not enough. You need to know where it is actually being served, who is responsible for renewing it, what system performs that renewal, and how you know when that process stops working. The certificate itself usually is not the surprising part. The surprising part is discovering the one forgotten endpoint that uses a completely different renewal path.
Cloudflare’s TypeScript CI work is interesting because it pushes in the opposite direction. Instead of taking something complicated and making it more constrained, it takes CI and makes it more programmable.
I can see the appeal immediately. Types, functions, libraries, tests, reuse, normal programming constructs. Anyone who has maintained a giant YAML pipeline has probably had the thought that this would be easier if it were just code.
But code is not automatically simpler. We have spent years proving that.
Once pipelines become arbitrary software, they inherit software problems. Dependency management, abstraction layers, shared libraries, version compatibility, testing, security review, and eventually some internal framework that only two people completely understand. The interesting question is not YAML versus TypeScript. It is whether we are finally willing to acknowledge that CI/CD has become application software and operate it accordingly.
That means ownership. Tests. Observability. Release discipline. Documentation. And probably a willingness to delete clever abstractions when they start making the system harder to understand than the problem they were supposed to solve.
Even the lightning stories fit this broader pattern. Dynatrace buying Arize shows AI observability getting absorbed into the normal observability stack. Dogwood is another attempt to put policy between an agent and the tools it can use. Pulumi is adding stronger credential protection around infrastructure configuration. And the AWS compromise detected through egress costs is a reminder that useful operational signals do not always come from the security product.
Sometimes the first indication that something is wrong is the bill.
That is why I like having FinOps, security, SRE, and platform engineering increasingly overlap. They are all looking at different symptoms of the same systems. A cost anomaly might be a deployment mistake. It might be a runaway workload. It might be credential abuse. The more those teams can share signals instead of treating them as separate domains, the faster somebody is likely to notice that the system is behaving differently than expected.
If I had to boil this episode down to one thing, it would be that boundaries and dependencies both need to be explicit.
Know which systems your recovery path depends on.
Know what authority an agent actually has.
Know who owns the certificate.
Know whether your CI pipeline is configuration or software.
And know which signals might tell you something is wrong before the obvious alarm fires.
The things that cause the biggest incidents are often not mysterious.
They are usually the dependencies everybody knew existed, but nobody realized had become critical.