Try reloading the player or open this episode directly on YouTube.
This episode of Ship It Weekly discusses a critical vulnerability in Atlassian's Data Center products, including Jira and Confluence. It also covers GitHub Copilot's new sandboxing feature, AWS Lambda SnapStart's operational risks, and the availability of Google Spanner Omni.
Now Playing
Critical Atlassian CVE Hits Jira, Confluence & Bitbucket, GitHub Copilot Sandboxing Goes GA, AWS Lambda SnapStart for Containers & Google Spanner Omni
Ship It Weekly
0:0017:56
Chapters
Jump to a section in this episode.
Speed & share
Transcript
Atlassian has a critical vulnerability affecting
eight of its Data Center products. GitHub is
putting Copilot agents behind local sandboxes
that enterprises can actually enforce. And Google
has taken Spanner out of Google Cloud and turned
it into a database you can run almost anywhere.
I'm Brian Teller from Teller's Tech, and this
is Ship It Weekly. Welcome back to Ship It Weekly,
the show about the DevOps, SRE, cloud, platform,
and security stories that matter when you are
the person keeping the thing running at three
in the morning. For the weekly story list and
source links, check out OnCallBrief.com. For
past episodes and show notes, head over to ShipItWeekly.fm.
The thread running through this week’s episode is control.
Not control in the abstract sense of “we should have more governance” or “we need better policies.” I mean the practical version. When something goes wrong, what can actually stop it? What can the system touch? Who owns the failure? And how much of that control is real versus something we assume is there because the product or platform feels managed?
The Atlassian vulnerability is a good example of why developer tooling belongs in the same security conversation as production infrastructure.
Jira, Confluence, Bitbucket, Bamboo, and the rest of that stack are not just collaboration tools. They hold source code, deployment information, internal documentation, credentials, architectural decisions, incident notes, and all kinds of operational context.
If somebody can get into the tooling your engineers use to build and operate production, that can become a production problem very quickly.
I think teams still mentally separate these things more than they should.
There is “production infrastructure,” and then there is “the stuff developers use.”
But the boundary between the two is increasingly fake.
Your source repository can trigger deployments.
Your CI system can assume cloud roles.
Your issue tracker can contain secrets somebody pasted into a ticket three years ago.
Your documentation can explain exactly how your network is built.
Your internal developer portal can tell somebody which service is critical, who owns it, and how it gets deployed.
The tooling around production is part of the attack surface around production.
That does not mean every Jira bug is a Sev-1. It means the blast radius of developer tooling deserves the same kind of thinking we already apply to cloud accounts, Kubernetes clusters, and production databases.
The Copilot sandboxing story gets at the same idea from a different angle.
I like agents. I use AI constantly. I think there is a lot of value in letting software do more work for us.
But I do not want the safety model to be:
“Hopefully the model behaves.”
That is not much of a control.
If an agent has access to my filesystem, Git credentials, SSH keys, GitHub token, internal network, and production tooling, I care a lot less about whether it is usually correct than I do about what happens the one time it is wrong.
That is why local sandboxing matters.
You are moving the boundary outside the model.
The model can make a mistake.
The sandbox can still say no.
That is much closer to how we already think about security everywhere else.
We do not give every engineer root on every production system and then ask them nicely not to use it.
We build RBAC.
We use short-lived credentials.
We scope IAM roles.
We separate environments.
We put guardrails around the action instead of assuming the actor will always make the right decision.
Agents should not be treated differently just because the thing issuing the command happens to be an LLM.
The question I keep coming back to is:
What can it touch while it is wrong?
That applies to AI agents, but it applies to automation generally.
A Terraform plan can be wrong.
A controller can be wrong.
A CI pipeline can be wrong.
A script can be wrong.
A human can definitely be wrong.
The safety of the system depends on what happens after that mistake.
If the answer is “it can destroy everything because it had permission,” then the interesting part is not the mistake.
The interesting part is the permission model.
That connects directly to the human closer this week.
The Google Cloud networking incident is a really useful example because there was no single cartoonishly bad decision.
One person correctly followed an incorrect procedure.
Another person incorrectly followed a correct procedure.
That is exactly the kind of thing real operations work looks like.
People are working from documentation.
They are moving through maintenance steps.
They are dealing with infrastructure that has redundancy, dependencies, and state spread across multiple components.
Eventually somebody is going to misunderstand something.
Eventually a procedure is going to be stale.
Eventually two individually reasonable actions are going to combine into something nobody expected.
If the system only stays up when every human follows every step perfectly, every time, then the human is not the only problem.
The system is brittle.
Training is useful.
Documentation matters.
Runbooks matter.
But “we will retrain the engineer” is not a substitute for a guardrail.
If disconnecting one more fiber path is going to remove the last healthy path for a region, I want the system screaming before that cable gets pulled.
Better yet, I want the system to make the unsafe action difficult or impossible unless somebody deliberately overrides it.
That is where reliability and security start looking very similar.
Both are really asking:
What happens when somebody or something does the wrong thing?
The Lambda SnapStart story also fits here, although it is less obvious.
SnapStart is a performance optimization.
You initialize the function ahead of time, snapshot the environment, and restore from that snapshot instead of doing a full cold start.
That is useful.
But now the lifecycle changed.
Something that used to be created fresh during startup might now be copied into every restored environment.
Random state.
Credentials.
Connections.
IDs.
Caches.
Anything initialized before the snapshot.
The feature did not remove the lifecycle.
It gave you a different lifecycle.
That is a pattern I see constantly in infrastructure.
We introduce an abstraction to remove one operational problem and end up with a different operational problem.
Containers did not eliminate servers.
Kubernetes did not eliminate infrastructure.
Managed databases did not eliminate database operations.
Serverless did not eliminate runtime behavior.
Snapshots do not eliminate initialization.
They move where you need to think about it.
Spanner Omni might be the clearest version of that.
People have wanted Spanner outside Google Cloud for years.
Now you can have it.
You can run the database engine in your own environment, on another cloud, on Kubernetes, or on infrastructure you manage yourself.
That gives you portability and control.
It also gives you the pager.
Google can provide the database technology.
Google cannot guarantee the network underneath it.
They cannot guarantee your disks.
They cannot guarantee your Kubernetes cluster.
They cannot guarantee that you configured replication correctly.
That is why there is no Google availability SLA attached to Spanner Omni.
You wanted ownership.
You got ownership.
And I do not mean that negatively.
There are plenty of legitimate reasons to want that tradeoff.
Regulatory requirements.
Data locality.
Multi-cloud strategy.
Existing infrastructure.
Cost.
Organizational constraints.
Sometimes operating it yourself is absolutely the right decision.
But portability and operational responsibility are usually tied together.
You cannot ask the cloud provider to stop owning the infrastructure and then expect them to keep owning the outcome.
That is probably the larger point across all four stories.
Abstractions are useful.
Managed services are useful.
Automation is useful.
AI agents are useful.
But none of them remove responsibility.
They move it.
Sometimes they move responsibility from the application team to the platform team.
Sometimes from the platform team to AWS.
Sometimes from AWS back to you.
Sometimes from a human to an agent.
Sometimes from an agent to a sandbox.
The important thing is knowing where it moved.
Because that is usually where the failure ends up too.
I have talked about this in the context of AI-generated infrastructure before. The dangerous part is rarely that the tool generated something obviously insane. The dangerous part is when the output looks reasonable enough that everybody assumes the surrounding controls will catch the mistake.
That is why I keep coming back to guardrails.
Not because guardrails prevent every failure.
They do not.
But they change the consequence of being wrong.
A bad Terraform plan that cannot touch production is annoying.
A bad Terraform plan with admin access to the entire AWS organization is an incident.
An AI agent that writes the wrong file inside a sandbox is annoying.
An AI agent that can read your credentials and reach production is a different problem.
An engineer following the wrong procedure while the network still has a hard safety boundary is recoverable.
An engineer following the wrong procedure where one more action can remove the last healthy path can take down a region.
The mistake may be the same.
The system around the mistake determines the blast radius.
That is the part we actually get to engineer.
📝 Notes
Show Notes
This week on Ship It Weekly: Atlassian disclosed a critical arbitrary file access vulnerability affecting eight Data Center products, including Jira, Confluence, Bitbucket, and Bamboo. GitHub Copilot local sandboxing is now generally available, giving teams enforceable controls around filesystem, network, credentials, and local tooling. AWS published a deeper look at Lambda SnapStart for container images, including the operational risks of restoring initialized state. And Google Spanner Omni is now generally available, bringing Spanner outside Google Cloud and putting more of the availability responsibility back on the customer.
The bigger theme this week is control. Developer tools are production infrastructure. AI agents need real execution boundaries, not just instructions. Faster startup through snapshots means understanding exactly what state is being restored. And moving a managed database onto your own infrastructure gives you portability, but also gives you the pager.
In the lightning round: Citrix has a NetScaler SAML vulnerability under active exploitation. Cloudflare is working toward becoming a publicly trusted certificate authority. And attackers are actively scanning for Rejetto HFS servers vulnerable to a flaw that can lead to account takeover and remote code execution.
And the human closer comes from Lorin Hochstein and a recent Google Cloud networking incident. One part of the outage involved someone correctly following an incorrect procedure, while another involved someone incorrectly following a correct one. Training matters, but systems also need guardrails that keep normal human mistakes from turning into large outages. If your system only works when nobody makes a mistake, the system is the problem.
Control Is the Part You Actually Own
The thread running through this week’s episode is control.
Not control in the abstract sense of “we should have more governance” or “we need better policies.” I mean the practical version. When something goes wrong, what can actually stop it? What can the system touch? Who owns the failure? And how much of that control is real versus something we assume is there because the product or platform feels managed?
The Atlassian vulnerability is a good example of why developer tooling belongs in the same security conversation as production infrastructure.
Jira, Confluence, Bitbucket, Bamboo, and the rest of that stack are not just collaboration tools. They hold source code, deployment information, internal documentation, credentials, architectural decisions, incident notes, and all kinds of operational context.
If somebody can get into the tooling your engineers use to build and operate production, that can become a production problem very quickly.
I think teams still mentally separate these things more than they should.
There is “production infrastructure,” and then there is “the stuff developers use.”
But the boundary between the two is increasingly fake.
Your source repository can trigger deployments.
Your CI system can assume cloud roles.
Your issue tracker can contain secrets somebody pasted into a ticket three years ago.
Your documentation can explain exactly how your network is built.
Your internal developer portal can tell somebody which service is critical, who owns it, and how it gets deployed.
The tooling around production is part of the attack surface around production.
That does not mean every Jira bug is a Sev-1. It means the blast radius of developer tooling deserves the same kind of thinking we already apply to cloud accounts, Kubernetes clusters, and production databases.
The Copilot sandboxing story gets at the same idea from a different angle.
I like agents. I use AI constantly. I think there is a lot of value in letting software do more work for us.
But I do not want the safety model to be:
“Hopefully the model behaves.”
That is not much of a control.
If an agent has access to my filesystem, Git credentials, SSH keys, GitHub token, internal network, and production tooling, I care a lot less about whether it is usually correct than I do about what happens the one time it is wrong.
That is why local sandboxing matters.
You are moving the boundary outside the model.
The model can make a mistake.
The sandbox can still say no.
That is much closer to how we already think about security everywhere else.
We do not give every engineer root on every production system and then ask them nicely not to use it.
We build RBAC.
We use short-lived credentials.
We scope IAM roles.
We separate environments.
We put guardrails around the action instead of assuming the actor will always make the right decision.
Agents should not be treated differently just because the thing issuing the command happens to be an LLM.
The question I keep coming back to is:
What can it touch while it is wrong?
That applies to AI agents, but it applies to automation generally.
A Terraform plan can be wrong.
A controller can be wrong.
A CI pipeline can be wrong.
A script can be wrong.
A human can definitely be wrong.
The safety of the system depends on what happens after that mistake.
If the answer is “it can destroy everything because it had permission,” then the interesting part is not the mistake.
The interesting part is the permission model.
That connects directly to the human closer this week.
The Google Cloud networking incident is a really useful example because there was no single cartoonishly bad decision.
One person correctly followed an incorrect procedure.
Another person incorrectly followed a correct procedure.
That is exactly the kind of thing real operations work looks like.
People are working from documentation.
They are moving through maintenance steps.
They are dealing with infrastructure that has redundancy, dependencies, and state spread across multiple components.
Eventually somebody is going to misunderstand something.
Eventually a procedure is going to be stale.
Eventually two individually reasonable actions are going to combine into something nobody expected.
If the system only stays up when every human follows every step perfectly, every time, then the human is not the only problem.
The system is brittle.
Training is useful.
Documentation matters.
Runbooks matter.
But “we will retrain the engineer” is not a substitute for a guardrail.
If disconnecting one more fiber path is going to remove the last healthy path for a region, I want the system screaming before that cable gets pulled.
Better yet, I want the system to make the unsafe action difficult or impossible unless somebody deliberately overrides it.
That is where reliability and security start looking very similar.
Both are really asking:
What happens when somebody or something does the wrong thing?
The Lambda SnapStart story also fits here, although it is less obvious.
SnapStart is a performance optimization.
You initialize the function ahead of time, snapshot the environment, and restore from that snapshot instead of doing a full cold start.
That is useful.
But now the lifecycle changed.
Something that used to be created fresh during startup might now be copied into every restored environment.
Random state.
Credentials.
Connections.
IDs.
Caches.
Anything initialized before the snapshot.
The feature did not remove the lifecycle.
It gave you a different lifecycle.
That is a pattern I see constantly in infrastructure.
We introduce an abstraction to remove one operational problem and end up with a different operational problem.
Containers did not eliminate servers.
Kubernetes did not eliminate infrastructure.
Managed databases did not eliminate database operations.
Serverless did not eliminate runtime behavior.
Snapshots do not eliminate initialization.
They move where you need to think about it.
Spanner Omni might be the clearest version of that.
People have wanted Spanner outside Google Cloud for years.
Now you can have it.
You can run the database engine in your own environment, on another cloud, on Kubernetes, or on infrastructure you manage yourself.
That gives you portability and control.
It also gives you the pager.
Google can provide the database technology.
Google cannot guarantee the network underneath it.
They cannot guarantee your disks.
They cannot guarantee your Kubernetes cluster.
They cannot guarantee that you configured replication correctly.
That is why there is no Google availability SLA attached to Spanner Omni.
You wanted ownership.
You got ownership.
And I do not mean that negatively.
There are plenty of legitimate reasons to want that tradeoff.
Regulatory requirements.
Data locality.
Multi-cloud strategy.
Existing infrastructure.
Cost.
Organizational constraints.
Sometimes operating it yourself is absolutely the right decision.
But portability and operational responsibility are usually tied together.
You cannot ask the cloud provider to stop owning the infrastructure and then expect them to keep owning the outcome.
That is probably the larger point across all four stories.
Abstractions are useful.
Managed services are useful.
Automation is useful.
AI agents are useful.
But none of them remove responsibility.
They move it.
Sometimes they move responsibility from the application team to the platform team.
Sometimes from the platform team to AWS.
Sometimes from AWS back to you.
Sometimes from a human to an agent.
Sometimes from an agent to a sandbox.
The important thing is knowing where it moved.
Because that is usually where the failure ends up too.
I have talked about this in the context of AI-generated infrastructure before. The dangerous part is rarely that the tool generated something obviously insane. The dangerous part is when the output looks reasonable enough that everybody assumes the surrounding controls will catch the mistake.
That is why I keep coming back to guardrails.
Not because guardrails prevent every failure.
They do not.
But they change the consequence of being wrong.
A bad Terraform plan that cannot touch production is annoying.
A bad Terraform plan with admin access to the entire AWS organization is an incident.
An AI agent that writes the wrong file inside a sandbox is annoying.
An AI agent that can read your credentials and reach production is a different problem.
An engineer following the wrong procedure while the network still has a hard safety boundary is recoverable.
An engineer following the wrong procedure where one more action can remove the last healthy path can take down a region.
The mistake may be the same.
The system around the mistake determines the blast radius.
That is the part we actually get to engineer.