Try reloading the player or open this episode directly on YouTube.
This episode of Ship It Weekly discusses Amazon Linux 2027's public preview featuring SELinux by default and updates for compatibility testing. It also covers GitHub Actions' new cache permissions to enhance security and the critical RCE vulnerability in N-able N-central.
Now Playing
Amazon Linux 2027, GitHub Actions Cache Security, Secret-Scanning Merge Blocks, N-central CVSS 10 RCE, Karmada Graduation, ShieldCrash, CodeQL ARM64 & When Observability Fails Too
Ship It Weekly
0:0014:47
Chapters
Jump to a section in this episode.
Speed & share
Transcript
Amazon Linux 2027 is now in public preview with
SELinux enforcing by default DNF5 and a newer
kernel. GitHub Actions now lets you explicitly
control cache permissions to reduce cache-poisoning
risk. And N-able N-central has a CVSS 10 pre-
auth remote code execution flaw that is already
being exploited. I'm Brian Teller from Teller's
Tech, and this is Ship It Weekly. Welcome back
to Ship It Weekly, the show about the DevOps,
SRE, cloud, platform, and security stories that
matter when you are the person keeping the thing
running at three in the morning. For the weekly
story list and source links, check out OnCallBrief.com. For past episodes and show notes, head
over to ShipItWeekly.fm. This week, Amazon Linux
2027 enters public preview with some changes
that are going to matter for compatibility. GitHub
tightens Actions cache permissions. GitHub can
now block pull requests that introduce exposed
secrets. And N-able N-central has a critical
pre-auth RCE under active exploitation. Then
we have a quick lightning round and a human closer
about a disk filling up at 2:47 in the morning
and taking observability down with it. Let's
get into it. First up, Amazon Linux 2027 is now
in public preview. This is more than a routine
version refresh. You get a newer kernel, DNF5,
newer language runtimes, updated Graviton support,
AWS-LC, and an x86-64-v3 baseline. SELinux is
also enforcing by default. That one is probably
the first thing I would test. If you have custom
agents, bootstrap scripts, monitoring software,
or anything that expects broad filesystem access,
this is where assumptions can start surfacing.
And that is really the point of the preview.
Operating system upgrades are rarely hard because
of the OS itself. They are hard because of everything
that quietly depended on the old one. Custom
AMIs, package installs, systemd units, cloud-
init, security agents, image pipelines. AWS is
explicitly telling teams to test their automation
now. Terraform, CloudFormation, CI/CD, custom
AMIs. There are also preview limitations. SSM
Patch Manager does not currently support Amazon
Linux 2027 preview. That is not a reason to avoid
testing it. It is exactly why you test it. The
x86-64-v3 baseline is also worth paying attention
to if you have older hardware assumptions or
odd build dependencies. The OS is part of your
platform whether developers think about it or
not. So I would treat this preview as a compatibility
test opportunity. Not a migration event yet.
Next, GitHub Actions now lets you explicitly
control cache permissions. And this is a better
security change than it probably looks like at
first glance. Build caches feel like a performance
The thing that stood out to me this week is how much of infrastructure security is really about making boundaries explicit before somebody finds out the hard way where they actually are.
The GitHub cache change is probably my favorite example.
Most of us think about CI caches as a performance optimization. You cache dependencies or build artifacts so the next job runs faster. Pretty boring.
But if an untrusted pull request can write something into a cache, and a privileged workflow restores that cache later, the cache just became a security boundary.
The attacker does not necessarily need your production credentials. They need control over something that eventually gets trusted by a job that does have those credentials.
That is why I like GitHub adding explicit read, write, write-only, and no-access modes. It is not some huge new security product. It is taking an existing capability and finally letting us describe who should actually be able to do what with it.
And I think there are probably a lot of things in CI/CD that we still treat as plumbing that should really be treated as security controls.
Artifacts are one.
Caches are one.
Runner selection is another.
Reusable workflows definitely are.
If your deployment process consumes something created earlier in the pipeline, you should probably know who was allowed to create it.
The secret-scanning change is similar.
I have always preferred controls that happen as close as possible to the developer doing the work.
Finding a leaked credential in a security dashboard three days later is useful.
Stopping the pull request from merging is much better.
Push protection is better still if you can stop it before the secret ever gets committed.
But none of those controls have to be mutually exclusive.
People bypass things.
Detection rules change.
Secrets get introduced in weird ways.
Having another enforcement point at merge gives you another chance to catch it.
And importantly, it does not depend on somebody remembering to go check another dashboard.
The platform just says no.
Fix this first.
That is what good guardrails should do.
The Amazon Linux story is a little different, but I think there is a similar operational lesson.
Nobody gets particularly excited about testing a new base operating system.
But Amazon Linux 2027 changing things like the kernel, DNF, crypto libraries, CPU baseline, and SELinux defaults is exactly the kind of change that exposes assumptions you forgot were assumptions.
Maybe your monitoring agent expects something that is no longer there.
Maybe some bootstrap script needs filesystem access SELinux now blocks.
Maybe an old binary does not like the new CPU baseline.
Maybe your patching process depends on tooling that does not support the preview yet.
None of these are particularly interesting problems.
They are just really annoying problems when you discover them during a migration.
That is why previews are useful.
You can take your existing image pipeline, swap the base image in a test environment, and see what catches fire.
If nothing does, great.
If something does, you have time to figure out why without anybody waiting for production to come back.
The N-central story is the one I would take most seriously this week.
A CVSS 10 pre-auth RCE under active exploitation is already bad.
Put it in an RMM platform and it gets considerably worse.
These tools exist specifically to control other machines.
They install software.
Run commands.
Manage endpoints.
Usually with pretty significant privileges.
That is the whole reason you bought the thing.
It is also exactly why an attacker wants it.
We sometimes talk about protecting the control plane in Kubernetes or cloud infrastructure like it is a special architectural concept.
But there are control planes all over an organization.
Your identity provider is a control plane.
Your CI/CD platform is one.
Your endpoint-management system is one.
Your virtualization management is one.
Your backup infrastructure can be one.
And your RMM absolutely is one.
If compromising one system gives somebody legitimate mechanisms to control a thousand other systems, that one system should probably not have the same security posture as a random application server.
That sounds obvious when you say it that way.
In practice, those management systems sometimes become the oldest things in the environment because everyone is afraid to touch them.
Which is exactly the opposite of what you want.
Then there is the disk failure from the closer.
I like that story because there is nothing exotic about it.
The disk filled.
That is probably one of the oldest infrastructure failures there is.
The interesting part is that the failure started taking observability with it.
And I have seen versions of that problem plenty of times.
You are investigating why something is broken, but the logs you need are on the machine that is broken.
Or the monitoring agent is starved for the same CPU or memory as the application.
Or the network path you use to collect telemetry is the network path that just failed.
Suddenly the incident is not just, “Why is production down?”
It is, “Why is production down, and why did all my graphs stop updating at exactly the moment I needed them?”
That is a much harder problem.
I do not think the answer is to build some completely separate, bulletproof observability environment that shares nothing with production. You could spend an absurd amount of money trying to eliminate every shared failure domain.
The useful question is simpler.
What failures can blind me?
If this node dies, do I still know it died?
If this disk fills, did the logs already leave the machine?
If this cluster goes away, can something outside the cluster tell me?
If this AWS account has a problem, is every tool I need to diagnose it inside that same account?
You do not need independence everywhere.
You need enough independence that your most important failure modes still leave evidence behind.
The disk story also gets into something I think we occasionally get wrong with alerting.
An alert that fires before the outage is not automatically a good alert.
Timing matters.
If the disk hits 90 percent and you have six hours before it fills, that is useful.
If it hits 99 percent and you have forty-five seconds, technically the monitoring system caught it before the outage.
Congratulations.
Nobody could do anything about it.
The useful metric is runway.
How long do I have before this becomes a problem?
Is that enough time for automation to handle it?
Is it enough time for a human to respond?
If not, move the threshold.
That is probably the connection I see across most of this week’s stories.
A lot of infrastructure work is deciding where you want to find out something is wrong.
I would rather find out an Amazon Linux dependency is broken in a preview environment.
I would rather find out a cache permission is wrong before an untrusted workflow writes to it.
I would rather find a secret while the pull request is still open.
I definitely want to find an N-central vulnerability before somebody is actively using my management plane against me.
And I want to know a disk is filling while I still have enough disk left to do something about it.
None of that is particularly flashy engineering.
It is mostly moving discovery earlier.
But moving discovery earlier is often the difference between fixing something during normal working hours and explaining at three in the morning why all the dashboards went blank at the same time production went down.
📝 Notes
Show Notes
This week on Ship It Weekly: Amazon Linux 2027 enters public preview with kernel 7.1+, SELinux enforcing by default, DNF5, newer language runtimes, AWS-LC, and an x86-64-v3 baseline. GitHub Actions adds explicit cache permissions to reduce cache-poisoning risk. GitHub can now block pull requests from merging when they introduce exposed secrets. And N-able N-central has a critical pre-auth RCE that Huntress says is being actively exploited in the wild.
The bigger theme this week is catching problems before they turn into incidents. Amazon Linux 2027 gives teams time to test AMIs, bootstrap scripts, agents, Terraform, CloudFormation, and CI/CD before the next platform generation becomes production reality. GitHub’s new cache controls make workflow trust boundaries explicit instead of leaving them implied. And secret-scanning rulesets move credential detection directly into the merge path, where developers can actually act on it.
In the lightning round: Karmada graduates from the CNCF as multi-cluster and distributed AI scheduling grow, ShieldCrash research claims another Microsoft Defender patch bypass with SYSTEM-level access, CodeQL 2.27 adds native Linux ARM64 support, and Dependabot can now read private GitHub Packages without another personal access token.
And the human closer is about what happens when observability shares the same failure domain as the thing it is watching. A full disk is bad enough. It gets worse when logs stop writing, monitoring data disappears, and the tools used to diagnose the outage start failing too. The takeaway is not that every monitoring component needs total isolation. It is that you should know what can blind you, and make sure at least one useful signal survives the failures you care about most.
The thing that stood out to me this week is how much of infrastructure security is really about making boundaries explicit before somebody finds out the hard way where they actually are.
The GitHub cache change is probably my favorite example.
Most of us think about CI caches as a performance optimization. You cache dependencies or build artifacts so the next job runs faster. Pretty boring.
But if an untrusted pull request can write something into a cache, and a privileged workflow restores that cache later, the cache just became a security boundary.
The attacker does not necessarily need your production credentials. They need control over something that eventually gets trusted by a job that does have those credentials.
That is why I like GitHub adding explicit read, write, write-only, and no-access modes. It is not some huge new security product. It is taking an existing capability and finally letting us describe who should actually be able to do what with it.
And I think there are probably a lot of things in CI/CD that we still treat as plumbing that should really be treated as security controls.
Artifacts are one.
Caches are one.
Runner selection is another.
Reusable workflows definitely are.
If your deployment process consumes something created earlier in the pipeline, you should probably know who was allowed to create it.
The secret-scanning change is similar.
I have always preferred controls that happen as close as possible to the developer doing the work.
Finding a leaked credential in a security dashboard three days later is useful.
Stopping the pull request from merging is much better.
Push protection is better still if you can stop it before the secret ever gets committed.
But none of those controls have to be mutually exclusive.
People bypass things.
Detection rules change.
Secrets get introduced in weird ways.
Having another enforcement point at merge gives you another chance to catch it.
And importantly, it does not depend on somebody remembering to go check another dashboard.
The platform just says no.
Fix this first.
That is what good guardrails should do.
The Amazon Linux story is a little different, but I think there is a similar operational lesson.
Nobody gets particularly excited about testing a new base operating system.
But Amazon Linux 2027 changing things like the kernel, DNF, crypto libraries, CPU baseline, and SELinux defaults is exactly the kind of change that exposes assumptions you forgot were assumptions.
Maybe your monitoring agent expects something that is no longer there.
Maybe some bootstrap script needs filesystem access SELinux now blocks.
Maybe an old binary does not like the new CPU baseline.
Maybe your patching process depends on tooling that does not support the preview yet.
None of these are particularly interesting problems.
They are just really annoying problems when you discover them during a migration.
That is why previews are useful.
You can take your existing image pipeline, swap the base image in a test environment, and see what catches fire.
If nothing does, great.
If something does, you have time to figure out why without anybody waiting for production to come back.
The N-central story is the one I would take most seriously this week.
A CVSS 10 pre-auth RCE under active exploitation is already bad.
Put it in an RMM platform and it gets considerably worse.
These tools exist specifically to control other machines.
They install software.
Run commands.
Manage endpoints.
Usually with pretty significant privileges.
That is the whole reason you bought the thing.
It is also exactly why an attacker wants it.
We sometimes talk about protecting the control plane in Kubernetes or cloud infrastructure like it is a special architectural concept.
But there are control planes all over an organization.
Your identity provider is a control plane.
Your CI/CD platform is one.
Your endpoint-management system is one.
Your virtualization management is one.
Your backup infrastructure can be one.
And your RMM absolutely is one.
If compromising one system gives somebody legitimate mechanisms to control a thousand other systems, that one system should probably not have the same security posture as a random application server.
That sounds obvious when you say it that way.
In practice, those management systems sometimes become the oldest things in the environment because everyone is afraid to touch them.
Which is exactly the opposite of what you want.
Then there is the disk failure from the closer.
I like that story because there is nothing exotic about it.
The disk filled.
That is probably one of the oldest infrastructure failures there is.
The interesting part is that the failure started taking observability with it.
And I have seen versions of that problem plenty of times.
You are investigating why something is broken, but the logs you need are on the machine that is broken.
Or the monitoring agent is starved for the same CPU or memory as the application.
Or the network path you use to collect telemetry is the network path that just failed.
Suddenly the incident is not just, “Why is production down?”
It is, “Why is production down, and why did all my graphs stop updating at exactly the moment I needed them?”
That is a much harder problem.
I do not think the answer is to build some completely separate, bulletproof observability environment that shares nothing with production. You could spend an absurd amount of money trying to eliminate every shared failure domain.
The useful question is simpler.
What failures can blind me?
If this node dies, do I still know it died?
If this disk fills, did the logs already leave the machine?
If this cluster goes away, can something outside the cluster tell me?
If this AWS account has a problem, is every tool I need to diagnose it inside that same account?
You do not need independence everywhere.
You need enough independence that your most important failure modes still leave evidence behind.
The disk story also gets into something I think we occasionally get wrong with alerting.
An alert that fires before the outage is not automatically a good alert.
Timing matters.
If the disk hits 90 percent and you have six hours before it fills, that is useful.
If it hits 99 percent and you have forty-five seconds, technically the monitoring system caught it before the outage.
Congratulations.
Nobody could do anything about it.
The useful metric is runway.
How long do I have before this becomes a problem?
Is that enough time for automation to handle it?
Is it enough time for a human to respond?
If not, move the threshold.
That is probably the connection I see across most of this week’s stories.
A lot of infrastructure work is deciding where you want to find out something is wrong.
I would rather find out an Amazon Linux dependency is broken in a preview environment.
I would rather find out a cache permission is wrong before an untrusted workflow writes to it.
I would rather find a secret while the pull request is still open.
I definitely want to find an N-central vulnerability before somebody is actively using my management plane against me.
And I want to know a disk is filling while I still have enough disk left to do something about it.
None of that is particularly flashy engineering.
It is mostly moving discovery earlier.
But moving discovery earlier is often the difference between fixing something during normal working hours and explaining at three in the morning why all the dashboards went blank at the same time production went down.