Cloudflare Saves 100TB of RAM, AI Drives Server Prices Up, AWS Adds a Fourth London AZ, Route 53 DNS Self-Service, AKS eBPF Routing, Go 1.27, and the Danger of Hidden Infrastructure Assumptions
Try reloading the player or open this episode directly on YouTube.
In this episode, Ship It Weekly discusses Cloudflare's 100TB RAM savings through low-level cache optimizations and the implications of AI-driven price hikes in memory. AWS adds a fourth London Availability Zone, revealing hidden infrastructure assumptions.
Now Playing
Cloudflare Saves 100TB of RAM, AI Drives Server Prices Up, AWS Adds a Fourth London AZ, Route 53 DNS Self-Service, AKS eBPF Routing, Go 1.27, and the Danger of Hidden Infrastructure Assumptions
Ship It Weekly
0:0016:48
Chapters
Jump to a section in this episode.
Speed & share
Transcript
Cloudflare shaved about 100 terabytes of memory
from the cache behind 1.1.1.1. OVHcloud says
AI demand has helped push memory costs up roughly
six-fold. And AWS added a fourth Availability
zone in London, exposing infrastructure that
quietly assumed there would always be three.
This week is mostly about assumptions that work
fine until scale or reality changes underneath
them. I'm Brian Teller from Teller's Tech, and
this is Ship It Weekly. Welcome back to Ship
It Weekly, the show about the DevOps, SRE, cloud,
platform, and security stories that matter when
you're the person that keeps the thing running
at three in the morning. For the weekly story
list and source links, check out OnCallBrief.com. For past episodes and show notes, head
over to ShipItWeekly.fm. This week, Cloudflare
shows what saving a few bytes means. when you
have more than 250 billion DNS cache entries.
OVHcloud is raising infrastructure prices as
AI changes the memory supply chain. AWS added
a fourth Availability Zone to London. And Route
53 Global Resolver is getting a better model
for cross-account DNS self-service. Then we
have four quick lightning round stories and a
human closer about something that almost every
Kubernetes engineer eventually worries about.
Running the right command against the wrong cluster.
Let's get into it. First up, Cloudflare published
a really good engineering deep dive into the
cache behind 1.1.1.1. Cloudflare's DNS services
hold more than 250 billion cached DNS entries
at any given time. At that scale, one wasted
byte per entry is more than 250 gigabytes of
memory across the fleet. So Cloudflare went looking
for bytes, not more servers, not a new caching
layer, bytes. They made five changes to how cache
entries are represented and stored in memory.
Together, those changes cut the memory footprint
of each entry by more than half. Across the fleet,
Cloudflare says that that freed roughly 100 terabytes
of RAM. And performance improved too. Insert
throughput went up 43% and lookup latency dropped
19%. That is what makes this one interesting.
We tend to think of optimization as a trade.
Spend more CPU to save memory. Add complexity
to reduce latency. But better memory layout and
fewer allocations can make the whole system faster
and smaller. It also shows how different optimization
looks at scale. If your application has 10,000
objects in memory, shaving eight bytes from each
one probably does not matter. If you have 250
billion of them, suddenly the exact layout of
a Rust struct matters a lot. Premature optimization
is still real. But once you find something that
exists billions of times, tiny inefficiencies
stop being tiny. Sometimes the big infrastructure
optimization is not a new architecture. It is
one field in a struct. Next, OVHcloud is raising
prices on some infrastructure starting in September.
And the reason is memory. OVHcloud says RAM costs
roughly six times more in June than it did a
year earlier. Suppliers are shifting manufacturing
capacity. toward high-bandwidth memory for AI
workloads which is affecting the supply and price
of more traditional RAM some newer servers are
reportedly going up around 40 to 59 percent certain
gaming servers are increasing even more this
is a good example of the second-order effects
of the ai infrastructure boom you do not need
to run an LLM You do not need GPUs. You do not
even need an AI project. You can still end up
paying for the demand AI creates further down
the supply chain. For FinOps teams, this changes
some assumptions. Right-sizing still matters.
Commit discounts still matter. Reducing idle
capacity still matters. But none of that completely
offsets the underlying hardware becoming dramatically
more expensive. It can also change whether an
optimization project is worth doing. Maybe nobody
cared about squeezing another 20% out of a memory-heavy database cluster last year. A 50% infrastructure
price increase can change that calculation quickly.
Cloud abstracts the hardware. It does not make
hardware economics disappear. Eventually, those
economics show up in the invoice. Third, AWS
added a fourth Availability Zone to the London
region. The new zone is eu-west-2d. More capacity
and more fault isolation are obviously good things.
But some customer automation immediately exposed
an assumption. London had three availability
zones. Maybe Terraform discovers all availability
zones and creates a subnet in each one. But the
module only has three CIDR blocks. Maybe there
are three NAT gateways, three route tables, three
entries in a configuration array. The code was
dynamic enough to discover the fourth zone. The
architecture was not dynamic enough to support
it. That is a subtle difference. Dynamic discovery
does not automatically mean dynamic compatibility.
If a module supports exactly three zones, validate
that and fail clearly. If it is supposed to support
any number of zones, make sure every dependent
resource actually scales with that number. Do
not let a cloud provider adding capacity accidentally
change your production architecture. Cloud APIs
are dynamic inputs. Regions change. Services expand.
Instance types appear. Availability Zones get
added. Sometimes the cloud provider changing underneath
you is not the outage. It is just your assumptions
finally getting tested. Fourth, AWS published
a useful pattern for shared DNS views with Route
53 Global Resolver. The cross-account capability
itself is not brand new. The operating model
is what I like. In a typical multi-account AWS
environment, a central networking team owns DNS
resolution. That makes sense. You probably do
not want every application team independently
changing the organization-wide resolver. But
application teams still own their services. So
when they create a private hosted zone, somebody
has to make that zone visible through central
DNS. Historically, that often becomes a ticket.
App team creates the zone. Networking team gets
the request. Networking team associates it. App
team waits. That works with five teams. It becomes
a bottleneck with 50. Shared DNS views let the
networking account keep control of the resolver,
while application accounts associate their own
private hosted zones through AWS Resource Access
Manager. The platform team keeps the guardrails.
The application team gets the self-service action
it actually needs. That is the balance I like
in platform engineering. Centralize the decisions
that need governance. Delegate the actions team
can safely own. You are not removing control.
You are removing a human scheduling dependency
from a routine deployment path. Quick lightning
round. First, some strange hardware security
research. A project called skitter-creek-bath-
salts demonstrates manipulation of memory-controller
translation registers in ways that can undermine
security assumptions higher in the CPU stack.
This mostly affects older AMD families and requires
significant privilege in practice, so this is
not a panic-and-patch story, but it is a useful
reminder that higher-level security boundaries
only protect what they can actually see. Second,
eBPF Host Routing is now generally available
in Advanced Container Networking Services for
AKS. Routing and masquerading can move away from
the traditional iptables path and into eBPF,
which can improve latency, throughput, and CPU
usage. But host-level iptables rules are bypassed.
So understand what your workloads and node tooling
depend on before you flip the switch. Third,
CloudFront Functions has a nice observability
improvement. Functions can now add custom context
directly into CloudFront access logs. That means
less work correlating separate Function logs
with the request that triggered them. Small feature,
much nicer debugging. And fourth, Go 1.27 is
out. The headline language feature is generic
methods. There are also improvements around type
inference, JSON, UUID support, runtime profiling,
and tooling. If Go is a big part of your environment,
this is worth actually reading the release notes
instead of just bumping the version in CI. The
human closer this week is about a mistake that
is painfully easy to make. Running kubectl against
the wrong cluster. The scenario is simple. You
have multiple terminal tabs open. In one tab,
you switch Kubernetes context. Later, in another
tab, you run a command believing you are still
pointed at the environment you were using before.
The command works perfectly. That is the problem.
kubectl context lives in the kubeconfig. It is
not tied to your terminal tab. So if two shells
use the same kubeconfig and one changes the current
context, the other silently sees that change
too. No warning. No authentication failure. Your
command just succeeds against a different cluster.
Those failures are dangerous because the system
behaves exactly as designed while violating the
operator's mental model. If kubectl crashed,
you would stop. If authentication failed, you
would investigate. But if a delete command succeeds
against production instead of staging, everything
technically worked. There are some simple protections.
Show the current cluster and namespace in your
shell prompt. Pass the context explicitly for
destructive actions. Use separate kubeconfigs
or session-scoped contexts. And add deliberate
friction around production. I like that last
one. We spend a lot of time removing friction
from engineering workflows. Usually, that is
good. But not all friction is waste. Typing the
production cluster name before a destructive
command costs five seconds. That is cheap compared
with restoring a namespace because muscle memory
moved faster than your attention. And the strongest
The thing I kept coming back to in this episode is how much infrastructure depends on assumptions that stop looking like assumptions after they have been true for long enough.
Cloudflare’s DNS cache is a good example from the performance side. Saving a few bytes in a data structure sounds almost pointless until you multiply it by more than 250 billion cache entries. Then suddenly those bytes turn into roughly one hundred terabytes of RAM. That is the kind of optimization work I really enjoy because it is not flashy. Nobody replaced the entire system or introduced another platform. They looked closely at how the existing thing behaved at scale and found places where small inefficiencies had become enormous.
It is also a useful reminder that “premature optimization” does not mean optimization is bad. It means you need to know where optimization actually matters. There is no reason to spend days shaving bytes from something instantiated a few thousand times. But if the same structure exists billions of times, memory layout, allocations, and cache locality become architecture decisions. At that scale, changing one field can matter more than adding another server.
The OVHcloud story gets at the same idea from a completely different direction. Most teams think about AI infrastructure costs in terms of GPUs, model APIs, or whatever AI workload they are actually running. But the demand is starting to affect the hardware market underneath everything else. If manufacturers shift capacity toward high-bandwidth memory because that is where AI demand is, then a company running normal databases and web servers can still end up paying more for RAM.
That is where FinOps gets more interesting than just finding idle instances. Right-sizing and commitment discounts are still useful, but they cannot completely solve a changing cost basis underneath the service. Sometimes the infrastructure itself becomes more expensive. A database optimization nobody could justify last year might suddenly make sense when the memory underneath it costs fifty percent more.
The AWS London story is probably the cleanest example of a hidden assumption finally getting tested. AWS added a fourth Availability Zone. That should be good news. More capacity and another failure domain are things we usually want.
But if your Terraform discovers every available AZ and another part of the module only has three subnet CIDRs, you have a problem. The code looked dynamic. It just was not dynamic all the way through.
I think that distinction matters a lot in infrastructure code. Dynamic discovery is not the same thing as supporting arbitrary change. Sometimes explicitly saying “this module supports exactly three zones” is safer than pretending everything is flexible. At least then the failure is obvious. The dangerous version is when the input changes and your automation quietly creates an architecture nobody intended.
That same tension shows up in the Route 53 story. Central networking teams usually own DNS for a reason. It is shared infrastructure, and letting every account independently control global resolution would be chaos. But forcing every private hosted-zone association through a networking ticket does not scale either.
The shared DNS view model is a pretty good example of platform engineering when it works well. Centralize the part that actually needs governance. Delegate the routine action the application team can safely perform itself. You are not giving up control. You are getting a person out of the middle of something that should not require scheduling.
That is usually what I want from an internal platform. Not a giant abstraction that hides everything. Not a portal for the sake of having a portal. Just guardrails around the decisions that matter and self-service around the ones that do not.
Even the lightning stories fit the same pattern. AKS moving more networking behavior into eBPF can improve performance, but it also changes assumptions around iptables. CloudFront Functions putting custom context directly into access logs removes one of those annoying observability gaps that seems small until you are debugging something under pressure. Go 1.27 adds language and runtime improvements that are easy to ignore until they intersect with something your codebase depends on. And the memory-isolation research is basically a reminder that every security boundary depends on what the layer underneath it is actually doing.
The kubectl closer is probably the most relatable version of all of this.
You have two terminals open. You switch context in one of them. Later you use the other one, assuming it is still pointed where it was before.
The command succeeds.
That is what makes the mistake dangerous.
The system did not fail. Your mental model did.
We spend a lot of time trying to remove friction from engineering workflows, and most of the time that is the right goal. But production is one place where a little friction can be healthy. Showing the current cluster in your prompt, explicitly passing a context for destructive commands, using separate kubeconfigs, or requiring stronger permissions in production may slow you down by a few seconds.
That is a pretty cheap trade compared with deleting something from the wrong cluster.
And the bigger lesson is not really about Kubernetes. It is about where we place safety.
If the only thing preventing a production mistake is an engineer remembering which terminal tab they used twenty minutes ago, the control is too weak. The same applies to the AWS AZ story. If the only thing keeping the module correct is AWS continuing to return exactly three zones forever, the control is too weak.
Good infrastructure engineering is not eliminating assumptions. You cannot do that.
It is deciding which assumptions are important enough to enforce, validate, or design around.
Sometimes that means validating the AZ count.
Sometimes it means putting the cluster name in the command.
Sometimes it means letting application teams manage their own DNS association inside a boundary the platform team owns.
And sometimes it means realizing that eight wasted bytes are not eight bytes anymore when you have 250 billion of them.
Most incidents are not caused by something completely unknowable.
A lot of them happen when reality changes and the system keeps behaving according to an assumption nobody realized was still there.
📝 Notes
Show Notes
This week on Ship It Weekly: Cloudflare explains how five low-level optimizations to the cache behind 1.1.1.1 freed roughly 100 terabytes of RAM while also improving performance. OVHcloud is raising infrastructure prices as AI demand reshapes the memory supply chain. AWS adds a fourth Availability Zone to London, exposing automation that quietly assumed there would always be three. And Route 53 Global Resolver gets a cleaner cross-account model for DNS self-service.
The bigger theme this week is assumptions. A few wasted bytes do not matter until you have 250 billion cache entries. A Region having three Availability Zones feels permanent until AWS adds a fourth. And centralized DNS governance works fine until every application team needs a networking ticket just to make a private zone resolvable.
In the lightning round: new research looks at manipulating DRAM controller translation registers and the assumptions that creates for memory isolation, AKS eBPF Host Routing reaches general availability, CloudFront Functions can now put custom context directly into access logs, and Go 1.27 lands generic methods along with runtime, tooling, and standard-library improvements.
And the human closer looks at an easy Kubernetes mistake: running kubectl against the wrong cluster. Because the active context belongs to the kubeconfig rather than a terminal tab, changing it in one shell can silently affect another. It is a good reminder that some friction is worth keeping around production, and that the safest guardrails live somewhere stronger than operator memory.
The thing I kept coming back to in this episode is how much infrastructure depends on assumptions that stop looking like assumptions after they have been true for long enough.
Cloudflare’s DNS cache is a good example from the performance side. Saving a few bytes in a data structure sounds almost pointless until you multiply it by more than 250 billion cache entries. Then suddenly those bytes turn into roughly one hundred terabytes of RAM. That is the kind of optimization work I really enjoy because it is not flashy. Nobody replaced the entire system or introduced another platform. They looked closely at how the existing thing behaved at scale and found places where small inefficiencies had become enormous.
It is also a useful reminder that “premature optimization” does not mean optimization is bad. It means you need to know where optimization actually matters. There is no reason to spend days shaving bytes from something instantiated a few thousand times. But if the same structure exists billions of times, memory layout, allocations, and cache locality become architecture decisions. At that scale, changing one field can matter more than adding another server.
The OVHcloud story gets at the same idea from a completely different direction. Most teams think about AI infrastructure costs in terms of GPUs, model APIs, or whatever AI workload they are actually running. But the demand is starting to affect the hardware market underneath everything else. If manufacturers shift capacity toward high-bandwidth memory because that is where AI demand is, then a company running normal databases and web servers can still end up paying more for RAM.
That is where FinOps gets more interesting than just finding idle instances. Right-sizing and commitment discounts are still useful, but they cannot completely solve a changing cost basis underneath the service. Sometimes the infrastructure itself becomes more expensive. A database optimization nobody could justify last year might suddenly make sense when the memory underneath it costs fifty percent more.
The AWS London story is probably the cleanest example of a hidden assumption finally getting tested. AWS added a fourth Availability Zone. That should be good news. More capacity and another failure domain are things we usually want.
But if your Terraform discovers every available AZ and another part of the module only has three subnet CIDRs, you have a problem. The code looked dynamic. It just was not dynamic all the way through.
I think that distinction matters a lot in infrastructure code. Dynamic discovery is not the same thing as supporting arbitrary change. Sometimes explicitly saying “this module supports exactly three zones” is safer than pretending everything is flexible. At least then the failure is obvious. The dangerous version is when the input changes and your automation quietly creates an architecture nobody intended.
That same tension shows up in the Route 53 story. Central networking teams usually own DNS for a reason. It is shared infrastructure, and letting every account independently control global resolution would be chaos. But forcing every private hosted-zone association through a networking ticket does not scale either.
The shared DNS view model is a pretty good example of platform engineering when it works well. Centralize the part that actually needs governance. Delegate the routine action the application team can safely perform itself. You are not giving up control. You are getting a person out of the middle of something that should not require scheduling.
That is usually what I want from an internal platform. Not a giant abstraction that hides everything. Not a portal for the sake of having a portal. Just guardrails around the decisions that matter and self-service around the ones that do not.
Even the lightning stories fit the same pattern. AKS moving more networking behavior into eBPF can improve performance, but it also changes assumptions around iptables. CloudFront Functions putting custom context directly into access logs removes one of those annoying observability gaps that seems small until you are debugging something under pressure. Go 1.27 adds language and runtime improvements that are easy to ignore until they intersect with something your codebase depends on. And the memory-isolation research is basically a reminder that every security boundary depends on what the layer underneath it is actually doing.
The kubectl closer is probably the most relatable version of all of this.
You have two terminals open. You switch context in one of them. Later you use the other one, assuming it is still pointed where it was before.
The command succeeds.
That is what makes the mistake dangerous.
The system did not fail. Your mental model did.
We spend a lot of time trying to remove friction from engineering workflows, and most of the time that is the right goal. But production is one place where a little friction can be healthy. Showing the current cluster in your prompt, explicitly passing a context for destructive commands, using separate kubeconfigs, or requiring stronger permissions in production may slow you down by a few seconds.
That is a pretty cheap trade compared with deleting something from the wrong cluster.
And the bigger lesson is not really about Kubernetes. It is about where we place safety.
If the only thing preventing a production mistake is an engineer remembering which terminal tab they used twenty minutes ago, the control is too weak. The same applies to the AWS AZ story. If the only thing keeping the module correct is AWS continuing to return exactly three zones forever, the control is too weak.
Good infrastructure engineering is not eliminating assumptions. You cannot do that.
It is deciding which assumptions are important enough to enforce, validate, or design around.
Sometimes that means validating the AZ count.
Sometimes it means putting the cluster name in the command.
Sometimes it means letting application teams manage their own DNS association inside a boundary the platform team owns.
And sometimes it means realizing that eight wasted bytes are not eight bytes anymore when you have 250 billion of them.
Most incidents are not caused by something completely unknowable.
A lot of them happen when reality changes and the system keeps behaving according to an assumption nobody realized was still there.