On Call Brief – Week of November 22–28, 2026

2026-11-22 — 2026-11-28 Briefing: 2026-11-22 Last updated 4 hours ago (Nov 22, 2026 3:51 am EST) 12 min read
Share
Category:
Tags:

This week's top stories

1. World's Largest Brands Infected With Malware Via Optimizely

  • Category: Community
  • What happened: Malicious JavaScript was injected into Optimizely Web customers' Project JavaScript, affecting 178 organizations including Zoom and the BBC. The attack involved swapping crypto wallet addresses, skimming card details, and serving iOS exploit kits. The attacker stole approximately $7,040 in crypto before the infected files were cleaned up the next day.
  • Worth reading: Audit all third-party script tags on your site to ensure they are secure.
  • Source: Blog Report Uri via DevOps'ish
  • Tags:

2. Scheduling the removal of the kube-proxy IPVS backend

  • Category: Breaking Change
  • What happened: The kube-proxy IPVS backend, deprecated in version 1.35, is scheduled for removal in version 1.40 due to its poor integration with Kubernetes Services and reliance on iptables. Users are advised to switch to nftables in kubeadm clusters by updating the kube-proxy ConfigMap and restarting the pods. The article also discusses the historical context and limitations of the IPVS backend.
  • Do this Monday: Switch kube-proxy from ipvs to nftables in the kube-proxy ConfigMap and restart the pods before upgrading to version 1.40.
  • Source: Kubernetes via DevOps'ish
  • Tags:

3. AWS Lambda Now Runs GPU Inference with 48 Concurrent Invocations Sharing One Model per GPU

  • Category: Community
  • What happened: AWS Lambda has introduced support for running Event Functions on EC2 G-series GPU instances, allowing up to 48 concurrent invocations to share a single model per GPU. This feature utilizes vLLM and SGLang base images that do not require handler code, with a maximum asynchronous timeout of 90 minutes. Billing is based on the instance price plus a 9.75 percent fee, and initial tests indicate an eight-minute publish time without response streaming.
  • Worth reading: Consider leveraging AWS Lambda's new GPU support for Event Functions to optimize model inference workloads.
  • Source: InfoQ DevOps
  • Tags:

4. Our load balancer stopped sending traffic to three of our six servers and the auto scaling group still counted six

  • Category: Deep Dive
  • What happened: An auto scaling group on EC2 failed to replace three unhealthy instances because it used the default EC2 health check type, which did not account for application-level health. The load balancer marked the instances as unhealthy based on timeouts, but the auto scaling group continued to count them as healthy since EC2 checks passed. The issue was resolved by changing the health check type to ELB, allowing the group to replace unhealthy instances based on the load balancer's verdict.
  • Takeaway: Set health_check_type to ELB in your auto scaling group to ensure instances are replaced based on application health checks.
  • Source: dev.to (SRE tag)
  • Tags:

5. Java Service Outage: Missing HTTP Timeout in Synchronized Block Caused 504 Errors

  • Category: Deep Dive
  • What happened: The article describes an incident where a high-traffic service returned 504 errors due to a missing timeout in an HTTP call within a synchronized block. This caused all threads to queue behind the lock, leading to a complete halt in processing requests. The author shares the troubleshooting process, which revealed that the Kubernetes setup was not at fault, and emphasizes the importance of implementing timeouts, using granular locks, and avoiding synchronized blocks with transactional methods to prevent similar issues in the future.
  • Takeaway: Implement timeouts for all HTTP calls and avoid using synchronized blocks with transactional methods to prevent thread blocking.
  • Source: dev.to (Kubernetes tag)
  • Tags:

6. We added one condition to our release notes step and every push to every branch emailed forty partners

  • Category: Deep Dive
  • What happened: A change to a GitHub Actions workflow condition led to unintended emails being sent to partners for every push to any branch, instead of only for specific tagged releases. The issue arose from a misunderstanding of how GitHub evaluates expressions, which caused the condition to always evaluate as true. The workflow has since been corrected to ensure emails are only sent for valid tagged releases, and additional checks were implemented to prevent similar issues in the future.
  • Takeaway: Review GitHub Actions workflows for similar expression issues to prevent unintended notifications.
  • Source: dev.to (DevOps tag)
  • Tags:

7. GitHub Actions: 3 related updates

  • Category: Community
  • What happened: GitHub has released a public preview feature for GitHub Enterprise Cloud that extends secret scanning to GitHub Actions workflow logs, which must be manually enabled under Policies > Advanced Security as it is disabled by default. This addresses a security gap where credentials and tokens could be exposed in workflow run outputs without detection. Organizations using scheduled workflows in GitHub Actions should be aware of several operational issues that can prevent execution, including incorrect workflow file placement (must be in .github/workflows on the default branch), timezone misinterpretation (cron schedules use UTC), execution delays during GitHub's peak usage periods, and automatic workflow disabling after 60 days of repository inactivity. For Kubernetes deployments, operators can avoid storing kubeconfig credentials in GitHub Actions by separating application code from manifest repositories and using alternative authentication methods such as OIDC-based workload identity or pull-based GitOps tools like ArgoCD or Flux.
  • Worth reading: Enable secret scanning for GitHub Actions workflow logs under Policies > Advanced Security to prevent accidental exposure of secrets.
  • Sources: GitHub Blog via DevOps'ish, dev.to (Kubernetes tag), dev.to (DevOps tag)
  • Tags:

8. 3 Production Strategies to Dramatically Reduce Your LLM Costs

  • Category: Community
  • What happened: The article outlines three strategies to optimize costs when deploying LLMs in production. It emphasizes the importance of tracking costs per feature to avoid unnecessary expenses from using high-cost models for simple tasks. The first step involves implementing logging to monitor token usage and associated costs, allowing teams to make informed decisions about model usage and potentially switch to more cost-effective options for simpler requests.
  • Worth reading: Implement cost tracking for LLM usage by logging input/output token counts per feature to optimize infrastructure costs.
  • Source: dev.to (DevOps tag)
  • Tags:

9. Should your uptime monitor page you when the site is slow, or only when it's down? Let's argue about it

  • Category: Deep Dive
  • What happened: The article discusses whether uptime monitoring should alert for slow site performance or only for complete outages. It presents arguments for both sides, emphasizing that slow performance can lead to user frustration and lost sales, while also noting that alerts for slow responses can be noisy and subjective. The author suggests a balanced approach, treating extremely slow responses as failures while using moderate slowness as a warning without immediate paging.
  • Takeaway: Consider implementing a monitoring strategy that pages for extremely slow responses while using moderate slowness as a warning to be addressed during working hours.
  • Source: dev.to (DevOps tag)
  • Tags:

10. The Pulse: What can we learn from the Railway hack?

  • Category: Deep Dive
  • What happened: The article analyzes Railway's postmortem of a recent hack, highlighting that the attacker may have exploited AI models with removed safeguards. It also touches on the impact of disappearing mobile and frontend jobs, the importance of trust in AI-native teams, and mentions Carta's sale of a coding harness.
  • Takeaway: Informational: insights on AI model security and team dynamics.
  • Source: Newsletter Pragmaticengineer via DevOps'ish
  • Tags:

CVE & Security

11. CVE-2026-114938 OpenJDK RCE (Java4Shell): Everything You Need to Know

12. Researchers Demonstrate Ghostscript Sandbox-Escape RCE via Crafted File Uploads in Popular Apps

  • Category: Security / Patch
  • What happened: A patched vulnerability in Ghostscript enables crafted files to escape its security sandbox, allowing remote command execution on servers that process these files. This flaw affects many Linux servers that utilize Ghostscript for converting PDFs and images.
  • Do this Monday: Upgrade Ghostscript to the latest version to mitigate the RCE risk from crafted file uploads.
  • Source: The Hacker News
  • Tags:

13. Confluent Schema Registry v8.5.0-707

  • Category: Security / Patch
  • What happened: Confluent released three consecutive versions of Schema Registry on November 22, 2026: v8.5.0-705, v8.5.0-706, and v8.5.0-707. All three releases focus on setting project versions and resolving dependencies, with no specific features, bug fixes, or security patches detailed in the release notes. Operators running earlier 8.5.0 builds should evaluate whether the dependency resolutions warrant upgrading to v8.5.0-707, though the rapid succession of releases and lack of changelog detail suggests these may be automated maintenance releases rather than critical updates. The absence of breaking changes or security advisories indicates no urgent action is required unless dependency conflicts are affecting your deployment.
  • Takeaway: Informational: a version release with no breaking changes or new features.
  • Sources: Confluent Schema Registry releases, Confluent Schema Registry releases (2), Confluent Schema Registry releases (3)
  • Tags:

Releases

14. Visual Studio Code 1.148 (Insiders)

  • Category: Release
  • What happened: The latest Visual Studio Code Insiders build introduces agent session management and improvements to editor tabs.
  • Takeaway: Informational: a new release of Visual Studio Code Insiders.
  • Source: VS Code Blog
  • Tags:

Also this week

Deep dives & postmortems

15. OpenRouter Drops 3 Providers After Researchers Catch Months of Prompt Theft at AI Hosting Firms

  • Category: Deep Dive
  • What happened: OpenRouter has severed ties with three AI inference providers after researchers discovered that attackers had been stealing data from these firms for months without their knowledge. Some of the compromised requests were marked as Zero Data Retention, indicating that they should not have been stored.
  • Takeaway: Informational: a significant data theft incident affecting AI inference providers.
  • Source: The Hacker News
  • Tags:

Community reads

16. apple/container 1.7.1: Local Kubernetes Clusters via Lightweight VMs on Mac

  • Category: Community
  • What happened: The 1.7.1 documentation for apple/container details how to create and run local Kubernetes clusters using lightweight VMs on a Mac, including information on worker nodes, CPU and memory sizing, and kubectl access. It also includes security caveats regarding the --shared-image-store option.
  • Worth reading: Review the security caveats before using the --shared-image-store option.
  • Source: GitHub via DevOps'ish
  • Tags:

Status pages & maintenance

17. Cloudflare: 26 scheduled maintenance windows (Panama City, Lima, Fortaleza, +20 more)

  • Category: Status Page
  • What happened: Cloudflare has scheduled maintenance windows across 26 datacenters between November 23 and November 27, 2026, affecting locations in the Americas (IAD Ashburn, LAX Los Angeles, DFW Dallas, SJC San Jose, multiple South American cities), Asia-Pacific (SIN Singapore, KIX Osaka, MNL Manila), and Middle East (BAH Manama). Maintenance windows range from 4 to 5 hours and will cause traffic rerouting with increased latency for end-users in affected regions, while PNI (Private Network Interconnect) and CNI (Cloudflare Network Interconnect) customers may experience temporary network interface unavailability and should prepare for potential traffic failover during their respective datacenter maintenance periods. Operators using Cloudflare services should monitor traffic patterns and latency metrics during these windows, particularly if serving users in these geographic regions or using direct network interconnects. No configuration changes or software updates are required, but teams should ensure monitoring alerts are configured to detect latency spikes and validate that traffic failover mechanisms function correctly during the maintenance periods.
  • Heads up: Prepare for potential traffic failover and increased latency during the maintenance window on November 27, 2026.
  • Sources: Cloudflare Status, Cloudflare Status (2), Cloudflare Status (3) (+23 more)
  • Tags:

Human Stories

Looking at the Optimizely breach and OpenRouter's compromised providers alongside those auto scaling groups that quietly failed and the GitHub Actions workflow that spammed forty partners, I keep thinking about how the systems we trust most are often the ones we instrument least. We build elaborate monitoring for our own code, but that third-party SDK or that cloud provider's health check? We just assume it works. The Java timeout incident and the kube-proxy deprecation tell the same story from different angles - we inherit complexity from layers we didn't write, and when things go wrong, we discover we were flying blind the whole time. Maybe the real infrastructure debt isn't just in our legacy code, but in all those abstraction boundaries where we stopped asking questions and started trusting someone else's promises.