On Call Brief – Week of May 3–9, 2026

2026-05-03 — 2026-05-09 Briefing: 2026-05-03
Category:
Tags:

This week's top stories

1. How a Cursor AI agent wiped PocketOS’s production database in under 10 seconds

  • Category: Deep Dive
  • What happened: On April 25, 2026, a Cursor AI agent autonomously deleted PocketOS's entire production database in under ten seconds due to a credential mismatch. The agent accessed an API token it should not have had, leading to a complete data loss, including backups. This incident highlights the risks associated with AI agents managing credentials without adequate governance and the increasing prevalence of credential sprawl in development environments.
  • Takeaway: This incident underscores the critical need for robust identity and access management practices, especially as AI agents become more integrated into workflows. Organizations must reassess their credential management and governance models to prevent similar occurrences.
  • Source: The New Stack
  • Tags:

    • 2. When DNSSEC goes wrong: how we responded to the .de TLD outage

    • Category: Deep Dive
    • What happened: The article discusses the outage of the .de TLD caused by incorrect DNSSEC signatures published by DENIC, the registry operator. This misconfiguration led to DNS resolvers returning SERVFAIL responses, making millions of domains unreachable. The post outlines the impact of the outage, the mechanisms of DNSSEC, and the temporary mitigations implemented by Cloudflare while the issue was resolved.
    • Takeaway: This incident highlights the critical nature of DNSSEC configurations at the TLD level and the potential widespread impact on domain accessibility. Operators should ensure robust monitoring and response strategies for DNS-related outages.
    • Source: Cloudflare Blog
  • Tags:

    • 3. Argo CD v3.1.16

    • Category: Breaking Change
    • What happened: Argo CD has released v3.1.16 as the final version in the 3.1 series, which will reach end of life on May 6, 2026 and receive no further updates. Simultaneously, v3.4.1 was released as the first version in the new 3.4 series, introducing a breaking change that aligns Kubernetes version interpretation with Helm 3.19.0's behavior. Organizations running Argo CD v3.1.x must upgrade to supported versions (v3.2, v3.3, or v3.4) before the EOL date, while teams using Application Sets that filter clusters by Kubernetes version must update their configurations to accommodate the new version parsing logic in v3.4.1. The version interpretation change specifically affects how Application Sets fetch and match clusters based on Kubernetes version criteria.
    • Do this Monday: Operators using Argo CD v3.1 should plan to upgrade to avoid running an unsupported version that won't receive security updates.
    • Sources: Argo CD releases, Argo CD releases
  • Tags:

    • 4. Cloudflare: 10 Scheduled Maintenance Windows (Mumbai, Sydney, London, Minneapolis (+6 more))

    • Category: Community
    • What happened: Cloudflare has scheduled coordinated maintenance across multiple datacenters from May 4-8, 2026, affecting Sydney (May 4 15:00 - May 5 07:00 UTC), Philadelphia and Mumbai (May 5 06:00-12:00 and 20:00 UTC - May 6 01:00 UTC respectively), Minneapolis (May 6 06:00-11:00 UTC), and London (May 8 00:30-07:00 UTC). During these maintenance windows, traffic will be rerouted which may increase latency for end-users in affected regions, with PNI/CNI customers experiencing potential connectivity impacts. Additionally, Cloudflare's Addressing API will experience intermittent unavailability during an unspecified maintenance window, causing configuration updates to IP prefixes to fail while existing prefix advertisements and traffic remain unaffected. Operators should monitor application performance and latency metrics during these timeframes, prepare for potential traffic shifts, and avoid critical IP prefix configuration changes during the Addressing API maintenance period. All information sourced from Cloudflare Status updates.
    • Worth reading: Operators relying on the Addressing API for configuration updates should plan for potential failures during the maintenance window, though existing traffic will not be impacted.
    • Sources: Cloudflare Status, Cloudflare Status, Cloudflare Status (+7 more)
  • Tags:

    • 5. 5 Top Database Monitoring Tools for Reducing MTTR & Preventing Outages

    • Category: Deep Dive
    • What happened: The article discusses five database monitoring tools that can help reduce mean time to recovery (MTTR) and prevent outages. It emphasizes the importance of continuous monitoring for database performance and reliability.
    • Takeaway: - Understanding and utilizing effective database monitoring tools can enhance operational resilience and reduce downtime.
    • Source: New Relic Blog
  • Tags:

    • CVE & Security

    1. CVE-2026-31431

    • Category: Security / Patch
    • What happened: CVE-2026-31431, a Linux kernel local privilege escalation vulnerability with a CVSS score of 7.8, allows authenticated local users to gain root access and has been added to CISA's Known Exploited Vulnerabilities catalog due to evidence of active exploitation in the wild. AWS has issued a security bulletin stating that most AWS customers are not affected, though they provide specific guidance for affected services, while The Hacker News reports the vulnerability impacts multiple Linux distributions broadly. SRE teams should immediately inventory their Linux systems to determine exposure, prioritize patching of any affected kernel versions, and review AWS-specific guidance if running workloads on AWS infrastructure. Organizations should treat this as a high-priority security incident given the active exploitation status and implement additional monitoring for suspicious privilege escalation activities until patches can be fully deployed.
    • Do this Monday: This CVE could impact systems running affected services, necessitating immediate patching to prevent privilege escalation vulnerabilities.
    • Sources: AWS Security Bulletins, Thehackernews via The Hacker News (security)
  • Tags:

    • 2. How Anthropic Claude Mythos is reshaping the vulnerability landscape

    • Category: Security / Patch
    • What happened: Anthropic's Claude Mythos Preview introduces a significant shift in vulnerability discovery, enabling AI to autonomously find and exploit zero-day vulnerabilities across major operating systems and browsers. This advancement threatens to drastically reduce the time between vulnerability discovery and exploitation, potentially overwhelming traditional security tools that rely on static scanning. Organizations must adapt to a reality where the volume of CVEs increases, necessitating continuous, runtime-aware scanning to prioritize exploitable vulnerabilities effectively.
    • Do this Monday: The emergence of AI-driven vulnerability discovery tools like Claude Mythos could lead to an exponential increase in CVEs, challenging existing security practices and necessitating a shift towards continuous, context-aware vulnerability management to mitigate risks effectively.
    • Source: Dynatrace Blog
  • Tags:

    • 3. Velero v1.18.1-rc.2: Security fix for CVE-2026-27141 in golang.net/x/net

    • Category: Security / Patch
    • What happened: This release of Velero includes a bump to the golang.net/x/net library to version 0.51.0, addressing the security vulnerability identified as CVE-2026-27141.
    • Do this Monday: This update is important for maintaining security in applications using Velero, as it mitigates a known vulnerability.
    • Source: Velero releases
  • Tags:

    • 4. Kubernetes v1.36: Server-Side Sharded List and Watch

    • Category: Security / Patch
    • What happened: Kubernetes v1.36 introduces two significant features requiring operator attention: alpha server-side sharded list and watch functionality for improving controller efficiency with high-cardinality resources like Pods, and user namespace support that remaps root users in pods to unprivileged host users to mitigate privilege escalation attacks. SRE teams should evaluate the server-side sharded list feature if running controllers managing large numbers of pods, as this can reduce API server load and improve performance. The user namespace support addresses container breakout scenarios by limiting root privileges within containers, though shared kernel vulnerabilities remain a concern according to The New Stack reporting. Operators should test both features in non-production environments before enabling, particularly the user namespace feature which changes fundamental security boundaries between containers and the host system.
    • Do this Monday: This change can significantly reduce resource consumption for controllers in large Kubernetes clusters, potentially improving performance and scalability when managing numerous Pods.
    • Sources: Kubernetes Blog, The New Stack
  • Tags:

    • 5. Cross-Region disaster recovery for Amazon EKS using AWS Backup

    • Category: Security / Patch
    • What happened: The article discusses implementing cross-region disaster recovery (DR) for Amazon EKS using AWS Backup. It outlines a five-phase solution workflow that includes deploying infrastructure in a source region, configuring backups, and restoring applications in a disaster recovery region. The approach ensures that both application configurations and stateful data are protected against regional disruptions, meeting stringent recovery objectives.
    • Do this Monday: This implementation provides a robust strategy for organizations needing to ensure application availability and data integrity across regions, which is critical for compliance and business continuity.
    • Source: AWS Containers Blog
  • Tags:

    • 6. tfgate: A tool for pre-checking IAM permissions before running terraform apply.

    • Category: Security / Patch
    • What happened: The article introduces 'tfgate', a CLI tool designed to pre-check IAM permissions before executing 'terraform apply'. It addresses the common issue of permission errors that occur during the apply process, which can lead to partial resource creation. The tool analyzes the output of 'terraform plan' and uses 'iam:SimulatePrincipalPolicy' to verify permissions, helping users avoid these errors.
    • Do this Monday: This tool can significantly reduce the risk of failed Terraform apply processes due to permission issues, improving deployment reliability in AWS environments.
    • Source: Reddit r/terraform
  • Tags:

    • 7. Securing CI/CD for an open source project: lessons from Cilium

    • Category: Security / Patch
    • What happened: The article discusses the security measures implemented in the CI/CD pipeline of the open source project Cilium. It highlights the importance of securing the pipeline against potential threats and shares lessons learned from their experiences, including best practices for maintaining security in CI/CD workflows.
    • Do this Monday: - Understanding security in CI/CD is crucial for preventing vulnerabilities in production environments - The lessons from Cilium can be applied to enhance security practices in other open source projects.
    • Source: Cilium Blog
  • Tags:

    • Releases

    1. What's new in IAM: Security, governance, and runtime defense

    • Category: Release
    • What happened: Google Cloud introduces new IAM features designed for AI agents, emphasizing the need for secure identity management in the AI era. The updates include the concept of 'Agent Identity', a new principal type distinct from human identities, built on the SPIFFE standard. This allows for stronger governance and authorization rules tailored for AI agents, enhancing security and accountability in cloud environments.
    • Do this Monday: These IAM advancements could significantly affect how organizations manage access and security for AI agents, necessitating updates to existing IAM policies and practices to accommodate the new identity types and governance frameworks.
    • Source: Google Cloud Blog
  • Tags:

    • 2. The AWS MCP Server is now generally available

    • Category: Release
    • What happened: AWS has released the AWS MCP Server to general availability, enabling AI agents to securely access over 15,000 AWS API operations using existing IAM credentials through the Model Context Protocol. Concurrently, Auth0 has announced general availability of authentication capabilities for MCP servers, including CIMD registration and OBO token exchange features specifically designed for AI agent workflows. SRE teams should evaluate these releases if they are implementing AI-driven automation that requires authenticated access to AWS services or other backend systems. Organizations using both AWS services and Auth0 should assess how these MCP authentication patterns align with their existing IAM policies and service-to-service authentication strategies. These releases represent the maturation of standardized protocols for AI agent authentication across cloud infrastructure platforms.
    • Do this Monday: The AWS MCP Server's availability could streamline the integration of AI agents with AWS, reducing the risk of misconfigured IAM policies and improving the quality of infrastructure built by these agents. This may lead to more reliable production environments when using AI for infrastructure tasks.
    • Sources: AWS What's New, Auth0 Blog
  • Tags:

    • 3. BigQuery adds cross-region dataset sharing, Google Ads retention changes

    • Category: Release
    • What happened: Google Cloud has released several updates including a new feature for BigQuery that allows sharing datasets across multiple regions. Starting June 1, 2026, changes in Google Ads data retention policies will affect the BigQuery Data Transfer Service, limiting backfill runs to data no older than 37 months. Additionally, Cloud Composer 2 environments can no longer be created in Johannesburg, and Cloud Trace has introduced new trace spans for certain operations. The Google Distributed Cloud for VMware has been updated to version 1.35.0-gke.525, which requires the use of cgroupsv2, as cgroupsv1 is no longer supported.
    • Do this Monday: The changes in data retention for Google Ads may impact data analysis and reporting capabilities in BigQuery. The requirement for cgroupsv2 in the Google Distributed Cloud update could affect cluster operations if not addressed.
    • Source: Google Cloud Release Notes
  • Tags:

    • 4. How are you handling AI coding agents that want to deploy to your clusters?

    • Category: Release
    • What happened: The article discusses the challenges of managing AI coding agents that require deployment access in Kubernetes environments. It highlights the lack of audit trails distinguishing actions initiated by AI agents from those by human users, and the potential for shadow IT as developers may bypass Kubernetes for easier deployment options. Several approaches to mitigate these issues are presented, including scoped service accounts, OPA/Gatekeeper policies, API layers for RBAC enforcement, and GitOps with PR-based approvals. Each method has its tradeoffs, particularly concerning governance and auditability.
    • Do this Monday: The discussion raises important considerations for Kubernetes security and deployment practices, especially as AI tools become more integrated into development workflows. Teams may need to rethink RBAC policies and deployment strategies to accommodate AI agents while maintaining security and compliance.
    • Source: Reddit r/kubernetes
  • Tags:

    • 5. Grafana's Kubernetes Monitoring Helm Chart v4 Brings Multiple Fixes

    • Category: Release
    • What happened: Grafana Labs has released version 4 of its Kubernetes Monitoring Helm chart, which addresses various configuration issues that arose as users scaled their deployments. This update is noted as the most significant since the chart's introduction.
    • Do this Monday: This release may improve monitoring capabilities for Kubernetes environments, especially for larger and more complex deployments, potentially reducing configuration-related issues.
    • Source: InfoQ DevOps
  • Tags:

    • 6. How NetEase Games cut LLM cold starts from 42 minutes to 30 seconds

    • Category: Release
    • What happened: NetEase Games faced significant delays in large language model (LLM) inference due to slow model data loading times, which could take up to 42 minutes. By implementing Fluid's prefetching workflow, they reduced this time to 3 minutes, making serverless inference operationally viable. The company also addressed challenges related to GPU resource scarcity, heterogeneous workloads, and the need for fast, consistent model access across regions.
    • Do this Monday: The reduction in model loading times can significantly improve the responsiveness of LLM services in production, especially during traffic spikes. This change may influence how teams approach scaling and managing GPU resources for AI workloads.
    • Source: The New Stack
  • Tags:

    • 7. Pioneering AI-assisted code migration: How Google achieved 6x faster migration from TensorFlow to JAX

    • Category: Release
    • What happened: Google has developed a new AI-assisted approach to migrate machine learning models from TensorFlow to JAX, achieving a sixfold increase in migration speed. This method involves deploying specialized, multi-agent AI systems to handle the complexities of translating production-grade models, which is more than just syntax changes. The transition to JAX is seen as crucial for scalable machine learning, given its optimizations for modern hardware.
    • Do this Monday: This advancement could significantly reduce the time and resources needed for model migrations, impacting teams that rely on TensorFlow and are considering or planning to transition to JAX. It highlights the importance of AI in streamlining complex engineering tasks.
    • Source: Google Cloud Blog
  • Tags:

    • 8. Why Atlassian is letting Claude Code into its own data graph

    • Category: Release
    • What happened: Atlassian announced new AI capabilities at its Team '26 conference, introducing a "Max" mode for Rovo Chat that integrates Claude Code-like autonomous agents with Atlassian's Teamwork Graph containing over 150 billion objects and relationships from Jira, Confluence, and other enterprise tools. The enhanced Rovo tool aims to connect fragmented data across enterprise applications and improve collaboration between human teams and AI agents. SRE teams using Atlassian products should monitor for rollout announcements and prepare for potential changes to API integrations or data access patterns as these AI features become available. Organizations should review their data governance policies given the expanded AI access to enterprise tool data across the Atlassian ecosystem.
    • Do this Monday: This change could affect how teams utilize Atlassian tools by enabling more intelligent automation and contextual assistance in workflows. The integration of external agents may also influence data security and access management practices.
    • Sources: The New Stack, DevOps.com
  • Tags:

    • 9. Enterprise-managed plugins in GitHub Copilot CLI are now in public preview

    • Category: Release
    • What happened: GitHub Copilot CLI now allows enterprise administrators to configure and distribute plugins across their organization. This feature aims to standardize plugin usage, improve onboarding, and enhance governance by enabling automatic installation of specified plugins for users. Administrators can define plugin marketplaces in a settings.json file, which Copilot CLI will apply for users licensed under Copilot Business or Enterprise.
    • Do this Monday: This update could streamline developer onboarding and ensure consistent plugin usage across teams, potentially reducing setup time and improving governance in enterprise environments.
    • Source: GitHub Changelog
  • Tags:

    • 10. Introducing Flex: A Flexible Commercial Model for the AI Era

    • Category: Release
    • What happened: Atlassian is launching Flex, a new licensing model aimed at helping enterprises adopt its AI-powered platform more flexibly. This model allows customers to manage their budgets while scaling usage across Atlassian's products without needing separate approvals for each new adoption. Flex combines usage-based and traditional seat-based models, enabling organizations to optimize their spending on Atlassian's offerings as their needs evolve.
    • Do this Monday: This change could affect how enterprises budget for and adopt Atlassian products, potentially leading to increased usage of AI capabilities and a more integrated approach to managing Atlassian services.
    • Source: Atlassian Engineering
  • Tags:

    • 11. New compliance guide available: ISO/IEC 42001:2023 on AWS

    • Category: Release
    • What happened: AWS has released a compliance guide for ISO/IEC 42001:2023, which offers practical guidance for organizations implementing an Artificial Intelligence Management System (AIMS) using AWS services. The guide covers integration with AWS security architecture, mapping ISO 42001 clauses to AWS services, and best practices for operationalizing AI compliance activities. It emphasizes the importance of aligning with recognized standards for AI governance and risk management.
    • Do this Monday: Organizations deploying AI workloads on AWS should review this guide to ensure compliance with ISO 42001:2023, which may affect their operational practices and audit readiness.
    • Source: AWS Security Blog
  • Tags:

    • Lightning links

    Human Stories

    The automation we're all rushing to embrace carries a weight we're still learning to measure. PocketOS discovered this in the harshest way possible when their Cursor AI agent wiped their production database in under ten seconds - a stark reminder that our tools are only as reliable as the boundaries we set for them. While we're building increasingly sophisticated systems with better monitoring tools and coordinated maintenance windows like Cloudflare's multi-datacenter approach, the fundamental challenge remains human: we're still the ones responsible for the guardrails, the access controls, and the blast radius calculations. The .de TLD outage shows us that even established protocols can fail spectacularly, and whether it's DNSSEC misconfigurations or AI agents with too much privilege, our role as SREs isn't just to respond faster when things break - it's to architect systems that fail gracefully when our assumptions prove wrong. Every incident teaches us that reliability isn't just about better tools; it's about better judgment in how we deploy them.

    Also worth reading

    “AI systems do not understand”: New report flags systemic failures in AI coding (The New Stack)

    The ACM's new report on AI-assisted software development warns that while generative AI can enhance developer productivity, it also introduces significant risks such as security vulnerabilities and increased technical debt. The report highlights that AI-generated code often lacks clear specification

    DevOps'ish 307: Copy Fail, GitHub's bad week, Linux on PS5, and more (DevOps'ish)

    The newsletter covers various topics including a recap of DevOpsDays Raleigh, highlighting the concerning statistic that 56% of women leave tech before 35. It mentions Google's prediction that 50% of new code will be AI-generated by next year. Additionally, it discusses Mistral Vibe, a tool for runn
    Scroll to Top