Making Deployments Safe at Meta: Health Checks for Continuous Change-Safety

Continuous deployment to large scale production systems creates a tension between release velocity and reliability. Every change is a potential reliability incident, yet every delay is a missed opportunity. This paper describes the deployment time health check infrastructure that Meta uses to mediate this tension across thousands of heterogeneous services. We summarize the architecture of this prevention based distributed system's service called Service Health Checker, explain how check authors compose templated metric queries, thresholds, and workflow predicates; and discuss how the system is integrated with tiered and phased rollouts so that regressions trigger automatic rollback. We then describe the operational problems that emerged at scale, such as noise, alert fatigue, drift, and uncovered regressions, and the program of measurement, tooling, and improved defaults we deployed to address them. We close with lessons learned from years of operating deployment health checks at Meta, and the directions we are exploring next, including AI assisted health check tuning. Index Terms: deployment safety, continuous deployment, monitoring, software reliability, release engineering, software reliability engineering, AIOps, anomaly detection

From Reactive to Autonomous: Evolution of AI Operations in Cloud Network Infrastructure

The operational model for cloud network infrastructure has undergone a fundamental transformation over the past decade. What began as manual, human-driven troubleshooting has evolved through scripted automation, rule-based systems, and AI-assisted operations into fully autonomous incident resolution. This paper traces the evolution of AI operations (AIOps) in cloud network infrastructure, identifying the architectural patterns, organizational challenges, and technical inflection points that enabled each generational transition. Drawing from production experience operating network infrastructure at hyperscale, we present a maturity model that characterizes five distinct operational generations, analyze the technical and organizational barriers that impede transitions between generations, and document the metrics that indicate readiness for increased autonomy. We show that the path from reactive to autonomous operations is not merely a technology problem but requires co-evolution of tooling, trust frameworks, knowledge management practices, and operational culture. Our findings provide a practical roadmap for infrastructure organizations seeking to adopt progressively autonomous AI operations.