Achieving True Scale-to-Zero in Kubernetes with Zero Lost Requests

The inherent variability of application traffic renders fixed replica counts inefficient, leading to either wasted resources or underprovisioning. The ideal solution involves applications scaling dynamically with demand, including the ultimate state of scaling down to zero replicas when idle. While scaling to zero is straightforward by setting replica counts, the critical challenge lies in seamlessly scaling back up without losing initial requests. A recently demonstrated methodology tackles this by integrating standard Kubernetes components to achieve true scale-to-zero with guaranteed request persistence.

The proposed architecture utilizes a suite of robust cloud-native tools: Envoy Gateway for advanced traffic ingress via Gateway API, KEDA (Kubernetes Event-driven Autoscaling) as the core scaling engine, Prometheus for real-time metric collection from the gateway and applications, and a Pod Monitor to bridge Envoy and Prometheus. Crossplane can orchestrate the provisioning and wiring of these components into a fully operational cluster. This setup enables KEDA to scale deployments from zero to multiple replicas based on Prometheus metrics for steady-state traffic, while its HTTP add-on interceptor handles cold starts. When an application is at zero replicas, incoming requests are routed to the KEDA HTTP add-on interceptor, which holds them while KEDA scales up the necessary pods. Once a pod is ready, the intercepted request is forwarded, ensuring no traffic is dropped. This approach offers significant resource efficiency, preserves standard Kubernetes primitives and avoids vendor lock-in, and provides highly responsive metrics-driven scaling. While introducing a cold start latency for the first request after an idle period and buffering limits on the interceptor, these trade-offs are generally considered minor given the substantial benefits for applications with variable and intermittent traffic patterns.