The Resource Allocation Problem in Cloud
Cloud infrastructure teams face a continuous optimization problem that traditional algorithms struggle with. You have got a pool of compute resources, a stream of workloads with varying requirements, and a cost function that penalizes both under-provisioning (SLA violations) and over-provisioning (wasted spend). The inputs are noisy, the action space is large, and the feedback is delayed. Classic RL territory.
I've been working on RL-based resource allocation for about three years now, and the honest truth is that it's harder than most research papers suggest. The gap between a policy that works in simulation and one that works in production is enormous. But when it works, the results are impressive. We've seen 15-25 percent cost reduction compared to rule-based autoscalers.
Formulating the Problem as an MDP
The first challenge is defining your state space, action space, and reward function. Getting these right matters more than the choice of RL algorithm.
import numpy as np
from dataclasses import dataclass
@dataclass
class ClusterState:
cpu_utilization: np.ndarray
memory_utilization: np.ndarray
request_rate: np.ndarray
latency_p99: np.ndarray
error_rate: np.ndarray
queue_depth: np.ndarray
total_cpu_allocated: float
total_cpu_available: float
pending_pods: int
node_count: int
hour_of_day: int
day_of_week: int
minutes_since_last_scaling: int
def to_vector(self):
per_service = np.concatenate([
self.cpu_utilization, self.memory_utilization,
self.request_rate, self.latency_p99,
self.error_rate, self.queue_depth
])
cluster = np.array([
self.total_cpu_allocated / self.total_cpu_available,
self.pending_pods / max(self.node_count, 1),
self.node_count / 100.0,
])
temporal = np.array([
self.hour_of_day / 24.0,
self.day_of_week / 7.0,
min(self.minutes_since_last_scaling, 60) / 60.0,
])
return np.concatenate([per_service, cluster, temporal])Action Space Design
This is where many teams get stuck. A naive action space of set replica count to N for each service is combinatorially explosive. With 20 services and a max of 50 replicas each, you have got 50 to the power of 20 possible actions. No RL algorithm is learning that.
We use a factored action space. The agent makes one scaling decision at a time: which service to scale, and by how much (typically -2, -1, 0, +1, +2 replicas). The agent is invoked every 60 seconds, and it picks the single most impactful scaling action. This reduces the action space to about 100 discrete actions for 20 services.
There's a constraint layer on top that prevents obviously bad actions: you can't scale below the minimum replica count, you can't scale above the cluster available capacity, and you can't scale a service that was scaled less than 5 minutes ago to avoid thrashing.
The Reward Function Problem
Designing the reward function is the single most important and most difficult part of this project. A reward function that only optimizes for cost will happily run everything at minimum replicas and accept SLA violations. A reward function that only optimizes for latency will max out every service. You need a carefully calibrated balance.
See also: Sparse Attention Patterns for Long-Sequence Transformers.
def compute_reward(state, action, next_state, sla_config):
reward = 0.0
for svc_idx in range(num_services):
p99 = next_state.latency_p99[svc_idx]
sla_target = sla_config[svc_idx].latency_p99_target
if p99 > sla_target:
violation_ratio = (p99 - sla_target) / sla_target
reward -= 10.0 * violation_ratio
if next_state.error_rate[svc_idx] > 0.01:
reward -= 5.0 * next_state.error_rate[svc_idx]
cpu_cost = next_state.total_cpu_allocated * COST_PER_CPU_HOUR
reward -= 0.1 * cpu_cost
if action != 0:
reward -= 0.05
return rewardTraining Infrastructure
You can't train an RL policy directly on your production cluster. The exploration phase will cause outages. Instead, we built a simulator that replays historical cluster traces and simulates the effect of scaling actions on latency and throughput.
The simulator needs to capture the relationship between replica count, request rate, and latency with reasonable fidelity. We use queuing theory models (M/M/c queues) calibrated against historical data from each service. It isn't perfect, real services have complex failure modes that queuing models don't capture, but it's good enough for the RL agent to learn useful policies.
We train with PPO (Proximal Policy Optimization) because it's stable and sample-efficient enough for our problem size. Training takes about 4 hours on a single GPU, replaying 6 months of cluster traces. The policy is retrained weekly to adapt to changing workload patterns.
Safe Deployment
Even after training, we don't give the RL agent direct control. It runs in a recommendation mode where its scaling suggestions are filtered through a safety layer. The safety layer enforces hard constraints and can veto any action that would reduce replicas for a service currently experiencing elevated error rates.
We also maintain a fallback policy, a traditional HPA configuration that takes over if the RL agent recommendations consistently result in worse outcomes than the baseline. The switchover is automatic and based on a 30-minute rolling comparison of SLA compliance and cost.
It took us about six months from initial prototype to production deployment, and the RL policy now manages autoscaling for 15 services. The biggest win has been during traffic ramp-up periods. The RL agent learned to pre-scale about 10 minutes before traffic spikes that follow predictable patterns, which the reactive HPA couldn't do.
We covered a related topic in Optical Character Recognition Pipeline Design for Noisy Docu.
Simulator Fidelity and Validation
The quality of your RL policy is bounded by the quality of your simulator. If the simulator doesn't accurately capture the relationship between scaling actions and system behavior, the policy will learn strategies that work in simulation but fail in production. This is the sim-to-real gap, and in cloud infrastructure it's a serious concern.
We validate the simulator against historical data by replaying past scaling events and comparing the predicted metrics (latency, error rate, throughput) against what actually happened. The simulator should correctly predict the direction of metric changes (scaling up reduces latency) even if the magnitude isn't exact.
Our acceptance criteria: the simulator predictions should correlate with real metrics at r=0.85 or higher for latency and throughput. Below that, the policy tends to learn strategies that exploit simulator inaccuracies rather than genuinely optimizing the real system.
For workloads with complex scaling dynamics (like services that warm up slowly or have connection pool limitations), we add calibration parameters to the simulator. These are tuned weekly using recent production data. The calibration step takes about 30 minutes of compute and significantly improves policy quality.
Multi-Service Coordination
The factored action space (one scaling action per timestep) works well for independent services, but many production systems have service dependencies that create coordination challenges. If service A calls service B, scaling up A without scaling up B can shift the bottleneck rather than eliminating it.
We handle this by adding a dependency graph to the reward function. When the agent scales up a service, the reward includes a penalty if any downstream dependency is already at high utilization. This implicitly teaches the agent to scale dependency chains from the bottom up, which matches the intuition of experienced SREs.
For a related perspective, see Weight Initialization Schemes and Their Impact on Convergenc.
In practice, the most important coordination patterns are:
Scale the database tier before the application tier. Database connections are the tightest resource in most architectures, and adding application replicas without database capacity causes connection pool exhaustion.
Scale cache layers before backend services. A cold cache after scaling up creates a thundering herd that can overwhelm the backend. We pre-warm caches as part of the scaling action, but this adds latency to the scaling operation.
Don't scale stateful services during peak hours unless absolutely necessary. Stateful scaling involves data rebalancing, which consumes resources that could serve production traffic. The RL agent learned this on its own from the reward signal, which was satisfying to observe.
Results and Lessons Learned
After 18 months in production, our RL autoscaler manages 15 services with a combined cost of about 40,000 dollars per month. The key metrics compared to the previous rule-based autoscaler:
Cost reduction: 22 percent average, with some services seeing up to 35 percent savings and others seeing only 8 percent. The biggest savings come from services with predictable diurnal patterns where the RL agent learned to pre-scale and pre-downscale.
SLA compliance: improved from 99.85 percent to 99.93 percent. The improvement comes primarily from predictive scaling. The rule-based autoscaler only reacted to threshold breaches, which meant a brief SLA violation on every traffic spike. The RL agent scales proactively for predictable spikes, eliminating those brief violations.
Scaling frequency: reduced by 40 percent. The RL agent makes fewer but larger scaling decisions compared to the rule-based approach, which reduces the overhead of pod scheduling and connection draining. This was an unexpected benefit. The no-op penalty in the reward function taught the agent that not scaling is often the best action.