Scheduler and warm pools

Start latency is the feature this subsystem exists to deliver. Everything here is in service of one number per tier: the probability that a session create can claim a pre-started runtime instead of booting one.

The claim path versus the cold path

A claim is a compare-and-swap on a Redis set of available pool instances, followed by a node instruction to bind the session. The CAS is what prevents two concurrent creates taking the same instance; a lost CAS retries against the next instance, and after three failures falls through to cold start rather than spinning.

The ratio is why pool hit rate, not raw capacity, is the metric the infra team watches.

Pool identity

A pool is keyed by four dimensions, and all four must match for a claim:

(tier, template@digest, size_class, zone)

• template@digest — the content hash, not the friendly name. A template update creates a new pool; the old pool drains as its instances are claimed or expire.
• size_class — a discrete bucket (s = 1 vCPU / 2 GiB, m = 2 / 4, l = 4 / 8, xl = 8 / 16, 2xl = 16 / 32), not an arbitrary request. Continuous sizing would make pools uncachable; requests are rounded up to the next class and billed on the class.
• zone — a session with an attached volume is constrained to the volume's zone, which is the most common cause of a miss in an otherwise healthy region.

Pool sizing

pool-mgr recomputes targets every 15 seconds per pool.

target = clamp(
    ceil( burn_rate × refill_latency × safety )  +  baseline,
    floor,
    ceiling
)

burn_rate        claims/second, EWMA over 5 min, plus a seasonal term from
                 the same slot last week and yesterday
refill_latency   p90 seconds to bring one instance to ready for this pool
safety           1.5 for tiers 1–2, 2.0 for tiers 3–4 (slower refill, costlier miss)
baseline         minimum depth so the first request of a quiet period is warm
floor            contractual reserved depth for orgs that bought it
ceiling          budget guard: pooled capacity cost cap per tier per zone

The seasonal term matters because agent traffic is extremely diurnal and weekly. A purely reactive EWMA always under-provisions the Monday morning ramp; the previous week's same-slot value gets the pool ahead of it.

The economics

Every pooled instance occupies real capacity. Pool cost is a direct platform expense, not a customer charge — except where a customer has bought reserved depth, which is billed as pool_reservation_s.

pool cost      ≈ depth × hourly cost of one idle instance
miss cost      ≈ p(miss) × creates × (cold latency − warm latency) × value of latency

Tier 4 makes this concrete: a pooled desktop costs roughly a running desktop, so depth 20 costs twenty desktops' worth of capacity continuously. That is why tier 4 pool depth beyond a small platform baseline is a purchased reservation rather than a free platform guarantee.

Refill

Refill is asynchronous and rate-limited. A claim schedules a refill immediately, but refills are throttled per host so a burst of claims does not saturate every host's NVMe and network simultaneously — which would slow the cold starts those same claims are falling back to.

Refill priority: pools below floor, then pools furthest below target as a fraction, then pre-emptive refill for predicted demand.

Placement

When there is no claim to make, placer chooses a host.

Filters — hard constraints

Scoring — soft preferences

Every decision is recorded with its scores and the filter that eliminated each rejected host, which is what makes "why did this cold start" answerable from the infra console.

Spread and affinity

Callers can request placement hints, which are preferences and never hard constraints unless the plan includes reserved capacity:

• spread: "zone" — place replicas of a workload across zones
• affinity: { session_id } — co-locate with another session, for low-latency inter-session work
• anti_affinity: { label } — keep labelled sessions apart

Oversubscription

Memory is barely oversubscribed anywhere because memory pressure is not gracefully degradable — it becomes an OOM kill, which to a customer is an unexplained failure. CPU oversubscription shows up as slower execution, which is recoverable and visible in cpu_ms.

Steal time per session is tracked and exposed. Sustained steal above a threshold triggers placement rebalancing for new sessions and marks the host as contended in the scoring function.

Eviction

Sessions can be evicted only when the plan marks them preemptible, or when the host is failing.

Non-preemptible sessions are never evicted for capacity. If the fleet is short, new creates are refused with 503 capacity_unavailable rather than killing running work — failing to start is recoverable, failing mid-run usually is not.

Metrics the infra team watches

pool_hit_ratio and pool_waste_ratio are the two numbers in tension. Tuning the fleet is the continuous process of holding the first above target while keeping the second low enough that pooled capacity is not the dominant cost line. See capacity planning.