Runbooks

Index

PoolHitRatioLow

Alert: pool_hit_ratio{tier} < 0.90 for 10 minutes Severity: SEV2 — the tier's start-latency SLO is breaking now Owner: Fleet on-call

Impact

Sessions at the affected tier are cold starting. Customers see start latency at the p99 column of the tier table instead of the p50 — 2.5 s instead of 180 ms at tier 2, 45 s instead of 2.5 s at tier 4. Error budget for start latency is burning.

Diagnose

1. 1

   Open Pools in the infra console. Sort by hit ratio ascending. Identify whether this is one pool, one zone, or the whole tier.

2. 2

   Read depth against target, and refill latency, together. The combination tells you which of three problems you have:

   depth/target	refill p90	Cause
   Low	Normal	Demand spike outrunning the pool. Go to Demand spike
   Low	High	Pool cannot refill. Go to Refill failure
   At target	Normal	Pool key mismatch. Go to Key mismatch

3. 3

   Check zone balance. A single zone dragging the tier's aggregate usually means volume-driven placement concentration, not a fleet-wide problem.

Demand spike

Depth low, refill latency normal — the pool is working, there is just more demand than target.

1  Check Placement for no_capacity failures.
     present  -> this is a capacity incident, not a pool incident.
                 Go to PlacementFailures.
     absent   -> capacity exists; the target is too low.

2  Identify the organization driving it (Pools -> drill-down -> claims by org).

3  Raise the pool floor for the affected pools:
     ops console -> Pools -> <pool> -> set floor
   This overrides the computed target until cleared. Set it to roughly
   1.5× observed burn_rate × refill_latency.

4  If one organization is driving a step change, contact them.
   A 10× ramp is a capacity planning conversation, not an incident.

5  Clear the floor override within 24 h once the seasonal term has learned
   the new level. Leave a note on the incident.

Refill failure

Depth low and refill latency high. Raising the target makes this worse.

1  Fleet -> filter to the affected class and zone.
   Look at NVMe utilisation and network throughput per host.

2  Common causes, in order of frequency:

   a  Template distribution storm
      A newly published template is being pulled by every host at once.
      Check Images -> distribution status.
      Fix: throttle distribution; pause the publish if it is still spreading.

   b  Storage backend degraded
      Snapshot reads are slow, so every restore is slow.
      Check regional object storage latency and error rate.
      Fix: fail over to the secondary endpoint; raise with the storage team.

   c  Host NVMe saturation
      Too many concurrent refills on the same hosts.
      Check per-host refill concurrency.
      Fix: lower the per-host refill rate limit; refills spread over more hosts.

   d  A bad host
      One or two hosts failing refills repeatedly and retrying.
      Check refill error rate per host.
      Fix: cordon those hosts.

3  Do NOT raise pool targets while refill latency is elevated.
   More targets means more concurrent refills means worse latency.

Key mismatch

Depth at target but hit ratio low — the pool holds instances nobody is claiming.

1  Compare the claimed key against the pooled key for a failing session:
     Placement -> paste a cold-started session ID -> read "pool lookup".

2  Usual causes:

   a  Template digest moved
      A republish created a new digest; the pool still holds the old one.
      Fix: pool-mgr drains the old pool and builds the new one within ~15 min.
           If urgent, force the new pool's floor.

   b  Size class drift
      Customers requesting a class with no pool (often "l" or "xl").
      Fix: add the class to the template's pooled size classes.

   c  Zone forced by a volume
      Sessions pinned to a zone whose pool is thin.
      Fix: raise that zone's floor; longer term, rebalance volume creation.

3  Confirm recovery: hit ratio should climb within two refill cycles.

Escalate

• No improvement after 30 minutes, or hit ratio below 0.75: page the secondary and declare SEV1.
• Refill failure traced to storage: page the storage on-call.
• Multiple regions affected simultaneously: this is not a pool problem. Check for a recent pool-mgr or placer deploy and consider a rollback.

Verify and close

Hit ratio back above target for 15 minutes, start-latency p95 within SLO, no floor overrides left set without a note, and an incident record with the cause and the remaining follow-up.

MeteringLagHigh

Alert: ingest lag > 10 minutes, or any node WAL depth > 6 hours Severity: SEV1 — approaching unbilled work and stalled spend-cap enforcement Owner: Billing on-call, with platform on-call as secondary

Impact

Three things degrade together: customers cannot see current usage; spend caps are evaluated against stale data so a runaway workload is not stopped; and if a node's WAL fills, usage is permanently lost, which is revenue loss and a reconciliation failure.

The 24-hour WAL ceiling is hard. A node past it refuses new placements.

Diagnose

1. 1

   Open Metering in the infra console. Identify which stage is lagging:

   emission -> ingest        node side, or the transport between
   ingest -> minute bucket   aggregation
   bucket -> rated usage     rating
   rated -> usage API        serving

2. 2

   Check events_rejected. It must be zero. Non-zero means malformed events, which usually means a node running a version that emits a schema the ingest does not accept. Find the version skew before anything else.

3. 3

   Check node WAL depth distribution. One node deep is a node problem. Every node deep is an ingest problem.

Ingest stalled

1  Check Kafka consumer lag for the metering ingest group.
     lag growing     -> consumers cannot keep up: scale out, check for a hot partition
     lag flat, high  -> consumers are stuck: check for a poison message

2  Poison message: the ingest log names the offset and the reason.
     Do NOT skip the offset blind. Capture the message first — it is billable usage.
     Route it to the dead-letter topic, record the session IDs, and replay
     after the fix. Skipping without capture is unbilled work.

3  Hot partition: partitioning is by organization_id. One very large organization
   can saturate a partition. Fix: increase partition count and rebalance; longer
   term, sub-partition large organizations by project.

Rating stalled

1  Check ClickHouse insert latency and merge backlog.
   A merge backlog slows inserts, which backs up rating.
   Fix: reduce insert batch frequency; let merges catch up.

2  Check for a price list lookup failure. Rating needs the org's price list
   version for the window. A missing version blocks rating for that org only —
   so a global lag is not this.

Nodes buffering

1  If one node: check its control-stream status. It may be isolated.
   Go to NodeIsolated.

2  If many nodes in one zone: network partition between the zone and
   the ingest endpoint. Check the regional network before anything else.

3  Nodes keep running sessions and keep buffering. Do not drain them to
   "fix" the lag — draining destroys sessions and does not help the WAL.

4  WAL above 12 h on any node: raise to SEV1 if not already, and prepare
   to extract the WAL manually. The procedure is in the billing team's
   internal runbook and requires a billing lead.

Do not

• Do not skip events to clear lag. Every skipped event is work the platform did unbilled, and a guaranteed reconciliation mismatch.
• Do not disable spend-cap enforcement to reduce load. Caps failing open is already happening; disabling them makes it permanent.
• Do not restart nodes. The WAL survives a restart, but running sessions do not.

Verify and close

Ingest lag under 2 minutes, all node WALs under 1 hour, events_rejected at zero, and the next reconciliation run showing zero mismatches. If any events went to the dead-letter topic, confirm they were replayed and the affected organizations' usage is correct before closing.

Outlined runbooks

PlacementFailures

Alert: placement_failures{reason=no_capacity} > 1% of creates for 5 minutes · SEV2

Customers are being refused. First: is it one class, one zone, one region? Check allocatable_ratio per class and zone. Likely causes: genuine capacity shortage (reduce pool targets to free capacity as a stopgap, start procurement); a zone failure concentrating load; a large customer ramp; or hosts stuck in draining. Escalate to capacity planning if headroom is genuinely exhausted — there is no operational fix for not enough machines.

EgressDenialSpike

Alert: denial rate > 20× baseline for one organization · SEV3

The strongest available signal that an agent has been prompt-injected. Do not kill the sessions — this is the customer's decision and their data. Identify the organization, the sessions, and the destinations being attempted. Contact the organization's security contacts with the denied destination list. If destinations match known-malicious indicators, escalate to the security lead, who may add them to the platform floor.

ControlPlaneErrorRate

Alert: 5xx > 1% for 5 minutes · SEV1

Check which service: gateway, identity, admission, placer, session-mgr. Check for a recent deploy first — rollback is faster than diagnosis. Then dependencies: Postgres primary health, Redis availability, global replication lag. Running sessions are unaffected; say so on the status page, because it changes what customers do.

NodeIsolated

Alert: node unreachable > 15 minutes with running sessions · SEV2

Sessions are running unobserved and unmetered. Determine whether the node is dead or merely unreachable. If unreachable, the sessions are still serving customers — do not fail them. If dead, the reconciler will fail the sessions; confirm customers got session.failed. Either way, recover or extract the metering WAL before retiring the node.

SpendCapEnforcementStalled

Alert: rated usage lag > 5 minutes · SEV2

Caps are failing open. Usually a downstream symptom of MeteringLagHigh — check that first. If metering is healthy, the cap evaluator itself is stuck. Identify organizations currently near their cap and monitor them manually until enforcement resumes. Do not disable caps.

ZoneDown

Alert: zone unreachable or > 50% of hosts unhealthy · SEV1

Sessions in the zone are lost; confirm session.failed webhooks were delivered. Verify surviving zones are absorbing placements and that the failure-domain reserve is doing its job. Rebalance pool targets to the surviving zones. Post to the status page immediately — affected customers are seeing failures right now. Do not attempt to migrate sessions; they are gone.

ReconciliationMismatch

Alert: any mismatch in the nightly reconciliation · SEV1

A billing correctness incident. Do not close it by re-running the job. Identify the affected sessions and whether billing was high or low. If customers were overbilled, issue credit notes before they notice. If underbilled, decide deliberately whether to back-bill — usually not, for amounts below a threshold, but record the decision. Root-cause the pipeline defect before closing; a mismatch that recurs is a systematic error, not a one-off.