Infra console

Screen-by-screen specification of ops.workspaces.build — the fleet view for platform SRE.

ops.workspaces.build — the whole fleet, every organization, metadata only.

Access requires platform SSO with a hardware security key. Every mutating action writes an audit entry, and entries touching a customer's resources are visible to that customer.

Fleet

Headroom below 15% on any class in any zone starts a procurement cycle. Steal above 3% means the class's oversubscription ratio is too aggressive for its workload mix.

Headroom below 15% on any class in any zone is the signal to add hardware. Steal time above 3% means the class's oversubscription ratio is too aggressive for its current workload mix.

Node detail

State, kernel and runtime versions, template cache contents, sessions by tier, resource graphs, device errors, the control-stream status, and metering WAL depth.

Actions: cordon · drain with a deadline · force-drain · reboot · retire. Draining shows how many sessions are affected and how many are preemptible before confirming.

Pools

The screen that predicts the next SLO breach.

Depth far below target AND refill latency far above normal is not demand outrunning the pool — it is the pool unable to refill. Raising the target would make it worse.

The highlighted row is the pattern to recognise: depth far below target and refill latency well above normal. That is not demand outrunning the pool, it is the pool unable to refill — storage or network pressure on the hosts backing it. Adding target depth would make it worse.

Per-pool drill-down: claim rate over time, target computation with every input shown, refill history, waste ratio, and the sessions currently holding claims.

Placement

Every decision is recorded with its scores and the filter that eliminated each rejected host. This turns an investigation into a lookup, and it is the most used screen during an incident.

Every placement decision is recorded with its scores and the filter that eliminated each rejected host. Paste a session ID, get the whole decision. This turns "why was it slow" from an investigation into a lookup, and it is the single most used screen during an incident.

Sessions

A cross-tenant list, metadata only: ID, organization, tier, state, node, age, resource use, cost. No filesystem, no logs, no screen, no network payloads.

Filters: organization, tier, state, node, zone, age, resource use, failure reason.

Actions: force-terminate (requires a reason, notifies the customer, writes an audit entry) · cordon the node it is on · request break-glass.

The most common use is finding the one organization whose runaway loop is consuming a zone's capacity, then talking to them — not killing their sessions.

Regions

Per-region: control-plane health, data-plane capacity, egress volume and denial rate, storage saturation, cross-region replication lag, and the active incident if there is one.

Metering

Billing correctness has its own screen because a silent failure here is worse than an outage.

ingest lag p99        42s      ✓
rating lag p99        28s      ✓
node WAL depth max    18m      ✓   (alert at 6h, critical at 24h)
events deduped        0.4%     ✓   (expected non-zero; spike = node retries)
events rejected       0        ✓   (must be zero)
reconciliation        last run 03:00 UTC · 10,000 sessions sampled · 0 mismatches

The reconciliation line is the one that matters. A nightly job replays sampled sessions from archived node WALs and compares against billed usage. Any mismatch is a billing incident at outage severity.

Images

Template build queue, build logs, vulnerability scan results, signing status, and per-region distribution: which regions have the layers, which have pre-booted snapshots, which pools are pinned.

A newly published template that is pinned to a pool but has no snapshots yet is shown as propagating, because sessions on it will cold start until it finishes.

Incidents

The status page is driven from here, by the same metrics the fleet view uses. A hand-maintained status page diverges from reality exactly when it matters most.

Organizations

The support view. Plan, quotas and current consumption, recent errors by code, spend trend, open incidents affecting them, and their session list in metadata form.

Visible: everything about shape and volume. Not visible: anything about content.

Actions: adjust quotas within plan · grant credits · open a support note · initiate break-glass.

Break-glass

Two people, scoped, time-boxed, logged immutably, visible to the customer, and the organization's owners are emailed at grant time.

Rules, enforced by the system rather than by policy:

Two people. The requester cannot approve. The approver must be in the on-call rotation.
Scoped. Logs, filesystem, screen and network payloads are separate grants.
Time-boxed. Maximum four hours, with no extension — a new request instead.
Logged immutably. Both the grant and every access under it.
Visible to the customer. The entries appear in their audit log, with the reason and both names.
Notified. The organization's owners get an email at grant time, not afterwards.

Break-glass without customer consent is possible only for an active abuse or security investigation, requires a security lead as the approver, and is reviewed monthly.

Admin

Price list management · plan definitions · region enablement · feature flags per organization · platform-floor policy (the non-overridable egress denies) · template base image promotion.

Changes here affect every customer, so each one is a four-eyes operation with a diff preview and a staged rollout.