Control plane

The control plane is authoritative for everything the platform knows and decides. It is deliberately kept off the data path, so its latency budget is generous and its availability requirement is lower than the data plane's.

Deployment shape

The control plane runs as cells. A cell is a complete, independently deployable copy of the control-plane services serving one region group, with its own regional Postgres and its own Redis. Global state — organizations, keys, templates, invoices — lives in a globally replicated Postgres that every cell reads and that accepts writes through a single primary.

A cell failure takes out new-session creation for its region group and nothing else. Cells never call each other synchronously on a request path.

Services

api-gateway

The only publicly reachable control-plane service.

Responsibilities, in request order:

1. Authenticate. Resolve Authorization to a principal: an API key, an end-user session token, or a dashboard OIDC session. Keys are verified against a hash; the lookup is Redis-cached with a 60-second TTL and invalidated on revocation.
2. Authorize. Check the principal's scope set against the route's required scope, then check object ownership — every object carries its project, every project its organization.
3. Rate limit. Token bucket per (organization, route class) plus a per-key burst bucket, both in Redis. See rate limits.
4. Idempotency. For unsafe methods carrying Idempotency-Key, claim the key in Redis with a 24-hour TTL, then either replay the stored response or run the handler and store it.
5. Validate. Request bodies are validated against the same OpenAPI schema the public reference is generated from. Divergence is a build failure, not a documentation bug.
6. Log. Every request produces one structured record with request_id, principal, route, status, latency and the usage it caused.

Latency target: p99 under 15 ms of gateway overhead, excluding the handler.

identity

Organizations, projects, members, roles, API keys, end-user records and token minting.

• RBAC is role-per-member-per-organization, with project-level overrides. Roles: owner, admin, developer, operator, billing, viewer.
• API keys are stored as a hash plus a short non-secret prefix used for display and lookup sharding. The plaintext is returned once.
• End-user tokens are short-lived JWTs minted from a publishable key, scoped to one end user and optionally one session, signed by a per-region key with a 10-minute default TTL.
• SSO for the dashboard is OIDC, with SCIM provisioning for enterprise plans.

Writes go to global Postgres; every cell holds a read model kept current by logical replication.

admission

The right to create. Called before placement, and the only thing standing between a runaway loop and a large invoice.

Checks, all of which must pass:

Admission reserves a concurrency slot that is released on any terminal session state. The reservation is a Redis entry with a TTL slightly longer than the maximum provisioning time, so a lost teardown cannot leak a slot forever.

placer

Chooses where a session runs. Input: tier, template, resource request, region preference, residency constraint, affinity hints. Output: a concrete host and either a warm-pool claim or a cold-start instruction.

Selection runs as filter-then-score:

filter   region allowed by residency policy
         zone has capacity in the tier's placement class
         host kernel/runtime version satisfies the template
         host not draining, not cordoned, health score above threshold
         host has the template layers cached (soft filter: cold hosts stay eligible)

score    + warm pool hit for (tier, template, size)        weight 100
         + template layers already local                   weight  30
         + headroom after placement                        weight  20
         + zone balance                                    weight  15
         - host at risk of noisy-neighbour contention      weight  25
         - host approaching a scheduled drain              weight  40

The warm-pool hit term dominates on purpose: claiming a pre-started instance is worth more to the user than any packing efficiency gained by placing elsewhere.

Placement decisions are recorded with their scores, which is what makes "why did this session cold start" answerable after the fact.

Detail: scheduler and warm pools.

session-mgr

Owns the session state machine and the reconciler.

It never issues an imperative "create this VM" call. It writes desired state to regional Postgres, and wbnode instances pull the desired state for their host over a long-lived bidirectional stream and report actual state back. The reconciler compares the two and acts:

Reconciliation is idempotent and runs continuously, so a dropped message or a restarted process cannot leave a session or a VM stranded.

Detail: session lifecycle.

policy

Turns declarative policy into a concrete artifact for the data plane.

Compilation happens once per session, at admission. The data plane enforces the compiled artifact with no callback to the control plane, which is what keeps enforcement available when the control plane is not. A policy change propagates to running sessions at the next enforcement interval — within 30 seconds for egress, immediately for new processes under the CLI allowlist.

The platform floor is the set of restrictions no customer policy can relax: link-local and metadata endpoints, internal platform CIDRs, known-malicious destinations, and the guest-to-host device surface.

Detail: policy engine.

image-svc

Template registry and build pipeline.

• Templates are content-addressed and immutable. tpl_base_ubuntu_24_04@sha256:… is the real identity; the friendly name is a moving pointer.
• Customer templates are built from a declarative spec in a sandboxed builder, scanned for known vulnerabilities, signed, and distributed to regional layer caches.
• Distribution is lazy per region with eager pre-pull for templates attached to a warm pool.
• For microVM tiers, a template also has a pre-booted snapshot per size class, built once and used for every restore. Producing that snapshot is part of the publish pipeline, which is why a freshly published template is cold until its snapshots exist.

pool-mgr

Computes and holds the warm-pool target for every (tier, template, size, zone) combination, and instructs nodes to pre-start or release instances. Detail: scheduler and warm pools.

metering

Ingests usage events from nodes, proxies and storage. Deduplicates, validates, aggregates to minute buckets and publishes to the rating pipeline. Detail: metering model.

billing

Rates aggregated usage against the organization's effective price list, maintains rollups, applies credits, produces invoices, and runs dunning. Detail: invoices and credits.

audit

An append-only, hash-chained record of every privileged action: key created, policy changed, member role changed, session force-terminated, snapshot exported, spend cap raised. Written synchronously with the action it records, exportable by the customer, retained for seven years.

webhooks

Signed delivery with exponential backoff, a per-endpoint circuit breaker, and 30 days of replay from the events API. Detail: webhooks.

Data stores

Consistency decisions worth knowing

Session state is regionally strong. A read of a session in its own region always reflects the latest write. A cross-region list may lag by replication delay.

Quota counters are optimistic with reconciliation. The Redis counter is authoritative for admission; a reconciler compares it against Postgres every 30 seconds and corrects drift. A lost Redis instance means quotas briefly over-admit, which is preferred over refusing every create.

Usage events are at-least-once, dedupe is at ingest. Nodes retry until acknowledged. The dedupe key (session_id, meter, window_start) makes retries harmless. A customer is never double-billed for a redelivery.

Billing is strongly consistent at the invoice boundary. Rating and rollups are eventually consistent during the period; invoice finalisation takes a consistent snapshot and is immutable thereafter.