Observability

What the platform measures, what customers can see about their own sessions, and the signals the infra team runs on.

Two audiences, one pipeline. Customers need to debug their agents; the infra team needs to run the fleet. Both are served from the same telemetry, filtered by ownership.

Session telemetry

Everything a session does is attached to a trace keyed by session_id.

Signal	Contents	Retention	Visible to customer
Lifecycle events	State transitions with reason, placement decision, cold or warm	90 days	Yes
Run records	Command or code, exit code, duration, truncated output	30 days	Yes
Logs	stdout and stderr per run, size-capped	7 days, configurable to 90	Yes
Egress log	Every connection: destination, rule matched, bytes, duration	30 days	Yes
Policy events	Every denial, with the rule	90 days	Yes
Resource samples	10 s CPU, memory, disk, network	7 days	Yes, aggregated
Recordings	Browser video, HAR, CDP trace, desktop video	Configurable	Yes, opt-in
Placement telemetry	Scores, rejected hosts, pool state at decision time	30 days	Summary only
Host telemetry	Steal time, memory bandwidth, thermal, device errors	30 days	No

The platform does not read session content

Log and recording bodies are stored encrypted with the organization's key and are not read by platform systems, including for debugging, except where an operator uses the break-glass flow — which requires two-person approval, is time-boxed, and writes an immutable audit entry the customer can see.

What a customer sees

Session trace

curl -sS "$WB_BASE_URL/sessions/$SESSION/trace" -H "Authorization: Bearer $WB_API_KEY"

{
  "session_id": "sess_2kQ9xwR4",
  "spans": [
    { "t": 0,    "event": "session.created",      "region": "bom1", "zone": "bom1-b" },
    { "t": 4,    "event": "placement.decided",    "host": "node-bom1b-0412", "pool_hit": true },
    { "t": 61,   "event": "session.running",      "cold_start": false, "start_ms": 61 },
    { "t": 1204, "event": "run.started",          "run_id": "run_8Hc2Vd", "command": "pnpm install" },
    { "t": 1310, "event": "egress.connect",       "host": "registry.npmjs.org", "rule": "allow:host=registry.npmjs.org" },
    { "t": 1402, "event": "egress.denied",        "host": "telemetry.vendor.io", "rule": "default_deny" },
    { "t": 18442,"event": "run.finished",         "run_id": "run_8Hc2Vd", "exit_code": 0 },
    { "t": 21003,"event": "session.stopping",     "reason": "client_delete" },
    { "t": 21890,"event": "session.stopped",      "usage": { "session_s": 21, "cpu_ms": 14320, "egress_bytes": 48211004 } }
  ]
}

The trace is the primary debugging artifact. "Why was my agent slow" is usually answered by pool_hit: false, and "why did my install fail" is usually an egress.denied line.

OpenTelemetry export

Customers can have session telemetry exported to their own collector as OTLP, so agent traces sit alongside their application traces.

{
  "otel_export": {
    "endpoint": "https://otlp.customer.com:4317",
    "protocol": "grpc",
    "headers": { "authorization": "Bearer …" },
    "signals": ["traces", "metrics", "logs"],
    "sampling": { "traces": 1.0, "logs": "errors_only" }
  }
}

Spans are emitted with wb.session.id, wb.run.id, wb.tier, wb.region, wb.end_user_id and wb.template as attributes, so a customer can group their own traces by tier or by end user without joining against the usage API.

Live streams

Stream	Transport	Use
Run output	SSE	Watch a command as it runs
PTY	WebSocket	Interactive shell, dashboard terminal
Desktop	WebRTC	Tier 4 live view and human handoff
Session events	SSE on `/sessions/{id}/events`	React to state changes without polling

What the infra team sees

Golden signals, per tier

Signal	Metric	SLO
Start latency	`session_start_ms{tier,cold}` p50 / p95	Per the tier table
Availability	`session_create_success_ratio{tier}`	99.9%
Pool health	`pool_hit_ratio{tier}`	> 0.97
Execution	`run_duration_ms{tier}` p50 / p95	No SLO — customer-determined
Failure	`session_failed_total{reason}`	`host_failure` < 0.1% of sessions
Saturation	`allocatable_ratio{class,zone}`	> 0.15 headroom

Fleet health

nodes            ready / cordoned / draining / unhealthy, per class per zone
pools            depth vs target, refill latency, waste ratio
placement        decisions/s, failures by reason, p99 decision latency
contention       steal time p95 per class, memory bandwidth saturation
storage          scratch utilisation, volume IOPS, snapshot upload backlog
egress           bytes/s per region, denial rate, proxy p99 added latency
metering         ingest lag, WAL depth per node, dedupe rejection rate
control plane    gateway p99, admission p99, reconciler loop lag

Alerts that page

Alert	Condition	Why it pages
`PoolHitRatioLow`	`pool_hit_ratio{tier} < 0.90` for 10 m	The tier's start-latency SLO is breaking now
`MeteringLagHigh`	Ingest lag > 10 m, or any node WAL > 6 h	Approaching unbilled work
`PlacementFailures`	`no_capacity` > 1% of creates for 5 m	Customers are being refused
`EgressDenialSpike`	Denial rate > 20× baseline for one org	Likely a compromised or injected agent
`ControlPlaneErrorRate`	5xx > 1% for 5 m	API is failing
`NodeIsolated`	Any node unreachable > 15 m with running sessions	Sessions are running unobserved
`SpendCapEnforcementStalled`	Rated usage lag > 5 m	Caps are not being enforced

Alerts that do not page

Pool waste above target, individual host degradation with sessions already drained, template build failures for unused templates, and anything a reconciler will fix within its loop interval. These open a ticket instead. Paging on self-healing conditions is how an on-call rotation stops reading pages.

Metering observability

Metering has its own dashboards because billing correctness is non-negotiable.

events_in          per node, per meter
events_deduped     redeliveries caught — expected non-zero, spikes mean node retries
events_rejected    schema or window violations — should be zero
ingest_lag         node emission to minute bucket available
rating_lag         minute bucket to rated usage
usage_api_lag      rated usage to customer-visible
reconciliation     sampled sessions: node-reported vs billed, must match exactly

The reconciliation check is the important one. A nightly job replays a sample of sessions from node WAL archives and compares against billed usage. A mismatch is a billing incident, handled under the same severity as an outage.

Audit

Separate from telemetry, with different guarantees: append-only, hash-chained, seven-year retention, written synchronously with the action.

Recorded: key created or revoked, policy changed, member role changed, spend cap changed, session force-terminated by an operator, snapshot exported or copied cross-region, break-glass access granted and exercised, TLS inspection enabled, public port exposed.

Exportable by the customer in full, because the entries about their organization are theirs.

Customer-facing status

status.workspaces.build publishes per-region, per-tier availability and start-latency percentiles from the same metrics the infra team runs on, not a separate hand-maintained set. Incidents are posted with the affected regions and tiers, and sessions failed by a declared incident are credited automatically under the SLOs.