Desktop control

GET  /v1/sessions/{id}/desktop/screenshot     observe
POST /v1/sessions/{id}/desktop/act            input batch
GET  /v1/sessions/{id}/desktop/windows        window tree
POST /v1/sessions/{id}/desktop/clipboard      read or write the clipboard
WS   /v1/sessions/{id}/desktop                live view and control (WebRTC signalling)
POST /v1/sessions/{id}/desktop/handoff        transfer control to a human

Tier 4 only.

Observe

curl -sS "$WB_BASE_URL/sessions/$SESSION/desktop/screenshot?format=webp&scale=0.5&include=windows,text" \
  -H "Authorization: Bearer $WB_API_KEY"

{
  "image_b64": "…",
  "format": "webp",
  "width": 1280, "height": 800,
  "native_width": 2560, "native_height": 1600,
  "scale": 0.5,
  "captured_at": "2026-10-06T09:22:11.402Z",
  "windows": [
    { "id": "w_3", "title": "Invoices — LedgerPro", "rect": [0, 48, 2560, 1552], "focused": true, "app": "ledgerpro" },
    { "id": "w_7", "title": "Save As", "rect": [820, 540, 920, 480], "focused": false, "modal": true }
  ],
  "text_regions": [
    { "text": "Export to CSV", "rect": [1624, 280, 208, 48], "confidence": 0.98 },
    { "text": "File name:",    "rect": [860,  620, 180, 36], "confidence": 0.99 }
  ]
}

windows comes from the guest window manager and text_regions from guest-side OCR. Both exist so the model does not have to locate everything from raw pixels, which is slow, costly and error-prone. Giving a model the window list and the OCR regions alongside a half-scale screenshot typically halves the number of turns a task takes.

Act

curl -sS "$WB_BASE_URL/sessions/$SESSION/desktop/act" \
  -H "Authorization: Bearer $WB_API_KEY" -H "Content-Type: application/json" \
  -d '{
    "actions": [
      { "type": "move",   "x": 1728, "y": 304 },
      { "type": "click",  "button": "left" },
      { "type": "wait",   "ms": 300 },
      { "type": "key",    "combo": "ctrl+s" },
      { "type": "type",   "text": "q3-invoices.csv", "delay_ms": 20 },
      { "type": "key",    "combo": "Return" }
    ],
    "observe_after": true,
    "coordinate_space": "native"
  }'

coordinate_space is native by default. Set it to observed to use the coordinate space of the last screenshot the agent saw, and the platform rescales — which removes a whole class of off-by-scale bugs when a model works from a downscaled image.

Batches stop at the first failure, and everything before it has already happened.

Windows

curl -sS "$WB_BASE_URL/sessions/$SESSION/desktop/windows" -H "Authorization: Bearer $WB_API_KEY"

{
  "object": "list",
  "data": [
    { "id": "w_3", "title": "Invoices — LedgerPro", "app": "ledgerpro", "rect": [0,48,2560,1552],
      "focused": true, "minimized": false, "modal": false },
    { "id": "w_7", "title": "Save As", "app": "ledgerpro", "rect": [820,540,920,480],
      "focused": false, "minimized": false, "modal": true }
  ]
}

The modal: true flag is worth branching on: a modal dialog means nothing else will accept input until it is dismissed, which is the most common reason a computer-use agent appears to hang.

Clipboard

curl -sS "$WB_BASE_URL/sessions/$SESSION/desktop/clipboard" -H "Authorization: Bearer $WB_API_KEY"

curl -sS -X POST "$WB_BASE_URL/sessions/$SESSION/desktop/clipboard" \
  -H "Authorization: Bearer $WB_API_KEY" -H "Content-Type: application/json" \
  -d '{ "text": "value to paste" }'

The clipboard is the reliable way to get a long string into a desktop application — typing it character by character is slow and susceptible to focus changes mid-type. Clipboard operations are mediated through the API and logged; there is no transparent host-guest clipboard channel.

Live view

wss://connect.workspaces.build/v1/sessions/{id}/desktop?token=wb_st_…

WebRTC signalling, then a media stream plus an input data channel.

{ "type": "offer",  "sdp": "…" }
{ "type": "answer", "sdp": "…" }
{ "type": "input",  "action": { "type": "click", "x": 1728, "y": 304 } }
{ "type": "quality", "max_fps": 15, "max_bitrate_kbps": 2000 }

Token scopes: desktop:view for watching, desktop:control for watching and driving. Mint desktop:view for an end user watching their agent work; mint desktop:control only when a human is meant to take over.

Human handoff

curl -sS "$WB_BASE_URL/sessions/$SESSION/desktop/handoff" \
  -H "Authorization: Bearer $WB_API_KEY" -H "Content-Type: application/json" \
  -d '{ "mode": "human", "reason": "captcha", "notify_webhook": true, "timeout_s": 600 }'

{
  "mode": "human",
  "agent_input": "disabled",
  "view_url": "https://console.workspaces.build/handoff/sess_2kQ9xwR4?t=…",
  "expires_at": "2026-10-06T09:34:11Z"
}

Handoff is a first-class flow, not a convention. The platform downgrades the agent's input channel to read-only, grants input to the human, emits a session.handoff webhook, and writes an audit entry for both the start and the return. POST :handoff with mode: "agent" hands control back.

This is the right answer for CAPTCHAs, unexpected MFA prompts and anything requiring a judgement the agent should not make alone.

Metering

Practical guidance

• Observe less often than you act. A screenshot per action triples vision token cost for little gain. Batch several actions, then observe once.
• Use scale: 0.5 or lower. Most desktop UI is legible at half scale, and vision tokens scale with pixels.
• Prefer keyboard to mouse. Keyboard shortcuts are deterministic; clicking a coordinate depends on the window being where it was when the screenshot was taken.
• Check modal before concluding the application is stuck.
• Snapshot after a long setup. Installing and configuring an application takes minutes at the most expensive tier; snapshot once and restore for every subsequent task.