Drive and record the desktop

Drive the desktop with a prompt

The coding agents of the sandbox drive the same screen that you stream. Each harness on a sandbox has built-in computer tools. These tools take screenshots, click, type, scroll and start apps. You only send a prompt:

boat prompt bx_f7k2q9hd "Open example.com in Chrome and tell me the page title"

curl -sS -X POST "$BOAT_API_BASE/sandboxes/bx_f7k2q9hd/prompt" \
  -H "Authorization: Bearer $BOAT_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"prompt":"Open example.com in Chrome and tell me the page title"}'

await sandbox.prompt({
  sandboxId: "bx_f7k2q9hd",
  prompt: "Open example.com in Chrome and tell me the page title",
});

sandbox.prompt("bx_f7k2q9hd", PromptRequest(
    prompt="Open example.com in Chrome and tell me the page title",
))

Open a stream in one window. Watch the agent work in it.

You install nothing.
You register nothing.
Each harness already has the tools.

For the full tool list, the tool names per harness and the health check, read Computer use.

Mix shell commands and prompts

Shell commands also work on the same screen. You can use them together with prompts. For example, open a page yourself, then give the sandbox a visual task:

boat exec bx_f7k2q9hd 'DISPLAY=:0 google-chrome-stable "https://example.com" & sleep 2; wmctrl -a Chrome'
boat prompt bx_f7k2q9hd "Complete the checkout flow in the open browser window"

curl -sS -X POST "$BOAT_API_BASE/sandboxes/bx_f7k2q9hd/commands" \
  -H "Authorization: Bearer $BOAT_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"command":"DISPLAY=:0 google-chrome-stable \"https://example.com\" & sleep 2; wmctrl -a Chrome"}'

await sandbox.command({
  sandboxId: "bx_f7k2q9hd",
  command: 'DISPLAY=:0 google-chrome-stable "https://example.com" & sleep 2; wmctrl -a Chrome',
});

sandbox.command("bx_f7k2q9hd", CommandRequest(
    command='DISPLAY=:0 google-chrome-stable "https://example.com" & sleep 2; wmctrl -a Chrome',
))

A sandbox has one screen. Each conversation on the sandbox drives that same screen. If two agents click at the same time, they interfere with each other. Give each parallel GUI task its own sandbox, or run the tasks one after the other.

Record the screen

ascii-record-desktop records the live desktop to an MP4 from inside the sandbox. Use it to make a demo or to keep evidence of a GUI run.

boat exec bx_f7k2q9hd "ascii-record-desktop start /home/user/demo.mp4"
# ... drive the desktop ...
boat exec bx_f7k2q9hd "ascii-record-desktop stop"
boat scp bx_f7k2q9hd:/home/user/demo.mp4 ./demo.mp4

curl -sS -X POST "$BOAT_API_BASE/sandboxes/bx_f7k2q9hd/commands" \
  -H "Authorization: Bearer $BOAT_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"command":"ascii-record-desktop start /home/user/demo.mp4"}'

curl -L -o demo.mp4 "$BOAT_API_BASE/sandboxes/bx_f7k2q9hd/artifacts?path=demo.mp4" \
  -H "Authorization: Bearer $BOAT_API_KEY"

await sandbox.command({
  sandboxId: "bx_f7k2q9hd",
  command: "ascii-record-desktop start /home/user/demo.mp4",
});
const video = await sandbox.artifact({ sandboxId: "bx_f7k2q9hd", path: "demo.mp4" });

sandbox.command("bx_f7k2q9hd", CommandRequest(
    command="ascii-record-desktop start /home/user/demo.mp4",
))
video = sandbox.artifact("bx_f7k2q9hd", "demo.mp4")

Run ascii-record-desktop status to see if a recording runs. To share a recording without a download, serve its folder from the sandbox. Then expose the folder with a URL that needs a token:

boat ssh bx_f7k2q9hd "cd ~ && (python3 -m http.server 8090 --bind 0.0.0.0 &) && host 8090 --private"

​Drive the desktop with a prompt

​Mix shell commands and prompts

​Record the screen

Drive the desktop with a prompt

Mix shell commands and prompts

Record the screen