> ## Documentation Index
> Fetch the complete documentation index at: https://docs.boat.dev/llms.txt
> Use this file to discover all available pages before exploring further.

# Evals and RL environments

> Run many eval or RL episodes on Boat: one named snapshot per environment, one fresh sandbox per episode, parallel fan-out within start limits, teardown, cost. Harbor and Terminal-Bench status.

An episode is one sandbox. Prepare the environment once, save it as a named snapshot, and start each episode from it. Delete the sandbox when the episode ends. The next episode starts from the same clean state.

```mermaid theme={null}
flowchart LR
  P[Prepare one sandbox] --> S[Save named snapshot]
  S --> E1[Episode 1: boat new --from]
  S --> E2[Episode 2: boat new --from]
  S --> E3[Episode N: boat new --from]
  E1 --> D[Score, then delete]
  E2 --> D
  E3 --> D
```

## 1. Prepare the environment

Install the task's dependencies and fixtures in one sandbox. Then save it under a name:

```bash theme={null}
id=$(boat new --json | jq -r 'select(.event == "ready") | .id')
boat exec "$id" --timeout 600 -- bash -lc 'git clone https://github.com/acme/eval-tasks && pip install -r eval-tasks/requirements.txt'
boat snapshot "$id" task-env-v1
boat delete "$id" --yes
```

* Use one named snapshot per environment version: `task-env-v1`, `task-env-v2`. A named snapshot does not change after you save it.
* Docker images: a named snapshot keeps only the images that a container uses. Run `docker create --name keep-<x> <image>` for each image. See [What a snapshot saves](/snapshots/whats-saved#docker).
* 10 named snapshots are free. Each one more costs \$1.70 a month. See [Named snapshots](/snapshots/templates).

## 2. Run episodes in parallel

This Python script starts one sandbox per episode from the named snapshot, runs the episode, reads the score and deletes the sandbox. It paces the starts to the minute limit of the \$20 plan.

```python title="run_episodes.py" theme={null}
import os
import time
import uuid
from concurrent.futures import ThreadPoolExecutor

from boat_sdk import ApiClient, Configuration, exec_command, wait_until_ready
from boat_sdk.api.boat_api import BoatApi
from boat_sdk.exceptions import ApiException
from boat_sdk.models import CreateSandboxRequest

STARTS_PER_MINUTE = 12      # $20 plan. See https://docs.boat.dev/limits
STARTS_PER_HOUR = 60        # $20 plan. The hour limit is the tighter one for long runs.
TEMPLATE = "task-env-v1"

config = Configuration(host="https://boat.dev/api/v1", access_token=os.environ["BOAT_API_KEY"])


def start(boat: BoatApi) -> str:
    request = CreateSandboxRequest(var_from=TEMPLATE, type="small", ttl_seconds=3600)
    key = str(uuid.uuid4())          # the same key on a retry never makes a second sandbox
    for attempt in range(6):
        try:
            return boat.create(idempotency_key=key, create_sandbox_request=request).sandbox.id
        except ApiException as exc:
            if exc.status not in (429, 503):
                raise
            time.sleep(min(60, 2 ** attempt * 5))
    raise RuntimeError("could not start a sandbox")


def episode(boat: BoatApi, n: int) -> tuple[int, int, str]:
    sandbox_id = start(boat)
    try:
        wait_until_ready(boat, sandbox_id)
        result = exec_command(boat, sandbox_id, f"python3 eval-tasks/run.py --episode {n}", timeout_seconds=600)
        out = result.actual_instance
        return n, out.exit_code, out.stdout
    finally:
        boat.delete_sandbox(sandbox_id=sandbox_id, x_ascii_confirm_delete=sandbox_id)


with ApiClient(config) as client:
    boat = BoatApi(client)
    with ThreadPoolExecutor(max_workers=50) as pool:
        futures = []
        for n in range(100):
            futures.append(pool.submit(episode, boat, n))
            # pace the starts under both the minute and the hour limit
            time.sleep(max(60 / STARTS_PER_MINUTE, 3600 / STARTS_PER_HOUR))
        for future in futures:
            n, code, stdout = future.result()
            print(n, code, stdout.strip()[-200:])
```

* `exec_command` waits at most 600 seconds. For a longer episode, start it with `detached=True` and poll `command_status`. See [Python SDK](/sdks/python#run-a-long-job-or-a-server-in-the-background).
* `ttl_seconds=3600` stops a sandbox after one hour if your script crashes before the delete.
* In asyncio code, wrap each call in `asyncio.to_thread`. The SDK is synchronous.

### Fork instead of a named snapshot

`boat fork <id>` (or `fork` in the API) also copies a sandbox. Use it to branch one episode mid-way, for example to try several actions from the same state.

* A fork copies the newest snapshot of its source. If the source keeps running, later forks get later states.
* A fork of a running sandbox uses its last background snapshot. Files written after it can be missing. Stop the source first for an exact copy.

For a fixed start state, use a named snapshot. See [Resume and fork](/snapshots/resume-and-fork).

## Limits

| Plan | Starts per minute / hour / day | Sandboxes at once |
| - | - | - |
| \$20 | 12 / 60 / 200 | 100 |
| \$100 | 30 / 210 / 840 | 300 |
| \$500 | 65 / 420 / 1,680 | 1,000 |
| \$2000 | 90 / 600 / 2,400 | 2,000 |
| [Free trial](/free-trial) | 5 / 25 / 75 | 2 |

* On the \$20 plan, the hour limit (60) sets the pace: 100 episodes take about 100 minutes to start. On the \$100 plan (210 per hour), they take about 29 minutes. The day limit caps a long run.
* A start above the limit fails with `429 rate_limited`, names the `window`, and does not count. The API sends no `Retry-After` header.
* An organization multiplies these numbers by its seats.
* `--fail-fast` (`fail_fast=True`) answers in about 1.5 seconds when no machine is free, instead of waiting.

All limits and how to raise them: [Limits](/limits).

## Cost

Boat bills each second that a sandbox runs. A deleted sandbox costs nothing.

| Run | Size | Cost |
| - | - | - |
| 100 episodes of 10 minutes | `small` (\$0.018 an hour) | about \$0.30 |
| 100 episodes of 10 minutes | `default` (\$0.036 an hour) | about \$0.60 |
| 1,000 episodes of 30 minutes | `default` | about \$18 |

Model calls are extra when an agent runs in the sandbox. See [Pricing](/pricing) and [AI credentials](/ai-credentials).

## Agents in the episode

To run Claude Code, Codex or another harness in each sandbox, send a prompt to the sandbox and wait for it. See [Integrated agents](/integrated-agents) and [Python SDK](/sdks/python#prompt-an-integrated-agent). To keep your model keys out of the sandboxes, use the [AI credentials proxy](/ai-credentials).

For episodes that other people or untrusted code control, create the sandboxes with `no_env=True`. They then get none of your secrets.

## Harbor and Terminal-Bench

* [Harbor](https://docs.harborframework.com) runs on Boat through the [Harbor adapter](/integrations/harbor) (`harbor run -e harbor_box_environment:BoxEnvironment`).
* **Terminal-Bench 2.0 does not run through that adapter today.** Each of its tasks names a prebuilt Docker image, and the adapter refuses tasks with a Docker image.
* To run a Terminal-Bench style task on Boat now, install its dependencies in a named snapshot (step 1) and run the task's test script with `boat exec`.

## Clean up

List what is still running, and delete what your run left:

```bash theme={null}
boat list --json | jq -r '.sandboxes[] | "\(.id) \(.name)"'
boat delete bx_... --yes
```


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.