Skip to main content
harbor-box runs Harbor trials in boat sandboxes instead of local Docker containers. It is a Harbor environment adapter: the class BoxEnvironment in the module harbor_box_environment.
Terminal-Bench 2.0 does not run on boat through this adapter today. Every Terminal-Bench 2.0 task names a prebuilt Docker image (docker_image), and harbor-box 0.1.2 refuses tasks with a Docker image. Use the adapter for your own tasks that run on the standard boat sandbox. For other ways to run evals on boat, see Evals and RL environments.
The package keeps its old name, harbor-box. Version 0.1.2 reads the key from BOX_API_KEY and calls the old Box API. That API sends a Sunset date of 31 October 2026. See Migrating from Box.

Set it up

You need Python 3.12 or later and a boat account. For the trial, see Free trial.
  1. Create an API key on the API keys tab of the dashboard, or run boat api-key create harbor --preset ci.
  2. Install the adapter. It also installs Harbor (harbor>=0.14.0), httpx and tenacity:
  3. Give the key to the adapter. It reads BOX_API_KEY:

Run a Harbor job on boat

Pass the adapter as the environment of harbor run. -e takes a custom environment as module:Class:
At the end of a trial, harbor-box 0.1.2 fails to delete its sandbox (the delete call has no confirmation header). The sandbox keeps running until its auto-stop, 24 hours by default. After each job, run boat list and delete the trial sandboxes with boat delete <id> --yes.

Check your setup

This script creates one sandbox through the adapter, writes a file, runs a command and stops the sandbox. boat tested it with harbor-box 0.1.2 and Harbor 0.24.
hello_sandbox.py
The output shows hello from Boat, the uname line of the sandbox and exit: 0.

What the adapter maps

Configuration

BoxEnvironment(...) takes the standard Harbor arguments (environment_dir, environment_name, session_id, trial_paths, task_env_config) and these options:

What the adapter does with your task

boat does not build Docker images. Each trial starts on the standard boat sandbox (Ubuntu with common toolchains). See Base image and tools. To start many trials from one prepared state, prepare one sandbox, save a saved snapshot, and create each trial sandbox from it. See Evals and RL environments.

Isolation

The adapter creates no-env sandboxes by default. A trial gets none of your secrets, credentials or private repositories, and it cannot act on your account or on other sandboxes. To use your environment, pass no_env=False.

Not available

The adapter refuses these task settings with an error:
  • A prebuilt Docker image (docker_image), Docker image builds and Compose.
  • Network policies: no network, or allow-lists.
  • GPUs, TPUs and Windows.
The adapter writes files as the sandbox user, not as root. A task that writes outside the work directory (for example /solution or /tests) fails.

Resources

PyPI package

harbor-box on PyPI.

Harbor: run a job

The harbor run flags.

Evals and RL environments

Run many eval episodes on boat: saved snapshots, forks, limits and cost.

API keys

Create the key that the adapter needs.