Skip to content

Run

trialdesignbench.run

Generate a Harbor job.yaml, resolve auth, and invoke harbor run.

Harbor is driven through files only: this module writes task copies and a job config, runs the harbor CLI, and records a tdb-run.json manifest. It never imports Harbor. Per-agent settings (hosts, credentials, closed-book kwargs) come from trialdesignbench.agents.

Layout for --jobs-dir J --job-name N:

J/N/                job directory owned by Harbor
  job.yaml          generated config (`harbor run -c`)
  tdb-run.json      RunManifest
J/N.tasks/          task copies with the agent allowlist filled in
  harbor-plugin/    tdb_harbor_agents.py, the agent classes Harbor imports

Task copies live beside the job directory because Harbor deletes any subdirectory of a job without a result.json when a job is resumed.

Agents are launched through Harbor's import_path, not its agent name: job.yaml names a class in tdb_harbor_agents, a copy of trialdesignbench/harbor_agents.py (Harbor's adapters with a file-based instruction transport, see that module), and harbor runs with the copy's directory on PYTHONPATH. The copy keeps the job runnable from any Harbor install, and its digest is recorded in the manifest.

PLUGIN_DIR = 'harbor-plugin' module-attribute

Subdirectory of the task copies holding the Harbor plugin module.

AgentRequest dataclass

effort = None class-attribute instance-attribute

Reasoning effort level; None or agents.DEFAULT_EFFORT keep the harness default.

RunError

Bases: RuntimeError

Raised before launch when a run cannot be configured safely.

RunPlan dataclass

env = field(default_factory=dict) class-attribute instance-attribute

Environment overrides for command (the plugin's PYTHONPATH).

agent_config(request, *, auth, skills, env)

Harbor AgentConfig dict plus its provenance record.

check_image(image, requests)

Problems with the local image (empty list when it matches the pins).

dump_job_yaml(config)

Serialize as JSON, which is valid YAML, so no YAML dependency is needed.

execute(plan)

Run harbor run -c job.yaml and record the exit code.

fill_allowed_hosts(task_toml, agent_hosts, environment_hosts)

Fill the two allowlist placeholders written by tdb build.

agent_hosts go to [agent] (during agent.run()), environment_hosts to the [environment] baseline (during agent setup).

harbor_env(plugin, env)

Environment overrides for the harbor process: the plugin directory first on PYTHONPATH.

install_harbor_plugin(tasks_copy)

Copy trialdesignbench/harbor_agents.py beside the task copies.

The copy is the module job.yaml imports (agents.HARBOR_PLUGIN_MODULE); harbor_env puts its directory on PYTHONPATH, so the harbor process finds it whether or not Harbor shares a Python environment with tdb.

materialize_tasks(tasks_dir, dest, agent_hosts, environment_hosts, task_ids)

Copy tasks beside the job dir and fill both allowlists.

parse_agent_pairs(agents, models, versions=(), efforts=())

Pair repeated --agent/--model with optional versions and efforts.

--effort is given once for every agent, or once per --agent.

plan_run(tasks_dir, requests, *, auth='api', n_attempts=1, n_concurrent=None, skills=(), job_name=None, jobs_dir=Path('jobs'), task_ids=None, canary=False, yes=False, env=None, dry_run=False)

Validate inputs, write task copies, job.yaml, and tdb-run.json.

read_build_manifest(tasks_dir)

Load tdb-build.json from a tasks directory.

regrade_command(job_dir, tasks_dir, *, job_name=None, jobs_dir=None, n_concurrent=None)

harbor job regrade command for a recorded job.

task_dirs(tasks_dir)

Harbor task directories under tasks_dir.

write_manifest(job_dir, manifest)

Write tdb-run.json.