Skip to content

Run

trialdesignbench.run

Generate a Harbor job.yaml, resolve auth, and invoke harbor run.

Harbor is driven through files only: this module writes task copies and a job config, runs the harbor CLI, and records a tdb-run.json manifest. It never imports Harbor. Per-agent settings (hosts, credentials, closed-book kwargs) come from trialdesignbench.agents.

Layout for --jobs-dir J --job-name N:

J/N/                job directory owned by Harbor
  job.yaml          generated config (`harbor run -c`)
  tdb-run.json      RunManifest
J/N.tasks/          task copies with the agent allowlist filled in

Task copies live beside the job directory because Harbor deletes any subdirectory of a job without a result.json when a job is resumed.

AgentRequest dataclass

effort = None class-attribute instance-attribute

Reasoning effort level; None or agents.DEFAULT_EFFORT keep the harness default.

RunError

Bases: RuntimeError

Raised before launch when a run cannot be configured safely.

agent_config(request, *, auth, skills, env)

Harbor AgentConfig dict plus its provenance record.

check_image(image, requests)

Problems with the local image (empty list when it matches the pins).

dump_job_yaml(config)

Serialize as JSON, which is valid YAML, so no YAML dependency is needed.

execute(plan)

Run harbor run -c job.yaml and record the exit code.

fill_allowed_hosts(task_toml, agent_hosts, environment_hosts)

Fill the two allowlist placeholders written by tdb build.

agent_hosts go to [agent] (during agent.run()), environment_hosts to the [environment] baseline (during agent setup).

materialize_tasks(tasks_dir, dest, agent_hosts, environment_hosts, task_ids)

Copy tasks beside the job dir and fill both allowlists.

parse_agent_pairs(agents, models, versions=(), efforts=())

Pair repeated --agent/--model with optional versions and efforts.

--effort is given once for every agent, or once per --agent.

plan_run(tasks_dir, requests, *, auth='api', n_attempts=1, n_concurrent=None, skills=(), job_name=None, jobs_dir=Path('jobs'), task_ids=None, canary=False, yes=False, env=None, dry_run=False)

Validate inputs, write task copies, job.yaml, and tdb-run.json.

read_build_manifest(tasks_dir)

Load tdb-build.json from a tasks directory.

regrade_command(job_dir, tasks_dir, *, job_name=None, jobs_dir=None, n_concurrent=None)

harbor job regrade command for a recorded job.

task_dirs(tasks_dir)

Harbor task directories under tasks_dir.

write_manifest(job_dir, manifest)

Write tdb-run.json.