Skip to content

Agents

trialdesignbench.agents

Agents tdb run can launch closed book, and why others are refused.

This module is the one place to edit when adding, changing, or removing an agent. Each AgentProfile in AGENTS records everything tdb run needs:

  • the model providers the agent may call (the provider/ prefix of --model), which fix the API key variable and the API host for --auth api;
  • optional subscription login for --auth subscription;
  • the pinned CLI version (an ImagePins field) and whether the CLI is preinstalled in the image or installed by Harbor during agent setup. In the latter case setup_hosts are reachable during setup only;
  • the Harbor kwargs and env vars that disable web tools and nonessential traffic, and the disabled tool names recorded for provenance;
  • how --effort reaches the agent (Effort): the Harbor kwarg that carries the reasoning effort level and the levels it accepts.

Harbor agents that are not in AGENTS are refused; REFUSED_AGENTS explains why for the notable ones. To add an agent, check its Harbor adapter (for example harbor agent schema <name>) for how it installs, authenticates, and exposes web tools, pin its version in ImagePins, add a profile here, and confirm the hosts with tdb env check --canary --agent <name> and a smoke run before marking them verified.

DEFAULT_EFFORT = 'default' module-attribute

--effort value that leaves the harness default in place (no kwarg).

EFFORT_LEVELS = ('none', 'minimal', 'low', 'medium', 'high', 'xhigh', 'max') module-attribute

Canonical reasoning effort levels, lowest to highest. Every agent accepts a subset (Effort.levels), and every model a subset of that.

AgentError

Bases: ValueError

Raised when an agent, model, or credential cannot be used closed book.

AgentProfile dataclass

How to run one Harbor agent closed book.

effort instance-attribute

How --effort reaches the agent.

image_label property

Image label recording the pinned version (see the Dockerfile).

preinstalled instance-attribute

True when the image ships the CLI and Harbor skips its install.

setup_hosts = () class-attribute instance-attribute

Hosts Harbor's install step needs; reachable during agent setup only.

verified = False class-attribute instance-attribute

True when the API and setup hosts were confirmed by the canary and a smoke run.

version_pin instance-attribute

ImagePins field holding the pinned CLI version.

Effort dataclass

How --effort sets an agent's reasoning effort through Harbor.

how instance-attribute

What Harbor turns the kwarg into.

kwarg instance-attribute

Harbor kwarg carrying the level (see harbor agent schema <name>).

levels instance-attribute

Levels the kwarg accepts, lowest to highest. Harbor validates these for reasoning_effort kwargs; for free-string kwargs the check here is the only one before the trial runs.

note = '' class-attribute instance-attribute

Default behavior and what happens with a level the model lacks.

Provider dataclass

A model API used with an API key.

Subscription dataclass

Subscription (OAuth) login supported by an agent's Harbor adapter.

default_file = None class-attribute instance-attribute

Credential file used when file_env is unset (~ is the home dir).

env instance-attribute

Agent env flags that make Harbor use the login instead of an API key.

file_env = None class-attribute instance-attribute

Host variable naming a credential file.

token_env = None class-attribute instance-attribute

Host variable holding a login token.

api_hosts(profile, auth, provider)

Model API hosts the agent reaches during agent.run().

effort_from_kwargs(agent, kwargs)

The reasoning effort level in a Harbor agent config, if one was set.

Used by tdb report to read the level back from each trial's result.json; None for the harness default or an unknown agent.

effort_kwargs(profile, level)

Harbor kwargs for a reasoning effort level.

None or DEFAULT_EFFORT leave the harness default in place and return no kwargs. Any other level must be one the agent accepts; this check runs before launch because Claude Code and OpenCode silently ignore a level they do not know.

get_profile(name)

The profile for a Harbor agent name, or AgentError with the reason.

hosts_verified(profile, auth)

Whether the host list for this agent and auth mode was smoke tested.

model_provider(profile, model)

The provider prefix of model, which must be one the agent may call.

resolve_auth(profile, auth, provider, env)

Check the host credentials and return the agent env flags for them.

Harbor's adapters read API keys and tokens from the host environment themselves; only the names are checked here, and values are never copied into job.yaml.