Skip to content

Agents

trialdesignbench.agents

Agents tdb run can launch closed book, and why others are refused.

This module is the one place to edit when adding, changing, or removing an agent. Each AgentProfile in AGENTS records everything tdb run needs:

  • the model providers the agent may call (the provider/ prefix of --model, from trialdesignbench.providers.PROVIDERS), which fix the API key variable and the API host for --auth api;
  • optional subscription login for --auth subscription;
  • the pinned CLI version (an ImagePins field) and whether the CLI is preinstalled in the image or installed by Harbor during agent setup. In the latter case setup_hosts are reachable during setup only;
  • the Harbor kwargs and env vars that disable web tools and nonessential traffic, and the disabled tool names recorded for provenance;
  • how --effort reaches the agent (Effort): the Harbor kwarg that carries the reasoning effort level and the levels it accepts;
  • the class in trialdesignbench.harbor_agents that tdb run launches through Harbor's import_path (harbor_class): Harbor's adapter for the agent with the file-based instruction transport, see that module.

Harbor agents that are not in AGENTS are refused; REFUSED_AGENTS explains why for the notable ones. To add an agent, check its Harbor adapter (for example harbor agent schema <name>) for how it installs, authenticates, and exposes web tools, pin its version in ImagePins, add a profile here, and confirm the hosts with tdb env check --canary --agent <name> and a smoke run before marking them verified.

DEFAULT_EFFORT = 'default' module-attribute

--effort value that leaves the harness default in place (no kwarg).

EFFORT_LEVELS = ('none', 'minimal', 'low', 'medium', 'high', 'xhigh', 'max') module-attribute

Canonical reasoning effort levels, lowest to highest. Every agent accepts a subset (Effort.levels), and every model a subset of that.

HARBOR_PLUGIN_MODULE = 'tdb_harbor_agents' module-attribute

Module name under which tdb run exposes trialdesignbench/harbor_agents.py to the harbor process (a copy beside the task copies, on PYTHONPATH).

AgentError

Bases: ValueError

Raised when an agent, model, or credential cannot be used closed book.

AgentProfile dataclass

How to run one Harbor agent closed book.

effort instance-attribute

How --effort reaches the agent.

harbor_class instance-attribute

Class in trialdesignbench.harbor_agents launched through Harbor's import_path: Harbor's adapter for name with the file-based instruction transport.

image_label property

Image label recording the pinned version (see the Dockerfile).

preinstalled instance-attribute

True when the image ships the CLI and Harbor skips its install.

setup_hosts = () class-attribute instance-attribute

Hosts Harbor's install step needs; reachable during agent setup only.

verified = False class-attribute instance-attribute

True when the API and setup hosts were confirmed by the canary and a smoke run.

version_pin instance-attribute

ImagePins field holding the pinned CLI version.

Effort dataclass

How --effort sets an agent's reasoning effort through Harbor.

how instance-attribute

What Harbor turns the kwarg into.

kwarg instance-attribute

Harbor kwarg carrying the level (see harbor agent schema <name>).

levels instance-attribute

Levels the kwarg accepts, lowest to highest. Harbor validates these for reasoning_effort kwargs; for free-string kwargs the check here is the only one before the trial runs.

note = '' class-attribute instance-attribute

Default behavior and what happens with a level the model lacks.

Subscription dataclass

Subscription (OAuth) login supported by an agent's Harbor adapter.

default_file = None class-attribute instance-attribute

Credential file used when file_env is unset (~ is the home dir).

env instance-attribute

Agent env flags that make Harbor use the login instead of an API key.

file_env = None class-attribute instance-attribute

Host variable naming a credential file.

token_env = None class-attribute instance-attribute

Host variable holding a login token.

api_hosts(profile, auth, provider)

Model API hosts the agent reaches during agent.run().

effort_from_kwargs(agent, kwargs)

The reasoning effort level in a Harbor agent config, if one was set.

Used by tdb report to read the level back from each trial's result.json; None for the harness default or an unknown agent.

effort_kwargs(profile, level)

Harbor kwargs for a reasoning effort level.

None or DEFAULT_EFFORT leave the harness default in place and return no kwargs. Any other level must be one the agent accepts; this check runs before launch because Claude Code and OpenCode silently ignore a level they do not know.

get_profile(name)

The profile for a Harbor agent name, or AgentError with the reason.

hosts_verified(profile, auth)

Whether the host list for this agent and auth mode was smoke tested.

import_path(profile)

Harbor import_path of the agent class tdb run launches.

tdb run writes this instead of Harbor's agent name: Harbor prefers a valid name over import_path, which would skip the plugin.

model_provider(profile, model)

The provider prefix of model, which must be one the agent may call.

resolve_auth(profile, auth, provider, env)

Check the host credentials and return the agent env entries for them.

Harbor's adapters read API keys and tokens from the host environment themselves; only the names are checked here, and values are never copied into job.yaml. A provider Harbor cannot map (Provider.harbor_credential) gets its key as a ${NAME} template that Harbor resolves at launch.