Skip to content

Judge

trialdesignbench.judge

Rubric judges.

A judge maps (question, criteria, agent artifacts) to one verdict per criterion. Judges never see other questions' rubrics. The API judges share one prompt, response contract, voting, retry, and logging scheme (ApiJudge) and differ only in the provider API they call. FakeJudge gives deterministic verdicts for tests and dry runs.

Naming rule

One rule names everything about a judge: the model provider it calls, by the provider's identifier in trialdesignbench.providers.PROVIDERS (the same <provider>/ prefix tdb run --model uses). For a provider p:

  • the judge model is p/<model>, for example anthropic/claude-opus-5-5 (tdb build --judge-model, TDB_JUDGE_MODEL, tdb grade --judge-model);
  • the backend name is p (tdb grade --judge, JudgeInfo.name);
  • the class is judge_class_name(p): p in PascalCase plus Judge, each hyphen-separated segment capitalized (AnthropicJudge, OpenaiJudge, XaiJudge, OpencodeGoJudge);
  • the provider's official SDK, when the judge uses one, is the optional extra judge-p (trialdesignbench[judge-anthropic]); judge installs them all;
  • the API key variable and the egress host are the provider's.

The judge model therefore selects the backend (judge_backend), and tdb build records it per task so the verifier receives exactly the API key and egress its judge needs. The tests enforce the rule for every judge in API_JUDGES; a new judge is an ApiJudge subclass named by the rule, with its JudgeBackend, added to that tuple.

SDKs are imported lazily, so the core package and conversion-only workflows never need them; the OpenCode Go judge uses the standard library.

API_JUDGES = (AnthropicJudge, OpenaiJudge, XaiJudge, OpencodeGoJudge) module-attribute

Judges that call a model API, in PROVIDERS order. A new backend is an ApiJudge subclass named by the naming rule, with its own JudgeBackend, added here.

GoProtocol = Literal['chat', 'responses'] module-attribute

OpenAI protocols of the OpenCode Go gateway: Chat Completions or Responses.

JUDGE_BACKENDS = {cls.backend.name: cls.backend for cls in API_JUDGES} module-attribute

Backends by name; judge_backend selects one from a judge model string.

AnthropicJudge

Bases: ApiJudge

Judge backed by the Anthropic Messages API with structured JSON output.

Needs the anthropic package (trialdesignbench[judge-anthropic]). Judge models are anthropic/<id>. Sends temperature=0 except for model families using API sampling defaults (supports_temperature), including Haiku 5.5, which rejects temperature=0.

model_id property

The judge model without its provider prefix, as the API expects.

sdk_version()

Installed version of the backend's SDK, if the judge uses one.

ApiJudge

Bases: ABC

Shared behavior of judges that call a model API.

All criteria of one question are batched into a single call. With votes > 1, the call is repeated and verdicts are majority-voted. Transient failures (_Retryable) are retried with exponential backoff; a call that still fails, or any other error, marks every criterion of the question error. Every attempt is recorded in the question's judge log.

Subclasses set backend, build the API request in _request, and make one call in _call_once.

model_id property

The judge model without its provider prefix, as the API expects.

sdk_version()

Installed version of the backend's SDK, if the judge uses one.

FakeJudge

Deterministic judge for tests.

verdicts maps criterion ids to verdicts; rule computes a verdict from (criterion, artifacts) for anything not in the mapping; otherwise default is used.

JudgeArtifacts dataclass

What the judge may see for one question.

JudgeBackend dataclass

A model provider the rubric judge can call, with the judge's defaults.

tdb build writes key_env as "${key_env}" into [verifier.env] and api_host into the verifier allowlist; tdb run checks the key exists.

api_host property

Hostname the judge reaches (exact, for the verifier allowlist).

default_model instance-attribute

<provider>/<model> used when neither --judge-model nor TDB_JUDGE_MODEL is set.

extra property

Optional extra installing the SDK, None when none is needed.

key_env property

Host variable holding the API key.

name property

Backend name: JudgeInfo.name, and a tdb grade --judge value.

provider instance-attribute

The provider that names the judge and supplies its key and host.

sdk = None class-attribute instance-attribute

Distribution name of the provider's official SDK the judge uses, installed by the extra judge-<provider>; None for the standard library.

model_id(model)

model without the <provider>/ prefix, as the API expects.

OpenaiJudge

Bases: ApiJudge

Judge backed by the OpenAI Responses API with structured JSON output.

Needs the openai package (trialdesignbench[judge-openai]). Judge models are openai/<id>. The request asks for strict JSON schema output and store=false, so OpenAI keeps no copy of the exchange; no sampling parameters are sent, because reasoning models reject them and the others then use the API defaults. max_output_tokens bounds reasoning and answer tokens together.

model_id property

The judge model without its provider prefix, as the API expects.

sdk_version()

Installed version of the backend's SDK, if the judge uses one.

OpencodeGoJudge

Bases: ApiJudge

Judge backed by the OpenCode Go gateway, an OpenAI-compatible API.

Same prompt, response contract, voting, and retries as the other judges, with the standard library only. Judge models are opencode-go/<id> for the ids in the gateway's model list.

The gateway serves each model over exactly one protocol: Chat Completions for most, the Responses API for the GPT, Grok, and Muse Spark families. A request in the other protocol fails with ModelProtocolUnsupported, so the judge starts with Chat Completions and switches once per instance. Every request carries a stable x-opencode-session header, which the gateway requires, and a trialdesignbench/<version> user agent, because generic HTTP-library user agents are blocked.

Some Go models train on request data (for example muse-spark-1.3-contributor), and the gateway refuses them with HTTP 400 unless a workspace privacy setting allows such endpoints. A judge sends the hidden rubrics, so that setting must stay off; the default opencode-go/grok-4.7 is served without it, and a refused model fails every criterion of the question loudly instead of leaking rubrics.

model_id property

The judge model without its provider prefix, as the API expects.

sdk_version()

Installed version of the backend's SDK, if the judge uses one.

XaiJudge

Bases: ApiJudge

Judge backed by the xAI API through the official xai-sdk.

Needs the xai-sdk package (trialdesignbench[judge-xai]). Judge models are xai/<id>. The SDK speaks gRPC to api.x.ai on port 443, the provider's API host, so the verifier allowlist is the same as for its REST API. Structured output is requested with the result JSON schema as the response format; no sampling parameters are sent. The SDK's own transport retries are off so that every attempt is recorded here.

model_id property

The judge model without its provider prefix, as the API expects.

sdk_version()

Installed version of the backend's SDK, if the judge uses one.

build_user_message(question, criteria, artifacts)

Judge input for one question. Never includes other questions.

error_results(question, criteria, message)

Mark every criterion of a question as error.

judge_backend(model)

The backend a <provider>/<model> judge model string selects.

A string without a provider, or with a provider that has no judge, is a ValueError, so a typo never sends the model to the wrong API.

judge_class_name(provider)

The class name the naming rule gives the judge of a provider.

Each hyphen-separated segment of the provider name is capitalized: opencode-go names OpencodeGoJudge.

judge_prompt_sha256()

Hash identifying the judge prompt and response contract.

make_judge(model=None, *, backend=None, votes=1)

The API judge for a model string.

model defaults to TDB_JUDGE_MODEL, then to the backend's default. backend names the judge explicitly (tdb grade --judge); the model must then belong to it. Unknown backends and mismatches are a ValueError.

supports_temperature(model)

Whether the Anthropic judge should send an explicit temperature.

Accepts the model id with or without its anthropic/ prefix.