Canary
trialdesignbench.canary
Network canary: a Harbor task that proves the egress policy is enforced.
The canary uses the same two-phase policy as benchmark tasks: an
[environment] baseline for agent setup (model API hosts plus any install
hosts) and an [agent] allowlist for agent.run() (model API hosts only).
- The setup probe runs as the environment healthcheck, inside the agent container after Harbor has applied the baseline. It records whether blocked URLs are unreachable and whether each model API host connects.
- With
agent_probe=True(Harbor'soracleagent,tdb env check --canary),solution/solve.shrepeats the probe during the agent phase and also requires the install hosts to be blocked there.
The verifier turns the probe results into reward 1 (policy holds) or 0.
tdb report refuses to report a job whose canary did not score 1.
canary_result(trial_dir)
Read a canary trial's verifier output: (passed, reasons).
write_canary_task(dest, *, image, agent_hosts, setup_hosts=(), agent_probe=False)
Write the canary task into dest/network-canary and return its path.
agent_hosts are the model API hosts allowed in both phases;
setup_hosts are added to the setup baseline only. agent_probe makes
solution/solve.sh probe the agent phase, which only Harbor's oracle
agent runs; the verifier then requires that result too.