Judge rounds start from scratch: let round N continue round N-1's session #282
Loading…
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
What happens
Each judge round starts a fresh session. Its brief lists what the last judge failed ("Check each again"), but it rebuilds everything else: how to serve and drive the app, its reference figures, the evidence for what passed. In run 134 round 1 took ~540 s of work in 95 turns. Round 2, after a fix that only changed how the script loads, took 903 s and 79 turns, re-checking everything from scratch.
Fix
agent.judge.rNforksagent.judge.r(N-1): a judge chain of its own, never the build session. Two guards:Done when
A judge round after a fix takes a fraction of round 1's time and still catches a regression in a criterion that passed before (a simulated run shows it).
From #279, run 134.
Verified on run 136 (llama.cpp, deployed
8c527d6). Judge round 1 failed 2 of 7 criteria. After the fix round, judge round 2 forked round 1's session (osf.session.forkedfromagent.judge #1, 35 messages inherited). It started on what had failed: the stale-results defect was fixed. It also found a new one, red errors on fields nobody had touched yet. It had used 18 turns when the run was stopped; vLLM's fresh round 2 in run 134 used 79.