Judge rounds start from scratch: let round N continue round N-1's session #282

Closed
opened 2026-10-28 16:49:47 -04:00 by cmoriarty · 1 comment
Owner

What happens

Each judge round starts a fresh session. Its brief lists what the last judge failed ("Check each again"), but it rebuilds everything else: how to serve and drive the app, its reference figures, the evidence for what passed. In run 134 round 1 took ~540 s of work in 95 turns. Round 2, after a fix that only changed how the script loads, took 903 s and 79 turns, re-checking everything from scratch.

Fix

agent.judge.rN forks agent.judge.r(N-1): a judge chain of its own, never the build session. Two guards:

  • Each round still opens a fresh browser context, with nothing from earlier rounds (#266).
  • The brief says the code has changed: re-check every failed criterion from a fresh page load, then sweep the ones that passed, judging from what is on screen now.

Done when

A judge round after a fix takes a fraction of round 1's time and still catches a regression in a criterion that passed before (a simulated run shows it).

From #279, run 134.

**What happens** Each judge round starts a fresh session. Its brief lists what the last judge failed ("Check each again"), but it rebuilds everything else: how to serve and drive the app, its reference figures, the evidence for what passed. In run 134 round 1 took ~540 s of work in 95 turns. Round 2, after a fix that only changed how the script loads, took 903 s and 79 turns, re-checking everything from scratch. **Fix** `agent.judge.rN` forks `agent.judge.r(N-1)`: a judge chain of its own, never the build session. Two guards: - Each round still opens a fresh browser context, with nothing from earlier rounds (#266). - The brief says the code has changed: re-check every failed criterion from a fresh page load, then sweep the ones that passed, judging from what is on screen now. **Done when** A judge round after a fix takes a fraction of round 1's time and still catches a regression in a criterion that passed before (a simulated run shows it). _From #279, run 134._
Author
Owner

Verified on run 136 (llama.cpp, deployed 8c527d6). Judge round 1 failed 2 of 7 criteria. After the fix round, judge round 2 forked round 1's session (osf.session.forked from agent.judge #1, 35 messages inherited). It started on what had failed: the stale-results defect was fixed. It also found a new one, red errors on fields nobody had touched yet. It had used 18 turns when the run was stopped; vLLM's fresh round 2 in run 134 used 79.

Verified on run 136 (llama.cpp, deployed 8c527d6). Judge round 1 failed 2 of 7 criteria. After the fix round, judge round 2 forked round 1's session (`osf.session.forked` from `agent.judge #1`, 35 messages inherited). It started on what had failed: the stale-results defect was fixed. It also found a new one, red errors on fields nobody had touched yet. It had used 18 turns when the run was stopped; vLLM's fresh round 2 in run 134 used 79.
Sign in to join this conversation.
No labels
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
cmoriarty/braid#282
No description provided.