Visual check rounds take as long as the first: later rounds are told to judge everything afresh and are not told what changed #379

Closed
opened 2026-11-04 15:40:18 -05:00 by cmoriarty · 1 comment
Owner

What happens
On run 172 (Scratch, thorough) each Visual check round took as long as the first, or longer, while the faults it was checking shrank:

Round Wall time Faults found Browser subagents Tool calls Context alarm
1 55 min 6 of 20 criteria 6 657 no
2 74 min 4 of 20 7 810 yes, respawned (fork)
3 75+ min 1 9 923 yes

The rounds are not fresh starts: round 2 forked round 1's session (45 messages inherited) and round 3 forked round 2's. But the judge behaves as if they were. Round 2 left one fault (the portrait reaction bar over the bid buttons), and round 3's subagents were "Full game, sounds, end, rematch", "Catch deal + trump flip animation", "Finish game, verify rematch restart", "Scoreboard + landing/waiting recheck" and five more.

Why

  1. The brief contradicts itself. _fresh_look_lines (src/osf/engine/prompts.py) says: "Check each criterion that failed again, from a fresh page load, then go over the ones that passed, quickly." Three paragraphs earlier _recheck_lines says: "Check each again under the same id, and judge everything else afresh too: a fix can break what worked." The judge follows the stronger one.
  2. The brief is 64k characters every round: 35k of delta specs for all four capabilities plus the original brief, and a list of 78 lane-covered scenarios. That filled the forked session's context in rounds 2 and 3 (osf.budget.alarm at 13:16 and again in round 3), and a respawn throws the inherited memory away, so the fork buys nothing.
  3. The judge is not told what changed. fixloop.same_code compares a digest to say whether the code changed since the last judge, but the brief never names the files, so the judge cannot tell which passed criteria a fix could have touched.
  4. Whole-screen criteria ("the table screen looks right during play: layout, readability, turn cues, no visual faults", B5-table) can only be rechecked by playing a game, so one such criterion forces a full playthrough every round.

Fix (proposal), for rounds 2 and later

  • Name the files the fix round changed since the last judge looked (the loop has the digest; give it the list, from git diff --stat between the two verify records).
  • Recheck a failed criterion from a fresh page load. Recheck a passed one only when a changed file could affect it, with one targeted look, not a full playthrough; carry every other passed criterion forward from the last report, marked carried: true with the round it was seen in.
  • Replay a full game only when the fix touched game flow (the files the judge is told about), at most once a round.
  • Drop "judge everything else afresh too" from _recheck_lines and keep the targeted wording in one place.
  • Send a later round only the delta specs of capabilities with open or affected criteria, so a continued session fits its context without a respawn.
  • Record each round's wall time and subagent count in the judge record, so a run can show the rounds getting shorter.

Done when
A round that follows one with a single fault rechecks that fault and the criteria its fix could touch, carries the rest forward, and finishes in a fraction of the first round's time, with the first round unchanged.

Observed on run 172 (run_01M6K8CA22F0FE8TNG6K6J6PG3), Visual check rounds 1-3, 2026-11-04; brief read at origin/main d53f9e9d. Related: braid#266, braid#282 (fresh browser per round), braid#165 (no-change claims), braid#378 (fork alarm wording).

**What happens** On run 172 (Scratch, thorough) each Visual check round took as long as the first, or longer, while the faults it was checking shrank: | Round | Wall time | Faults found | Browser subagents | Tool calls | Context alarm | |---|---|---|---|---|---| | 1 | 55 min | 6 of 20 criteria | 6 | 657 | no | | 2 | 74 min | 4 of 20 | 7 | 810 | yes, respawned (fork) | | 3 | 75+ min | 1 | 9 | 923 | yes | The rounds are not fresh starts: round 2 forked round 1's session (45 messages inherited) and round 3 forked round 2's. But the judge behaves as if they were. Round 2 left one fault (the portrait reaction bar over the bid buttons), and round 3's subagents were "Full game, sounds, end, rematch", "Catch deal + trump flip animation", "Finish game, verify rematch restart", "Scoreboard + landing/waiting recheck" and five more. **Why** 1. The brief contradicts itself. `_fresh_look_lines` (`src/osf/engine/prompts.py`) says: "Check each criterion that failed again, from a fresh page load, then go over the ones that passed, quickly." Three paragraphs earlier `_recheck_lines` says: "Check each again under the same id, and judge everything else afresh too: a fix can break what worked." The judge follows the stronger one. 2. The brief is 64k characters every round: 35k of delta specs for all four capabilities plus the original brief, and a list of 78 lane-covered scenarios. That filled the forked session's context in rounds 2 and 3 (`osf.budget.alarm` at 13:16 and again in round 3), and a respawn throws the inherited memory away, so the fork buys nothing. 3. The judge is not told what changed. `fixloop.same_code` compares a digest to say *whether* the code changed since the last judge, but the brief never names the files, so the judge cannot tell which passed criteria a fix could have touched. 4. Whole-screen criteria ("the table screen looks right during play: layout, readability, turn cues, no visual faults", `B5-table`) can only be rechecked by playing a game, so one such criterion forces a full playthrough every round. **Fix (proposal), for rounds 2 and later** - Name the files the fix round changed since the last judge looked (the loop has the digest; give it the list, from `git diff --stat` between the two verify records). - Recheck a failed criterion from a fresh page load. Recheck a passed one only when a changed file could affect it, with one targeted look, not a full playthrough; carry every other passed criterion forward from the last report, marked `carried: true` with the round it was seen in. - Replay a full game only when the fix touched game flow (the files the judge is told about), at most once a round. - Drop "judge everything else afresh too" from `_recheck_lines` and keep the targeted wording in one place. - Send a later round only the delta specs of capabilities with open or affected criteria, so a continued session fits its context without a respawn. - Record each round's wall time and subagent count in the judge record, so a run can show the rounds getting shorter. **Done when** A round that follows one with a single fault rechecks that fault and the criteria its fix could touch, carries the rest forward, and finishes in a fraction of the first round's time, with the first round unchanged. _Observed on run 172 (`run_01M6K8CA22F0FE8TNG6K6J6PG3`), Visual check rounds 1-3, 2026-11-04; brief read at origin/main d53f9e9d. Related: braid#266, braid#282 (fresh browser per round), braid#165 (no-change claims), braid#378 (fork alarm wording)._
Author
Owner

Shipped in PR #390, production ddd4b6b.

  • A later round is told which files changed and what passed. It rechecks what a changed file could affect, and carries the rest.
  • A carry no earlier round passed fails the round, and fixloop check-report catches it while the judge can still put it right.
  • The brief's bullet on playing the whole game is left out once what changed is known.

Evidence: tests/pipeline/test_later_rounds.py and test_check_report.py. Seven Free replays of run 172's round 3 took 14.5 to 35.3 minutes, against 75 or more. They ran on other models, so not like for like.

  • Four passed. The last, on the final brief: 19 of 19 in 20.1 minutes and 56 tool calls.
  • Three failed on earlier briefs. The faults they showed are fixed, except one that failed its step's check for a reason the replay did not print. It prints it now, and that cause was never seen.

Not proved: whether carrying passed criteria ever hides a real regression. Watch that on the next full run.

**Shipped** in PR #390, production ddd4b6b. - A later round is told which files changed and what passed. It rechecks what a changed file could affect, and carries the rest. - A carry no earlier round passed fails the round, and `fixloop check-report` catches it while the judge can still put it right. - The brief's bullet on playing the whole game is left out once what changed is known. **Evidence:** `tests/pipeline/test_later_rounds.py` and `test_check_report.py`. Seven Free replays of run 172's round 3 took 14.5 to 35.3 minutes, against 75 or more. They ran on other models, so not like for like. - Four passed. The last, on the final brief: 19 of 19 in 20.1 minutes and 56 tool calls. - Three failed on earlier briefs. The faults they showed are fixed, except one that failed its step's check for a reason the replay did not print. It prints it now, and that cause was never seen. **Not proved:** whether carrying passed criteria ever hides a real regression. Watch that on the next full run.
Sign in to join this conversation.
No labels
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
cmoriarty/braid#379
No description provided.