Run 182: findings from the verifying run after the run 181 batch #425
Loading…
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Run 182 is the verifying run after the run 181 batch (PR #421, #423), the Scratch brief on
cmoriarty/scratch-vllm-421, thorough, judge atmedium(braid#420), every other step atxhigh. Baselines: run 174 (judge at medium, 225 agent min) and run 181 (judge at xhigh, 362 agent min). Findings land here as comments while it runs; the local log isdocs/design/measurements/work/run-182/findings.md.Scorecard for the batch (filled in as the steps land):
../screenshots: ?medium: ?So far (planning, 16 min)
window.__apphook as "not asked for by the brief". The brief's rule asks the proposal for it (#407); the lens wrote a note, the revise step left it, so the hook survived. A should-fix on the same ground would strip the judge's way in: the review brief should name the hook rule as Braid's.Depends on:lines (the propose brief could ask up front) and a random-first-dealer scenario that passed without randomness.openspec.apply[all]attempt 1 (about 70 min, 35+ turns) to deploy #423, and resumed at 03:04 on attempt 2. Apply time and attempts are not comparable with run 181's; the judge rounds and the review are.Apply, attempt 2 at 55 min (after the resume): 163 turns, context 202k with one compaction, 115k reasoning tokens. Attempt 1 had done 100 turns in 45 min before the stop.
/var/lib/openspecflow/runs/182/screenshots/<name>on the first try (landing, table-bidding, table-waiting-play); no read under the worktree, no search.browsersubagent was started; the primary made 111browser_evaluate, 24 screenshots, 24tabscalls and 15 navigates, and its last words are "Driver re-launched with the fixed parser. Waiting for the game to complete." The brief says one look at what changed, not a test campaign, and to delegate tobrowserby default (braid#383); run 172's lesson was that the primary's looks fill its own context. Candidate: the apply brief says that a whole session by script is the judge's, and that the apply's own look is a screenshot per changed state, throughbrowser.close), and one page script was cut at 180 s with "the script did not settle; check its loop ends" (braid#329) instead of holding the step.game.tsand the rooms tests; each cost a turn, none stalled it.Steps 19 of 45; apply still running.
Apply done: 115 agent minutes over the two attempts (45 before the stop, 70 after), 356 turns, one compaction at 202k with 16 tasks left. Run 181's apply took 233 min over two attempts (the steer ended its first). All 38 tasks ticked,
npm test62/62, typecheck clean, ban check clean on 15 test files, 28 scenarios left to the Visual check invisual.json, 5 untested with reasons. The agent's own in-browser verification covered the table, the scoreboard, the end screen, help, a 390×844 portrait game played by taps, rematch, mute, keyboard play and reduced motion: thorough, and the reason for finding 7 (the primary drove all of it itself, through whole games).Code review running next: #418's check (reads of pasted files, 181: 12 in 28 reads) comes with it.
Code review: 6 min, 19 turns, context 52k → 117k (181: 13.9 min, 36 turns, 165k). Three small server fixes (a path-containment check, a dead re-export, open-games listing rooms whose players all left); 15 server tests and the typecheck green after them.
#418 did not bind. The brief pasted 54 new files whole (146,583 characters) and named 25 more as too long to paste. The review still read 24 distinct files in 30 reads, every one of them a pasted file, and not one read was answered by the guard. Either the record under
<run>/tmp/pasted.jsonwas not written, or the guard's match (path, mtime, size) fails on production. Inspecting the run's folder next; a fix follows.The judge's brief is as the batch meant it (53,922 characters): the motion rule ("never write a freeze of your own") sits before the probes block, which sits before "Write the report as you go" and "How to end" (#416); the browser paragraph says it cannot read the source and asks for the hook's names (#419); the screenshots line names
../screenshots(#413). Its requests carrymedium(#420 holds).Correction to finding 11: #418 held. Of the review's 24 distinct reads, 0 were of the 29 files the brief pasted whole; all 24 were among the 25 files the brief listed as "not pasted: read it". The guard had nothing to answer (the record on disk holds the 29, with matching mtimes). What the finding really is:
_new_file_blockspastes ingit statusorder until 80,000 characters, so.dockerignore,Dockerfile,README.md,package.jsonand every client component were pasted whole, and the engine (game.ts427 lines,deck.ts74,rounds.ts13,types.ts118), the server (rooms.ts393,index.ts45,static.ts61) and all their tests were left to read,rounds.tsand the tsconfigs included though they are tiny. The review then read exactly those. Candidate: paste the source before configuration and documentation, always paste a file under a few hundred lines, and leave onlypackage-lock.json-sized files to a read. Filed as the next batch's issue.Judge round 1: attempt 1 lost to a stall the batch itself made possible; attempt 2 running (137 min on the step so far).
A page script that pauses the clock has no time limit. At 04:28 the judge wrote its own
browser_evaluate:clock.pause(); const ack = await …(a socket round trip). The clock's new gate (#415) holds that message while the clock is paused, so the promise never settled; and the 180 s bound on page scripts (braid#329) is asetTimeoutin the page, which now runs on Braid's clock, paused. The call sat 1,201 s until the watchdog ended the attempt as "silent for 1198 s while busy" and redid the step: 27 minutes and the judge's first look gone. Fix in hand: the clock exposes the machine's timer (realTimeout) and the bound uses it, so the script is cut at 180 s whatever the clock does. Proved by a journey; PR next.The first
browsercheck spent 60 turns waiting.Lobby check: landing, create, joinmade 26wait_forcalls (22 of them 30 s) and 28finds for a players list that never grew (its brief promised bots the judge had not started yet), took 2 screenshots, and ran to its 60-turn cap in 13.9 min; the continuation did the whole check in 1.6 min. No file reads at all (#419's change holds), and its requests carrymedium. So a check's length can be waits as well as reads: a candidate rule is a cap onwait_forper check, or a brief that forbids waiting for what the caller has not started.2273743and Portainer's last stack update was 03:04. Braid's restart recovery did what it should: attempt 2 (56 min of judging, fourbrowserchecks done) was adjudicated orphaned at 05:44 and attempt 3 continued it; the reboot cut attempt 3 at 05:54, and attempt 4 continued at 05:55 from Braid's note. Cost: about 15 minutes and the judge's working context twice. Not a Braid finding, but a run started on a Saturday night meets the Sunday update window; worth scheduling runs or the window around each other.Judge checks so far (attempt 2, all at
medium, none reading a file): lobby 13.9 min (the waits, finding 15) + continuation 1.6 min, table 6.2 min, scoreboard/help/mute 3.0 min.Attempts 2 to 4 of this round have cost 56 + 10 + 26 minutes; the round's four
browserchecks are done and in the report, so attempt 5 continues from them.Round 1 passed on the judge's first round: 34 criteria pass, the session ran end to end, no console errors, verdict pass. No fix round, so the loop closed at round 1 (run 181 needed two rounds and 156 judge minutes; run 174 two rounds and 50). The feature is committed and the summary step is running.
browserchecks), 5 (58 min, 201 turns) and 6 (26 min, 98 turns, the report). Tenbrowserchecks in all, none reading a file (#419's change holds); their lengths ran from 1.6 to 13.9 min, the long one all waits (finding 15).osf.engine.filmstripruns), the sweep and the scratch flash as held frames.getAnimations()and the clock, and ran the pasted text probe once. The rule says "never write a freeze of your own"; it did anyway, because its own freeze handles the server round trip the probe does not. The probe should take an action that awaits a reply, or the rule should say when a freeze of one's own is right.medium, and the round's criteria read sound.Run 182 finished: succeeded, its pull request merged and the change archived, 348 min wall, 345 agent minutes (run 181: 363 wall, 362 agent; run 174: 225 agent).
Without the two stalls and the reboot the run would have taken about 260 agent minutes with a passing first round; with them it matched run 181's wall and did more (no fix round needed).
Scorecard, final
New issues from this run: #424 (helper tab), #426 (review lens and the hook), #427 (paste budget), #429 (no awaiting the server under a pause). Fixed during the run: Ask's delivery (PR #423), the page-script bound on the paused clock (PR #428).
Finding 7 withdrawn. The apply's own in-browser verification (whole sessions by script, 111 page scripts, no
browsersubagent) was read as a cost. It is the likelier reason this run passed the judge's first round with no fix round, where runs 174 and 181 each needed a second round. The developer decides how to check its own work; no restriction follows. The one thing to carry: the apply brief's wording that says to delegate looks tobrowserby default is contradicted by this outcome and should be struck, not enforced. The 180 s bound on a page script that does not settle stays as the only guard.Correction to finding 18 and the scorecard: no filmstrip was ever composed. The judge's one run of
osf.engine.filmstrip(attempt 6, the deal's five frames) failed withCannot find package 'playwright': the helper resolves the Playwright CLI symlink (/opt/playwright/bin/playwright -> ../lib/node_modules/playwright/cli.js) before walking up to the package and so looks in the wrong folder; the script run by hand with the package folder set composes a strip fine. The judge read four frames one by one, as the fallback told it, and never tried the trump flip's strip. So #415 held only in that the clock held the server's state; the frames were the judge's owngetAnimations()freezes and the strip did not exist.Where the findings went. Finding 7 withdrawn (above). Findings 2, 11–13: #426, #427. Findings 14, 15, 17: #419's wait cap; #429 is moot under #433. Finding 3: #430. Finding 16: the reboot was handled as designed, nothing to do. Finding 18 and the strip bug: the motion capability is reverted whole in #433, and a method is researched with a person judging alongside in #434. The wall-time analysis (
docs/design/measurements/2026-11-08-runs-172-182-wall-time.md): the Visual check's list doubled because the apply left 11 state-change scenarios to it on the ground that it had looked at them itself, #431; the Visual check's browser-only framing, #432. #414 waits for a run with a fix round.Closed: every finding here was filed or fixed. #426, #427, #430 (reopened and fixed again in PR 446), #431, #432 shipped in PRs 435 to 438 and 446 to 449; #433 reverted the motion capability and #434 holds the research; #419 closed on its measurement; #414 held on run 183 (its round 3 looked at 6 and carried 53). Run 183's own findings were braid#439, now closed too.