Run 183: findings from the verifying run after the run 182 batch #439
Loading…
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Run 183 (
run_01M6Y92EB0QPHBWHJGJYR5B0V0) is the verifying run after the run 182 review: the motion revert (PR #435) and the batch of six (PR #437), oncmoriarty/scratch-vllm-437, the Scratch brief, thorough, judge atmedium, every other step atxhigh, auto-approve on. Baselines: run 174 (225 agent min), run 181 (362), run 182 (345, about 220 clean). Findings land here as comments while it runs; the local log isdocs/design/measurements/work/run-183/findings.md.Predictions, to be scored against run 182
visual.json)browserchecksbrowsercheckDepends on:window.__appWhat would say I was wrong:
visual.jsonstill over 25 entries or "verified" reasons let through (tighten the word lists); the judge's criteria count not falling with its list (its own added criteria); abrowsercheck at the 60-turn cap on page scripts (the next lever, not this batch); any review finding about the hook, the README or the test command.Also watched: the Ask tab (#424) if Ask is used; the wait cap's refusal, if a check hits it, and what the check does next; the review brief's size against the 200k alarm; #414's look list, if a fix round happens.
Planning: explore 0.5 min, propose 4.7 min (13 turns, 36k peak context), review running.
Specs:and noDepends on:line, and #430's fix never reached it. The thorough pipeline'sopenspec.proposeis thepropose-changebrief (build_propose_change), which carries the proposal rules and one sentence on tasks ("checkboxes, grouped under numbered headings"). TheSpecs:andDepends on:rules, and the example #430 added, live in_ARTIFACT_RULES["tasks"], which only the per-artifactopenspec-artifactbrief pastes, and this pipeline does not use it. Run 182's archivedtasks.mdhasDepends on:lines only because the revise step added them after the review's five findings, and noSpecs:line at all. So the prediction for #430 (0 findings) will fail for a reason the batch did not touch; the fix is to givebuild_propose_changethe tasks rules. Reopening #430.Combined review: 7.9 min, 7 turns, 57k peak context (run 182: 8.6 min). Four should-fix findings; revise running.
game-tablespec ask forwindow.__app(a requirement with two scenarios), and the scope lens raised nothing about it. Run 182's lens had written a note.Depends on:omission came back as one scope finding, "tasks.md declares no Depends on lines" (run 182: five), the consequence of finding 1; revise will add the lines as it did in 182.Scorecard so far: #426 ✓, #430 ✗ (finding 1, reopened).
Apply done: 93.3 min (run 182: 115), 573 turns, two compactions (182: one), 6 tool errors, 55 of 104 scenarios tagged (182: 44 of 77). No
browsersubagent; 135 page scripts, 104 tab calls, 41 navigates of its own across three tabs. Code review running.Both hand-over files are in a shape Braid does not read, so 49 scenarios are uncovered and nothing is left to the Visual check. The brief asks for
{"scenarios": [{"id": "<capability>/<scenario>", "why": "..."}]}. The apply wrotevisual.jsonas{"summary": ..., "visual": [{"spec", "requirement", "scenario", "reason"}]}(14 entries, all appearance: the five animations, reduced motion, tally marks, the trump card in view, …) anduntested.jsonas{"summary": ..., "untested": [{"spec", "requirement", "scenarios": [...], "reason"}]}(18 entries covering 49 scenarios: every lobby, table and end-screen scenario, each "verified in a real browser", under a design decision that "the React client has no automated test suite by design")._listed_scenarioslooks forscenariosand returns nothing for both, so the record says 49 uncovered, none visual, none untested; the code review's brief got no untested list to read; and the Visual check's brief will say to judge every scenario a person could see that no passing test covers, about 45 of them. Run 182 wrote the right shape.#431's check never ran, and would not have caught this. The 11 state-change scenarios of run 182 went to
visual.json; this run's 49 went tountested.json, with the same "verified in a browser" reasons, where no reason check exists, and the client was declared untestable by design. Two gaps: a hand-over file in the wrong shape should failscript.scenariosnaming the expected shape, not count for nothing; and the untested hatch needs the same rule as the visual one, that having looked is no reason and a scenario with a testable outcome keeps its test. Filing both.Prediction scorecard so far: #426 ✓; #430 ✗ (finding 1); the Visual check's list will be about 45, not 12 to 18, for the reason in finding 6.
Code review: 10.1 min, 25 turns, 147k peak context (run 182: 6 min, 19 turns, 117k), 1 note, 5 fixed. Then round 1's unit lane went red and a fix round started; the Visual check is skipped this round.
#427 holds. The review read 3 files in all (run 182: 24 files in 30 reads), none of them a pasted one. The brief was larger (147k peak at the review's end against 182's 117k) and the review took 4 minutes longer, with five fixes of its own against 182's three; the reads are gone, as predicted.
A test the apply wrote is red in the lane and was green for the apply: it depends on chance.
packages/server/test/game.test.ts, "a card of another suit is rejected when the player holds the led suit", drives two bots over a random deal for up to 30 s and asserts an illegal play was attempted along the way; whether the follower ever holds both the led suit and another card is the deal's. The round fails on it, the fix round gets it, and the Visual check waits for round 2. The weak-test lint (ban-check) has no rule for a test whose outcome depends on an unseeded random source; a candidate for it. (The engine takes an injectable RNG; the test did not inject one.)Note: the fix round's brief will list 49 uncovered scenarios (finding 6) and ask for tests; what it does with that is the next thing to read.
Fix round 1: 6.3 min, 10 turns; it made the deal-dependent test wait for the rejection itself instead of round 2, no assertion weakened. Round 2 green at 16:14. Visual check (as round 2, round 1 having been skipped) started 16:15; brief 59k chars, no motion probe or filmstrip, the wait cap and one-state sentences present, 55 scenarios marked covered, the other 49 to judge where visible.
browserchecks, 0 criteria, report unfinished. It symlinked the worktree'snode_modulesundertmp/scratch, wrotesolver.mjs,dryrun.mjsand two debug scripts that import the engine's modules, and has run and edited them 27 times ("10% success rate, too low; try multiple T values"): a planner that searches for a deal and a line of play that ends in a chosen outcome. This is run 62's pattern (braid#170), and the brief's rule, "a script does not stand in for looking: no bundling of the app's modules to run them", is not enforced by anything. The cause is upstream: the game-end scenarios (shared win, tie-break, sole winner) are state changes that should have unit tests and were left in the mis-shapeduntested.json(finding 6), so they reached the judge as "uncovered", and the judge is trying to stage a tie by hand. Not intervening; the time is the measurement. Candidate: the guard refuses abashin the judge's step that runs a script under the run's tmp importing from the worktree (as thebrowsersubagent is refused file tools), with the words to judge from what a person sees or mark the criterion unclear.Visual check at 52 minutes: still no look at the app, no criterion judged.
The solver consumed a whole context window before the first look. At 36 min the step hit the 200k alarm and compacted: 82 turns, 46 edits and 32 shell runs on five scripts under
tmp/scratch(solver.mjs,dryrun.mjs, three debug scripts), 0browserchecks, 0 criteria. After the compaction it set out to "reconstruct what actually happened": the summary named adriver.mjsthe disk does not have, and afindover/was refused by the guard.It then read Braid's own source to learn the report's schema. Through
$OSF_PYTHONit printedinspect.getsource(osf.pipeline.fixloop)in four slices andJUDGE_SCHEMAfromosf.pipeline.nodes, 18 reads and 23 shell calls since the compaction, though the brief gives it a command that checks the report (fixloop check-report) and says to use it rather than write a validator. The agent's interpreter importsosfbecause the run's Python is Braid's venv; nothing in the shell guard stops a read of Braid's package (braid#201 territory).Both are the judge's brief not binding where the guard could: a script under the run's tmp that imports the worktree's modules, and a read of Braid's own package, are refusable calls. Run 182's judge did neither; the difference is the 49 uncovered scenarios (finding 6), tie-break scenarios among them, that this judge is trying to stage by hand.
Visual check done: 93.4 min, 465 turns, two compactions, 59 criteria (53 pass, 4 fail, 2 unclear), verdict fail; fix round 2 next. Run 182's clean judge was 78 min for 34 criteria; run 181's 110 for 38.
It judged alone, and wrote its report at the end. 0
browserchecks in 93 minutes: 102 page scripts, 54 clicks, 35 screenshots, 14 waits of its own across three tabs, so the wait cap (which counts the subagent's sessions) touched nothing. The report had 0 criteria at 67 and 88 minutes and 59 at the close: "write the report as you go" was not followed, and the guard that refuses the first look until the file exists was satisfied by the empty file written at the start. Of the 59, none is aBcriterion; it wrotelook/…ids instead, and slugs (game-table/portrait-play) rather than the scenarios' names.It found two real defects, which is the point of the step. The Leave button does nothing in the waiting room and on the end screen: the client passes its ack callback in the payload slot of
room:leave, so the server never acks. And at 375 px the fanned hand is 474 px wide and the end cards are off screen. Both are the fix round's.The tie it spent its first hour staging never came.
game-end/shared-winisunclear: "four complete 3-player games were played, driven by a custom planner that targets an exact tie; all four ended" otherwise. A unit test of the scoring with a chosen tally would have settled it in a second; the scenario reached the judge through the mis-shapeduntested.json(finding 6). Sound is the otherunclear(headless).The revert held. No motion probe, no filmstrip, no clock in the brief or the transcript; the deal, flip, slide, sweep and scratch animations were judged from screenshots and the state hook, all pass.
Scorecard: criteria 59 (predicted about 20); clean judge minutes 93 (predicted 45 to 55);
browserchecks 0 (predicted 6 to 8); longest check and waits: no checks to measure. Every miss traces to finding 6, the hand-over files Braid could not read.Fix round 2: 52.3 min, 79 turns (run 181's fix round: 23 min). Round 3 green; Visual check round 3 started 18:39 in a fresh session.
game-table/portrait-play,look/waiting-b2,look/end-b2,brief/production-quality, and the two unclear ones), carry 53 by id, two files changed (store.ts,Table.tsx), do not go through a session again. That is the mechanism working on its first real round. But the brief is 65,929 characters, larger than the first look's 59,375: a 24k section "What the last judge found" pastes every one of the 53 passed criteria with its evidence, though the judge is told to carry them by id and that the evidence is in.osf/fix/r2/judge.json; 12k of delta specs; and about 20k of the first-look rulebook (Judge what you see, Plan the states, Probes, thebrowserbriefing) with one line saying this round's list holds over it. The instruction that matters starts 21,500 characters into its own section. Candidate: a later round's brief leads with the look list and the changed files, names the carried ids in one line, drops the passed criteria's evidence (it is on disk), and leaves out the first-look sections the round does not use. Whether the judge obeys the lists is the next thing to read.Visual check round 3 at 64 minutes: the look list is obeyed, and one criterion is eating the round.
#414 works on a real round. The judge took the six looks it was given and nothing else: four are written and pass (
game-table/portrait-play,look/waiting-b2,look/end-b2,game-table/sound-on-events), with fivebrowserchecks delegated this time and the report written as it goes. The two fixes are confirmed: Leave works in the waiting room and on the end screen, and the hand fits at 375 px.The last half hour is
game-end/shared-winalone. It wasunclearin round 2, so it is on the look list, and the judge is staging a tie again: adriver.mjsunder the run's tmp that plays whole games through a coordinator, on its fourth run ("Bob still hasn't played in this fresh room either… continuing (run 4)"), withsleep 120between polls. 222 turns, context at 198k, a compaction due. A scenario whose outcome is a tally comparison belongs to a unit test (finding 6 and #441); once it reaches the Visual check asunclear, nothing bounds what a round may spend on one criterion. Candidates: anunclearthat names what could not be staged goes to the fix round as "write the test" rather than back on the look list; and a per-criterion budget in a later round (one targeted look means one, thenunclearstands).The brief audit is #443.
Run 183 finished: succeeded, pull request merged and the change archived, 355 min wall, 354 agent minutes (run 182: 345, about 220 clean; run 181: 362; run 174: 225). No incident this time: no stall, no reboot, no deploy. Every minute is the pipeline's own.
Scorecard against the predictions
browserchecks, first lookbrowsercheckDepends on:window.__appWhat the run says. The changes that could be measured held: the review reads (#427), the hook (#426), the revert (#433), the look and carry lists (#414, first real round). The headline prediction failed on one upstream fact: the apply wrote both hand-over files in a shape Braid does not read (finding 6), so 49 scenarios, tie-breaks among them, reached the Visual check as uncovered, #431's check never ran, and the judge spent its first hour, and half of round 3, staging a tie that a unit test settles in a second. Two fix rounds and a second Visual check followed, where run 182 had none.
Filed from this run: #430 reopened (the tasks rules never reach the thorough pipeline's planner), #440 (a mis-shaped hand-over file fails the scenarios step), #441 (the untested hatch takes "verified in a browser" and a whole client as untestable by design), #443 (audit of every brief, with research).
Candidates not yet filed: the planner writes MAY in probabilistic scenarios (finding 4, also in run 182); a test on an unseeded random source passes the weak-test lint (9); the judge may run scripts that import the worktree's modules and read Braid's own package (10 to 12); "write the report as you go" is satisfied by an empty file (13); an
unclearthat could not be staged goes back on the look list with no bound on one criterion (19); the apply compacts twice (3). Fix round 2's 52 minutes for two small fixes is unexamined.Closed: every finding here is fixed or has its own ticket. braid#430, #440, #441 and #443 shipped on 2026-11-08 (PRs 446, 448); braid#434 holds the animation research.