Run 183: findings from the verifying run after the run 182 batch #439

Closed
opened 2026-11-08 14:10:17 -05:00 by cmoriarty · 11 comments
Owner

Run 183 (run_01M6Y92EB0QPHBWHJGJYR5B0V0) is the verifying run after the run 182 review: the motion revert (PR #435) and the batch of six (PR #437), on cmoriarty/scratch-vllm-437, the Scratch brief, thorough, judge at medium, every other step at xhigh, auto-approve on. Baselines: run 174 (225 agent min), run 181 (362), run 182 (345, about 220 clean). Findings land here as comments while it runs; the local log is docs/design/measurements/work/run-183/findings.md.

Predictions, to be scored against run 182

what run 182 expected change
scenarios left to the Visual check (visual.json) 28 (11 state changes) 12 to 18, none reasoned "verified in task N" #431
Visual check criteria, round 1 34 about 20 #431
browser checks 10, three at the 60-turn cap 6 to 8, none at the cap for waiting #419 #431
longest browser check 13.9 min, 26 waits under 5 min, no check over 6 waits #419
Visual check round 1, clean about 78 min 45 to 55 min #431 #419
code review reads of new files 24 files in 30 reads under 5 #427
code review brief 147k chars about 230k chars #427
should-fix findings about Depends on: 5 0 #430
finding or note about window.__app 1 0 #426
judge stalls on a paused clock 2 (50 min) 0 #433
apply 115 min about the same; more tests, still looks itself #431
clean agent minutes about 220 170 to 190

What would say I was wrong: visual.json still over 25 entries or "verified" reasons let through (tighten the word lists); the judge's criteria count not falling with its list (its own added criteria); a browser check at the 60-turn cap on page scripts (the next lever, not this batch); any review finding about the hook, the README or the test command.

Also watched: the Ask tab (#424) if Ask is used; the wait cap's refusal, if a check hits it, and what the check does next; the review brief's size against the 200k alarm; #414's look list, if a fix round happens.

Run 183 (`run_01M6Y92EB0QPHBWHJGJYR5B0V0`) is the verifying run after the run 182 review: the motion revert (PR #435) and the batch of six (PR #437), on `cmoriarty/scratch-vllm-437`, the Scratch brief, thorough, judge at `medium`, every other step at `xhigh`, auto-approve on. Baselines: run 174 (225 agent min), run 181 (362), run 182 (345, about 220 clean). Findings land here as comments while it runs; the local log is `docs/design/measurements/work/run-183/findings.md`. **Predictions, to be scored against run 182** | what | run 182 | expected | change | |---|---:|---:|---| | scenarios left to the Visual check (`visual.json`) | 28 (11 state changes) | 12 to 18, none reasoned "verified in task N" | #431 | | Visual check criteria, round 1 | 34 | about 20 | #431 | | `browser` checks | 10, three at the 60-turn cap | 6 to 8, none at the cap for waiting | #419 #431 | | longest `browser` check | 13.9 min, 26 waits | under 5 min, no check over 6 waits | #419 | | Visual check round 1, clean | about 78 min | 45 to 55 min | #431 #419 | | code review reads of new files | 24 files in 30 reads | under 5 | #427 | | code review brief | 147k chars | about 230k chars | #427 | | should-fix findings about `Depends on:` | 5 | 0 | #430 | | finding or note about `window.__app` | 1 | 0 | #426 | | judge stalls on a paused clock | 2 (50 min) | 0 | #433 | | apply | 115 min | about the same; more tests, still looks itself | #431 | | clean agent minutes | about 220 | 170 to 190 | | **What would say I was wrong:** `visual.json` still over 25 entries or "verified" reasons let through (tighten the word lists); the judge's criteria count not falling with its list (its own added criteria); a `browser` check at the 60-turn cap on page scripts (the next lever, not this batch); any review finding about the hook, the README or the test command. **Also watched:** the Ask tab (#424) if Ask is used; the wait cap's refusal, if a check hits it, and what the check does next; the review brief's size against the 200k alarm; #414's look list, if a fix round happens.
Author
Owner

Planning: explore 0.5 min, propose 4.7 min (13 turns, 36k peak context), review running.

  1. The planner wrote 8 task groups with no Specs: and no Depends on: line, and #430's fix never reached it. The thorough pipeline's openspec.propose is the propose-change brief (build_propose_change), which carries the proposal rules and one sentence on tasks ("checkboxes, grouped under numbered headings"). The Specs: and Depends on: rules, and the example #430 added, live in _ARTIFACT_RULES["tasks"], which only the per-artifact openspec-artifact brief pastes, and this pipeline does not use it. Run 182's archived tasks.md has Depends on: lines only because the revise step added them after the review's five findings, and no Specs: line at all. So the prediction for #430 (0 findings) will fail for a reason the batch did not touch; the fix is to give build_propose_change the tasks rules. Reopening #430.
**Planning: explore 0.5 min, propose 4.7 min (13 turns, 36k peak context), review running.** 1. **The planner wrote 8 task groups with no `Specs:` and no `Depends on:` line, and #430's fix never reached it.** The thorough pipeline's `openspec.propose` is the `propose-change` brief (`build_propose_change`), which carries the proposal rules and one sentence on tasks ("checkboxes, grouped under numbered headings"). The `Specs:` and `Depends on:` rules, and the example #430 added, live in `_ARTIFACT_RULES["tasks"]`, which only the per-artifact `openspec-artifact` brief pastes, and this pipeline does not use it. Run 182's archived `tasks.md` has `Depends on:` lines only because the revise step added them after the review's five findings, and no `Specs:` line at all. So the prediction for #430 (0 findings) will fail for a reason the batch did not touch; the fix is to give `build_propose_change` the tasks rules. Reopening #430.
Author
Owner

Combined review: 7.9 min, 7 turns, 57k peak context (run 182: 8.6 min). Four should-fix findings; revise running.

  1. #426 holds. The proposal and the game-table spec ask for window.__app (a requirement with two scenarios), and the scope lens raised nothing about it. Run 182's lens had written a note.
  2. The Depends on: omission came back as one scope finding, "tasks.md declares no Depends on lines" (run 182: five), the consequence of finding 1; revise will add the lines as it did in 182.
  3. The random-dealer finding is a repeat. Spec lens: "Random-dealer scenario uses MAY, not checkable" ("the first dealer MAY differ between the two games"). Run 182's review found the same scenario written so it passed without randomness. Twice on the same brief says the planner reaches for MAY on anything probabilistic; a sentence in the proposal rules (a scenario states what SHALL be observable, and a random outcome is checked by a seed or by a run of many) would spare the revise pass. Candidate issue after the run.
  4. The two animations findings (the scenarios omit the trump flip and the card slide the requirement names; the test would pass with the flip omitted) are ordinary review work.

Scorecard so far: #426 ✓, #430 ✗ (finding 1, reopened).

**Combined review: 7.9 min, 7 turns, 57k peak context (run 182: 8.6 min). Four should-fix findings; revise running.** 2. **#426 holds.** The proposal and the `game-table` spec ask for `window.__app` (a requirement with two scenarios), and the scope lens raised nothing about it. Run 182's lens had written a note. 3. **The `Depends on:` omission came back as one scope finding**, "tasks.md declares no Depends on lines" (run 182: five), the consequence of finding 1; revise will add the lines as it did in 182. 4. **The random-dealer finding is a repeat.** Spec lens: "Random-dealer scenario uses MAY, not checkable" ("the first dealer MAY differ between the two games"). Run 182's review found the same scenario written so it passed without randomness. Twice on the same brief says the planner reaches for MAY on anything probabilistic; a sentence in the proposal rules (a scenario states what SHALL be observable, and a random outcome is checked by a seed or by a run of many) would spare the revise pass. Candidate issue after the run. 5. The two animations findings (the scenarios omit the trump flip and the card slide the requirement names; the test would pass with the flip omitted) are ordinary review work. Scorecard so far: #426 ✓, #430 ✗ (finding 1, reopened).
Author
Owner

Apply done: 93.3 min (run 182: 115), 573 turns, two compactions (182: one), 6 tool errors, 55 of 104 scenarios tagged (182: 44 of 77). No browser subagent; 135 page scripts, 104 tab calls, 41 navigates of its own across three tabs. Code review running.

  1. Both hand-over files are in a shape Braid does not read, so 49 scenarios are uncovered and nothing is left to the Visual check. The brief asks for {"scenarios": [{"id": "<capability>/<scenario>", "why": "..."}]}. The apply wrote visual.json as {"summary": ..., "visual": [{"spec", "requirement", "scenario", "reason"}]} (14 entries, all appearance: the five animations, reduced motion, tally marks, the trump card in view, …) and untested.json as {"summary": ..., "untested": [{"spec", "requirement", "scenarios": [...], "reason"}]} (18 entries covering 49 scenarios: every lobby, table and end-screen scenario, each "verified in a real browser", under a design decision that "the React client has no automated test suite by design"). _listed_scenarios looks for scenarios and returns nothing for both, so the record says 49 uncovered, none visual, none untested; the code review's brief got no untested list to read; and the Visual check's brief will say to judge every scenario a person could see that no passing test covers, about 45 of them. Run 182 wrote the right shape.

  2. #431's check never ran, and would not have caught this. The 11 state-change scenarios of run 182 went to visual.json; this run's 49 went to untested.json, with the same "verified in a browser" reasons, where no reason check exists, and the client was declared untestable by design. Two gaps: a hand-over file in the wrong shape should fail script.scenarios naming the expected shape, not count for nothing; and the untested hatch needs the same rule as the visual one, that having looked is no reason and a scenario with a testable outcome keeps its test. Filing both.

Prediction scorecard so far: #426 ✓; #430 ✗ (finding 1); the Visual check's list will be about 45, not 12 to 18, for the reason in finding 6.

**Apply done: 93.3 min (run 182: 115), 573 turns, two compactions (182: one), 6 tool errors, 55 of 104 scenarios tagged (182: 44 of 77). No `browser` subagent; 135 page scripts, 104 tab calls, 41 navigates of its own across three tabs. Code review running.** 6. **Both hand-over files are in a shape Braid does not read, so 49 scenarios are uncovered and nothing is left to the Visual check.** The brief asks for `{"scenarios": [{"id": "<capability>/<scenario>", "why": "..."}]}`. The apply wrote `visual.json` as `{"summary": ..., "visual": [{"spec", "requirement", "scenario", "reason"}]}` (14 entries, all appearance: the five animations, reduced motion, tally marks, the trump card in view, …) and `untested.json` as `{"summary": ..., "untested": [{"spec", "requirement", "scenarios": [...], "reason"}]}` (18 entries covering 49 scenarios: every lobby, table and end-screen scenario, each "verified in a real browser", under a design decision that "the React client has no automated test suite by design"). `_listed_scenarios` looks for `scenarios` and returns nothing for both, so the record says 49 uncovered, none visual, none untested; the code review's brief got no untested list to read; and the Visual check's brief will say to judge every scenario a person could see that no passing test covers, about 45 of them. Run 182 wrote the right shape. 7. **#431's check never ran, and would not have caught this.** The 11 state-change scenarios of run 182 went to `visual.json`; this run's 49 went to `untested.json`, with the same "verified in a browser" reasons, where no reason check exists, and the client was declared untestable by design. Two gaps: a hand-over file in the wrong shape should fail `script.scenarios` naming the expected shape, not count for nothing; and the untested hatch needs the same rule as the visual one, that having looked is no reason and a scenario with a testable outcome keeps its test. Filing both. Prediction scorecard so far: #426 ✓; #430 ✗ (finding 1); the Visual check's list will be about 45, not 12 to 18, for the reason in finding 6.
Author
Owner

Code review: 10.1 min, 25 turns, 147k peak context (run 182: 6 min, 19 turns, 117k), 1 note, 5 fixed. Then round 1's unit lane went red and a fix round started; the Visual check is skipped this round.

  1. #427 holds. The review read 3 files in all (run 182: 24 files in 30 reads), none of them a pasted one. The brief was larger (147k peak at the review's end against 182's 117k) and the review took 4 minutes longer, with five fixes of its own against 182's three; the reads are gone, as predicted.

  2. A test the apply wrote is red in the lane and was green for the apply: it depends on chance. packages/server/test/game.test.ts, "a card of another suit is rejected when the player holds the led suit", drives two bots over a random deal for up to 30 s and asserts an illegal play was attempted along the way; whether the follower ever holds both the led suit and another card is the deal's. The round fails on it, the fix round gets it, and the Visual check waits for round 2. The weak-test lint (ban-check) has no rule for a test whose outcome depends on an unseeded random source; a candidate for it. (The engine takes an injectable RNG; the test did not inject one.)

Note: the fix round's brief will list 49 uncovered scenarios (finding 6) and ask for tests; what it does with that is the next thing to read.

**Code review: 10.1 min, 25 turns, 147k peak context (run 182: 6 min, 19 turns, 117k), 1 note, 5 fixed. Then round 1's unit lane went red and a fix round started; the Visual check is skipped this round.** 8. **#427 holds.** The review read 3 files in all (run 182: 24 files in 30 reads), none of them a pasted one. The brief was larger (147k peak at the review's end against 182's 117k) and the review took 4 minutes longer, with five fixes of its own against 182's three; the reads are gone, as predicted. 9. **A test the apply wrote is red in the lane and was green for the apply: it depends on chance.** `packages/server/test/game.test.ts`, "a card of another suit is rejected when the player holds the led suit", drives two bots over a random deal for up to 30 s and asserts an illegal play was attempted along the way; whether the follower ever holds both the led suit and another card is the deal's. The round fails on it, the fix round gets it, and the Visual check waits for round 2. The weak-test lint (`ban-check`) has no rule for a test whose outcome depends on an unseeded random source; a candidate for it. (The engine takes an injectable RNG; the test did not inject one.) Note: the fix round's brief will list 49 uncovered scenarios (finding 6) and ask for tests; what it does with that is the next thing to read.
Author
Owner

Fix round 1: 6.3 min, 10 turns; it made the deal-dependent test wait for the rejection itself instead of round 2, no assertion weakened. Round 2 green at 16:14. Visual check (as round 2, round 1 having been skipped) started 16:15; brief 59k chars, no motion probe or filmstrip, the wait cap and one-state sentences present, 55 scenarios marked covered, the other 49 to judge where visible.

  1. The Visual check is 28 minutes in with no look taken: it is building a solver over the engine. 50 turns, 0 browser checks, 0 criteria, report unfinished. It symlinked the worktree's node_modules under tmp/scratch, wrote solver.mjs, dryrun.mjs and two debug scripts that import the engine's modules, and has run and edited them 27 times ("10% success rate, too low; try multiple T values"): a planner that searches for a deal and a line of play that ends in a chosen outcome. This is run 62's pattern (braid#170), and the brief's rule, "a script does not stand in for looking: no bundling of the app's modules to run them", is not enforced by anything. The cause is upstream: the game-end scenarios (shared win, tie-break, sole winner) are state changes that should have unit tests and were left in the mis-shaped untested.json (finding 6), so they reached the judge as "uncovered", and the judge is trying to stage a tie by hand. Not intervening; the time is the measurement. Candidate: the guard refuses a bash in the judge's step that runs a script under the run's tmp importing from the worktree (as the browser subagent is refused file tools), with the words to judge from what a person sees or mark the criterion unclear.
**Fix round 1: 6.3 min, 10 turns; it made the deal-dependent test wait for the rejection itself instead of round 2, no assertion weakened. Round 2 green at 16:14. Visual check (as round 2, round 1 having been skipped) started 16:15; brief 59k chars, no motion probe or filmstrip, the wait cap and one-state sentences present, 55 scenarios marked covered, the other 49 to judge where visible.** 10. **The Visual check is 28 minutes in with no look taken: it is building a solver over the engine.** 50 turns, 0 `browser` checks, 0 criteria, report unfinished. It symlinked the worktree's `node_modules` under `tmp/scratch`, wrote `solver.mjs`, `dryrun.mjs` and two debug scripts that import the engine's modules, and has run and edited them 27 times ("10% success rate, too low; try multiple T values"): a planner that searches for a deal and a line of play that ends in a chosen outcome. This is run 62's pattern (braid#170), and the brief's rule, "a script does not stand in for looking: no bundling of the app's modules to run them", is not enforced by anything. The cause is upstream: the game-end scenarios (shared win, tie-break, sole winner) are state changes that should have unit tests and were left in the mis-shaped `untested.json` (finding 6), so they reached the judge as "uncovered", and the judge is trying to stage a tie by hand. Not intervening; the time is the measurement. Candidate: the guard refuses a `bash` in the judge's step that runs a script under the run's tmp importing from the worktree (as the `browser` subagent is refused file tools), with the words to judge from what a person sees or mark the criterion unclear.
Author
Owner

Visual check at 52 minutes: still no look at the app, no criterion judged.

  1. The solver consumed a whole context window before the first look. At 36 min the step hit the 200k alarm and compacted: 82 turns, 46 edits and 32 shell runs on five scripts under tmp/scratch (solver.mjs, dryrun.mjs, three debug scripts), 0 browser checks, 0 criteria. After the compaction it set out to "reconstruct what actually happened": the summary named a driver.mjs the disk does not have, and a find over / was refused by the guard.

  2. It then read Braid's own source to learn the report's schema. Through $OSF_PYTHON it printed inspect.getsource(osf.pipeline.fixloop) in four slices and JUDGE_SCHEMA from osf.pipeline.nodes, 18 reads and 23 shell calls since the compaction, though the brief gives it a command that checks the report (fixloop check-report) and says to use it rather than write a validator. The agent's interpreter imports osf because the run's Python is Braid's venv; nothing in the shell guard stops a read of Braid's package (braid#201 territory).

Both are the judge's brief not binding where the guard could: a script under the run's tmp that imports the worktree's modules, and a read of Braid's own package, are refusable calls. Run 182's judge did neither; the difference is the 49 uncovered scenarios (finding 6), tie-break scenarios among them, that this judge is trying to stage by hand.

**Visual check at 52 minutes: still no look at the app, no criterion judged.** 11. **The solver consumed a whole context window before the first look.** At 36 min the step hit the 200k alarm and compacted: 82 turns, 46 edits and 32 shell runs on five scripts under `tmp/scratch` (`solver.mjs`, `dryrun.mjs`, three debug scripts), 0 `browser` checks, 0 criteria. After the compaction it set out to "reconstruct what actually happened": the summary named a `driver.mjs` the disk does not have, and a `find` over `/` was refused by the guard. 12. **It then read Braid's own source to learn the report's schema.** Through `$OSF_PYTHON` it printed `inspect.getsource(osf.pipeline.fixloop)` in four slices and `JUDGE_SCHEMA` from `osf.pipeline.nodes`, 18 reads and 23 shell calls since the compaction, though the brief gives it a command that checks the report (`fixloop check-report`) and says to use it rather than write a validator. The agent's interpreter imports `osf` because the run's Python is Braid's venv; nothing in the shell guard stops a read of Braid's package (braid#201 territory). Both are the judge's brief not binding where the guard could: a script under the run's tmp that imports the worktree's modules, and a read of Braid's own package, are refusable calls. Run 182's judge did neither; the difference is the 49 uncovered scenarios (finding 6), tie-break scenarios among them, that this judge is trying to stage by hand.
Author
Owner

Visual check done: 93.4 min, 465 turns, two compactions, 59 criteria (53 pass, 4 fail, 2 unclear), verdict fail; fix round 2 next. Run 182's clean judge was 78 min for 34 criteria; run 181's 110 for 38.

  1. It judged alone, and wrote its report at the end. 0 browser checks in 93 minutes: 102 page scripts, 54 clicks, 35 screenshots, 14 waits of its own across three tabs, so the wait cap (which counts the subagent's sessions) touched nothing. The report had 0 criteria at 67 and 88 minutes and 59 at the close: "write the report as you go" was not followed, and the guard that refuses the first look until the file exists was satisfied by the empty file written at the start. Of the 59, none is a B criterion; it wrote look/… ids instead, and slugs (game-table/portrait-play) rather than the scenarios' names.

  2. It found two real defects, which is the point of the step. The Leave button does nothing in the waiting room and on the end screen: the client passes its ack callback in the payload slot of room:leave, so the server never acks. And at 375 px the fanned hand is 474 px wide and the end cards are off screen. Both are the fix round's.

  3. The tie it spent its first hour staging never came. game-end/shared-win is unclear: "four complete 3-player games were played, driven by a custom planner that targets an exact tie; all four ended" otherwise. A unit test of the scoring with a chosen tally would have settled it in a second; the scenario reached the judge through the mis-shaped untested.json (finding 6). Sound is the other unclear (headless).

  4. The revert held. No motion probe, no filmstrip, no clock in the brief or the transcript; the deal, flip, slide, sweep and scratch animations were judged from screenshots and the state hook, all pass.

Scorecard: criteria 59 (predicted about 20); clean judge minutes 93 (predicted 45 to 55); browser checks 0 (predicted 6 to 8); longest check and waits: no checks to measure. Every miss traces to finding 6, the hand-over files Braid could not read.

**Visual check done: 93.4 min, 465 turns, two compactions, 59 criteria (53 pass, 4 fail, 2 unclear), verdict fail; fix round 2 next.** Run 182's clean judge was 78 min for 34 criteria; run 181's 110 for 38. 13. **It judged alone, and wrote its report at the end.** 0 `browser` checks in 93 minutes: 102 page scripts, 54 clicks, 35 screenshots, 14 waits of its own across three tabs, so the wait cap (which counts the subagent's sessions) touched nothing. The report had 0 criteria at 67 and 88 minutes and 59 at the close: "write the report as you go" was not followed, and the guard that refuses the first look until the file exists was satisfied by the empty file written at the start. Of the 59, none is a `B` criterion; it wrote `look/…` ids instead, and slugs (`game-table/portrait-play`) rather than the scenarios' names. 14. **It found two real defects, which is the point of the step.** The Leave button does nothing in the waiting room and on the end screen: the client passes its ack callback in the payload slot of `room:leave`, so the server never acks. And at 375 px the fanned hand is 474 px wide and the end cards are off screen. Both are the fix round's. 15. **The tie it spent its first hour staging never came.** `game-end/shared-win` is `unclear`: "four complete 3-player games were played, driven by a custom planner that targets an exact tie; all four ended" otherwise. A unit test of the scoring with a chosen tally would have settled it in a second; the scenario reached the judge through the mis-shaped `untested.json` (finding 6). Sound is the other `unclear` (headless). 16. **The revert held.** No motion probe, no filmstrip, no clock in the brief or the transcript; the deal, flip, slide, sweep and scratch animations were judged from screenshots and the state hook, all pass. Scorecard: criteria 59 (predicted about 20); clean judge minutes 93 (predicted 45 to 55); `browser` checks 0 (predicted 6 to 8); longest check and waits: no checks to measure. Every miss traces to finding 6, the hand-over files Braid could not read.
Author
Owner

Fix round 2: 52.3 min, 79 turns (run 181's fix round: 23 min). Round 3 green; Visual check round 3 started 18:39 in a fresh session.

  1. #414's lists are right, and buried. The round 3 brief says: look at 6 (game-table/portrait-play, look/waiting-b2, look/end-b2, brief/production-quality, and the two unclear ones), carry 53 by id, two files changed (store.ts, Table.tsx), do not go through a session again. That is the mechanism working on its first real round. But the brief is 65,929 characters, larger than the first look's 59,375: a 24k section "What the last judge found" pastes every one of the 53 passed criteria with its evidence, though the judge is told to carry them by id and that the evidence is in .osf/fix/r2/judge.json; 12k of delta specs; and about 20k of the first-look rulebook (Judge what you see, Plan the states, Probes, the browser briefing) with one line saying this round's list holds over it. The instruction that matters starts 21,500 characters into its own section. Candidate: a later round's brief leads with the look list and the changed files, names the carried ids in one line, drops the passed criteria's evidence (it is on disk), and leaves out the first-look sections the round does not use. Whether the judge obeys the lists is the next thing to read.
**Fix round 2: 52.3 min, 79 turns (run 181's fix round: 23 min). Round 3 green; Visual check round 3 started 18:39 in a fresh session.** 17. **#414's lists are right, and buried.** The round 3 brief says: look at 6 (`game-table/portrait-play`, `look/waiting-b2`, `look/end-b2`, `brief/production-quality`, and the two unclear ones), carry 53 by id, two files changed (`store.ts`, `Table.tsx`), do not go through a session again. That is the mechanism working on its first real round. But the brief is 65,929 characters, larger than the first look's 59,375: a 24k section "What the last judge found" pastes every one of the 53 passed criteria with its evidence, though the judge is told to carry them by id and that the evidence is in `.osf/fix/r2/judge.json`; 12k of delta specs; and about 20k of the first-look rulebook (Judge what you see, Plan the states, Probes, the `browser` briefing) with one line saying this round's list holds over it. The instruction that matters starts 21,500 characters into its own section. Candidate: a later round's brief leads with the look list and the changed files, names the carried ids in one line, drops the passed criteria's evidence (it is on disk), and leaves out the first-look sections the round does not use. Whether the judge obeys the lists is the next thing to read.
Author
Owner

Visual check round 3 at 64 minutes: the look list is obeyed, and one criterion is eating the round.

  1. #414 works on a real round. The judge took the six looks it was given and nothing else: four are written and pass (game-table/portrait-play, look/waiting-b2, look/end-b2, game-table/sound-on-events), with five browser checks delegated this time and the report written as it goes. The two fixes are confirmed: Leave works in the waiting room and on the end screen, and the hand fits at 375 px.

  2. The last half hour is game-end/shared-win alone. It was unclear in round 2, so it is on the look list, and the judge is staging a tie again: a driver.mjs under the run's tmp that plays whole games through a coordinator, on its fourth run ("Bob still hasn't played in this fresh room either… continuing (run 4)"), with sleep 120 between polls. 222 turns, context at 198k, a compaction due. A scenario whose outcome is a tally comparison belongs to a unit test (finding 6 and #441); once it reaches the Visual check as unclear, nothing bounds what a round may spend on one criterion. Candidates: an unclear that names what could not be staged goes to the fix round as "write the test" rather than back on the look list; and a per-criterion budget in a later round (one targeted look means one, then unclear stands).

The brief audit is #443.

**Visual check round 3 at 64 minutes: the look list is obeyed, and one criterion is eating the round.** 18. **#414 works on a real round.** The judge took the six looks it was given and nothing else: four are written and pass (`game-table/portrait-play`, `look/waiting-b2`, `look/end-b2`, `game-table/sound-on-events`), with five `browser` checks delegated this time and the report written as it goes. The two fixes are confirmed: Leave works in the waiting room and on the end screen, and the hand fits at 375 px. 19. **The last half hour is `game-end/shared-win` alone.** It was `unclear` in round 2, so it is on the look list, and the judge is staging a tie again: a `driver.mjs` under the run's tmp that plays whole games through a coordinator, on its fourth run ("Bob still hasn't played in this fresh room either… continuing (run 4)"), with `sleep 120` between polls. 222 turns, context at 198k, a compaction due. A scenario whose outcome is a tally comparison belongs to a unit test (finding 6 and #441); once it reaches the Visual check as `unclear`, nothing bounds what a round may spend on one criterion. Candidates: an `unclear` that names what could not be staged goes to the fix round as "write the test" rather than back on the look list; and a per-criterion budget in a later round (one targeted look means one, then `unclear` stands). The brief audit is #443.
Author
Owner

Run 183 finished: succeeded, pull request merged and the change archived, 355 min wall, 354 agent minutes (run 182: 345, about 220 clean; run 181: 362; run 174: 225). No incident this time: no stall, no reboot, no deploy. Every minute is the pipeline's own.

step run 183 run 182
explore, propose, review, revise 14 18
apply 93 (two compactions) 115
code review 10 6
fix, round 1 (a chance-dependent test) 6 none
Visual check, first look 93 (two compactions, 59 criteria, fail on 4) 204 with 120 lost; 78 clean, 34 criteria, pass
fix, round 2 52 none
Visual check, round 3 83 (look at 6, carry 53; pass, 1 unclear) none
summarize 1 3

Scorecard against the predictions

what run 182 predicted run 183
scenarios left to the Visual check 28 12 to 18 14 written, 0 read; 49 uncovered ✗ finding 6
Visual check criteria, first look 34 about 20 59 ✗
browser checks, first look 10 6 to 8 0 (it drove three tabs itself) ✗
waits per browser check up to 26 at most 6 3, 3, 2, 1 (round 3); cap never reached ✓ untested
Visual check first look, clean 78 min 45 to 55 93 ✗
code review reads of new files 24 in 30 reads under 5 3 ✓ #427
code review brief 147k chars about 230k 253k as designed
findings about Depends on: 5 0 1 ✗ finding 1, #430 reopened
finding about window.__app 1 0 0 ✓ #426
stalls on a paused clock 2 0 0 ✓ #433
apply 115 about the same 93 ✓
agent minutes 345 (220 clean) 170 to 190 354 ✗

What the run says. The changes that could be measured held: the review reads (#427), the hook (#426), the revert (#433), the look and carry lists (#414, first real round). The headline prediction failed on one upstream fact: the apply wrote both hand-over files in a shape Braid does not read (finding 6), so 49 scenarios, tie-breaks among them, reached the Visual check as uncovered, #431's check never ran, and the judge spent its first hour, and half of round 3, staging a tie that a unit test settles in a second. Two fix rounds and a second Visual check followed, where run 182 had none.

Filed from this run: #430 reopened (the tasks rules never reach the thorough pipeline's planner), #440 (a mis-shaped hand-over file fails the scenarios step), #441 (the untested hatch takes "verified in a browser" and a whole client as untestable by design), #443 (audit of every brief, with research).

Candidates not yet filed: the planner writes MAY in probabilistic scenarios (finding 4, also in run 182); a test on an unseeded random source passes the weak-test lint (9); the judge may run scripts that import the worktree's modules and read Braid's own package (10 to 12); "write the report as you go" is satisfied by an empty file (13); an unclear that could not be staged goes back on the look list with no bound on one criterion (19); the apply compacts twice (3). Fix round 2's 52 minutes for two small fixes is unexamined.

**Run 183 finished: succeeded, pull request merged and the change archived, 355 min wall, 354 agent minutes (run 182: 345, about 220 clean; run 181: 362; run 174: 225). No incident this time: no stall, no reboot, no deploy. Every minute is the pipeline's own.** | step | run 183 | run 182 | |---|---:|---:| | explore, propose, review, revise | 14 | 18 | | apply | 93 (two compactions) | 115 | | code review | 10 | 6 | | fix, round 1 (a chance-dependent test) | 6 | none | | Visual check, first look | 93 (two compactions, 59 criteria, fail on 4) | 204 with 120 lost; 78 clean, 34 criteria, pass | | fix, round 2 | 52 | none | | Visual check, round 3 | 83 (look at 6, carry 53; pass, 1 unclear) | none | | summarize | 1 | 3 | **Scorecard against the predictions** | what | run 182 | predicted | run 183 | | |---|---:|---:|---:|---| | scenarios left to the Visual check | 28 | 12 to 18 | 14 written, 0 read; 49 uncovered | ✗ finding 6 | | Visual check criteria, first look | 34 | about 20 | 59 | ✗ | | `browser` checks, first look | 10 | 6 to 8 | 0 (it drove three tabs itself) | ✗ | | waits per `browser` check | up to 26 | at most 6 | 3, 3, 2, 1 (round 3); cap never reached | ✓ untested | | Visual check first look, clean | 78 min | 45 to 55 | 93 | ✗ | | code review reads of new files | 24 in 30 reads | under 5 | 3 | ✓ #427 | | code review brief | 147k chars | about 230k | 253k | as designed | | findings about `Depends on:` | 5 | 0 | 1 | ✗ finding 1, #430 reopened | | finding about `window.__app` | 1 | 0 | 0 | ✓ #426 | | stalls on a paused clock | 2 | 0 | 0 | ✓ #433 | | apply | 115 | about the same | 93 | ✓ | | agent minutes | 345 (220 clean) | 170 to 190 | 354 | ✗ | **What the run says.** The changes that could be measured held: the review reads (#427), the hook (#426), the revert (#433), the look and carry lists (#414, first real round). The headline prediction failed on one upstream fact: the apply wrote both hand-over files in a shape Braid does not read (finding 6), so 49 scenarios, tie-breaks among them, reached the Visual check as uncovered, #431's check never ran, and the judge spent its first hour, and half of round 3, staging a tie that a unit test settles in a second. Two fix rounds and a second Visual check followed, where run 182 had none. **Filed from this run:** #430 reopened (the tasks rules never reach the thorough pipeline's planner), #440 (a mis-shaped hand-over file fails the scenarios step), #441 (the untested hatch takes "verified in a browser" and a whole client as untestable by design), #443 (audit of every brief, with research). **Candidates not yet filed:** the planner writes MAY in probabilistic scenarios (finding 4, also in run 182); a test on an unseeded random source passes the weak-test lint (9); the judge may run scripts that import the worktree's modules and read Braid's own package (10 to 12); "write the report as you go" is satisfied by an empty file (13); an `unclear` that could not be staged goes back on the look list with no bound on one criterion (19); the apply compacts twice (3). Fix round 2's 52 minutes for two small fixes is unexamined.
Author
Owner

Closed: every finding here is fixed or has its own ticket. braid#430, #440, #441 and #443 shipped on 2026-11-08 (PRs 446, 448); braid#434 holds the animation research.

Closed: every finding here is fixed or has its own ticket. braid#430, #440, #441 and #443 shipped on 2026-11-08 (PRs 446, 448); braid#434 holds the animation research.
Sign in to join this conversation.
No labels
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
cmoriarty/braid#439
No description provided.