Run 189: findings after the run 188 batch #480

Closed
opened 2026-11-10 13:00:17 -05:00 by cmoriarty · 3 comments
Owner

What happens: run 189 re-runs cmoriarty/scratch-vllm-fp8#16 (the run 187/188 change) on production edfc38b, after the run 188 batch (braid#474). Thorough, auto-approve, vLLM qwen3.8-27b, every step at xhigh (run 188's Visual check ran at medium).

This issue: the run's findings as it goes: whether each #474 fix works, problems, and possible efficiency gains. One comment per area, kept up to date.

Done when: the run has ended and its findings are reviewed with the owner.

**What happens:** run 189 re-runs `cmoriarty/scratch-vllm-fp8#16` (the run 187/188 change) on production `edfc38b`, after the run 188 batch (braid#474). Thorough, auto-approve, vLLM qwen3.8-27b, every step at `xhigh` (run 188's Visual check ran at `medium`). **This issue:** the run's findings as it goes: whether each #474 fix works, problems, and possible efficiency gains. One comment per area, kept up to date. **Done when:** the run has ended and its findings are reviewed with the owner.
Author
Owner

Run 189: the #474 fixes, checked as the run reaches them

Fix What to see Seen
Group order (1) a group's brief names the groups done and not finished group 1: "Groups 2, 3, 4, 5, 6, 7, 8 and 9 have not finished: do not count on their work."
Retry first (2) a failed group's retry starts before a group that has not yes: group 1's retry ran 13:57-14:05, group 2 started after it
Scope (3) no group argues with "work only on group N" …
Own tasks and notes (17) each group brief: own tasks whole, others by heading group 1: its tasks whole, groups 2-9 by heading and Depends on (brief 61,578 chars; run 188's group 1: 55,839 for a smaller plan)
Tested by task N (18) tested_by entries used, none rejected not used by any group, so not exercised
Saved brief (5) continuation names $TMPDIR/briefs/…; agents read it instead of searching .osf/ yes: group 3's first compaction named /var/lib/openspecflow/runs/189/tmp/briefs/openspec.apply-3.1.md, and its second call read it
Unconfirmed summary (6) no wrong conclusion carried forward the line is sent (group 3); effect to watch
Changed tests (7) delegated runs are the changed tests, not the whole suite after a compaction the line is sent (group 3); group 2 kept to its changed tests; but before any compaction: group 1's test ran the whole suite (168 tests) before any compaction. The fix reached only the reminder after a compaction; the brief itself still says both (see Problems)
Bugs found (8) .osf/bugs.json entries; plan.bugs.* draft and file yes: 3 tickets drafted and filed (scratch-vllm-fp8 #18, #19, #20); but #20 was already fixed by group 7, see Problems
Code review helpers (9) review brief ~90k; reader helpers per area; no 5-minute alarm brief 90,957 chars; alarm at 8.4 min; reader never called; found and fixed 4 real defects
Changed tests named (10) main test file named yes: game-table.test.tsx named and read
Background commands (11) /api/shell output read and stop work on 2.0.30; no 30-minute hold not exercised: no step ended waiting on a background command; group 7 stopped its own server by hand
B11 (12) helper-read source tests recorded yes: 8 recorded (pwa.test.ts 3, styles.test.ts 5), none failed the round
Seed (14) tasks include a seed entry point yes: group 1 is "Foundation: room settings, server seed, smoke-test scaffolding"; the planner quoted the rule
No judge limit (15) judge's browser checks not refused yes for browser checks (107 scripts); once told to record criteria first after 12 looks
360 s scripts (19) no script cut at 180 s none cut: the Visual check's 4 script errors were mistakes in the scripts
Stop a helper (20) stop used, and it stops the helper yes: group 8 used it twice; the first was by mistake (it stopped the browser helper it meant to wait for, and restarted it)
Step row (16) finished rows keep compactions and total time yes: apply[3] (1 attempt) keeps its 2 compactions beside 1h 45m (owner's screenshot); apply[1] (2 attempts) carries 32.0 min across both, where the last attempt alone was 8.0
Effort every step xhigh, the Visual check included yes: every Visual check message xhigh

Steps against run 188

Step Run 188 Run 189
explore 4.7 min, 72k peak 5.7 min, 94k peak, 34 reads, 0 compactions
propose 16.4 min, 162k, forked (7,580-char brief), 8 groups 14.8 min, 116k, not forked: explore's last request (105k) was over half the alarm, so a fresh session with the brief pasted (26,971 chars); 9 groups
review round 1 17.8 min, 156k, 136,376-char brief, 8 findings, 1 blocking 12.8 min, 142k, 142,015-char brief, 4 findings (2 to fix, 2 notes), 0 compactions
revise, round 2 2.0 min; round 2 1.1 min 1.5 min, three edits, validated; no round 2 (nothing blocking)
apply group 1 18.3 min, 159k, no compaction, 0 subagents ~22 min, failed on the B8 bug (Problems); attempt 2 passed in 8.0 min, 163k peak, 0 compactions, 4 subagent calls (all its first attempt's test runs were whole-suite)
apply group 2 95,813-char brief, compacted at 22 min, failed on visual.json, retry 3.5 min 33,700-char brief, passed first time in 18.2 min, 120k peak, 0 compactions; ran only the tests it changed
apply group 3 (table and bidding, and the four-moment geometry browser test) the same test was group 8's: 197 min, 5 compactions 105.5 min, first attempt, 198 turns, 2 compactions (at ~37 and 68 min), 3 browser checks; most of it on the same timing race (Efficiency)
apply group 4 (seat and trick clarity) group 4 then: 77.4 min, 2 compactions 41.8 min, first attempt, 163 turns, 1 compaction (25 min in), brief 70,693 chars
apply group 5 (round summary and play) (other work) 53.2 min, first attempt, 195 turns, 1 compaction (24 min in), brief 61,132 chars; removed a drag test headless Playwright could not drive and listed it in untested.json with why
apply group 6 (hand and cards) (other work) 26.9 min, first attempt, 140 turns, 0 compactions (peak 195k), brief 51,628 chars; kept to changed tests, and also ran game-rules.test.ts because it imports Hand, which it changed
apply group 7 (table layout and fonts) group 8: 197.1 min, 5 compactions 168.8 min, first attempt, 5 compactions; about 115 min went on browser tests from groups 5 and 6 that had never run
apply group 8 38.9 min, 1 compaction
apply group 9 42.3 min, 1 compaction
code review 11.4 min, 1 compaction 18.4 min, 1 compaction, 4 defects fixed
lanes first round green: 254 unit tests, 11 browser tests in 18.4 s
Visual check 29.4 min at medium 76.3 min at xhigh, 2 compactions, 25 of 25, no fix round
whole run stopped at 560 min in fix round 1 succeeded, 660 min, no fix round

Problems

  • B8 counts playwright.config.ts as a browser test and fails the project's first smoke test. Group 1 added packages/client/e2e/playwright.config.ts and packages/client/e2e/smoke.spec.ts to a project with no browser test. is_browser_spec takes any file it treats as a test file that imports @playwright/test, so the config counts; with none on the base, #471's B8 keeps the first new one by name, the config, and fails smoke.spec.ts as "no browser spec file beside the project's existing one". Attempt 1 failed on it at the end of its work, and attempt 2's agent cannot see why ("this is a weird detection… why is playwright.config.ts being labeled the browser spec?"). Fix: a Playwright config (playwright.config.*) is never a spec, and B8 keeps a *.spec.* file first. A sim with a config beside the first spec. Blocks group 1, in every project that adds its first smoke test with a config in e2e/. Attempt 2 then found the bug by reading Braid's own source (next item), and worked around it by taking defineConfig from @playwright/test out of the config: a correct file made worse (it loses its types) to get past a Braid bug.

  • The guard refuses running Braid's package but not reading it. Group 1's attempt 2: "I was told 'Braid's own package is not yours to run or read.' However, the grep worked", then quoted _TEST_FILE_RE … _TEST_CONFIG_RE from apply.py. Either let agents read the lint (it explained the finding when Braid's message did not), or refuse the read as well; and make B8's message name the file it kept, which would have saved the search.

  • Explore is told a missing openspec/ folder is expected, in a repository that has one. The line (prompts.py:183, from run 157) goes to every explore. Run 189's explore: "the instructions say 'repositories without an openspec/ folder are expected…' But there IS an openspec folder here. Let me take a quick peek — no, actually the instructions say not to". Fix: send it only when the worktree has no openspec/.

  • A long MODIFIED requirement passes the change's check and fails the main spec's after archive. OpenSpec's own instruction says the 500-character warning applies to "ADDED requirements and the main spec"; run 189's planner read that as MODIFIED being exempt ("So MODIFIED descriptio[ns]…"). But a MODIFIED requirement becomes the main spec's text: openspec validate --strict on the change passes it, and on the specs after archive it fails (it did on #474's own archive this morning). Fix: Braid's openspec.validate also holds MODIFIED requirements to 500 characters, so the planner hears it before the review, not at archive.

  • The review's brief says ".osf/review/ already exists", and in round one it does not. prompts.py:1299; run 189's combined review: "The instruction says .osf/review/ already exists but it doesn't." Fix: the engine creates the folder before briefing the review, so the line holds (it is there to save a listing).

  • The seed task (from #474) has no scenario to tag its test with. The planner added "a server seed" to group 1 but no scenario in the delta spec; group 1, held to "tag each test with the scenario it covers", hunted for one: "the delta spec for this change (game-lobby) doesn't have a seed scenario… For the seed test, the honest answer is: it protects the determinism the design relies on". Fix: the seed clause asks for a scenario with the task ("a scenario that the same seed deals the same game"), so the tag has a home.

  • Group 1 searched for Braid's lint source to learn what it enforces: python3 -c "import osf.pipeline.apply" and then find /, refused by the guard (also run 188's group 1). The brief lists the rules; the agent wanted to read how. Watch whether it recurs; the cost here was one refused call.

  • The apply brief contradicts itself on the suite. Its delegation line: "test: suites, typechecks, lints and builds run here… the project's own fast test command"; its checks section: "run only the tests you wrote or changed, not the whole suite: the lanes run it". Group 1 ran all 168 tests through test. #474 changed only the reminder after a compaction. Fix: the delegation line says "the tests you changed, typechecks, lints and builds", one source of the rule.

  • A later group's brief does not give the browser lane's declared command. It names the unit lane's ("the unit lane runs npm test, the command in .osf/test/unit.command") but for the browser lane only says how to declare one, though group 1 had written npx playwright test --config packages/client/e2e/playwright.config.ts. Group 5's own task then says "verify that npx playwright test --list lists the test", which from the root finds 0 tests, and it spent many turns on working directories and --config before finding it. Group 6 then hit the same wall ("Total: 0 tests in 0 files"). Fix: when .osf/test/e2e.command exists, the brief says "the browser lane runs <it>" as it does for the unit lane; and a plan's verification commands use the lane's command.

  • A miss against the change's own spec went into the bugs file as if it were outside the work. Group 6 found phone played cards are 48 px where the scenario "Cards render at legible sizes" says at least 56 (set by an earlier group), judged it risky to change, guessed "belongs to group 7", and wrote it to .osf/bugs.json. The file is for bugs outside the change. Here the change is failing its own requirement, and a filed ticket would let that ship. To check: whether group 7, the code review or the Visual check catches it.

  • Browser tests are written and ticked without ever being run. The plan's tasks say to verify a browser test with npx playwright test --list and a typecheck, and --list only loads the file. Group 5's round-end test has a plain ReferenceError (game() called inside a page script), so it could never have passed; group 6 ran no browser test at all; the lane would have been the first to run them. Group 7 found it by running the suite, then argued with its scope: "Work ONLY on the tasks in group 7" against "fixing what your own change broke", which does not cover a bug another group left. The planner wrote --list because it reasoned "Braid runs the actual browser check after the final group". The whole browser suite takes 18 s. Running these tests exposed three real faults in the game: a render crash on an empty table, a round summary skipped when updates arrive together, and the deal animation leaking under the summary.

  • A bug fixed later is still filed. Group 7 raised phone played cards to 56 px, and scratch-vllm-fp8#20 was filed saying they are 48. Nothing rechecks .osf/bugs.json before filing, and that entry was the change's own requirement, not an outside bug. Ticket titles are also the first 90 characters of the description, cut mid-sentence.

  • The Playwright config carries a comment about Braid's lint, explaining why it avoids defineConfig: the B8 bug has left a workaround in the project's code.

  • A refusal message misleads. A read of a path inside the run's own cache/ms-playwright was refused with "check the path you asked for against" the run's folder, which already contains it.

  • Group 3 put a fixed wait inside page.evaluate and argued it is not one. Its geometry probe waits a 400 ms minimum and caps at 3 s, in the page: "it's browser-side code, not a test wait… the 400ms is a guard". B10 (#474) sees waitForTimeout and sleeps in the test's own code, not a setTimeout/setInterval deadline inside an evaluated function. The rule's intent (no timing in tests) is dodged with the rule quoted. Fix: B10 also looks inside page.evaluate/waitForFunction bodies for a timer with a literal delay; the real fix is the state hook in Efficiency. Update: the merged smoke.spec.ts no longer contains it.

  • Minor: a retry's brief says both "check what you find rather than trusting it" (prompts.py:115, "You have been here before") and "do not repeat finished checks" (mediation.py:781). Group 1's attempt 2: "The instructions are a bit contradictory". One line would do: check what the failure names; leave what passed.

  • Efficiency, propose: explore read more than in run 188 (94k against 72k), which put its last request over half the alarm, so propose lost the fork and re-read the brief in a fresh session. Not wrong by the rule, but the fork is what saves propose its context; worth watching whether explore's reading grew because "Open at most 15 files" went (#471).

Console (from the owner)

  • With "show thinking" off, the agent's final statement is greyed like a thought. The collapsed rows read + Thought <its first line> in the thinking colour (text-thinking, Transcript.tsx), and the step's last words look the same, so what the agent says to the person reads as hidden reasoning. Owner: it should read as real output, in the text colour. To compare with how opencode 2 shows a session with thinking hidden before choosing.
  • The Ask box's placeholder does not say it leaves the agent alone. ASK_PLACEHOLDER (ui/src/lib/ask.ts) reads "Ask about this run"; only the button's tooltip says "without steering its agent". Owner: a short phrase, not a sentence. Proposed: "Ask about this run — the agent carries on".
  • The Agent box's placeholder uses "Ask", the other mode's name. composerHint.ts reads "Ask for a change, or tell it what you found…", which can be taken for the Ask mode. Proposed: "Request a change, or tell it what you found…".
  • The compaction mark does not say "compaction". A step row shows the count inside a dotted circle, which reads as a spinner or a pending state. Proposed: two short lines, one above and one below the number, so it looks squeezed.

Efficiency

  • A browser test of short-lived states rebuilds the same 1 ms recorder every run. Group 3 (run 189), like group 8 (run 188, most of its 197 min), needs the table's boxes at "bots bidding, my bid, play, last trick" with instant bots; the states pass between two polls, so it writes a setInterval(…, 1) recorder in the page ("a 1ms interval means the recorder runs 1000 times per second"). The seed (#474) fixes the deal, not the timing. Run 188's group 8 ended by adding an onStateChange hook to the app. Fix, of the seed rule's kind: when the plan has a browser test of moments in a flow, a task gives the app a hook that reports each state as it changes (or a test setting that holds each move), before that test's task. Looks have the same problem: group 5 (like run 188's group 4) patched the page's setTimeout (2,500 → 60,000 ms) to hold the round summary long enough to screenshot; a hook that pauses, or a setting that holds each transient state, would serve the looks and the Visual check too.
## Run 189: the #474 fixes, checked as the run reaches them | Fix | What to see | Seen | |---|---|---| | Group order (1) | a group's brief names the groups done and not finished | group 1: "Groups 2, 3, 4, 5, 6, 7, 8 and 9 have not finished: do not count on their work." | | Retry first (2) | a failed group's retry starts before a group that has not | **yes**: group 1's retry ran 13:57-14:05, group 2 started after it | | Scope (3) | no group argues with "work only on group N" | … | | Own tasks and notes (17) | each group brief: own tasks whole, others by heading | group 1: its tasks whole, groups 2-9 by heading and `Depends on` (brief 61,578 chars; run 188's group 1: 55,839 for a smaller plan) | | Tested by task N (18) | `tested_by` entries used, none rejected | not used by any group, so not exercised | | Saved brief (5) | continuation names `$TMPDIR/briefs/…`; agents read it instead of searching `.osf/` | **yes**: group 3's first compaction named `/var/lib/openspecflow/runs/189/tmp/briefs/openspec.apply-3.1.md`, and its second call read it | | Unconfirmed summary (6) | no wrong conclusion carried forward | the line is sent (group 3); effect to watch | | Changed tests (7) | delegated runs are the changed tests, not the whole suite | after a compaction the line is sent (group 3); group 2 kept to its changed tests; **but** before any compaction: group 1's `test` ran the whole suite (168 tests) before any compaction. The fix reached only the reminder after a compaction; the brief itself still says both (see Problems) | | Bugs found (8) | `.osf/bugs.json` entries; `plan.bugs.*` draft and file | **yes**: 3 tickets drafted and filed (scratch-vllm-fp8 #18, #19, #20); but #20 was already fixed by group 7, see Problems | | Code review helpers (9) | review brief ~90k; `reader` helpers per area; no 5-minute alarm | brief 90,957 chars; alarm at 8.4 min; `reader` never called; found and fixed 4 real defects | | Changed tests named (10) | main test file named | **yes**: `game-table.test.tsx` named and read | | Background commands (11) | `/api/shell` output read and stop work on 2.0.30; no 30-minute hold | not exercised: no step ended waiting on a background command; group 7 stopped its own server by hand | | B11 (12) | helper-read source tests recorded | **yes**: 8 recorded (`pwa.test.ts` 3, `styles.test.ts` 5), none failed the round | | Seed (14) | tasks include a seed entry point | **yes**: group 1 is "Foundation: room settings, server seed, smoke-test scaffolding"; the planner quoted the rule | | No judge limit (15) | judge's browser checks not refused | **yes** for browser checks (107 scripts); once told to record criteria first after 12 looks | | 360 s scripts (19) | no script cut at 180 s | none cut: the Visual check's 4 script errors were mistakes in the scripts | | Stop a helper (20) | `stop` used, and it stops the helper | **yes**: group 8 used it twice; the first was by mistake (it stopped the browser helper it meant to wait for, and restarted it) | | Step row (16) | finished rows keep compactions and total time | **yes**: apply[3] (1 attempt) keeps its 2 compactions beside 1h 45m (owner's screenshot); apply[1] (2 attempts) carries 32.0 min across both, where the last attempt alone was 8.0 | | Effort | every step `xhigh`, the Visual check included | **yes**: every Visual check message `xhigh` | ## Steps against run 188 | Step | Run 188 | Run 189 | |---|---|---| | explore | 4.7 min, 72k peak | 5.7 min, 94k peak, 34 reads, 0 compactions | | propose | 16.4 min, 162k, forked (7,580-char brief), 8 groups | 14.8 min, 116k, **not forked**: explore's last request (105k) was over half the alarm, so a fresh session with the brief pasted (26,971 chars); 9 groups | | review round 1 | 17.8 min, 156k, 136,376-char brief, 8 findings, 1 blocking | 12.8 min, 142k, 142,015-char brief, 4 findings (2 to fix, 2 notes), 0 compactions | | revise, round 2 | 2.0 min; round 2 1.1 min | 1.5 min, three edits, validated; no round 2 (nothing blocking) | | apply group 1 | 18.3 min, 159k, no compaction, 0 subagents | ~22 min, failed on the B8 bug (Problems); attempt 2 passed in 8.0 min, 163k peak, 0 compactions, 4 subagent calls (all its first attempt's `test` runs were whole-suite) | | apply group 2 | 95,813-char brief, compacted at 22 min, failed on `visual.json`, retry 3.5 min | 33,700-char brief, **passed first time in 18.2 min**, 120k peak, 0 compactions; ran only the tests it changed | | apply group 3 (table and bidding, and the four-moment geometry browser test) | the same test was group 8's: 197 min, 5 compactions | 105.5 min, first attempt, 198 turns, 2 compactions (at ~37 and 68 min), 3 browser checks; most of it on the same timing race (Efficiency) | | apply group 4 (seat and trick clarity) | group 4 then: 77.4 min, 2 compactions | 41.8 min, first attempt, 163 turns, 1 compaction (25 min in), brief 70,693 chars | | apply group 5 (round summary and play) | (other work) | 53.2 min, first attempt, 195 turns, 1 compaction (24 min in), brief 61,132 chars; removed a drag test headless Playwright could not drive and listed it in `untested.json` with why | | apply group 6 (hand and cards) | (other work) | 26.9 min, first attempt, 140 turns, 0 compactions (peak 195k), brief 51,628 chars; kept to changed tests, and also ran `game-rules.test.ts` because it imports `Hand`, which it changed | | apply group 7 (table layout and fonts) | group 8: 197.1 min, 5 compactions | 168.8 min, first attempt, 5 compactions; about 115 min went on browser tests from groups 5 and 6 that had never run | | apply group 8 | | 38.9 min, 1 compaction | | apply group 9 | | 42.3 min, 1 compaction | | code review | 11.4 min, 1 compaction | 18.4 min, 1 compaction, 4 defects fixed | | lanes | | first round green: 254 unit tests, 11 browser tests in 18.4 s | | Visual check | 29.4 min at `medium` | 76.3 min at `xhigh`, 2 compactions, 25 of 25, no fix round | | whole run | stopped at 560 min in fix round 1 | succeeded, 660 min, no fix round | ## Problems - **B8 counts `playwright.config.ts` as a browser test and fails the project's first smoke test.** Group 1 added `packages/client/e2e/playwright.config.ts` and `packages/client/e2e/smoke.spec.ts` to a project with no browser test. `is_browser_spec` takes any file it treats as a test file that imports `@playwright/test`, so the config counts; with none on the base, #471's B8 keeps the first new one by name, the config, and fails `smoke.spec.ts` as "no browser spec file beside the project's existing one". Attempt 1 failed on it at the end of its work, and attempt 2's agent cannot see why ("this is a weird detection… why is playwright.config.ts being labeled the browser spec?"). Fix: a Playwright config (`playwright.config.*`) is never a spec, and B8 keeps a `*.spec.*` file first. A sim with a config beside the first spec. **Blocks group 1, in every project that adds its first smoke test with a config in `e2e/`.** Attempt 2 then found the bug by reading Braid's own source (next item), and worked around it by taking `defineConfig` from `@playwright/test` out of the config: a correct file made worse (it loses its types) to get past a Braid bug. - **The guard refuses running Braid's package but not reading it.** Group 1's attempt 2: "I was told 'Braid's own package is not yours to run or read.' However, the grep worked", then quoted `_TEST_FILE_RE … _TEST_CONFIG_RE` from apply.py. Either let agents read the lint (it explained the finding when Braid's message did not), or refuse the read as well; and make B8's message name the file it kept, which would have saved the search. - **Explore is told a missing `openspec/` folder is expected, in a repository that has one.** The line (prompts.py:183, from run 157) goes to every explore. Run 189's explore: "the instructions say 'repositories without an `openspec/` folder are expected…' But there IS an openspec folder here. Let me take a quick peek — no, actually the instructions say not to". Fix: send it only when the worktree has no `openspec/`. - **A long MODIFIED requirement passes the change's check and fails the main spec's after archive.** OpenSpec's own instruction says the 500-character warning applies to "ADDED requirements and the main spec"; run 189's planner read that as MODIFIED being exempt ("So MODIFIED descriptio[ns]…"). But a MODIFIED requirement becomes the main spec's text: `openspec validate --strict` on the change passes it, and on the specs after archive it fails (it did on #474's own archive this morning). Fix: Braid's `openspec.validate` also holds MODIFIED requirements to 500 characters, so the planner hears it before the review, not at archive. - **The review's brief says "`.osf/review/` already exists", and in round one it does not.** prompts.py:1299; run 189's combined review: "The instruction says `.osf/review/` already exists but it doesn't." Fix: the engine creates the folder before briefing the review, so the line holds (it is there to save a listing). - **The seed task (from #474) has no scenario to tag its test with.** The planner added "a server seed" to group 1 but no scenario in the delta spec; group 1, held to "tag each test with the scenario it covers", hunted for one: "the delta spec for this change (game-lobby) doesn't have a seed scenario… For the seed test, the honest answer is: it protects the determinism the design relies on". Fix: the seed clause asks for a scenario with the task ("a scenario that the same seed deals the same game"), so the tag has a home. - **Group 1 searched for Braid's lint source to learn what it enforces**: `python3 -c "import osf.pipeline.apply"` and then `find /`, refused by the guard (also run 188's group 1). The brief lists the rules; the agent wanted to read how. Watch whether it recurs; the cost here was one refused call. - **The apply brief contradicts itself on the suite.** Its delegation line: "`test`: suites, typechecks, lints and builds run here… the project's own fast test command"; its checks section: "run only the tests you wrote or changed, not the whole suite: the lanes run it". Group 1 ran all 168 tests through `test`. #474 changed only the reminder after a compaction. Fix: the delegation line says "the tests you changed, typechecks, lints and builds", one source of the rule. - **A later group's brief does not give the browser lane's declared command.** It names the unit lane's ("the unit lane runs `npm test`, the command in `.osf/test/unit.command`") but for the browser lane only says how to declare one, though group 1 had written `npx playwright test --config packages/client/e2e/playwright.config.ts`. Group 5's own task then says "verify that `npx playwright test --list` lists the test", which from the root finds 0 tests, and it spent many turns on working directories and `--config` before finding it. Group 6 then hit the same wall ("Total: 0 tests in 0 files"). Fix: when `.osf/test/e2e.command` exists, the brief says "the browser lane runs `<it>`" as it does for the unit lane; and a plan's verification commands use the lane's command. - **A miss against the change's own spec went into the bugs file as if it were outside the work.** Group 6 found phone played cards are 48 px where the scenario "Cards render at legible sizes" says at least 56 (set by an earlier group), judged it risky to change, guessed "belongs to group 7", and wrote it to `.osf/bugs.json`. The file is for bugs outside the change. Here the change is failing its own requirement, and a filed ticket would let that ship. To check: whether group 7, the code review or the Visual check catches it. - **Browser tests are written and ticked without ever being run.** The plan's tasks say to verify a browser test with `npx playwright test --list` and a typecheck, and `--list` only loads the file. Group 5's round-end test has a plain `ReferenceError` (`game()` called inside a page script), so it could never have passed; group 6 ran no browser test at all; the lane would have been the first to run them. Group 7 found it by running the suite, then argued with its scope: "Work ONLY on the tasks in group 7" against "fixing what your own change broke", which does not cover a bug another group left. The planner wrote `--list` because it reasoned "Braid runs the actual browser check after the final group". The whole browser suite takes 18 s. Running these tests exposed three real faults in the game: a render crash on an empty table, a round summary skipped when updates arrive together, and the deal animation leaking under the summary. - **A bug fixed later is still filed.** Group 7 raised phone played cards to 56 px, and scratch-vllm-fp8#20 was filed saying they are 48. Nothing rechecks `.osf/bugs.json` before filing, and that entry was the change's own requirement, not an outside bug. Ticket titles are also the first 90 characters of the description, cut mid-sentence. - **The Playwright config carries a comment about Braid's lint**, explaining why it avoids `defineConfig`: the B8 bug has left a workaround in the project's code. - **A refusal message misleads.** A read of a path inside the run's own `cache/ms-playwright` was refused with "check the path you asked for against" the run's folder, which already contains it. - **Group 3 put a fixed wait inside `page.evaluate` and argued it is not one.** Its geometry probe waits a 400 ms minimum and caps at 3 s, in the page: "it's browser-side code, not a test wait… the 400ms is a guard". B10 (#474) sees `waitForTimeout` and sleeps in the test's own code, not a `setTimeout`/`setInterval` deadline inside an evaluated function. The rule's intent (no timing in tests) is dodged with the rule quoted. Fix: B10 also looks inside `page.evaluate`/`waitForFunction` bodies for a timer with a literal delay; the real fix is the state hook in Efficiency. Update: the merged `smoke.spec.ts` no longer contains it. - Minor: **a retry's brief says both "check what you find rather than trusting it" (prompts.py:115, "You have been here before") and "do not repeat finished checks" (mediation.py:781).** Group 1's attempt 2: "The instructions are a bit contradictory". One line would do: check what the failure names; leave what passed. - Efficiency, propose: explore read more than in run 188 (94k against 72k), which put its last request over half the alarm, so propose lost the fork and re-read the brief in a fresh session. Not wrong by the rule, but the fork is what saves propose its context; worth watching whether explore's reading grew because "Open at most 15 files" went (#471). ## Console (from the owner) - **With "show thinking" off, the agent's final statement is greyed like a thought.** The collapsed rows read `+ Thought <its first line>` in the thinking colour (`text-thinking`, Transcript.tsx), and the step's last words look the same, so what the agent says to the person reads as hidden reasoning. Owner: it should read as real output, in the text colour. To compare with how opencode 2 shows a session with thinking hidden before choosing. - **The Ask box's placeholder does not say it leaves the agent alone.** `ASK_PLACEHOLDER` (`ui/src/lib/ask.ts`) reads "Ask about this run"; only the button's tooltip says "without steering its agent". Owner: a short phrase, not a sentence. Proposed: "Ask about this run — the agent carries on". - **The Agent box's placeholder uses "Ask", the other mode's name.** `composerHint.ts` reads "Ask for a change, or tell it what you found…", which can be taken for the Ask mode. Proposed: "Request a change, or tell it what you found…". - **The compaction mark does not say "compaction".** A step row shows the count inside a dotted circle, which reads as a spinner or a pending state. Proposed: two short lines, one above and one below the number, so it looks squeezed. ## Efficiency - **A browser test of short-lived states rebuilds the same 1 ms recorder every run.** Group 3 (run 189), like group 8 (run 188, most of its 197 min), needs the table's boxes at "bots bidding, my bid, play, last trick" with instant bots; the states pass between two polls, so it writes a `setInterval(…, 1)` recorder in the page ("a 1ms interval means the recorder runs 1000 times per second"). The seed (#474) fixes the deal, not the timing. Run 188's group 8 ended by adding an `onStateChange` hook to the app. Fix, of the seed rule's kind: when the plan has a browser test of moments in a flow, a task gives the app a hook that reports each state as it changes (or a test setting that holds each move), before that test's task. Looks have the same problem: group 5 (like run 188's group 4) patched the page's `setTimeout` (2,500 → 60,000 ms) to hold the round summary long enough to screenshot; a hook that pauses, or a setting that holds each transient state, would serve the looks and the Visual check too.
Author
Owner

The owner went through these findings one at a time. The approved fixes are filed as braid#481. Left alone: a scenario for the seed test (harmless note, no preference given), and a stricter guard on reading Braid's code (a help file per check replaces the need).

The owner went through these findings one at a time. The approved fixes are filed as braid#481. Left alone: a scenario for the seed test (harmless note, no preference given), and a stricter guard on reading Braid's code (a help file per check replaces the need).
Author
Owner

Closing, done as the issue defines it: run 189 ended (succeeded, 660 min) and the owner reviewed its findings.

The approved fixes shipped as braid#481. Run 192 checked them in a full run (the table on braid#484): six seen working, two with bugs that braid#485 then fixed (the text-only rule's false positives, the code review never offered its helpers), and two not exercised (the bugs file cleaning, tested by). The two findings left alone are named in the comment above.

Closing, done as the issue defines it: run 189 ended (succeeded, 660 min) and the owner reviewed its findings. The approved fixes shipped as braid#481. Run 192 checked them in a full run (the table on braid#484): six seen working, two with bugs that braid#485 then fixed (the text-only rule's false positives, the code review never offered its helpers), and two not exercised (the bugs file cleaning, `tested by`). The two findings left alone are named in the comment above.
Sign in to join this conversation.
No labels
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
cmoriarty/braid#480
No description provided.