Run 189: findings after the run 188 batch #480
Loading…
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
What happens: run 189 re-runs
cmoriarty/scratch-vllm-fp8#16(the run 187/188 change) on productionedfc38b, after the run 188 batch (braid#474). Thorough, auto-approve, vLLM qwen3.8-27b, every step atxhigh(run 188's Visual check ran atmedium).This issue: the run's findings as it goes: whether each #474 fix works, problems, and possible efficiency gains. One comment per area, kept up to date.
Done when: the run has ended and its findings are reviewed with the owner.
Run 189: the #474 fixes, checked as the run reaches them
Depends on(brief 61,578 chars; run 188's group 1: 55,839 for a smaller plan)tested_byentries used, none rejected$TMPDIR/briefs/…; agents read it instead of searching.osf//var/lib/openspecflow/runs/189/tmp/briefs/openspec.apply-3.1.md, and its second call read ittestran the whole suite (168 tests) before any compaction. The fix reached only the reminder after a compaction; the brief itself still says both (see Problems).osf/bugs.jsonentries;plan.bugs.*draft and filereaderhelpers per area; no 5-minute alarmreadernever called; found and fixed 4 real defectsgame-table.test.tsxnamed and read/api/shelloutput read and stop work on 2.0.30; no 30-minute holdpwa.test.ts3,styles.test.ts5), none failed the roundstopused, and it stops the helperxhigh, the Visual check includedxhighSteps against run 188
testruns were whole-suite)visual.json, retry 3.5 minuntested.jsonwith whygame-rules.test.tsbecause it importsHand, which it changedmediumxhigh, 2 compactions, 25 of 25, no fix roundProblems
B8 counts
playwright.config.tsas a browser test and fails the project's first smoke test. Group 1 addedpackages/client/e2e/playwright.config.tsandpackages/client/e2e/smoke.spec.tsto a project with no browser test.is_browser_spectakes any file it treats as a test file that imports@playwright/test, so the config counts; with none on the base, #471's B8 keeps the first new one by name, the config, and failssmoke.spec.tsas "no browser spec file beside the project's existing one". Attempt 1 failed on it at the end of its work, and attempt 2's agent cannot see why ("this is a weird detection… why is playwright.config.ts being labeled the browser spec?"). Fix: a Playwright config (playwright.config.*) is never a spec, and B8 keeps a*.spec.*file first. A sim with a config beside the first spec. Blocks group 1, in every project that adds its first smoke test with a config ine2e/. Attempt 2 then found the bug by reading Braid's own source (next item), and worked around it by takingdefineConfigfrom@playwright/testout of the config: a correct file made worse (it loses its types) to get past a Braid bug.The guard refuses running Braid's package but not reading it. Group 1's attempt 2: "I was told 'Braid's own package is not yours to run or read.' However, the grep worked", then quoted
_TEST_FILE_RE … _TEST_CONFIG_REfrom apply.py. Either let agents read the lint (it explained the finding when Braid's message did not), or refuse the read as well; and make B8's message name the file it kept, which would have saved the search.Explore is told a missing
openspec/folder is expected, in a repository that has one. The line (prompts.py:183, from run 157) goes to every explore. Run 189's explore: "the instructions say 'repositories without anopenspec/folder are expected…' But there IS an openspec folder here. Let me take a quick peek — no, actually the instructions say not to". Fix: send it only when the worktree has noopenspec/.A long MODIFIED requirement passes the change's check and fails the main spec's after archive. OpenSpec's own instruction says the 500-character warning applies to "ADDED requirements and the main spec"; run 189's planner read that as MODIFIED being exempt ("So MODIFIED descriptio[ns]…"). But a MODIFIED requirement becomes the main spec's text:
openspec validate --stricton the change passes it, and on the specs after archive it fails (it did on #474's own archive this morning). Fix: Braid'sopenspec.validatealso holds MODIFIED requirements to 500 characters, so the planner hears it before the review, not at archive.The review's brief says "
.osf/review/already exists", and in round one it does not. prompts.py:1299; run 189's combined review: "The instruction says.osf/review/already exists but it doesn't." Fix: the engine creates the folder before briefing the review, so the line holds (it is there to save a listing).The seed task (from #474) has no scenario to tag its test with. The planner added "a server seed" to group 1 but no scenario in the delta spec; group 1, held to "tag each test with the scenario it covers", hunted for one: "the delta spec for this change (game-lobby) doesn't have a seed scenario… For the seed test, the honest answer is: it protects the determinism the design relies on". Fix: the seed clause asks for a scenario with the task ("a scenario that the same seed deals the same game"), so the tag has a home.
Group 1 searched for Braid's lint source to learn what it enforces:
python3 -c "import osf.pipeline.apply"and thenfind /, refused by the guard (also run 188's group 1). The brief lists the rules; the agent wanted to read how. Watch whether it recurs; the cost here was one refused call.The apply brief contradicts itself on the suite. Its delegation line: "
test: suites, typechecks, lints and builds run here… the project's own fast test command"; its checks section: "run only the tests you wrote or changed, not the whole suite: the lanes run it". Group 1 ran all 168 tests throughtest. #474 changed only the reminder after a compaction. Fix: the delegation line says "the tests you changed, typechecks, lints and builds", one source of the rule.A later group's brief does not give the browser lane's declared command. It names the unit lane's ("the unit lane runs
npm test, the command in.osf/test/unit.command") but for the browser lane only says how to declare one, though group 1 had writtennpx playwright test --config packages/client/e2e/playwright.config.ts. Group 5's own task then says "verify thatnpx playwright test --listlists the test", which from the root finds 0 tests, and it spent many turns on working directories and--configbefore finding it. Group 6 then hit the same wall ("Total: 0 tests in 0 files"). Fix: when.osf/test/e2e.commandexists, the brief says "the browser lane runs<it>" as it does for the unit lane; and a plan's verification commands use the lane's command.A miss against the change's own spec went into the bugs file as if it were outside the work. Group 6 found phone played cards are 48 px where the scenario "Cards render at legible sizes" says at least 56 (set by an earlier group), judged it risky to change, guessed "belongs to group 7", and wrote it to
.osf/bugs.json. The file is for bugs outside the change. Here the change is failing its own requirement, and a filed ticket would let that ship. To check: whether group 7, the code review or the Visual check catches it.Browser tests are written and ticked without ever being run. The plan's tasks say to verify a browser test with
npx playwright test --listand a typecheck, and--listonly loads the file. Group 5's round-end test has a plainReferenceError(game()called inside a page script), so it could never have passed; group 6 ran no browser test at all; the lane would have been the first to run them. Group 7 found it by running the suite, then argued with its scope: "Work ONLY on the tasks in group 7" against "fixing what your own change broke", which does not cover a bug another group left. The planner wrote--listbecause it reasoned "Braid runs the actual browser check after the final group". The whole browser suite takes 18 s. Running these tests exposed three real faults in the game: a render crash on an empty table, a round summary skipped when updates arrive together, and the deal animation leaking under the summary.A bug fixed later is still filed. Group 7 raised phone played cards to 56 px, and scratch-vllm-fp8#20 was filed saying they are 48. Nothing rechecks
.osf/bugs.jsonbefore filing, and that entry was the change's own requirement, not an outside bug. Ticket titles are also the first 90 characters of the description, cut mid-sentence.The Playwright config carries a comment about Braid's lint, explaining why it avoids
defineConfig: the B8 bug has left a workaround in the project's code.A refusal message misleads. A read of a path inside the run's own
cache/ms-playwrightwas refused with "check the path you asked for against" the run's folder, which already contains it.Group 3 put a fixed wait inside
page.evaluateand argued it is not one. Its geometry probe waits a 400 ms minimum and caps at 3 s, in the page: "it's browser-side code, not a test wait… the 400ms is a guard". B10 (#474) seeswaitForTimeoutand sleeps in the test's own code, not asetTimeout/setIntervaldeadline inside an evaluated function. The rule's intent (no timing in tests) is dodged with the rule quoted. Fix: B10 also looks insidepage.evaluate/waitForFunctionbodies for a timer with a literal delay; the real fix is the state hook in Efficiency. Update: the mergedsmoke.spec.tsno longer contains it.Minor: a retry's brief says both "check what you find rather than trusting it" (prompts.py:115, "You have been here before") and "do not repeat finished checks" (mediation.py:781). Group 1's attempt 2: "The instructions are a bit contradictory". One line would do: check what the failure names; leave what passed.
Efficiency, propose: explore read more than in run 188 (94k against 72k), which put its last request over half the alarm, so propose lost the fork and re-read the brief in a fresh session. Not wrong by the rule, but the fork is what saves propose its context; worth watching whether explore's reading grew because "Open at most 15 files" went (#471).
Console (from the owner)
+ Thought <its first line>in the thinking colour (text-thinking, Transcript.tsx), and the step's last words look the same, so what the agent says to the person reads as hidden reasoning. Owner: it should read as real output, in the text colour. To compare with how opencode 2 shows a session with thinking hidden before choosing.ASK_PLACEHOLDER(ui/src/lib/ask.ts) reads "Ask about this run"; only the button's tooltip says "without steering its agent". Owner: a short phrase, not a sentence. Proposed: "Ask about this run — the agent carries on".composerHint.tsreads "Ask for a change, or tell it what you found…", which can be taken for the Ask mode. Proposed: "Request a change, or tell it what you found…".Efficiency
setInterval(…, 1)recorder in the page ("a 1ms interval means the recorder runs 1000 times per second"). The seed (#474) fixes the deal, not the timing. Run 188's group 8 ended by adding anonStateChangehook to the app. Fix, of the seed rule's kind: when the plan has a browser test of moments in a flow, a task gives the app a hook that reports each state as it changes (or a test setting that holds each move), before that test's task. Looks have the same problem: group 5 (like run 188's group 4) patched the page'ssetTimeout(2,500 → 60,000 ms) to hold the round summary long enough to screenshot; a hook that pauses, or a setting that holds each transient state, would serve the looks and the Visual check too.The owner went through these findings one at a time. The approved fixes are filed as braid#481. Left alone: a scenario for the seed test (harmless note, no preference given), and a stricter guard on reading Braid's code (a help file per check replaces the need).
Closing, done as the issue defines it: run 189 ended (succeeded, 660 min) and the owner reviewed its findings.
The approved fixes shipped as braid#481. Run 192 checked them in a full run (the table on braid#484): six seen working, two with bugs that braid#485 then fixed (the text-only rule's false positives, the code review never offered its helpers), and two not exercised (the bugs file cleaning,
tested by). The two findings left alone are named in the comment above.