Runs 174-179: the hurdles Braid puts in front of its agents, in one change #407

Closed
opened 2026-11-07 14:54:45 -05:00 by cmoriarty · 1 comment
Owner

What happens. Runs 174 to 179 took 621 agent minutes: apply 233, Visual check 168. Reading every transcript (docs/design/measurements/2026-11-07-runs-174-179-efficiency.md) shows the time goes to hurdles Braid puts in the way, not to the work: a session forked at 82-134k that restarts fresh, a judge that reads 15 files to learn what Braid could tell it, a contrast probe that fails every card face, one browser profile shared by every step and person, screenshots reported at a path read cannot open, a lint that fails an attempt over tag placement, and briefs that say "game" and "play" to a project that is neither.

Fix, one change, reviewed proposal by proposal with the owner on 2026-11-07:

  1. openspec.apply starts fresh, not forked from revise: the brief already carries everything.
  2. The developer's own testing stays unrestricted; the environment around it gets cheap: the real screenshots folder in the brief, which command hot-reloads, read hook state after a flush.
  3. window.__app settled: the code review keeps it, the proposal template asks for it.
  4. The judge's facts in its brief: the hook surface, the start command, the routes, the screenshots folder; no source reads outside the delta.
  5. Text probe: composite rgba backgrounds, read SVG fill, skip disabled elements; a glyph the screenshot shows plainly passes.
  6. The browser reset between steps; one isolated context per person.
  7. Round 1 goes through a whole session only when the change touches the app's main flow; round 2's trigger tightened. Every brief software-agnostic: the sweep table in docs/design/measurements/work/runs-174-179/brief-sweep.md, and the judge's play field becomes session.
  8. Animations judged from a frozen filmstrip: a fake clock in the Playwright init script, five frames per named animation on one labelled image, the probe's timing record, animations disabled on every other screenshot (.../animation-judging.md).
  9. The // spec: tag is documentation: accepted on the test, its describe or the file top; B6 stops blocking; the write-spec-tests fan-out goes; B1, B2 and B8 stay (relates to #268).
  10. The known-model table says 32,000 output for qwen3.8-27b (production already is); the inert efforts: map goes.
  11. node_modules/.bin on the app tool's PATH; $TMPDIR scripts can find modules; the path guard allows glob on the run folder it names; pasted code in the write-spec-tests and code-review briefs.

Done when ./tools/test.sh and ./tools/sim.sh --smoke are green, a scripted replay of run 174's judge and apply steps shows the hurdles gone, and one vLLM run shows: apply starting under 20k, no screenshot path hunting, no probe entries on a card face, a round-1 judge of a change off the main flow not going through a whole session, and a filmstrip in a judge report.

**What happens.** Runs 174 to 179 took 621 agent minutes: apply 233, Visual check 168. Reading every transcript (`docs/design/measurements/2026-11-07-runs-174-179-efficiency.md`) shows the time goes to hurdles Braid puts in the way, not to the work: a session forked at 82-134k that restarts fresh, a judge that reads 15 files to learn what Braid could tell it, a contrast probe that fails every card face, one browser profile shared by every step and person, screenshots reported at a path `read` cannot open, a lint that fails an attempt over tag placement, and briefs that say "game" and "play" to a project that is neither. **Fix**, one change, reviewed proposal by proposal with the owner on 2026-11-07: 1. `openspec.apply` starts fresh, not forked from revise: the brief already carries everything. 2. The developer's own testing stays unrestricted; the environment around it gets cheap: the real screenshots folder in the brief, which command hot-reloads, read hook state after a flush. 3. `window.__app` settled: the code review keeps it, the proposal template asks for it. 4. The judge's facts in its brief: the hook surface, the start command, the routes, the screenshots folder; no source reads outside the delta. 5. Text probe: composite rgba backgrounds, read SVG `fill`, skip disabled elements; a glyph the screenshot shows plainly passes. 6. The browser reset between steps; one isolated context per person. 7. Round 1 goes through a whole session only when the change touches the app's main flow; round 2's trigger tightened. Every brief software-agnostic: the sweep table in `docs/design/measurements/work/runs-174-179/brief-sweep.md`, and the judge's `play` field becomes `session`. 8. Animations judged from a frozen filmstrip: a fake clock in the Playwright init script, five frames per named animation on one labelled image, the probe's timing record, animations disabled on every other screenshot (`.../animation-judging.md`). 9. The `// spec:` tag is documentation: accepted on the test, its describe or the file top; B6 stops blocking; the write-spec-tests fan-out goes; B1, B2 and B8 stay (relates to #268). 10. The known-model table says 32,000 output for qwen3.8-27b (production already is); the inert `efforts:` map goes. 11. `node_modules/.bin` on the `app` tool's PATH; `$TMPDIR` scripts can find modules; the path guard allows `glob` on the run folder it names; pasted code in the write-spec-tests and code-review briefs. **Done when** `./tools/test.sh` and `./tools/sim.sh --smoke` are green, a scripted replay of run 174's judge and apply steps shows the hurdles gone, and one vLLM run shows: apply starting under 20k, no screenshot path hunting, no probe entries on a card face, a round-1 judge of a change off the main flow not going through a whole session, and a filmstrip in a judge report.
Author
Owner

Shipped in PR #408 (merge 14a3a2db), archived as 2026-11-07-run-hurdles-174-179.

What shipped

  • openspec.apply starts a session of its own; the build chain begins at it.
  • The Visual check's brief carries the app's facts (the window.__app names and their file, the start command and whether it reloads, the screenshots folder) and reads no source outside the delta. Round 1 goes through a whole session only for a change on the app's main flow; round 2 replays only for a file round 1's session evidence named.
  • Every session starts in a fresh browser: the guard refuses a session's first Playwright call unless it is browser_close; each person is an isolatedContext.
  • Braid's clock in every page (the browser server's init script); the motion probe steps and pauses it with the animations; an animation the specs name is judged from a five-frame filmstrip (osf.engine.filmstrip).
  • The text probe composites translucent backgrounds, reads SVG fill and skips disabled controls.
  • Software-agnostic briefs, held by a test; the judge report's play field is session.
  • The spec tag is documentation: B6 recorded, file-top tags count, ban-check --base, no agent.write-spec-tests.
  • Every built-in step at the high tier under Auto; qwen3.8-27b recommended output 32,000 (production raised the same day).
  • node_modules/.bin on the app tool's PATH, NODE_PATH for $TMPDIR scripts, run-root globs land in screenshots or tmp, new files pasted into the code review's brief, the hook kept by fix and review.

Evidence from the laptop

  • ./tools/test.sh green (501 backend tests, 42 journeys); new tests for the lint, the guard, PATH, the run env, the judge's facts, the init script, the filmstrip, the clock, and the briefs' wording and size, each failing before its fix.
  • ./tools/sim.sh --smoke and the judge scenario green; four scenarios rewritten for the fresh apply.
  • Scripted replay of run 174's apply step: 25 tool calls, 0 errors. Its judge step's replay meets the new guard on its first browser call, as designed.

The next thorough run on vLLM is read for the five signs in Done when; anything it exposes is a new issue.

Shipped in PR #408 (merge 14a3a2db), archived as `2026-11-07-run-hurdles-174-179`. **What shipped** - `openspec.apply` starts a session of its own; the build chain begins at it. - The Visual check's brief carries the app's facts (the `window.__app` names and their file, the start command and whether it reloads, the screenshots folder) and reads no source outside the delta. Round 1 goes through a whole session only for a change on the app's main flow; round 2 replays only for a file round 1's session evidence named. - Every session starts in a fresh browser: the guard refuses a session's first Playwright call unless it is `browser_close`; each person is an `isolatedContext`. - Braid's clock in every page (the browser server's init script); the motion probe steps and pauses it with the animations; an animation the specs name is judged from a five-frame filmstrip (`osf.engine.filmstrip`). - The text probe composites translucent backgrounds, reads SVG `fill` and skips disabled controls. - Software-agnostic briefs, held by a test; the judge report's `play` field is `session`. - The spec tag is documentation: B6 recorded, file-top tags count, `ban-check --base`, no `agent.write-spec-tests`. - Every built-in step at the high tier under Auto; `qwen3.8-27b` recommended output 32,000 (production raised the same day). - `node_modules/.bin` on the `app` tool's PATH, `NODE_PATH` for `$TMPDIR` scripts, run-root globs land in screenshots or tmp, new files pasted into the code review's brief, the hook kept by fix and review. **Evidence from the laptop** - `./tools/test.sh` green (501 backend tests, 42 journeys); new tests for the lint, the guard, PATH, the run env, the judge's facts, the init script, the filmstrip, the clock, and the briefs' wording and size, each failing before its fix. - `./tools/sim.sh --smoke` and the judge scenario green; four scenarios rewritten for the fresh apply. - Scripted replay of run 174's apply step: 25 tool calls, 0 errors. Its judge step's replay meets the new guard on its first browser call, as designed. The next thorough run on vLLM is read for the five signs in Done when; anything it exposes is a new issue.
Sign in to join this conversation.
No labels
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
cmoriarty/braid#407
No description provided.