The hurdles Braid put in front of its agents in runs 174-179 go (#407) #408

Merged
cmoriarty merged 2 commits from issue-407-run-hurdles into main 2026-11-07 16:22:03 -05:00
Owner

Closes nothing yet: braid#407 closes after the vLLM run named below.

Runs 174 to 179 took 621 agent minutes, apply 233 and the Visual check 168. Every transcript was read (docs/design/measurements/2026-11-07-runs-174-179-efficiency.md); the minutes went to hurdles, not the work. The owner reviewed the proposals one by one; this is the kept set, as the OpenSpec change run-hurdles-174-179.

What changes

  • openspec.apply starts a session of its own; the build chain begins at it.
  • The Visual check's brief carries the app's facts (the window.__app names, the start command and whether it reloads, the screenshots folder), and says to read no source outside the delta's files. Round 1 goes through a whole session only for a change on the app's main flow; round 2 replays only for a file round 1's session evidence named.
  • Every session starts in a fresh browser: Braid's guard refuses a session's first Playwright call unless it is browser_close (the server already runs isolated, so a close is a wipe); each person is opened in an isolatedContext.
  • Braid's clock runs in every page through the browser server's init script (OSF_BRAID_CLOCK=0 leaves it out). The motion probe steps and pauses it with the animations, so a frame holds however many turns a screenshot takes. An animation the specs name is judged from a filmstrip of five captioned frames (osf.engine.filmstrip).
  • The text probe composites translucent backgrounds, reads SVG fill and skips disabled controls.
  • The briefs name no kind of software; the judge report's play field is session (the console reads either); a source-level test holds the text to that and to a size ceiling.
  • The spec tag is documentation: B6 is recorded only, a file-top tag counts, the brief's ban-check is scoped to the run's lines, and agent.write-spec-tests and its three sim scenarios go.
  • Every built-in step runs at the high tier under Auto; qwen3.8-27b's recommended output is 32,000, where production already is.
  • node_modules/.bin on the app tool's PATH; NODE_PATH for $TMPDIR scripts; a glob from the run root into screenshots or tmp lands there; new files pasted into the code review's brief; the hook kept by the fix and review steps.

Verified

  • ./tools/test.sh green: 501 backend tests, 42 journeys, 24 s. New tests for the lint, the guard, the app tool's PATH, the run env, the judge's facts, the browser server's init script, the filmstrip, the clock (journeys), and the briefs' wording and size.
  • ./tools/sim.sh --smoke and judge-requirement-by-name green; solo, apply-single, apply-single-no-progress and free-forks rewritten for the fresh apply.
  • Scripted replay of run 174's apply step: 25 tool calls, 0 errors. Its judge step's replay meets the new guard: 72 of 78 errors are the refusal of a first call that is not a close, as designed; a scripted replay cannot show the new brief's effect.

After merge: one vLLM run, read for apply starting under 20k, no screenshot path hunting, no probe entries on a card face, a round-1 judge of a change off the main flow not going through a whole session, and a filmstrip in a judge report. Then archive and close braid#407.

🤖 Generated with Claude Code

Closes nothing yet: braid#407 closes after the vLLM run named below. Runs 174 to 179 took 621 agent minutes, apply 233 and the Visual check 168. Every transcript was read (`docs/design/measurements/2026-11-07-runs-174-179-efficiency.md`); the minutes went to hurdles, not the work. The owner reviewed the proposals one by one; this is the kept set, as the OpenSpec change `run-hurdles-174-179`. **What changes** - `openspec.apply` starts a session of its own; the build chain begins at it. - The Visual check's brief carries the app's facts (the `window.__app` names, the start command and whether it reloads, the screenshots folder), and says to read no source outside the delta's files. Round 1 goes through a whole session only for a change on the app's main flow; round 2 replays only for a file round 1's session evidence named. - Every session starts in a fresh browser: Braid's guard refuses a session's first Playwright call unless it is `browser_close` (the server already runs isolated, so a close is a wipe); each person is opened in an `isolatedContext`. - Braid's clock runs in every page through the browser server's init script (`OSF_BRAID_CLOCK=0` leaves it out). The motion probe steps and pauses it with the animations, so a frame holds however many turns a screenshot takes. An animation the specs name is judged from a filmstrip of five captioned frames (`osf.engine.filmstrip`). - The text probe composites translucent backgrounds, reads SVG `fill` and skips disabled controls. - The briefs name no kind of software; the judge report's `play` field is `session` (the console reads either); a source-level test holds the text to that and to a size ceiling. - The spec tag is documentation: B6 is recorded only, a file-top tag counts, the brief's `ban-check` is scoped to the run's lines, and `agent.write-spec-tests` and its three sim scenarios go. - Every built-in step runs at the high tier under Auto; `qwen3.8-27b`'s recommended output is 32,000, where production already is. - `node_modules/.bin` on the `app` tool's PATH; `NODE_PATH` for `$TMPDIR` scripts; a glob from the run root into screenshots or tmp lands there; new files pasted into the code review's brief; the hook kept by the fix and review steps. **Verified** - `./tools/test.sh` green: 501 backend tests, 42 journeys, 24 s. New tests for the lint, the guard, the app tool's PATH, the run env, the judge's facts, the browser server's init script, the filmstrip, the clock (journeys), and the briefs' wording and size. - `./tools/sim.sh --smoke` and `judge-requirement-by-name` green; `solo`, `apply-single`, `apply-single-no-progress` and `free-forks` rewritten for the fresh apply. - Scripted replay of run 174's apply step: 25 tool calls, 0 errors. Its judge step's replay meets the new guard: 72 of 78 errors are the refusal of a first call that is not a close, as designed; a scripted replay cannot show the new brief's effect. **After merge:** one vLLM run, read for apply starting under 20k, no screenshot path hunting, no probe entries on a card face, a round-1 judge of a change off the main flow not going through a whole session, and a filmstrip in a judge report. Then archive and close braid#407. 🤖 Generated with [Claude Code](https://claude.com/claude-code)
feat(engine): the hurdles Braid put in front of its agents in runs 174-179 go (#407)
Some checks failed
test / suite (pull_request) Failing after 1m8s
123639ed43
Six runs took 621 agent minutes, 233 in apply and 168 in the Visual check, and the
transcripts show the time going to hurdles rather than the work. One change, reviewed
proposal by proposal with the owner:

- openspec.apply starts a session of its own: its brief carries the change, and the
  forked chain started it at 82-134k, three times over the alarm.
- The Visual check is given the app's facts (the window.__app names, the start command
  and whether it reloads, the screenshots folder) instead of reading 15 files for them;
  round 1 goes through a whole session only for a change on the app's main flow, and
  round 2 replays only for a file round 1's session evidence named.
- Every session starts in a fresh browser: the guard refuses a session's first
  Playwright call unless it is browser_close, which the isolated server makes a wipe;
  each person is opened in an isolated context.
- Braid's clock runs in every page (the browser server's init script), so the motion
  probe holds a frame however many turns a screenshot takes, and an animation the specs
  name is judged from a filmstrip (osf.engine.filmstrip) of five captioned frames.
- The text probe composites translucent backgrounds, reads SVG fill and skips disabled
  controls, which cleared every card face it used to list.
- The briefs name no kind of software: the judge report's play field is session, the
  examples span kinds of software, and a test holds the text to that and to a ceiling.
- The spec tag is documentation: B6 is recorded, a file-top tag counts, the brief's
  ban-check is scoped to the run's lines, and agent.write-spec-tests goes.
- Every built-in step runs at the high tier under Auto; qwen3.8-27b's recommended
  output is 32,000, where production already is.
- node_modules/.bin on the app tool's PATH, NODE_PATH for $TMPDIR scripts, globs from
  the run root into screenshots or tmp land there, new files pasted into the code
  review's brief, the hook kept by the fix and review steps.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
test(filmstrip): the helper's test keeps the machine's environment
All checks were successful
test / suite (pull_request) Successful in 45s
b45b0aa9cb
CI's container names its browsers through PLAYWRIGHT_BROWSERS_PATH, and the test
handed the helper only NODE_PATH and PATH, so Chromium was looked for under
/root/.cache and not found (actions run 364).

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
cmoriarty deleted branch issue-407-run-hurdles 2026-11-07 16:22:03 -05:00
Sign in to join this conversation.
No reviewers
No labels
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
cmoriarty/braid!408
No description provided.