Run the standard test on the fixes of 2026-11-12, and settle what they left open #536

Closed
opened 2026-11-13 15:07:32 -05:00 by cmoriarty · 3 comments
Owner

What happens

Run 195 (braid#505) is the last standard run, and it predates braid#506 and braid#509. No real run has exercised: the unit lane's command joined at a merge, the note for groups side by side, queued helpers, the guard's refusal of Braid's own code, or the several-people browser guidance.

Fix

Start the standard run (./tools/standard-run.sh <server> 4) and measure against run 195 (4 h 49 min, 428 agent minutes):

  • The unit lane runs both suites.
  • The code review works beside one helper: run 195's took 32.4 min, 22 of them waiting.
  • Groups side by side that add to one file: conflicts or none. If the one browser spec file still conflicts, revisit the lint's one-spec-file rule (owner chose guidance alone on 2026-11-12).
  • "Close the browser first": now named in the brief; refusals should stop.
  • The planner's shape: run 195's planner left all browser tests to the last group by its own choice.

Smaller things to settle from the same runs:

  • writes-not-ignored is on the apply steps only; a fix round or a quick fix can still write a code file git ignores.
  • The stall watchdog redoes a step that is silent for 480 s while busy. In run 193 that was a healthy step waiting on the model server (531 s).
  • stop takes a helper's name, not one helper.

Done when

The run has ended, its record is beside run 195's, and each smaller item is fixed or dropped.

**What happens** Run 195 (braid#505) is the last standard run, and it predates braid#506 and braid#509. No real run has exercised: the unit lane's command joined at a merge, the note for groups side by side, queued helpers, the guard's refusal of Braid's own code, or the several-people browser guidance. **Fix** Start the standard run (`./tools/standard-run.sh <server> 4`) and measure against run 195 (4 h 49 min, 428 agent minutes): - The unit lane runs both suites. - The code review works beside one helper: run 195's took 32.4 min, 22 of them waiting. - Groups side by side that add to one file: conflicts or none. If the one browser spec file still conflicts, revisit the lint's one-spec-file rule (owner chose guidance alone on 2026-11-12). - "Close the browser first": now named in the brief; refusals should stop. - The planner's shape: run 195's planner left all browser tests to the last group by its own choice. Smaller things to settle from the same runs: - `writes-not-ignored` is on the apply steps only; a fix round or a quick fix can still write a code file git ignores. - The stall watchdog redoes a step that is silent for 480 s while busy. In run 193 that was a healthy step waiting on the model server (531 s). - `stop` takes a helper's name, not one helper. **Done when** The run has ended, its record is beside run 195's, and each smaller item is fixed or dropped.
Author
Owner

The smaller items of this issue are in pull request 550 (change console-fixes-and-session-cap): the ignored-file check on fix rounds and quick fixes, stop by id, and the stall watchdog on vLLM.

About the watchdog. A request waiting on vLLM already kept a silent step from being redone, so run 193's 531 s silence was a request the server was running, most likely a long prompt it was reading. Run 193 has been deleted from production, so its metric samples could not be read back to confirm it. The fix therefore uses vLLM's own behaviour: a silent step is spared while vLLM's prompt counter rises or its cache fills as it runs a request, with the same three-window bound braid#295 uses for llama.cpp.

The issue stays open for its standard run, which measures this change once it ships.

The smaller items of this issue are in pull request 550 (change `console-fixes-and-session-cap`): the ignored-file check on fix rounds and quick fixes, `stop` by id, and the stall watchdog on vLLM. **About the watchdog.** A request waiting on vLLM already kept a silent step from being redone, so run 193's 531 s silence was a request the server was running, most likely a long prompt it was reading. Run 193 has been deleted from production, so its metric samples could not be read back to confirm it. The fix therefore uses vLLM's own behaviour: a silent step is spared while vLLM's prompt counter rises or its cache fills as it runs a request, with the same three-window bound braid#295 uses for llama.cpp. The issue stays open for its standard run, which measures this change once it ships.
Author
Owner

Run 202 (braid#557) was this issue's run. The owner stopped it at 4 h 15 min, during apply, after it turned up four findings, so it measured only part of what this issue asks:

  • The unit lane runs both suites: its command does. The merge joined the groups' commands into (cd backend && uv run pytest) && (cd frontend && npm run test). No lane ran it before the stop.
  • The code review beside one helper: not reached.
  • Groups side by side that add to one file: they conflict. Two of the four game merges met the spec file and the stylesheet, costing 12.6 min of agent work and a compaction. Lint B8's one-spec-file rule puts every game in that file (braid#557, finding 3).
  • "Close the browser first": 1 refusal, against 3 in run 195.
  • The planner's shape: each game group extended the spec file itself. A chain of three groups held all five games until 76 min into apply (run 195: 33 min).
  • Batch A: the server never had more than 2 requests, with 97.2% of 95.5M prompt tokens from cache. The stall watchdog redid a step that was waiting on its own shell command at 1200 s, before the command's own 1800 s limit, and the redo threw away 72 min of work (braid#557, finding 4).

Still open here: the code review beside one helper, and a lane running both suites. Both need the next standard run, after braid#557's fixes.

Run 202 (braid#557) was this issue's run. The owner stopped it at 4 h 15 min, during apply, after it turned up four findings, so it measured only part of what this issue asks: - **The unit lane runs both suites:** its command does. The merge joined the groups' commands into `(cd backend && uv run pytest) && (cd frontend && npm run test)`. No lane ran it before the stop. - **The code review beside one helper:** not reached. - **Groups side by side that add to one file:** they conflict. Two of the four game merges met the spec file and the stylesheet, costing 12.6 min of agent work and a compaction. Lint B8's one-spec-file rule puts every game in that file (braid#557, finding 3). - **"Close the browser first":** 1 refusal, against 3 in run 195. - **The planner's shape:** each game group extended the spec file itself. A chain of three groups held all five games until 76 min into apply (run 195: 33 min). - **Batch A:** the server never had more than 2 requests, with 97.2% of 95.5M prompt tokens from cache. The stall watchdog redid a step that was waiting on its own shell command at 1200 s, before the command's own 1800 s limit, and the redo threw away 72 min of work (braid#557, finding 4). **Still open here:** the code review beside one helper, and a lane running both suites. Both need the next standard run, after braid#557's fixes.
Author
Owner

Run 203 (braid#565) was this issue's run, after braid#559 fixed run 202's findings. It succeeded in 6 h 1 min on 421 agent minutes (run 195: 4 h 49 min on 428). Its record is on braid#565, beside run 195's.

  • The unit lane runs both suites: yes. Its command, (cd frontend && npx vitest run) && (cd backend && uv run python -m pytest), ran 41 front-end and 51 back-end tests in 11.6 s.
  • The code review beside one helper: yes.
    • It asked for nine helpers in one turn: one started, eight were queued, and it worked on.
    • It never waited inside a helper call. In run 195, that wait took 22 of its 32 minutes.
    • It took 44.9 min (run 195: 32.4), with 12 helpers and 13 fixes.
    • Twice, a second helper ran beside the first, so 3 sessions ran on a backend held to 2 (braid#565, finding 2).
  • Groups side by side adding to one file: none did. The planner chained every group that writes the browser spec file. The merges met one conflict: a README edit against a rewrite, which cost 2.8 min. The join braid#559 added has not met a real conflict yet.
  • "Close the browser first": 1 refusal (run 195: 3).
  • The planner's shape: 8 groups, with two games to a group and those groups chained. The last 142 min of apply ran one group at a time.

The smaller items shipped in pull request 550. Every question here now has its measurement, so I am closing this. What the run found is on braid#565.

Run 203 (braid#565) was this issue's run, after braid#559 fixed run 202's findings. It succeeded in 6 h 1 min on 421 agent minutes (run 195: 4 h 49 min on 428). Its record is on braid#565, beside run 195's. - **The unit lane runs both suites:** yes. Its command, `(cd frontend && npx vitest run) && (cd backend && uv run python -m pytest)`, ran 41 front-end and 51 back-end tests in 11.6 s. - **The code review beside one helper:** yes. - It asked for nine helpers in one turn: one started, eight were queued, and it worked on. - It never waited inside a helper call. In run 195, that wait took 22 of its 32 minutes. - It took 44.9 min (run 195: 32.4), with 12 helpers and 13 fixes. - Twice, a second helper ran beside the first, so 3 sessions ran on a backend held to 2 (braid#565, finding 2). - **Groups side by side adding to one file:** none did. The planner chained every group that writes the browser spec file. The merges met one conflict: a README edit against a rewrite, which cost 2.8 min. The join braid#559 added has not met a real conflict yet. - **"Close the browser first":** 1 refusal (run 195: 3). - **The planner's shape:** 8 groups, with two games to a group and those groups chained. The last 142 min of apply ran one group at a time. The smaller items shipped in pull request 550. Every question here now has its measurement, so I am closing this. What the run found is on braid#565.
Sign in to join this conversation.
No labels
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
cmoriarty/braid#536
No description provided.