A spec-test attempt discarded at 22 min for an npm install the brief forbids, then a cold retry and a merge conflict on the shared vitest config line #384

Closed
opened 2026-11-04 17:31:20 -05:00 by cmoriarty · 1 comment
Owner

What happens
agent.write-spec-tests[game-presentation] in run 172 took 40 min for 14 tests. Attempt 1 reasoned "Infrastructure work (allowed: ... dependencies into manifests)", the opposite of the brief ("When a test needs more (a dependency...), leave that test out"), ran npm install -D jsdom @testing-library/... at 12.6 min, spent 2.5 min debugging jsdom's localStorage under Node 26, and was rejected by tests-only at 22.2 min for package.json/package-lock.json. Attempt 2 opened in a new session and spent 10 min re-reading everything, then rewrote all four files DOM-free with a 168-line regex-over-HTML helper. Then a merge conflict: game-rooms had added *.test.ts and presentation *.test.{ts,tsx} to the same line of vitest.config.ts (1.1 min).

Fix

  • Guard manifest writes at the time of the call (refuse npm install/uv pip install and edits to manifests in a tests-only step), instead of discovering them at the end.
  • On a predicate retry, fork the failed session with the rejection, as revise forks propose, so the retry keeps the model's reading.
  • Put the client test glob (*.test.{ts,tsx}) and a Node 26 localStorage shim in the scaffold the apply step writes, so parallel test steps stop editing the same line.

Done when
A tests-only step cannot spend 20 min on a change its check will reject, and a retry starts from what the first attempt read. ~32 min on run 172 (braid#376 covers what the tests should be).

From the run 172 transcript review (Scratch on vLLM + MTP k=4, 2026-11-04; report in the owner's ~/repos/vLLM/reports/). Related: braid#378, braid#379.

**What happens** `agent.write-spec-tests[game-presentation]` in run 172 took 40 min for 14 tests. Attempt 1 reasoned "Infrastructure work (allowed: ... dependencies into manifests)", the opposite of the brief ("When a test needs more (a dependency...), leave that test out"), ran `npm install -D jsdom @testing-library/...` at 12.6 min, spent 2.5 min debugging jsdom's `localStorage` under Node 26, and was rejected by `tests-only` at 22.2 min for `package.json`/`package-lock.json`. Attempt 2 opened in a new session and spent 10 min re-reading everything, then rewrote all four files DOM-free with a 168-line regex-over-HTML helper. Then a merge conflict: game-rooms had added `*.test.ts` and presentation `*.test.{ts,tsx}` to the same line of `vitest.config.ts` (1.1 min). **Fix** - Guard manifest writes at the time of the call (refuse `npm install`/`uv pip install` and edits to manifests in a tests-only step), instead of discovering them at the end. - On a predicate retry, fork the failed session with the rejection, as revise forks propose, so the retry keeps the model's reading. - Put the client test glob (`*.test.{ts,tsx}`) and a Node 26 `localStorage` shim in the scaffold the apply step writes, so parallel test steps stop editing the same line. **Done when** A tests-only step cannot spend 20 min on a change its check will reject, and a retry starts from what the first attempt read. ~32 min on run 172 (braid#376 covers what the tests should be). _From the run 172 transcript review (Scratch on vLLM + MTP k=4, 2026-11-04; report in the owner's `~/repos/vLLM/reports/`). Related: braid#378, braid#379._
Author
Owner

Shipped in PR #390, production ddd4b6b.

  • A tests-only step is refused a package install or a manifest edit at the call, not after the attempt.
  • Its retry continues the rejected session, so it keeps what it read.

Evidence: tests/agent/test_path_guard.py, and sim spec-tests-edit-restored now expects the fork. None of the three Free replays of the spec-test step tried to add a dependency, so the refusal did not fire there and rests on its test.

**Shipped** in PR #390, production ddd4b6b. - A tests-only step is refused a package install or a manifest edit at the call, not after the attempt. - Its retry continues the rejected session, so it keeps what it read. **Evidence:** `tests/agent/test_path_guard.py`, and sim `spec-tests-edit-restored` now expects the fork. None of the three Free replays of the spec-test step tried to add a dependency, so the refusal did not fire there and rests on its test.
Sign in to join this conversation.
No labels
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
cmoriarty/braid#384
No description provided.