A spec-test step that touches a source file ends the run, with no retry #368

Closed
opened 2026-11-03 10:31:40 -05:00 by cmoriarty · 1 comment
Owner

What happens
Run 170 (Scratch, on Free) failed about 64 minutes in, with 45 steps skipped, because of one word. agent.write-spec-tests[game-rules] added export to a private helper in shared/rules.ts so a test could call it. The tests-only check failed, and the step was not tried again. Its work was otherwise good: 22 tests for 21 scenarios, one left failing on purpose for a real gap (createGame doesn't validate player count).

Why
The step has mediate: missing, so a failed check goes to a person unless the report is missing or malformed, or it only left files behind (#150). A source edit in this step goes to a person, and with auto-approve on the run just ends. This isn't specific to Free: any backend's run can end this way.

Fix

  • A step that fails only tests-only gets one retry, whose brief names the file and says to restore it and test through the public interface.
  • The test-writing briefs say that plainly up front: don't export a helper to test it.

Done when

  • A test: a tests-only failure gets a retry under mediate: missing, and other failed checks still go to a person.
  • A sim: a step edits a source file, restores it on the retry, and the run goes on.
  • A replay of this step on Free (it needs capturing while run 170 exists).
**What happens** Run 170 (Scratch, on Free) failed about 64 minutes in, with 45 steps skipped, because of one word. `agent.write-spec-tests[game-rules]` added `export` to a private helper in `shared/rules.ts` so a test could call it. The `tests-only` check failed, and the step was not tried again. Its work was otherwise good: 22 tests for 21 scenarios, one left failing on purpose for a real gap (`createGame` doesn't validate player count). **Why** The step has `mediate: missing`, so a failed check goes to a person unless the report is missing or malformed, or it only left files behind (#150). A source edit in this step goes to a person, and with auto-approve on the run just ends. This isn't specific to Free: any backend's run can end this way. **Fix** - A step that fails only `tests-only` gets one retry, whose brief names the file and says to restore it and test through the public interface. - The test-writing briefs say that plainly up front: don't export a helper to test it. **Done when** - A test: a `tests-only` failure gets a retry under `mediate: missing`, and other failed checks still go to a person. - A sim: a step edits a source file, restores it on the retry, and the run goes on. - A replay of this step on Free (it needs capturing while run 170 exists).
Author
Owner

Shipped in PR 371.

  • A spec-test step that edits code no longer fails the run. The engine puts the code back when tests-only fails, retries, and tells the agent what it restored.
  • The brief says not to change code in order to test it, an added export included.

Evidence:

  • sim/spec-tests-edit-restored and sim/spec-tests-edit-retry-skipped failed without the fix and pass with it. The second is why a plain retry wasn't enough: its condition skipped the step with the edit left in.
  • Run 170's own step, replayed on Free: succeeded in 1.6 min, 22 requests, no errors, no code changed. The model wrote 20 tests. Nineteen pass, and one fails because createGame doesn't reject a bad player count, and it left that test in.

The replay tool had a bug of its own, fixed in the same PR: a project on another Playwright failed its dependency step on the image's read-only browsers.

**Shipped in [PR 371](https://forgejo.underthere.xyz/cmoriarty/braid/pulls/371).** - A spec-test step that edits code no longer fails the run. The engine puts the code back when `tests-only` fails, retries, and tells the agent what it restored. - The brief says not to change code in order to test it, an added `export` included. **Evidence:** - `sim/spec-tests-edit-restored` and `sim/spec-tests-edit-retry-skipped` failed without the fix and pass with it. The second is why a plain retry wasn't enough: its condition skipped the step with the edit left in. - Run 170's own step, replayed on Free: succeeded in 1.6 min, 22 requests, no errors, no code changed. The model wrote 20 tests. Nineteen pass, and one fails because `createGame` doesn't reject a bad player count, and it left that test in. The replay tool had a bug of its own, fixed in the same PR: a project on another Playwright failed its dependency step on the image's read-only browsers.
Sign in to join this conversation.
No labels
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
cmoriarty/braid#368
No description provided.