Agents write too many tests #255

Closed
opened 2026-10-27 15:30:58 -04:00 by cmoriarty · 1 comment
Owner

What happens

  • The planned pinball repo has 2.3 lines of test per line of code (4,263 to 1,819).
  • There's a test file per scenario, and a new browser test with every change.
  • All three second fix rounds came from those browser tests, never from a bug in the game.

What we want

  • Put the rules from docs/testing.md into the agents' prompts: a test is written for a risk, names the requirement it protects, and never sleeps, retries or reaches the network.
  • One browser smoke test per project, not one per change.
  • No file per scenario.

Done when
The prompts carry the rules, and a rerun of the pinball brief has fewer lines of test than of code.

From the Braid vs Claude Code pinball test, 2026-10-27 (runs 110–120).

**What happens** - The planned pinball repo has 2.3 lines of test per line of code (4,263 to 1,819). - There's a test file per scenario, and a new browser test with every change. - All three second fix rounds came from those browser tests, never from a bug in the game. **What we want** - Put the rules from `docs/testing.md` into the agents' prompts: a test is written for a risk, names the requirement it protects, and never sleeps, retries or reaches the network. - One browser smoke test per project, not one per change. - No file per scenario. **Done when** The prompts carry the rules, and a rerun of the pinball brief has fewer lines of test than of code. _From the Braid vs Claude Code pinball test, 2026-10-27 (runs 110–120)._
Author
Owner

What shipped
Every brief that writes tests now says:

  • test a real risk and name the requirement it protects
  • never sleep, retry or reach the network
  • one test may cover several scenarios
  • a scenario about look or wording is the judge's
  • extend the project's one browser smoke test instead of adding one per change

Noted on the baseline in #260: the one-shot kept its tests lean, and the bloat came with planning.

Shipped in 5598ddf (OpenSpec change checks-that-match-the-brief, archived).

**What shipped** Every brief that writes tests now says: - test a real risk and name the requirement it protects - never sleep, retry or reach the network - one test may cover several scenarios - a scenario about look or wording is the judge's - extend the project's one browser smoke test instead of adding one per change Noted on the baseline in #260: the one-shot kept its tests lean, and the bloat came with planning. _Shipped in 5598ddf (OpenSpec change `checks-that-match-the-brief`, archived)._
Sign in to join this conversation.
No labels
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
cmoriarty/braid#255
No description provided.