Spec-test steps test for coverage; Braid's own testing standard is not in the briefs #376

Closed
opened 2026-11-04 11:12:46 -05:00 by cmoriarty · 1 comment
Owner

What happens
Watching run 172 (Scratch on the MTP vLLM stack), agent.write-spec-tests spent a long time writing elaborate tests, and the game-presentation step took two attempts. The briefs already say the right words (_test_lint_rules in src/osf/engine/prompts.py: "Keep the suite small and fast; a suite people trust is worth more than a large one", "write a test only for a real risk", "ask what would have to break for it to go red"), and the lint rejects mock-only and existence-only tests. Three things work against those words:

  1. The step is driven by scenario coverage. script.scenarios lists every scenario with no tagged test and fans out one session per capability told to "write a test for each scenario listed below as having none". The only exemption is appearance-only scenarios (visual.json). An agent that follows the brief writes a test per scenario whether or not it protects a risk; "one test may cover several scenarios" is the only counterweight. This is coverage for its own sake, by construction.
  2. The generated project gets no run-time budget. Braid's own suite fails CI past 180 s (openspec/specs/test-suite/spec.md); a project Braid builds has no equivalent, so a slow suite is fine as long as no single test sleeps.
  3. Braid's own standard is stricter than what it tells its agents. docs/testing.md (braid#241) and openspec/config.yaml hold Braid's tests to rules the briefs never state: prefer the real thing over fakes; assert on state and output, not on which function was called; a test is small (no subprocess, network or sleep; 1 s) unless marked; every test names what it protects; never a test that restates the code or exists for a scenario count or for coverage.

#268 and #255 are about the suite's size (browser spec per change, test lines per source line); this is about what the tests are for and how long they take.

Fix (proposal)

  • Put the docs/testing.md rules into _test_lint_rules so every test-writing brief carries them: real thing over fakes, assert on state and output, 1 s per test, no subprocess.
  • Let the spec-tests brief say that a scenario may be left untested with a stated reason (it restates the code, or an existing test already fails when it breaks), recorded like visual.json, so coverage stops fanning out a step for it.
  • Give the generated suite a budget: the unit lane's wall time is recorded in the lane report, and the brief names a target (a minute, say), so an agent sees a slow suite as a finding.

Done when
A run's spec-test steps write tests that each name a risk or a use case, the unit lane's time is in the report, and a scenario can be marked covered-by-design without a test.

Observed on run 172 (run_01M6K8CA22F0FE8TNG6K6J6PG3), 2026-11-04; briefs read at origin/main d53f9e9d.

**What happens** Watching run 172 (Scratch on the MTP vLLM stack), `agent.write-spec-tests` spent a long time writing elaborate tests, and the `game-presentation` step took two attempts. The briefs already say the right words (`_test_lint_rules` in `src/osf/engine/prompts.py`: "Keep the suite small and fast; a suite people trust is worth more than a large one", "write a test only for a real risk", "ask what would have to break for it to go red"), and the lint rejects mock-only and existence-only tests. Three things work against those words: 1. **The step is driven by scenario coverage.** `script.scenarios` lists every scenario with no tagged test and fans out one session per capability told to "write a test for each scenario listed below as having none". The only exemption is appearance-only scenarios (`visual.json`). An agent that follows the brief writes a test per scenario whether or not it protects a risk; "one test may cover several scenarios" is the only counterweight. This is coverage for its own sake, by construction. 2. **The generated project gets no run-time budget.** Braid's own suite fails CI past 180 s (`openspec/specs/test-suite/spec.md`); a project Braid builds has no equivalent, so a slow suite is fine as long as no single test sleeps. 3. **Braid's own standard is stricter than what it tells its agents.** `docs/testing.md` (braid#241) and `openspec/config.yaml` hold Braid's tests to rules the briefs never state: prefer the real thing over fakes; assert on state and output, not on which function was called; a test is small (no subprocess, network or sleep; 1 s) unless marked; every test names what it protects; never a test that restates the code or exists for a scenario count or for coverage. #268 and #255 are about the suite's size (browser spec per change, test lines per source line); this is about what the tests are for and how long they take. **Fix (proposal)** - Put the `docs/testing.md` rules into `_test_lint_rules` so every test-writing brief carries them: real thing over fakes, assert on state and output, 1 s per test, no subprocess. - Let the spec-tests brief say that a scenario may be left untested with a stated reason (it restates the code, or an existing test already fails when it breaks), recorded like `visual.json`, so coverage stops fanning out a step for it. - Give the generated suite a budget: the unit lane's wall time is recorded in the lane report, and the brief names a target (a minute, say), so an agent sees a slow suite as a finding. **Done when** A run's spec-test steps write tests that each name a risk or a use case, the unit lane's time is in the report, and a scenario can be marked covered-by-design without a test. _Observed on run 172 (`run_01M6K8CA22F0FE8TNG6K6J6PG3`), 2026-11-04; briefs read at origin/main d53f9e9d._
Author
Owner

Shipped in PR #390, production ddd4b6b.

  • The briefs that write tests carry Braid's own standard.
  • A scenario can go untested with a reason, which the code review and the PR description show.
  • The PR description names a lane past 60 s.

Evidence: tests/pipeline/test_untested_reaches_people.py. Three Free replays of run 172's agent.write-spec-tests[game-presentation]: 3.4, 4.4 and 5.4 minutes and 4, 2 and 4 tests, against 40 minutes and 14 tests, on a much faster model. Each test named a scenario with a real state change.

**Shipped** in PR #390, production ddd4b6b. - The briefs that write tests carry Braid's own standard. - A scenario can go untested with a reason, which the code review and the PR description show. - The PR description names a lane past 60 s. **Evidence:** `tests/pipeline/test_untested_reaches_people.py`. Three Free replays of run 172's `agent.write-spec-tests[game-presentation]`: 3.4, 4.4 and 5.4 minutes and 4, 2 and 4 tests, against 40 minutes and 14 tests, on a much faster model. Each test named a scenario with a real state change.
Sign in to join this conversation.
No labels
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
cmoriarty/braid#376
No description provided.