Spec-test steps test for coverage; Braid's own testing standard is not in the briefs #376
Loading…
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
What happens
Watching run 172 (Scratch on the MTP vLLM stack),
agent.write-spec-testsspent a long time writing elaborate tests, and thegame-presentationstep took two attempts. The briefs already say the right words (_test_lint_rulesinsrc/osf/engine/prompts.py: "Keep the suite small and fast; a suite people trust is worth more than a large one", "write a test only for a real risk", "ask what would have to break for it to go red"), and the lint rejects mock-only and existence-only tests. Three things work against those words:script.scenarioslists every scenario with no tagged test and fans out one session per capability told to "write a test for each scenario listed below as having none". The only exemption is appearance-only scenarios (visual.json). An agent that follows the brief writes a test per scenario whether or not it protects a risk; "one test may cover several scenarios" is the only counterweight. This is coverage for its own sake, by construction.openspec/specs/test-suite/spec.md); a project Braid builds has no equivalent, so a slow suite is fine as long as no single test sleeps.docs/testing.md(braid#241) andopenspec/config.yamlhold Braid's tests to rules the briefs never state: prefer the real thing over fakes; assert on state and output, not on which function was called; a test is small (no subprocess, network or sleep; 1 s) unless marked; every test names what it protects; never a test that restates the code or exists for a scenario count or for coverage.#268 and #255 are about the suite's size (browser spec per change, test lines per source line); this is about what the tests are for and how long they take.
Fix (proposal)
docs/testing.mdrules into_test_lint_rulesso every test-writing brief carries them: real thing over fakes, assert on state and output, 1 s per test, no subprocess.visual.json, so coverage stops fanning out a step for it.Done when
A run's spec-test steps write tests that each name a risk or a use case, the unit lane's time is in the report, and a scenario can be marked covered-by-design without a test.
Observed on run 172 (
run_01M6K8CA22F0FE8TNG6K6J6PG3), 2026-11-04; briefs read at origin/maind53f9e9d.Shipped in PR #390, production
ddd4b6b.Evidence:
tests/pipeline/test_untested_reaches_people.py. Three Free replays of run 172'sagent.write-spec-tests[game-presentation]: 3.4, 4.4 and 5.4 minutes and 4, 2 and 4 tests, against 40 minutes and 14 tests, on a much faster model. Each test named a scenario with a real state change.