A planned project still adds a browser spec, and dozens of tests, per change #268
Loading…
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
What happens
#255 told every test-writing brief to extend the project's one browser smoke test instead of adding one per change. The one-shots kept their tests lean (0.3–0.4 test lines per source line). The planned project in the #260 re-run did not:
e2e/play.spec.ts, five later changes each added their own browser spec: achievements, gamepad, online scores, tables, touch.Better than the first test (2.3 per line, 8 specs), but still a suite that grows with every change.
Fix
Done when
A planned project of several changes ends with one browser spec and fewer test lines than source lines.
From the #260 re-run, runs 124–132.
Partly addressed by #275 (PR #277): the build session now writes the scenario tests and lints them, and
agent.write-spec-testscontinues that session, so it sees the tests already there instead of starting cold. Whether a planned project still adds a browser spec per change needs a planned run to tell. Leaving this open until one does.Shipped in #301 (
5355515), partly seen on production. Leaving this open: the done-when needs a planned project of several changes, and the only one so far was stopped after its first item.agent.write-spec-testsread "The project's tests now: 33 in 5 file(s). Add to these files before you make a new one", with each file and its count. When the project has a browser spec, the brief names it to extend, and lint rule B8 (blocking) refuses a second one.Done when, unchanged: a planned project of several changes ends with one browser spec and fewer test lines than source lines.
Run 147 (the Breakout project, one change, vLLM): half met. It stays open.
node --testonly.Where they came from:
openspec.apply) wrote 538 test lines beside its source: about 0.9 per line.agent.write-spec-testssteps, one per capability, added about 317 more: two new files (arcade-look,responsive-display), +120 lines insound.test.js, +50 inpaddle.test.js.The browser spec per change is gone. What grows the suite now is two stages each writing tests for the same behaviour.
Next fix (to decide):
Also: #310's brief line says the other capabilities' steps "write theirs at the same time". They run one after another, so fix that reason too.
Can't show here: the plan now holds one change (#308), so "several changes" needs a longer brief.
The visual list is used as an escape hatch (run 152, production
1fc0f22). The build session listed 28 of the change's 54 scenarios in.osf/test/visual.json, so only one capability had a scenario left for a spec-test step.That hands logic to the Visual check, which missed 2 of 3 visual faults tonight. Wrong trade.
Fix: a scenario is the Visual check's only when its outcome is appearance alone: look, layout, fit, a sound heard. One whose outcome is a change of state (score, lives, phase, a saved value, a direction) keeps a unit test, however it is shown. The brief says so, and Braid flags a visual entry whose scenario names a state change.
Stays open.
And the Visual check skips most of them (run 152). Of the 28 scenarios the build session listed in
visual.json, its Visual check judged only 12 by name. Its brief names all 28. The other 16 have no test and no check: among them "Arrow keys steer the paddle", "Third ball lost ends the game", "Starting the game", "Muting", "Un-muting" and the three sounds.So
visual.jsonas shipped in PR #319 can leave scenarios with no coverage at all.Fix:
Part shipped in PR 362. Staying open: the done-when isn't met or measured.
Shipped (the Visual check half, from run 152, which left 16 scenarios with no test and no check):
Evidence: tests in
tests/pipeline/test_fix_stuck_and_visual_gaps.py, and 2 of 2 Free replays ofagent.write-spec-tests[farewell]succeeded, each writing one test. That scenario is plain logic, so it shows the brief still works, not that the new line changes a game's split.Not done: neither fix in this issue's body (a change adds to the existing browser spec; the step is shown the existing tests and their count), if they are still wanted. Nothing here measures one browser spec or fewer test lines than source lines. That takes a planned project of several changes on a real model.
A correction to my 11-03 comment. It said neither fix in this issue's body had shipped. Both had: a browser spec added beside the project's existing one is blocked (B8), and the test-writing step is shown the project's tests and their count (
_project_tests_lines).Next step, shipped in PR #390: the briefs carry Braid's testing standard, and a scenario can go untested with a reason. Three Free replays of run 172's spec-test step wrote 4, 2 and 4 tests, against 14.
Staying open: the done-when takes a planned project of several changes on a real model, and that has not been run.
Closing as superseded.
Both fixes in the body shipped: a browser spec added beside the project's existing one is blocked (B8, PR #301), and the test-writing brief listed the project's tests and their count. Since #407 (PR #408) no step writes tests to a scenario count:
agent.write-spec-testsis gone, and a change's tests come from the build session under Braid's testing standard (PR #390), with a scenario allowed to go untested for a stated reason.The done-when (a planned project of several changes ending with one browser spec and fewer test lines than source lines) was never measured on a planned project. The next planned project's run is where to measure it; a miss there is a new ticket against what Braid does now, not this one.