A planned project still adds a browser spec, and dozens of tests, per change #268

Closed
opened 2026-10-27 20:20:31 -04:00 by cmoriarty · 8 comments
Owner

What happens
#255 told every test-writing brief to extend the project's one browser smoke test instead of adding one per change. The one-shots kept their tests lean (0.3–0.4 test lines per source line). The planned project in the #260 re-run did not:

  • After the first item's e2e/play.spec.ts, five later changes each added their own browser spec: achievements, gamepad, online scores, tables, touch.
  • 6,727 test lines for 4,771 source lines (1.4 per line), 342 unit tests in 56 files over 9 changes.

Better than the first test (2.3 per line, 8 specs), but still a suite that grows with every change.

Fix

  • A change in a project that has a browser smoke test adds its steps to that file. A new spec file needs a reason the brief names.
  • The test-writing step is shown the project's existing tests and their count.

Done when
A planned project of several changes ends with one browser spec and fewer test lines than source lines.

From the #260 re-run, runs 124–132.

**What happens** #255 told every test-writing brief to extend the project's one browser smoke test instead of adding one per change. The one-shots kept their tests lean (0.3–0.4 test lines per source line). The planned project in the #260 re-run did not: - After the first item's `e2e/play.spec.ts`, five later changes each added their own browser spec: achievements, gamepad, online scores, tables, touch. - 6,727 test lines for 4,771 source lines (1.4 per line), 342 unit tests in 56 files over 9 changes. Better than the first test (2.3 per line, 8 specs), but still a suite that grows with every change. **Fix** - A change in a project that has a browser smoke test adds its steps to that file. A new spec file needs a reason the brief names. - The test-writing step is shown the project's existing tests and their count. **Done when** A planned project of several changes ends with one browser spec and fewer test lines than source lines. _From the #260 re-run, runs 124–132._
Author
Owner

Partly addressed by #275 (PR #277): the build session now writes the scenario tests and lints them, and agent.write-spec-tests continues that session, so it sees the tests already there instead of starting cold. Whether a planned project still adds a browser spec per change needs a planned run to tell. Leaving this open until one does.

Partly addressed by #275 (PR #277): the build session now writes the scenario tests and lints them, and `agent.write-spec-tests` continues that session, so it sees the tests already there instead of starting cold. Whether a planned project still adds a browser spec per change needs a planned run to tell. Leaving this open until one does.
Author
Owner

Shipped in #301 (5355515), partly seen on production. Leaving this open: the done-when needs a planned project of several changes, and the only one so far was stopped after its first item.

  • The brief: the test-writing brief now lists the project's tests. Run 139's agent.write-spec-tests read "The project's tests now: 33 in 5 file(s). Add to these files before you make a new one", with each file and its count. When the project has a browser spec, the brief names it to extend, and lint rule B8 (blocking) refuses a second one.
  • After run 139 (cmoriarty/breakout-planned, item 1): 922 source lines in 5 files, 679 test lines in 5 files (0.74 per source line; the pinball planned project ended at 1.4), and no browser spec.
  • A finding: a few lines below the new instruction, the same brief still says to write tests "in test files of your own named after the capability". The two pull against each other.

Done when, unchanged: a planned project of several changes ends with one browser spec and fewer test lines than source lines.

**Shipped in #301 (5355515), partly seen on production.** Leaving this open: the done-when needs a planned project of several changes, and the only one so far was stopped after its first item. - **The brief:** the test-writing brief now lists the project's tests. Run 139's `agent.write-spec-tests` read "The project's tests now: 33 in 5 file(s). Add to these files before you make a new one", with each file and its count. When the project has a browser spec, the brief names it to extend, and lint rule B8 (blocking) refuses a second one. - **After run 139** (cmoriarty/breakout-planned, item 1): 922 source lines in 5 files, 679 test lines in 5 files (0.74 per source line; the pinball planned project ended at 1.4), and no browser spec. - **A finding:** a few lines below the new instruction, the same brief still says to write tests "in test files of your own named after the capability". The two pull against each other. Done when, unchanged: a planned project of several changes ends with one browser spec and fewer test lines than source lines.
Author
Owner

Run 147 (the Breakout project, one change, vLLM): half met. It stays open.

  • Browser specs: none. The change uses node --test only.
  • Test lines: 861 for 610 source lines (1.41 per line). That is worse than run 139's 0.74, and as high as the pinball project this issue came from. 36 tests in 6 files.

Where they came from:

  • The build session (openspec.apply) wrote 538 test lines beside its source: about 0.9 per line.
  • The five agent.write-spec-tests steps, one per capability, added about 317 more: two new files (arcade-look, responsive-display), +120 lines in sound.test.js, +50 in paddle.test.js.

The browser spec per change is gone. What grows the suite now is two stages each writing tests for the same behaviour.

Next fix (to decide):

  • a spec-test step adds only scenarios that no test covers yet, and says which ones it skipped and why; or
  • one owner for scenario tests.

Also: #310's brief line says the other capabilities' steps "write theirs at the same time". They run one after another, so fix that reason too.

Can't show here: the plan now holds one change (#308), so "several changes" needs a longer brief.

**Run 147 (the Breakout project, one change, vLLM): half met.** It stays open. - **Browser specs: none.** The change uses `node --test` only. - **Test lines: 861 for 610 source lines (1.41 per line).** That is worse than run 139's 0.74, and as high as the pinball project this issue came from. 36 tests in 6 files. **Where they came from:** - The build session (`openspec.apply`) wrote 538 test lines beside its source: about 0.9 per line. - The five `agent.write-spec-tests` steps, one per capability, added about 317 more: two new files (`arcade-look`, `responsive-display`), +120 lines in `sound.test.js`, +50 in `paddle.test.js`. The browser spec per change is gone. What grows the suite now is two stages each writing tests for the same behaviour. **Next fix (to decide):** - a spec-test step adds only scenarios that no test covers yet, and says which ones it skipped and why; or - one owner for scenario tests. Also: #310's brief line says the other capabilities' steps "write theirs at the same time". They run one after another, so fix that reason too. **Can't show here:** the plan now holds one change (#308), so "several changes" needs a longer brief.
Author
Owner

The visual list is used as an escape hatch (run 152, production 1fc0f22). The build session listed 28 of the change's 54 scenarios in .osf/test/visual.json, so only one capability had a scenario left for a spec-test step.

  • Some are right: the neon look, a high-density display, a sound being audible.
  • Many are plain logic a unit test checks cheaply and reliably: "Third ball lost ends the game", "Starting the game", "Muting", "Mute does not persist", "Score updates on break", "Wall at game start", and "Arrow keys steer the paddle". That last one is the fault a unit test would have caught; see #269's seeded faults.

That hands logic to the Visual check, which missed 2 of 3 visual faults tonight. Wrong trade.

Fix: a scenario is the Visual check's only when its outcome is appearance alone: look, layout, fit, a sound heard. One whose outcome is a change of state (score, lives, phase, a saved value, a direction) keeps a unit test, however it is shown. The brief says so, and Braid flags a visual entry whose scenario names a state change.

Stays open.

**The visual list is used as an escape hatch (run 152, production 1fc0f22).** The build session listed 28 of the change's 54 scenarios in `.osf/test/visual.json`, so only one capability had a scenario left for a spec-test step. - **Some are right:** the neon look, a high-density display, a sound being audible. - **Many are plain logic** a unit test checks cheaply and reliably: "Third ball lost ends the game", "Starting the game", "Muting", "Mute does not persist", "Score updates on break", "Wall at game start", and "Arrow keys steer the paddle". That last one is the fault a unit test would have caught; see #269's seeded faults. That hands logic to the Visual check, which missed 2 of 3 visual faults tonight. Wrong trade. **Fix:** a scenario is the Visual check's only when its outcome is appearance alone: look, layout, fit, a sound heard. One whose outcome is a change of state (score, lives, phase, a saved value, a direction) keeps a unit test, however it is shown. The brief says so, and Braid flags a visual entry whose scenario names a state change. Stays open.
Author
Owner

And the Visual check skips most of them (run 152). Of the 28 scenarios the build session listed in visual.json, its Visual check judged only 12 by name. Its brief names all 28. The other 16 have no test and no check: among them "Arrow keys steer the paddle", "Third ball lost ends the game", "Starting the game", "Muting", "Un-muting" and the three sounds.

So visual.json as shipped in PR #319 can leave scenarios with no coverage at all.

Fix:

  • the narrower rule above: appearance only, and a state change keeps its unit test;
  • a visual scenario the round's report does not judge counts as uncovered again: it goes back to a spec-test step, or the round fails on it.
**And the Visual check skips most of them (run 152).** Of the 28 scenarios the build session listed in `visual.json`, its Visual check judged only 12 by name. Its brief names all 28. The other 16 have no test and no check: among them "Arrow keys steer the paddle", "Third ball lost ends the game", "Starting the game", "Muting", "Un-muting" and the three sounds. So `visual.json` as shipped in PR #319 can leave scenarios with no coverage at all. **Fix:** - the narrower rule above: appearance only, and a state change keeps its unit test; - a visual scenario the round's report does not judge counts as uncovered again: it goes back to a spec-test step, or the round fails on it.
Author
Owner

Part shipped in PR 362. Staying open: the done-when isn't met or measured.

Shipped (the Visual check half, from run 152, which left 16 scenarios with no test and no check):

  • Only a scenario whose outcome is appearance alone goes to the Visual check. A change of state keeps its unit test, even in a game.
  • A scenario left to the Visual check and not judged now fails the round.

Evidence: tests in tests/pipeline/test_fix_stuck_and_visual_gaps.py, and 2 of 2 Free replays of agent.write-spec-tests[farewell] succeeded, each writing one test. That scenario is plain logic, so it shows the brief still works, not that the new line changes a game's split.

Not done: neither fix in this issue's body (a change adds to the existing browser spec; the step is shown the existing tests and their count), if they are still wanted. Nothing here measures one browser spec or fewer test lines than source lines. That takes a planned project of several changes on a real model.

Part shipped in PR 362. Staying open: the done-when isn't met or measured. **Shipped** (the Visual check half, from run 152, which left 16 scenarios with no test and no check): - Only a scenario whose outcome is appearance alone goes to the Visual check. A change of state keeps its unit test, even in a game. - A scenario left to the Visual check and not judged now fails the round. **Evidence:** tests in `tests/pipeline/test_fix_stuck_and_visual_gaps.py`, and 2 of 2 Free replays of `agent.write-spec-tests[farewell]` succeeded, each writing one test. That scenario is plain logic, so it shows the brief still works, not that the new line changes a game's split. **Not done:** neither fix in this issue's body (a change adds to the existing browser spec; the step is shown the existing tests and their count), if they are still wanted. Nothing here measures one browser spec or fewer test lines than source lines. That takes a planned project of several changes on a real model.
Author
Owner

A correction to my 11-03 comment. It said neither fix in this issue's body had shipped. Both had: a browser spec added beside the project's existing one is blocked (B8), and the test-writing step is shown the project's tests and their count (_project_tests_lines).

Next step, shipped in PR #390: the briefs carry Braid's testing standard, and a scenario can go untested with a reason. Three Free replays of run 172's spec-test step wrote 4, 2 and 4 tests, against 14.

Staying open: the done-when takes a planned project of several changes on a real model, and that has not been run.

**A correction to my 11-03 comment.** It said neither fix in this issue's body had shipped. Both had: a browser spec added beside the project's existing one is blocked (B8), and the test-writing step is shown the project's tests and their count (`_project_tests_lines`). **Next step, shipped in PR #390:** the briefs carry Braid's testing standard, and a scenario can go untested with a reason. Three Free replays of run 172's spec-test step wrote 4, 2 and 4 tests, against 14. Staying open: the done-when takes a planned project of several changes on a real model, and that has not been run.
Author
Owner

Closing as superseded.

Both fixes in the body shipped: a browser spec added beside the project's existing one is blocked (B8, PR #301), and the test-writing brief listed the project's tests and their count. Since #407 (PR #408) no step writes tests to a scenario count: agent.write-spec-tests is gone, and a change's tests come from the build session under Braid's testing standard (PR #390), with a scenario allowed to go untested for a stated reason.

The done-when (a planned project of several changes ending with one browser spec and fewer test lines than source lines) was never measured on a planned project. The next planned project's run is where to measure it; a miss there is a new ticket against what Braid does now, not this one.

**Closing as superseded.** Both fixes in the body shipped: a browser spec added beside the project's existing one is blocked (B8, PR #301), and the test-writing brief listed the project's tests and their count. Since #407 (PR #408) no step writes tests to a scenario count: `agent.write-spec-tests` is gone, and a change's tests come from the build session under Braid's testing standard (PR #390), with a scenario allowed to go untested for a stated reason. The done-when (a planned project of several changes ending with one browser spec and fewer test lines than source lines) was never measured on a planned project. The next planned project's run is where to measure it; a miss there is a new ticket against what Braid does now, not this one.
Sign in to join this conversation.
No labels
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
cmoriarty/braid#268
No description provided.