The untested hatch takes "verified in a browser" as a reason, and a whole client as untestable by design #441

Closed
opened 2026-11-08 15:59:41 -05:00 by cmoriarty · 1 comment
Owner

What happens. Run 183's apply left 49 scenarios, every lobby, table and end-screen scenario, in untested.json with reasons of the form "Verified in a real browser: …", under a design decision that "the React client has no automated unit or integration test suite by design". Run 182 put 11 such scenarios in visual.json; #431 added a reason check there. The untested hatch has none, so the same reasons pass through it (braid#439, finding 7). The apply brief's untested rule says a scenario may be left untested with a reason; it does not say what a reason is.

Fix. The untested rule says what the visual one now says: having looked at it yourself is not a reason, a scenario with a testable outcome keeps its test, and a design decision does not excuse a whole client (a component test or a scripted check through the state hook is a test). untested_scenarios applies the same reason check as visual_lists, and a rejected entry stays uncovered and is named in "Where the scenarios stand". The design brief says the same about a decision not to test a layer.

Done when. A unit test on run 183's untested.json rejects its 18 entries and keeps one reasoned by what a test would restate; the brief tests hold the sentences; the next Scratch run's apply tags its lobby and table scenarios.

From run 183 (braid#439). Follows #431.

**What happens.** Run 183's apply left 49 scenarios, every lobby, table and end-screen scenario, in `untested.json` with reasons of the form "Verified in a real browser: …", under a design decision that "the React client has no automated unit or integration test suite by design". Run 182 put 11 such scenarios in `visual.json`; #431 added a reason check there. The untested hatch has none, so the same reasons pass through it (braid#439, finding 7). The apply brief's untested rule says a scenario may be left untested with a reason; it does not say what a reason is. **Fix.** The untested rule says what the visual one now says: having looked at it yourself is not a reason, a scenario with a testable outcome keeps its test, and a design decision does not excuse a whole client (a component test or a scripted check through the state hook is a test). `untested_scenarios` applies the same reason check as `visual_lists`, and a rejected entry stays uncovered and is named in "Where the scenarios stand". The design brief says the same about a decision not to test a layer. **Done when.** A unit test on run 183's `untested.json` rejects its 18 entries and keeps one reasoned by what a test would restate; the brief tests hold the sentences; the next Scratch run's apply tags its lobby and table scenarios. From run 183 (braid#439). Follows #431.
Author
Owner

Fixed in PR 448.

What changed.

  • The untested file gets a reason check of its own. A reason that says someone looked (verified, confirmed, inspected… in a browser, by screenshot, in tabs, by script, by hand) and names no test counts for nothing. The visual file's check could not be reused: it excuses a reason that names an appearance, and run 183's untested reasons say "renders", "visible" and "screen".
  • A rejected entry stays uncovered, and the round's record lists it under untested_rejected.
  • The apply steps and the fix rounds do not finish on a scenario set aside because they looked, in either file. The failed check names the scenarios and the two reasons that stand, and the step is tried again in its own session. That is where the apply is told to write the test.
  • The apply's brief says that having looked is not a reason and that every layer of the app keeps its tests. The planner's rules say the design leaves no layer without tests.

Evidence from the laptop. A unit test on run 183's untested.json: all 18 reasons rejected, and four reasons about a test kept (another test fails when it breaks, a test would only restate the code, a named test file, a named e2e spec). Run 183's 14 visual reasons all stand, as braid#431 intends. A second test fails the check on ten of the entries and on run 182's "Browser-verified in task 5.6".

Not done: no real model has met the check. The next Scratch run shows whether the apply tags its lobby and table scenarios.

Fixed in PR 448. **What changed.** - The untested file gets a reason check of its own. A reason that says someone looked (verified, confirmed, inspected… in a browser, by screenshot, in tabs, by script, by hand) and names no test counts for nothing. The visual file's check could not be reused: it excuses a reason that names an appearance, and run 183's untested reasons say "renders", "visible" and "screen". - A rejected entry stays uncovered, and the round's record lists it under `untested_rejected`. - The apply steps and the fix rounds do not finish on a scenario set aside because they looked, in either file. The failed check names the scenarios and the two reasons that stand, and the step is tried again in its own session. That is where the apply is told to write the test. - The apply's brief says that having looked is not a reason and that every layer of the app keeps its tests. The planner's rules say the design leaves no layer without tests. **Evidence from the laptop.** A unit test on run 183's `untested.json`: all 18 reasons rejected, and four reasons about a test kept (another test fails when it breaks, a test would only restate the code, a named test file, a named e2e spec). Run 183's 14 visual reasons all stand, as braid#431 intends. A second test fails the check on ten of the entries and on run 182's "Browser-verified in task 5.6". Not done: no real model has met the check. The next Scratch run shows whether the apply tags its lobby and table scenarios.
Sign in to join this conversation.
No labels
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
cmoriarty/braid#441
No description provided.