A browser test written as a script of its own costs two fix rounds though every test passes #459

Closed
opened 2026-11-09 10:52:30 -05:00 by cmoriarty · 3 comments
Owner

What happens

Run 185's apply wrote its browser tests as a hand-rolled script on the Playwright library (e2e/greeter.check.js), not a Playwright test file, to keep the page free of dependencies. Two fix rounds, 9 of the run's 54 minutes, went on that, with every test passing throughout.

  • Round 1 red. Braid declares the browser-test lane itself only for a Playwright test file. For the script it declared none, so 18 scenarios were "tested only by Playwright files that no lane ran".
  • Round 2 red. The fix round declared node e2e/run.js. It exited 0 and printed "all phases passed". The lane passes only with playwright-report/ or test-results/ on disk, or an output that counts passed tests in a form Braid knows, so it failed.

The apply's brief says the lane needs Playwright's results "which its default reporters do". It does not say to use the Playwright test runner, and a script on the library looks like it qualifies.

Fix

To decide, one or both:

  1. The briefs that write tests say it plainly: a browser test is a Playwright test file run by npx playwright test; a script of your own on the library is not one.
  2. The lane accepts other proof that tests ran: an output that names each check and its result, in a form the brief gives.

Done when

A build that writes its browser tests as the brief says gets a green first round. A test or sim covers a hand-rolled script: either it is told at the apply's own check, or its lane passes.

From run 185 (braid#457).

## What happens Run 185's apply wrote its browser tests as a hand-rolled script on the Playwright library (`e2e/greeter.check.js`), not a Playwright test file, to keep the page free of dependencies. Two fix rounds, 9 of the run's 54 minutes, went on that, with every test passing throughout. - **Round 1 red.** Braid declares the browser-test lane itself only for a Playwright test file. For the script it declared none, so 18 scenarios were "tested only by Playwright files that no lane ran". - **Round 2 red.** The fix round declared `node e2e/run.js`. It exited 0 and printed "all phases passed". The lane passes only with `playwright-report/` or `test-results/` on disk, or an output that counts passed tests in a form Braid knows, so it failed. The apply's brief says the lane needs Playwright's results "which its default reporters do". It does not say to use the Playwright test runner, and a script on the library looks like it qualifies. ## Fix To decide, one or both: 1. The briefs that write tests say it plainly: a browser test is a Playwright test file run by `npx playwright test`; a script of your own on the library is not one. 2. The lane accepts other proof that tests ran: an output that names each check and its result, in a form the brief gives. ## Done when A build that writes its browser tests as the brief says gets a green first round. A test or sim covers a hand-rolled script: either it is told at the apply's own check, or its lane passes. From run 185 (braid#457).
Author
Owner

Decided by the owner, 2026-11-09: fix 1. The briefs say a browser test is a Playwright test file run by the Playwright test runner; the lane keeps its rule.

Decided by the owner, 2026-11-09: fix 1. The briefs say a browser test is a Playwright test file run by the Playwright test runner; the lane keeps its rule.
Author
Owner

Fixed in PR 460, archived in PR 462.

The browser-test lane's rule in the briefs now says: a browser test is a Playwright test file (*.spec.ts or *.spec.js) run by the Playwright test runner, npx playwright test, with @playwright/test as a development dependency; a script of your own that drives a browser through the library is not one, gets no lane, and cannot pass one. The planner's rules ask for "an end-to-end test (a Playwright test file)" where they said "an end-to-end or scripted-UI test".

Evidence from the laptop. A test builds the planner's and the apply's briefs and finds both sentences, and no "scripted-UI test". The lane's rule is unchanged.

Not done. No real model has read the sentence yet: the Greeter run started after this merge is the check, and its first round of lanes should be green.

Fixed in PR 460, archived in PR 462. The browser-test lane's rule in the briefs now says: a browser test is a Playwright test file (`*.spec.ts` or `*.spec.js`) run by the Playwright test runner, `npx playwright test`, with `@playwright/test` as a development dependency; a script of your own that drives a browser through the library is not one, gets no lane, and cannot pass one. The planner's rules ask for "an end-to-end test (a Playwright test file)" where they said "an end-to-end or scripted-UI test". **Evidence from the laptop.** A test builds the planner's and the apply's briefs and finds both sentences, and no "scripted-UI test". The lane's rule is unchanged. **Not done.** No real model has read the sentence yet: the Greeter run started after this merge is the check, and its first round of lanes should be green.
Author
Owner

Seen on a real model: run 186 (Greeter, production at 4c6f9e4). The build wrote e2e/greeter.spec.js with a playwright.config.js, declared npm run test:e2e, and the first round of lanes was green: unit and e2e both passed, 42 of 42 scenarios covered by tagged tests. Run 185, before the fix, spent two fix rounds here.

**Seen on a real model: run 186** (Greeter, production at 4c6f9e4). The build wrote `e2e/greeter.spec.js` with a `playwright.config.js`, declared `npm run test:e2e`, and the first round of lanes was green: unit and e2e both passed, 42 of 42 scenarios covered by tagged tests. Run 185, before the fix, spent two fix rounds here.
Sign in to join this conversation.
No labels
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
cmoriarty/braid#459
No description provided.