A browser test written as a script of its own costs two fix rounds though every test passes #459
Loading…
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
What happens
Run 185's apply wrote its browser tests as a hand-rolled script on the Playwright library (
e2e/greeter.check.js), not a Playwright test file, to keep the page free of dependencies. Two fix rounds, 9 of the run's 54 minutes, went on that, with every test passing throughout.node e2e/run.js. It exited 0 and printed "all phases passed". The lane passes only withplaywright-report/ortest-results/on disk, or an output that counts passed tests in a form Braid knows, so it failed.The apply's brief says the lane needs Playwright's results "which its default reporters do". It does not say to use the Playwright test runner, and a script on the library looks like it qualifies.
Fix
To decide, one or both:
npx playwright test; a script of your own on the library is not one.Done when
A build that writes its browser tests as the brief says gets a green first round. A test or sim covers a hand-rolled script: either it is told at the apply's own check, or its lane passes.
From run 185 (braid#457).
Decided by the owner, 2026-11-09: fix 1. The briefs say a browser test is a Playwright test file run by the Playwright test runner; the lane keeps its rule.
Fixed in PR 460, archived in PR 462.
The browser-test lane's rule in the briefs now says: a browser test is a Playwright test file (
*.spec.tsor*.spec.js) run by the Playwright test runner,npx playwright test, with@playwright/testas a development dependency; a script of your own that drives a browser through the library is not one, gets no lane, and cannot pass one. The planner's rules ask for "an end-to-end test (a Playwright test file)" where they said "an end-to-end or scripted-UI test".Evidence from the laptop. A test builds the planner's and the apply's briefs and finds both sentences, and no "scripted-UI test". The lane's rule is unchanged.
Not done. No real model has read the sentence yet: the Greeter run started after this merge is the check, and its first round of lanes should be green.
Seen on a real model: run 186 (Greeter, production at
4c6f9e4). The build wrotee2e/greeter.spec.jswith aplaywright.config.js, declarednpm run test:e2e, and the first round of lanes was green: unit and e2e both passed, 42 of 42 scenarios covered by tagged tests. Run 185, before the fix, spent two fix rounds here.