Playwright tool defects the judge works around every round: ignored screenshot filename, no media emulation, scripts that hang 180 s, animation capture reinvented #382

Closed
opened 2026-11-04 17:31:20 -05:00 by cmoriarty · 1 comment
Owner

What happens
Across run 172's three judge rounds (247 evaluate calls against 71 clicks):

  • take_screenshot ignores filename and writes timestamped names, so the judge's read of the file it named failed 8 times (landing.png, waiting-bots.png, ...).
  • No emulateMedia, so "reduced motion" was unclear in all three rounds and the brief's criterion could never be judged.
  • Judge-written scripts in IIFE or (opts) => form failed 6 times ("{(intermediate value)} is not a function", "reading 'ms'"), and two evaluates hung 180 s each ("the script did not settle in 180 s").
  • Catching the sub-second deal and trump-flip animations was reinvented by every judge: five subagents in round 3 alone, about 35 min over the run.

Fix

  • Honour filename on screenshots, or return the path written so the judge reads the right file.
  • Add emulateMedia (reduced motion, colour scheme) to the browser toolset, or drop reduced motion from what a judge is asked to verify.
  • Cap evaluate at 60 s and say in the tool description that the function takes no arguments.
  • Ship a canned capture for animations: pause document.getAnimations() at a given currentTime, screenshot, resume; offer it in the judge brief instead of having each round invent one.

Done when
A judge round reads the screenshot it took, judges reduced motion, and captures an animation frame in one call. ~12 + 35 min on run 172.

From the run 172 transcript review (Scratch on vLLM + MTP k=4, 2026-11-04; report in the owner's ~/repos/vLLM/reports/). Related: braid#378, braid#379.

**What happens** Across run 172's three judge rounds (247 `evaluate` calls against 71 clicks): - `take_screenshot` ignores `filename` and writes timestamped names, so the judge's `read` of the file it named failed 8 times (`landing.png`, `waiting-bots.png`, ...). - No `emulateMedia`, so "reduced motion" was `unclear` in all three rounds and the brief's criterion could never be judged. - Judge-written scripts in IIFE or `(opts) =>` form failed 6 times ("{(intermediate value)} is not a function", "reading 'ms'"), and two evaluates hung 180 s each ("the script did not settle in 180 s"). - Catching the sub-second deal and trump-flip animations was reinvented by every judge: five subagents in round 3 alone, about 35 min over the run. **Fix** - Honour `filename` on screenshots, or return the path written so the judge reads the right file. - Add `emulateMedia` (reduced motion, colour scheme) to the browser toolset, or drop reduced motion from what a judge is asked to verify. - Cap `evaluate` at 60 s and say in the tool description that the function takes no arguments. - Ship a canned capture for animations: pause `document.getAnimations()` at a given `currentTime`, screenshot, resume; offer it in the judge brief instead of having each round invent one. **Done when** A judge round reads the screenshot it took, judges reduced motion, and captures an animation frame in one call. ~12 + 35 min on run 172. _From the run 172 transcript review (Scratch on vLLM + MTP k=4, 2026-11-04; report in the owner's `~/repos/vLLM/reports/`). Related: braid#378, braid#379._
Author
Owner

Shipped in PR #390, production ddd4b6b.

  • Named screenshots work, browser_emulate_media is allowed, and a page script may end in ;.
  • A new probe holds an animation frame in one call.

Measured on run 172, against what this issue says:

  • The defects are Braid's own: braid_guards.js deleted filename on purpose, boundEvaluate adds the wrapper and the 180 s, and emulate_media was left off the tool list.
  • The 60 s cap is dropped. Of 309 page scripts, 15 ran past 60 s, and 14 of those returned a result.
  • The trailing ; was 2 of the 14 scripts that failed.

Evidence: tests/agent/test_path_guard.py (failed first) and ui/e2e/judge-probes.spec.ts. In Free replays of round 3, judges set reduced motion with browser_emulate_media and judged it, which run 172 could not. Those replays carried the animation criteria, so the probe rests on its browser test.

**Shipped** in PR #390, production ddd4b6b. - Named screenshots work, `browser_emulate_media` is allowed, and a page script may end in `;`. - A new probe holds an animation frame in one call. **Measured on run 172, against what this issue says:** - The defects are Braid's own: `braid_guards.js` deleted `filename` on purpose, `boundEvaluate` adds the wrapper and the 180 s, and `emulate_media` was left off the tool list. - The 60 s cap is dropped. Of 309 page scripts, 15 ran past 60 s, and 14 of those returned a result. - The trailing `;` was 2 of the 14 scripts that failed. **Evidence:** `tests/agent/test_path_guard.py` (failed first) and `ui/e2e/judge-probes.spec.ts`. In Free replays of round 3, judges set reduced motion with `browser_emulate_media` and judged it, which run 172 could not. Those replays carried the animation criteria, so the probe rests on its browser test.
Sign in to join this conversation.
No labels
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
cmoriarty/braid#382
No description provided.