The Visual check can't see an animation shorter than a model turn: its screenshots land after it ends #346

Closed
opened 2026-10-30 13:34:54 -04:00 by cmoriarty · 2 comments
Owner

What happens
Run 160's brief asks for an "exciting" Hello World animation, which lasts 1.5 s. The Visual check's browser subagent clicked, waited and took the screenshot in separate turns. On fastllm each turn took about 10 s, so every screenshot showed the settled text. Its report: "I could not capture the mid-animation (coloured/scaled) state because the browser tool's own latency exceeded the 1.5 s animation duration". It judged the animation from the app's state hook without seeing it. Apply's own browser check hit the same thing, and worked around it by restarting the animation inside one evaluate and pausing it. On vLLM a turn takes 1–3 s, still longer than most UI animations.

Fix
Tell the browser subagent how to see motion: act and capture in one call. For example, one evaluate that clicks, waits N ms and pauses document.getAnimations(), then a screenshot. Or give it a tool that clicks and takes a screenshot after a delay.

Done when
On a brief with a short animation, the Visual check's screenshots show the animation mid-flight, on vLLM and on fastllm.

**What happens** Run 160's brief asks for an "exciting" Hello World animation, which lasts 1.5 s. The Visual check's browser subagent clicked, waited and took the screenshot in separate turns. On fastllm each turn took about 10 s, so every screenshot showed the settled text. Its report: "I could not capture the mid-animation (coloured/scaled) state because the browser tool's own latency exceeded the 1.5 s animation duration". It judged the animation from the app's state hook without seeing it. Apply's own browser check hit the same thing, and worked around it by restarting the animation inside one `evaluate` and pausing it. On vLLM a turn takes 1–3 s, still longer than most UI animations. **Fix** Tell the browser subagent how to see motion: act and capture in one call. For example, one `evaluate` that clicks, waits N ms and pauses `document.getAnimations()`, then a screenshot. Or give it a tool that clicks and takes a screenshot after a delay. **Done when** On a brief with a short animation, the Visual check's screenshots show the animation mid-flight, on vLLM and on fastllm.
Author
Owner

Shipped in PR #356 (c00c62f1, live on production).

  • The fix: the browser subagent's prompt, and the rules of a judge that drives the browser itself, now say how to catch short motion. In one browser_evaluate: click, wait N ms, then document.getAnimations().forEach(a => a.pause()), and only then take the screenshot. For a canvas or requestAnimationFrame loop, read the app's state in the same call.
  • Test: tests/engine/test_see_motion.py checks that both briefs say it.
  • Stays open: the done-when needs screenshots of a real animation mid-flight on vLLM and fastllm, and trogdor is off the network. It also needs a local replay of a Visual check on a short-animation brief.
**Shipped** in PR #356 (c00c62f1, live on production). - **The fix:** the browser subagent's prompt, and the rules of a judge that drives the browser itself, now say how to catch short motion. In one `browser_evaluate`: click, wait N ms, then `document.getAnimations().forEach(a => a.pause())`, and only then take the screenshot. For a canvas or `requestAnimationFrame` loop, read the app's state in the same call. - **Test:** `tests/engine/test_see_motion.py` checks that both briefs say it. - **Stays open:** the done-when needs screenshots of a real animation mid-flight on vLLM and fastllm, and trogdor is off the network. It also needs a local replay of a Visual check on a short-animation brief.
Author
Owner

Done in batch A, merged as #356 (c00c62f1), and live on production: Braid reports commit c00c62f, which includes it. It was checked with the repo's tests and sims before the merge. A live check on vLLM is still to come: trogdor is off the network. Closed at the owner's request.

Done in batch A, merged as #356 (`c00c62f1`), and live on production: Braid reports commit `c00c62f`, which includes it. It was checked with the repo's tests and sims before the merge. A live check on vLLM is still to come: trogdor is off the network. Closed at the owner's request.
Sign in to join this conversation.
No labels
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
cmoriarty/braid#346
No description provided.