Research: a method for the Visual check to judge animations reliably in reasonable time #434

Open
opened 2026-11-08 12:50:31 -05:00 by cmoriarty · 1 comment
Owner

What happens. Braid has no proven way for the Visual check to judge an animation. Run 174's judge chased single frames for seven minutes; the filmstrip and page clock of #407/#415 were built from that and reverted in #433 after run 182 showed the strip never composed, the probe unused and the clock stalling the step twice. Whether a model can judge motion from frames at all, and in what time, was never tested apart from a live run.

Research first, with a person judging alongside.

  • Candidates: a filmstrip from a probe that waits for the server before it freezes; Playwright's video recording with frames pulled afterwards; sampling document.getAnimations() and the app's own state over time; asking the app to expose its animation state on its hook; judging only start and end states and leaving motion to a person.
  • A test set: a handful of animations in a throwaway app, some right and some deliberately wrong (wrong direction, clipped, too slow, never finishes), each with a person's verdict written down first.
  • Measure for each candidate: does the agent's verdict match the person's, how many turns and minutes does it take, and how often does it stall or mislead. Try it in the Visual check's own session and through browser.

Fix. Decide from the results. Ship only a method that matches the person's verdicts and fits a few minutes per animation; otherwise the Visual check keeps judging start and end states and says so.

Done when. A measurement note under docs/design/measurements/ with the test set, each candidate's matches and minutes, and the decision; a change proposed from it, or the decision that there is none to make.

Follows #433. From the review of run 182's findings (braid#425).

**What happens.** Braid has no proven way for the Visual check to judge an animation. Run 174's judge chased single frames for seven minutes; the filmstrip and page clock of #407/#415 were built from that and reverted in #433 after run 182 showed the strip never composed, the probe unused and the clock stalling the step twice. Whether a model can judge motion from frames at all, and in what time, was never tested apart from a live run. **Research first, with a person judging alongside.** - Candidates: a filmstrip from a probe that waits for the server before it freezes; Playwright's video recording with frames pulled afterwards; sampling `document.getAnimations()` and the app's own state over time; asking the app to expose its animation state on its hook; judging only start and end states and leaving motion to a person. - A test set: a handful of animations in a throwaway app, some right and some deliberately wrong (wrong direction, clipped, too slow, never finishes), each with a person's verdict written down first. - Measure for each candidate: does the agent's verdict match the person's, how many turns and minutes does it take, and how often does it stall or mislead. Try it in the Visual check's own session and through `browser`. **Fix.** Decide from the results. Ship only a method that matches the person's verdicts and fits a few minutes per animation; otherwise the Visual check keeps judging start and end states and says so. **Done when.** A measurement note under `docs/design/measurements/` with the test set, each candidate's matches and minutes, and the decision; a change proposed from it, or the decision that there is none to make. Follows #433. From the review of run 182's findings (braid#425).
Author
Owner

Scope, widened by the owner on 2026-11-12: not only animations but anything on a page that changes over time: a timed transition, a loop that runs on the real clock, a state that passes before it can be looked at.

Why: run 194's apply group for one part spent 45 of its 83.7 minutes on a loop that ran about 40 times too fast (braid#501, finding 5). Its tests froze the clock and stepped by hand, so they could not see it; it was found by looking, through 37 one-off page inspections, each a round trip with the agent's own delay between a state and the look at it. The same tool that lets a check judge an animation would have shown this in one look.

For the research to cover as well:

  • Building steps, not only the Visual check: the agent that writes the code needs it first.
  • Recording state over a short interval in one call, against the page's own clock.
  • A stale page after a rebuild (the browser's cache), which cost turns twice in the same hunt.
**Scope, widened by the owner on 2026-11-12:** not only animations but anything on a page that changes over time: a timed transition, a loop that runs on the real clock, a state that passes before it can be looked at. **Why:** run 194's apply group for one part spent 45 of its 83.7 minutes on a loop that ran about 40 times too fast (braid#501, finding 5). Its tests froze the clock and stepped by hand, so they could not see it; it was found by looking, through 37 one-off page inspections, each a round trip with the agent's own delay between a state and the look at it. The same tool that lets a check judge an animation would have shown this in one look. **For the research to cover as well:** - Building steps, not only the Visual check: the agent that writes the code needs it first. - Recording state over a short interval in one call, against the page's own clock. - A stale page after a rebuild (the browser's cache), which cost turns twice in the same hunt.
Sign in to join this conversation.
No labels
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
cmoriarty/braid#434
No description provided.