The judge never looks at what it built, so a person finds obvious faults it passed #269
Loading…
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
What happens
The judge checks an app through what it does: clicks, keys,
window.__appstate and its criteria. It never looks at the result the way a person would. In the #260 re-run every game passed, and then a person found within a minute:Fix
Done when
A fault a person would spot at a glance, seeded into a simulated run (a mirrored control, an unreadable screen), fails the judge. Models judge visuals unreliably, so measure this on a few seeded faults before relying on it.
From the #260 hands-on play. Reworded 2026-10-29: the first version asked for pinball rules.
The judge passes games with a backwards flipper and no drainto The judge never looks at what it built, so a person finds obvious faults it passedMeasured with seeded faults (production
1fc0f22, vLLM qwen3.8-27b): 1 of 3 caught. Each run was a quick-fix on a branch ofbreakout-planned-2with one obvious fault planted. The brief added a line of text under the game, so the Visual check ran and played the whole game.window.__app: holding left moved the paddle +270, holding right moved it -270. It failed 2 of 6, the fix round swapped them back, and round 2 passed.#141a21on#0b0f14(run 154)// bright cyan.What this says
Fix direction
getImageData), and compute the contrast ratio of every text element against what is behind it. Fail below a floor (e.g. WCAG 3:1 for large text).Stays open.
Shipped in PR #356 (
c00c62f1, live on production).A whole-screen criterion: the judge now has a criterion for each main screen, judged as someone who knows this kind of app would see it. It covers the whole screen, a fault the base branch already had included, and is never described from the code, its comments, the README or the specs.
Two probes, pasted into the prompt from
src/osf/engine/probes/:The probes on run 154's and 155's faults, in Chromium (
ui/e2e/judge-probes.spec.ts):#141a21on#0b0f14Stays open: whether a model now uses the probes and fails these faults still has to be measured. That means re-running the three seeded-fault quick-fixes (runs 153–155), which waits on trogdor.
Done in batch A, merged as #356 (
c00c62f1), and live on production: Braid reports commitc00c62f, which includes it. It was checked with the repo's tests and sims before the merge. A live check on vLLM is still to come: trogdor is off the network. Closed at the owner's request.