04 · Playtest service: headless browser #6

New issue

Closed

opened 2026-07-23 00:07:09 -04:00 by cmoriarty · 1 comment

cmoriarty commented

2026-07-23 00:07:09 -04:00

Owner

Run the web game (Phaser default) in a headless browser; capture screenshots, console output, and video. Results feed graph gates and the studio vitals pane.

Adapt from agentic-game-dev/playtest — the web-native target should make this simpler than the Löve setup it replaces.

Run the web game (Phaser default) in a headless browser; capture **screenshots, console output, and video**. Results feed graph gates and the studio vitals pane. Adapt from `agentic-game-dev/playtest` — the web-native target should make this simpler than the Löve setup it replaces.

cmoriarty added this to the v1 - single user autonomous milestone

2026-07-23 00:07:09 -04:00

cmoriarty changed title from ~~Playtest service: headless browser~~ to 04 · Playtest service: headless browser

2026-07-23 00:10:06 -04:00

cmoriarty referenced this issue

2026-07-24 10:27:38 -04:00

05.1 · Asset file service on trogdor: intermediates, preview, cross-game library #14

cmoriarty referenced this issue

2026-07-24 10:32:44 -04:00

05.1 · Asset file service on trogdor: intermediates, preview, cross-game library #14

cmoriarty referenced this issue from a commit

2026-07-25 18:40:12 -04:00

feat: playtest service — headless web games with render gates (#6)

cmoriarty referenced this issue from a commit

2026-07-25 18:43:08 -04:00

feat: playtest fixture tier + mid-hold screenshots (#6)

cmoriarty referenced this issue from a commit

2026-07-25 18:45:13 -04:00

docs: playtest service — gates, scripted runs, the sentinel fixture (#6)

cmoriarty commented

2026-07-25 18:45:35 -04:00

Author

Owner

04 complete — playtest service is live

Docs: docs/playtest.md · service: playtest/ in this repo · fixture: trog-games/playtest-fixture

The web target made it much simpler, as the ticket predicted

The predecessor ran LÖVE under Xvfb, injected input with xdotool, and grabbed frames with ffmpeg's x11grab. Playwright does screenshots, console capture, real input events, and video natively, so all of that apparatus is gone. What I kept from the old service is its good judgment: the strict input-script validator (a script that silently skips a bad step produces screenshots of the wrong state) and the passive/scripted split.

Why it's a gate, not a screenshot service

A broken web game still "loads" — the old runner would report success and attach a picture of an empty canvas. Every run now judges: loaded, no_page_errors, no_console_errors, no_failed_requests (404 sprite = invisible player), page_responsive, and rendered_something (final-frame pixel variety — a one-color canvas is the classic silent failure).

Two things I deliberately did NOT gate:

page_responsive is a floor, not proof the game runs. I caught this by reading my own broken fixture's result: it scored 60fps while rendering one flat color, because the injected requestAnimationFrame probe keeps ticking even when the game's loop never starts. I renamed the check rather than let it imply something false.
motion is reported, never gated. Pixel change between first and last screenshot is a useful signal (0.0 next to a passing run deserves a look), but a static menu or turn-based board must not fail a gate for holding still.

Also found by using it

A scripted run's screenshot showed keys: - — because the shot lands after the key is released, which answers the wrong question for a movement test. Added {"key": "right", "hold": 1.2, "shot": "holding right"} to capture mid-hold; verified the screenshot then reads keys: ArrowRight.

The standing fixture (both directions)

trog-games/playtest-fixture holds a real canvas game and a broken sentinel that fails the way real games do: 404 script → ReferenceError → dead loop → one-color canvas.

trog test playtest      # 15s, asserts BOTH directions
good:   PASS (60.2 fps; 1 canvas; 523 colors; motion 1.05)
broken: FAIL (1 colors; motion 0.0; failed: no_page_errors, no_console_errors,
        no_failed_requests, rendered_something; first error: MissingEngine is not defined)

If the sentinel ever passes, the gates regressed — that's why a game that's broken on purpose is worth keeping.

Wiring

Standing CPU service (SwiftShader GL — never competes for a card, like the analyzer), built from this repo per the vendoring decision. POST /playtest on the orchestrator, trog playtest <game>, and a trog test playtest tier in the quick suite. Verdicts key to the game's HEAD commit — same staleness contract as asset critiques, so a playtest result can't outlive the build it judged. Screenshots and video serve from /output/playtest/<date>/<run>/. 117 tests green.

Known gap, deliberately deferred: the runner serves static files, so a project with package.json and no built entry gets a 422 telling it to commit dist/. Wire a build step when the framework choice settles in the studio tickets (#7).

Ready to close.

## 04 complete — playtest service is live Docs: `docs/playtest.md` · service: `playtest/` in this repo · fixture: trog-games/playtest-fixture ### The web target made it much simpler, as the ticket predicted The predecessor ran LÖVE under Xvfb, injected input with xdotool, and grabbed frames with ffmpeg's x11grab. Playwright does screenshots, console capture, real input events, and video natively, so all of that apparatus is gone. What I kept from the old service is its good judgment: the strict input-script validator (a script that silently skips a bad step produces screenshots of the wrong state) and the passive/scripted split. ### Why it's a gate, not a screenshot service A broken web game still "loads" — the old runner would report success and attach a picture of an empty canvas. Every run now judges: `loaded`, `no_page_errors`, `no_console_errors`, `no_failed_requests` (404 sprite = invisible player), `page_responsive`, and `rendered_something` (final-frame pixel variety — a one-color canvas is the classic silent failure). **Two things I deliberately did NOT gate:** - **`page_responsive` is a floor, not proof the game runs.** I caught this by reading my own broken fixture's result: it scored **60fps while rendering one flat color**, because the injected requestAnimationFrame probe keeps ticking even when the game's loop never starts. I renamed the check rather than let it imply something false. - **`motion` is reported, never gated.** Pixel change between first and last screenshot is a useful signal (0.0 next to a passing run deserves a look), but a static menu or turn-based board must not fail a gate for holding still. ### Also found by using it A scripted run's screenshot showed `keys: -` — because the shot lands after the key is released, which answers the wrong question for a movement test. Added `{"key": "right", "hold": 1.2, "shot": "holding right"}` to capture **mid-hold**; verified the screenshot then reads `keys: ArrowRight`. ### The standing fixture (both directions) `trog-games/playtest-fixture` holds a real canvas game and a **broken sentinel** that fails the way real games do: 404 script → ReferenceError → dead loop → one-color canvas. ``` trog test playtest # 15s, asserts BOTH directions good: PASS (60.2 fps; 1 canvas; 523 colors; motion 1.05) broken: FAIL (1 colors; motion 0.0; failed: no_page_errors, no_console_errors, no_failed_requests, rendered_something; first error: MissingEngine is not defined) ``` If the sentinel ever passes, the gates regressed — that's why a game that's broken on purpose is worth keeping. ### Wiring Standing CPU service (SwiftShader GL — never competes for a card, like the analyzer), built from this repo per the vendoring decision. `POST /playtest` on the orchestrator, `trog playtest <game>`, and a `trog test playtest` tier in the quick suite. Verdicts key to the game's **HEAD commit** — same staleness contract as asset critiques, so a playtest result can't outlive the build it judged. Screenshots and video serve from `/output/playtest/<date>/<run>/`. 117 tests green. **Known gap, deliberately deferred:** the runner serves static files, so a project with `package.json` and no built entry gets a 422 telling it to commit `dist/`. Wire a build step when the framework choice settles in the studio tickets (#7). Ready to close.

cmoriarty closed this issue

2026-07-25 18:47:59 -04:00