Bake off pixel-art generation by whether the brain can build a playable slice with it #37
Labels
No labels
bug
duplicate
enhancement
help wanted
invalid
question
wontfix
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set.
Reference
cmoriarty/trog#37
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
The question
Is a diffusion image model the right tool for pixel-art sprites at all?
Suspicion: the low-quality feel of games out of both agentic-game-dev and
trog traces to asking an image generator for pixel art and getting
pixel-art-styled illustration, which
pixel_post.pythen downscales andquantises onto a grid. That correction is lossy by construction — the model
never drew on the grid. True pixel art is placed pixel by pixel with intent:
discrete, symbolic, closer to code than to painting.
The test: a playable slice, not a sprite sheet
An image generator is proven by whether the brain can build a Phaser game
with its output. Not by whether an asset scores well alone.
This is the whole point of the redesign. We have run asset-only bake-offs
repeatedly —
sprite-bench,asset-bench,animate-bench, the #20 eye pick— and every one of them passed while the games kept coming out looking
amateur. A bench that keeps saying "good" about a pipeline that keeps
producing bad-looking games is measuring the wrong thing.
What an asset-only bench cannot see:
(
a-one-screen-arcade-2607281432: the "background" had a salmon, a troutand a bucket painted into it, and the critic passed it — correctly, against
its brief)
48px it is drawn at
six different light directions
that puts the character's feet in the wrong place, a sheet whose frames
don't divide evenly
beige rectangle for the entire game; run 10 loaded an 11MB music track and
never played it
Every one of those is invisible until something tries to build a game with
it. So the unit of measurement is a slice you can play, and the verdict
is a screenshot of it running.
Harness: change one thing
The confound to beat is that a full production run varies everything —
different concept, different GDD, different slice spec, different milestone
plan — so two runs are never comparable. So the harness fixes everything
upstream and varies only the generator.
A bake-off fixture repo, committed once, carrying identical
docs/00-brief.md,01-concept.md,02-gdd.md,03-slice.mdand08-assets.mdfor every arm. Then, per arm:08-assets.md(
slice_build's existing path — no new build machinery)playtest.drive_script)One variable: where the images came from. Everything else is held.
Reuse rather than build:
codeseatfor the build,playtest.run_zip/runfor the play,criticsfor the machine verdict, and the ledger forprovenance. The new code is a runner that loops arms — a
trogCLI tier(
trog test bakeoff), not a make target (#21).The fixed brief
One screen, and chosen to stress exactly what pixel art is judged on:
directional sprite where one is available)
generated UI dies)
Small enough to build in one session, broad enough that a generator good at
one thing and bad at another cannot hide.
Arms
A. Current seat — pixel-art model +
pixel_postdownscale/quantise.The incumbent and the control.
B. Editor MCP — the agent authors pixel art directly through a real
editor: willibrandon/pixel-mcp
(animation, retro palettes, dithering, shading, spritesheet export),
rkdfx/aseprite-mcp (Lua),
diivi/aseprite-mcp.
docs/toolbelt-audit.mdalready logged these under ticket 05, but scoped aspost-processing; this tries them as authoring. Headless Lua scripting
fits the seat model: CPU-only, no GPU contention with the brain,
deterministic. Licensing: Aseprite's binary is paid and its source non-free
(self-compile, personal use); LibreSprite
is the GPL fork.
C. Drawing commands — the brain emits palette ramps, shapes, symmetry and
dither regions; a deterministic renderer rasterises to an exact grid. The
StrokeNUWA direction. Raw-pixel emission
is not the variant to test: pixel-llm
concludes small local models cannot hold the format, and a 64×64 sprite is
4096 cells against our 27B dense / 35B-A3B MoE. Commands play to what the
brain is actually good at, and cost no GPU.
D. Pixel-native diffusion (reference ceiling, not adoptable) —
Retro Diffusion,
PixelLab + MCP
(360° rotation, skeleton animation, Wang tilesets). Both hosted SaaS with
accounts and job queues, which fails the self-hosted bar that ruled out the
Phaser Game Agent MCP. Worth running once, by hand, purely to see what
the ceiling looks like — if A is close to D, the generator was never the
problem.
Scoring
Machine, per arm — cheap and already built:
motion, failed requestscheck_phasebuild checks, includingassets_usedto use, placeholder fallbacks. A generator whose output the agent cannot
wire up loses even if the art is pretty.
Human, and decisive — the #20 eye pick:
background in a quarter-second
The critic does not get the final say here. It passed the ocean background
that ruined a game; it is a filter, not a judge.
What would make each arm win
Stated up front so the result cannot be rationalised afterwards:
problem was always integration and briefing, and #37 closes in favour of
fixing those (which is most of what 2026-07-28 turned out to be).
a held ramp, no antialiased mush — at comparable or better wiring cost.
self-hosted pixel art has a ceiling and we should decide whether we care.
Honest counter-evidence
Much of the "bad art" this week was not the generator:
55c05fde35e814`bucket`.pngd863509d24c8c8fcbd255The fish sprites in
a-one-screen-arcade-2607281432are genuinely good. Thepipeline has been losing and misusing art it already made correctly at least
as often as it has made bad art. Run the bake-off only after those fixes
have a clean run behind them, or it will measure integration bugs and call
them generator quality.
Boundary
If an editor MCP wins it knows nothing about trog's domain, so by the MCP
boundary rule (#13) it belongs in common-mcps with its own stack, the way
searxng does — not in trog compose.
Not blocking #35
#35's remaining scope is moving the milestone walk off batching. Separate
line of attack; do not fold in.
Bake off pixel-art authoring: diffusion vs an editor MCP vs drawing commandsto Bake off pixel-art generation by whether the brain can build a playable slice with itHandoff: the bake-off has a verdict — the brain drawing pixels beat the image generator
Session of 2026-07-28. Everything below is pushed, CI green and deployed.
The result
Same fixture, same brief, same 64x64 grid, same developer agent, same
headless browser. Only the generator differed.
The numbers understate it. Arm C's robot is a robot: symmetrical boxy
chassis, two cyan eye panels, antennae with yellow tips, orange shoulder
pads, a panelled torso, two legs with darker feet, one consistent navy
palette with lighter edge highlights, a clean 1px outline, filling its
frame. Arm A's is a faint fragment blob. First asset this pipeline has made
that is good pixel art without qualification — and it cost no GPU.
Play them (note
index.html; a bare directory serves a file listing):Why it works: diffusion samples an image and downscales it onto the
grid, so below ~64px there is not enough left to be a thing. A program
places every pixel deliberately, so it does not degrade with size — it is
the only approach here that gets BETTER as the grid shrinks, because 32x32
is 1024 decisions rather than a 16x reduction.
The harness (#37's actual deliverable)
trog test bakeoff [--arms current,commands]→trog_lib/bakeoff.py+POST /bakeoff. Hand-written fixture documents inserver/templates/bakeoff/are the control; every arm generates from thesame
08-assets.md, builds the same slice through the same code agent witha frozen brief, gets driven by the same script, and mirrors to a playable
URL. The slice brief lives in bakeoff.py deliberately — a control that
moves when production's prompt is tuned is not a control.
Arms are one dict entry:
ARMS = {name: Arm(name, description, generate)}.Pixel-art fixes that landed on the way
Each of these was a real defect found by looking at artifacts:
3cb602fgrid floor (32 short side, clamped up keeping aspect) +a deterministic coverage gate — a subject under 18% of its frame is
refused before a vision call, beside #34's alpha check.
7bcd7a1small grids auto-route to #20's Fine lane. See below.8ba4c3bone palette per game: the background seeds it, everythingelse snaps in OKLab. Six assets were getting six independent palettes,
which is exactly why sets read as a collage.
6320e18despeckle after the cutout — the colour cleanup ran BEFOREit, so stranded background was never swept. The rule is relative (a
fragment under a quarter of the main form), because the debris was bars
of twenty-odd pixels, not specks.
d24c8c8the asset-spec phase is told the pipeline makes pixel art(188be3a's fix, one layer up — a run wrote "Grid: Vector-style" into a
spec and sent it to a pixel-art seat).
d863509asset ids shed markdown — a whole run committedassets/sprite/`bucket`.pngand every sane path 404ed.Measured effect in isolation: the heart went from 16x16, 1.6% coverage,
one colour (a 2x2 dot) to 32x32, 27.4%, five colours — a properly
shaded heart. So the floors and routing are worth keeping regardless of
which arm wins.
The #20 lesson, which is the important one
#20 did this research in July: benched models, adopted craft prompting,
built
pixel_post, wired a Z-Image Fine lane specifically for small andintricate subjects. Then it wrote the conclusion as advice to a person:
Sound for a human picking assets. But the spec author in an unattended run
is a model that has never read that document — so no asset ever set
Fine: yes, and the lane built to fix this case sat unused for months.A decision left to human judgement in an autonomous system is a decision
left to nobody. That is why it felt like #20 was for nothing; the work was
right, the hand-off was to someone who is not in the loop.
#20 also wrote a revisit trigger — "revisit gen-at-256 if a real game's 32px
assets still disappoint through the Fine lane" — which has now fired and
is recorded as fired in
docs/asset-gen-research.md. gen-at-256 itself isstill not done.
Open: why was arm A that bad?
Genuinely unresolved, and the leading suspect is ours, not the model's.
Arm A's robot came back 3 colours at 3.1% coverage on a dark-navy dusk
palette. That is the signature of the cutout eating the subject:
cutout_backgroundflood-fills from the border through same-colouredpixels, and a slate-grey robot against a dark backdrop is exactly the case
where "background" and "subject" quantise close enough to merge. The heart
(bright red on its own) survived; the dark robot did not.
Cheap way to test: run the same brief with the cutout disabled and compare
coverage. If that is it, the fix is a cutout that refuses to remove more
than N% of the frame, or a light backdrop forced in the prompt so the
subject always contrasts with what gets flood-filled.
Other suspects worth eliminating in the same pass: gen-at-512 downscaling
16x to a 32 grid; PixelOE's contrast downscale on dark low-contrast art;
the craft suffix's "centered on a plain background" fighting "fill the
frame".
Next
New ticket for arm B (a real editor MCP) plus the craft research — arm C
was the cheap proxy for that hypothesis and it won, so paying for the real
thing is now justified rather than speculative.
Carried forward to #38 — the editor-MCP arm plus the craft research, with the "why was arm A that bad" investigation attached so we stop not understanding our own tools.
The harness (
trog test bakeoff) stays here and is reusable: a new arm is one dict entry introg_lib/bakeoff.py.