Asset pipeline cannot produce transparent sprites (no alpha, ever)

cmoriarty commented

2026-07-27 20:53:31 -04:00

Owner

Found during the first real trog run (#11), by looking at the artifacts rather than the verdict counts.

What is wrong

Every sprite and UI asset the pipeline generates is RGB with no alpha channel — the subject painted onto an opaque background. In a Phaser scene each of these composites as a solid rectangle.

From the failed run (trog-games/a-cozy-fishing-game-2607272154):

sprite/bird.png        40x24    RGB (NO ALPHA)
sprite/bobber.png      32x32    RGB (NO ALPHA)
sprite/dawn_minnow.png 64x48    RGB (NO ALPHA)
ui/calm_coin.png       24x24    RGB (NO ALPHA)
ui/journal_panel.png   320x120  RGB (NO ALPHA)

Why it has never been caught

Two reasons, and both are now closed or understood:

The visual critic cannot see it. It passed bird.png — a bird on an opaque dusk sky — reasoning that the body had "clean separation against the lighter, warm-toned background bands". The brief said "clean separation from sky", meaning a cutout. Fixed in 089724d with a deterministic cutout_problem gate that runs before the model.
Nothing downstream consumed the assets in a game engine until #11. The review page shows them on a page background, where an opaque backdrop looks fine.

Root cause

server/trog_lib/pixel_post.py converts to RGB at every stage, so alpha is discarded by construction:

pixel_post.py:49   arr = np.asarray(img.convert("RGB"), dtype=np.float64) / 255.0
pixel_post.py:55   return img.convert("RGB")
pixel_post.py:85   a = np.asarray(img.convert("RGB")).copy()
pixel_post.py:125  bgr = ... np.asarray(img.convert("RGB"))[:, :, ::-1]
pixel_post.py:146  small = img.convert("RGB").resize(...)

And no workflow in server/trog_lib/workflows/ does background removal or emits RGBA.

The decision to make

Two approaches, with real trade-offs — worth choosing deliberately rather than by whichever is easier:

Workflow-level: the ComfyUI graph emits RGBA directly (chroma-key on a known flat backdrop, or a matting node). Cleanest pixels, no post-hoc guessing at edges, but it is per-workflow work and the pixel-art seats would each need it.
Post-pass: a background-removal step before pixel_post, then pixel_post preserves alpha instead of flattening. One place to change, but automatic matting on 32x32 pixel art is exactly where it is worst — a hue-shifted ramp against a dusk sky has no clean edge to find.

Either way pixel_post must stop calling .convert("RGB") unconditionally, and the quantise/dither path needs to carry alpha through.

Done when

A generated sprite has an alpha channel with the subject cut out.
The cutout_problem gate (089724d) passes on real generated output rather than rejecting all of it.
trog test asset covers it, so this cannot regress silently again.

Related: #11 (found here), #20 (sprite/pixel-art quality), #18 (critic calibration).

Found during the first real `trog run` (#11), by looking at the artifacts rather than the verdict counts. ## What is wrong Every sprite and UI asset the pipeline generates is **RGB with no alpha channel** — the subject painted onto an opaque background. In a Phaser scene each of these composites as a solid rectangle. From the failed run (`trog-games/a-cozy-fishing-game-2607272154`): ``` sprite/bird.png 40x24 RGB (NO ALPHA) sprite/bobber.png 32x32 RGB (NO ALPHA) sprite/dawn_minnow.png 64x48 RGB (NO ALPHA) ui/calm_coin.png 24x24 RGB (NO ALPHA) ui/journal_panel.png 320x120 RGB (NO ALPHA) ``` ## Why it has never been caught Two reasons, and both are now closed or understood: 1. **The visual critic cannot see it.** It passed `bird.png` — a bird on an opaque dusk sky — reasoning that the body had *"clean separation against the lighter, warm-toned background bands"*. The brief said "clean separation from sky", meaning a cutout. Fixed in 089724d with a deterministic `cutout_problem` gate that runs before the model. 2. **Nothing downstream consumed the assets in a game engine until #11.** The review page shows them on a page background, where an opaque backdrop looks fine. ## Root cause `server/trog_lib/pixel_post.py` converts to RGB at every stage, so alpha is discarded by construction: ``` pixel_post.py:49 arr = np.asarray(img.convert("RGB"), dtype=np.float64) / 255.0 pixel_post.py:55 return img.convert("RGB") pixel_post.py:85 a = np.asarray(img.convert("RGB")).copy() pixel_post.py:125 bgr = ... np.asarray(img.convert("RGB"))[:, :, ::-1] pixel_post.py:146 small = img.convert("RGB").resize(...) ``` And no workflow in `server/trog_lib/workflows/` does background removal or emits RGBA. ## The decision to make Two approaches, with real trade-offs — worth choosing deliberately rather than by whichever is easier: - **Workflow-level:** the ComfyUI graph emits RGBA directly (chroma-key on a known flat backdrop, or a matting node). Cleanest pixels, no post-hoc guessing at edges, but it is per-workflow work and the pixel-art seats would each need it. - **Post-pass:** a background-removal step before `pixel_post`, then `pixel_post` preserves alpha instead of flattening. One place to change, but automatic matting on 32x32 pixel art is exactly where it is worst — a hue-shifted ramp against a dusk sky has no clean edge to find. Either way `pixel_post` must stop calling `.convert("RGB")` unconditionally, and the quantise/dither path needs to carry alpha through. ## Done when - A generated sprite has an alpha channel with the subject cut out. - The `cutout_problem` gate (089724d) passes on real generated output rather than rejecting all of it. - `trog test asset` covers it, so this cannot regress silently again. Related: #11 (found here), #20 (sprite/pixel-art quality), #18 (critic calibration).

cmoriarty referenced this issue

2026-07-27 20:57:47 -04:00

10 · E2E walking skeleton: brief → finished game, fully autonomous #11

cmoriarty referenced this issue

2026-07-27 21:13:11 -04:00

Process spine: generate assets per milestone, and two trogdor profiles (quality / speed) #35

cmoriarty referenced this issue

2026-07-27 21:16:10 -04:00

Process spine: generate assets per milestone, and two trogdor profiles (quality / speed) #35

cmoriarty referenced this issue from a commit

2026-07-27 23:41:40 -04:00

fix: sprites come out as cutouts, not opaque rectangles (#34)

cmoriarty commented

2026-07-27 23:46:03 -04:00

Author

Owner

Fixed in 2c1a9d1, verified end to end on the live rig.

Neither of the two proposed approaches, quite. The ticket weighed workflow-level RGBA against a post-pass and distrusted the post-pass because matting 32x32 pixel art is where matting is worst. That objection is about the raw render — and it stops applying if the cutout runs last. By that point the palette pass has already flattened the backdrop into one or two exact colours, so this is a flood fill over equal pixels, not a guess at an edge. No matting model, no per-workflow work, one place to change.

pixel_post.cutout_background is border-connected, never colour-matched: a pixel is background because it is reachable from the edge through background-coloured pixels. The sky between a bird's wing and its body goes; the identical blue inside its eye stays. Colour-matching would punch a hole through anything sharing a tone with the backdrop, which is exactly the failure this ticket was right to fear.

Details worth knowing:

Sheets seed each frame from its own edges (tile_w) — a sheet's border is not each frame's border, and seeding from the sheet alone cuts out the first and last frames only.
Backgrounds are exempt. A background IS the backdrop; clearing its edges would punch holes in the sky.

How it got caught. Not by looking — by wiring the per-item critic in #35. run_job had never called critics, so everything the new asset tool generated was committed unjudged. The first asset judged after that gap closed came back:

verdict: fail
critique: unusable as a game asset: no alpha channel — this is a picture
          with a background baked in, and would composite as an opaque rectangle

Measured across every sprite generated that session: chest.png, ws-potion.png, speedknight.png — all RGB, 0.0% transparent.

Verification, same prompt before and after, through the full production path (MCP tool → orchestrator → seat → critic → commit):

	before	after
mode	RGB	RGBA
transparent	0.0%	67.2%
`cutout_problem`	rejects	accepts
critic verdict	fail	pass

Committed: trog-games/seat-scratch/assets/sprite/alpha-chest.png — corner alpha 0, centre alpha 255, subject intact.

Regression cover in tests/test_pixel_post.py: background cleared, subject kept, the enclosed background-coloured pixel survives, every frame of a sheet is cut, cutout=False still yields RGB for backgrounds, and a round-trip asserting cutout_problem accepts what the pass produces — the gate that rejected all of this is now the test.

Done-when, checked: sprite has alpha with the subject cut out ✅ · the gate passes on real generated output ✅ · covered by tests ✅.

Fixed in `2c1a9d1`, verified end to end on the live rig. **Neither of the two proposed approaches, quite.** The ticket weighed workflow-level RGBA against a post-pass and distrusted the post-pass because matting 32x32 pixel art is where matting is worst. That objection is about the *raw render* — and it stops applying if the cutout runs **last**. By that point the palette pass has already flattened the backdrop into one or two exact colours, so this is a flood fill over equal pixels, not a guess at an edge. No matting model, no per-workflow work, one place to change. `pixel_post.cutout_background` is **border-connected, never colour-matched**: a pixel is background because it is reachable from the edge *through* background-coloured pixels. The sky between a bird's wing and its body goes; the identical blue inside its eye stays. Colour-matching would punch a hole through anything sharing a tone with the backdrop, which is exactly the failure this ticket was right to fear. Details worth knowing: - **Sheets seed each frame from its own edges** (`tile_w`) — a sheet's border is not each frame's border, and seeding from the sheet alone cuts out the first and last frames only. - **Backgrounds are exempt.** A background IS the backdrop; clearing its edges would punch holes in the sky. **How it got caught.** Not by looking — by wiring the per-item critic in #35. `run_job` had never called `critics`, so everything the new asset tool generated was committed unjudged. The first asset judged after that gap closed came back: ``` verdict: fail critique: unusable as a game asset: no alpha channel — this is a picture with a background baked in, and would composite as an opaque rectangle ``` Measured across every sprite generated that session: `chest.png`, `ws-potion.png`, `speedknight.png` — all RGB, 0.0% transparent. **Verification**, same prompt before and after, through the full production path (MCP tool → orchestrator → seat → critic → commit): | | before | after | |---|---|---| | mode | RGB | RGBA | | transparent | 0.0% | 67.2% | | `cutout_problem` | rejects | accepts | | critic verdict | **fail** | **pass** | Committed: `trog-games/seat-scratch/assets/sprite/alpha-chest.png` — corner alpha 0, centre alpha 255, subject intact. Regression cover in `tests/test_pixel_post.py`: background cleared, subject kept, the enclosed background-coloured pixel survives, every frame of a sheet is cut, `cutout=False` still yields RGB for backgrounds, and a round-trip asserting `cutout_problem` accepts what the pass produces — the gate that rejected all of this is now the test. Done-when, checked: sprite has alpha with the subject cut out ✅ · the gate passes on real generated output ✅ · covered by tests ✅.

cmoriarty commented

2026-07-27 23:48:16 -04:00

Author

Owner

further testing on #35 integrated with other changes, closing this as complete implementation

cmoriarty closed this issue

2026-07-27 23:48:16 -04:00

cmoriarty referenced this issue

2026-07-27 23:51:44 -04:00

Process spine: generate assets per milestone, and two trogdor profiles (quality / speed) #35

cmoriarty referenced this issue from a commit

2026-07-28 15:06:44 -04:00