Asset editing tools: pixel editor, basic image editor, basic audio editor #32

New issue

Open

opened 2026-07-27 12:19:35 -04:00 by cmoriarty · 0 comments

cmoriarty commented

2026-07-27 12:19:35 -04:00

Owner

Today the only lever a human has over a generated asset is regeneration: the tweak box in the files pane re-prompts the seat and commits whatever comes back (studio/src/panes/FilesPane.tsx, /orch/jobs). There is no way to fix a single wrong pixel, crop a sprite, or trim 200ms of silence off a sfx without spinning up a GPU and rolling the dice again. That is the wrong tool for small, deterministic corrections — and it is the difference between "the AI made something close" and "the game has the asset it needs".

Add direct-manipulation editors to the studio: a full-featured pixel editor for pixel art, a basic image editor for non-pixel art, and a basic audio editor for sfx/music/vocals.

Ground rules

Editors are studio-side. Pure browser (canvas / WebAudio), no GPU, no seat, no brain. Editing must work while a run is generating.
Git is truth: a save is a commit. Save goes through the existing PUT /orch/repo/{slug}/file/{path} (base64 body, commit message; server/trog_lib/orchestrator.py:565). Message convention studio: edit assets/…. Same-origin, no new service.
Honest concurrency. A save must handle the stale-sha 409 the same way the tweak flow now does (c40bac8), and must refresh the viewer + top bar on success rather than leaving a stale image on screen (the #31 2a bug — don't reintroduce it in a second code path).
No external CDN/network deps (nginx serves the SPA same-origin; keep the bundle self-contained). Prefer small vendored code over a heavyweight editor framework; the app already carries CodeMirror 6 for text — asset editors should be equally native, not an iframe'd third-party app.
Undo/redo is per-editor and in-memory; git history is the durable undo (a save is a commit, so reverting is a git operation, not an editor feature).

1. Pixel editor (full featured)

For anything the pixel pipeline produced — pixel_post.pixelize_png output: grid-aligned, palette-quantized PNG (server/trog_lib/pixel_post.py).

Must have:

Zoom/pan canvas with nearest-neighbour rendering, pixel grid overlay, checkerboard for alpha, 1:1 preview swatch alongside the zoomed view.
Tools: pencil, eraser, eyedropper, paint bucket (contiguous + global), line, rectangle, ellipse (filled/outline), rectangular + freeform + magic-wand select, move/cut/copy/paste of a selection, mirror/flip/rotate 90°.
Palette panel seeded from the asset's actual palette (the OKLab k-means palette baked in by pixel_post), with add/remove/reorder, palette swap across the whole sprite, and re-quantize-to-N after editing so a hand-edit can't silently blow the color count the art direction promised.
Per-pixel alpha, brush size, symmetry (x/y) toggle, dithering pattern fill.
Layers (at minimum: two — the sprite and a scratch/reference layer), plus onion skin against the neighbouring frame.
Spritesheet awareness. Sheets are a first-class output (comfy_item sheet path). The editor must read a frame grid (rows/cols or frame size), let you edit a single frame in isolation, and play the frames back as an animation at a settable fps so the edit is judged in motion.
Undo/redo stack, keyboard shortcuts matching common pixel-editor muscle memory (b/e/g/i, [/], space-drag pan, ctrl+z/ctrl+shift+z).

2. Basic image editor (non-pixel art)

For the HD/modern art direction of #24, and for any non-quantized image (backgrounds, UI, promo). Not Photoshop — the useful 10%:

Crop (free + fixed ratio), resize/scale (with resample choice), rotate/flip, canvas resize/pad.
Brightness/contrast, levels or curves (one is enough), saturation/hue shift, per-channel opacity.
Soft brush + eraser, alpha eraser / background knockout (flood-fill by color tolerance is enough — no ML matting).
Simple compositing: paste another repo asset in as a layer, move/scale it, flatten.
Export back as PNG (keep alpha) or WebP, with the byte size shown before saving.

3. Audio editor (basic)

For assets/{music,sfx,vocal}/*.wav (server/trog_lib/audio_item.py). Waveform-first, WebAudio decode, edits applied to the PCM buffer and re-encoded as WAV on save:

Waveform view with zoom, scrub, playhead, selection range, play selection / play loop.
Trim to selection, delete selection, silence-trim head/tail, cut/copy/paste.
Gain on selection, normalize, fade in/out (and crossfade at a splice point).
Loop-point editing with an audible loop preview, wired to the existing seamless-loop helper (audio_item.make_seamless_loop) so a music track edited by hand still loops cleanly.
Speed/pitch (resample is acceptable; no need for a phase vocoder), mono/stereo convert, sample-rate display.
Undo/redo, and a hard cap on in-memory buffer length so a long music track doesn't wedge the tab.

Choosing the right editor

The files pane already sniffs by extension (IMAGE_EXT / AUDIO_EXT, studio/src/panes/FilesPane.tsx:23). Audio is unambiguous. Pixel-vs-not needs a signal, in priority order:

the game's art direction (once #24 lands, this is authoritative);
path convention (assets/sprite/… vs assets/image/…);
heuristic (low unique-color count + grid alignment);
a manual toggle in the editor header — the human always wins.

Done when

Opening a pixel asset in the files pane offers "Edit", which opens the pixel editor in a pane; a hand edit saves and appears in the viewer without a reload.
Same for a non-pixel image (basic image tools) and for a wav (audio tools).
Every save produces exactly one commit in the game repo with a studio: edit … message, and survives a concurrent tweak (409 → refetch → re-apply or clearly refuse, never silently drop the edit).
Spritesheet frames can be edited individually and previewed as an animation.
Unit/typecheck coverage: npm run typecheck --prefix studio clean, plus tests for the non-UI cores (pixel ops, palette quantize, PCM trim/fade/normalize, WAV encode) so the math isn't only tested by eyeballing it.

Out of scope

Vector/SVG authoring and tilemap/level editing (separate concerns; the map pane owns level structure).
Multi-track DAW features (mixing several sources, effects racks, MIDI).
Anything that regenerates via a seat — that's the tweak flow (#31), not the editor.

Related: #24 (art-direction modes decide which editor is the default), #31 (tweak flow + viewer refresh bug), #28 (studio files pane follow-ups).

Today the only lever a human has over a generated asset is *regeneration*: the tweak box in the files pane re-prompts the seat and commits whatever comes back (`studio/src/panes/FilesPane.tsx`, `/orch/jobs`). There is no way to fix a single wrong pixel, crop a sprite, or trim 200ms of silence off a sfx without spinning up a GPU and rolling the dice again. That is the wrong tool for small, deterministic corrections — and it is the difference between "the AI made something close" and "the game has the asset it needs". Add direct-manipulation editors to the studio: a **full-featured pixel editor** for pixel art, a **basic image editor** for non-pixel art, and a **basic audio editor** for sfx/music/vocals. ## Ground rules - **Editors are studio-side.** Pure browser (canvas / WebAudio), no GPU, no seat, no brain. Editing must work while a run is generating. - **Git is truth: a save is a commit.** Save goes through the existing `PUT /orch/repo/{slug}/file/{path}` (base64 body, commit message; `server/trog_lib/orchestrator.py:565`). Message convention `studio: edit assets/…`. Same-origin, no new service. - **Honest concurrency.** A save must handle the stale-sha 409 the same way the tweak flow now does (c40bac8), and must refresh the viewer + top bar on success rather than leaving a stale image on screen (the #31 2a bug — don't reintroduce it in a second code path). - **No external CDN/network deps** (nginx serves the SPA same-origin; keep the bundle self-contained). Prefer small vendored code over a heavyweight editor framework; the app already carries CodeMirror 6 for text — asset editors should be equally native, not an iframe'd third-party app. - **Undo/redo is per-editor and in-memory**; git history is the durable undo (a save is a commit, so reverting is a git operation, not an editor feature). ## 1. Pixel editor (full featured) For anything the pixel pipeline produced — `pixel_post.pixelize_png` output: grid-aligned, palette-quantized PNG (`server/trog_lib/pixel_post.py`). Must have: - Zoom/pan canvas with nearest-neighbour rendering, pixel grid overlay, checkerboard for alpha, 1:1 preview swatch alongside the zoomed view. - Tools: pencil, eraser, eyedropper, paint bucket (contiguous + global), line, rectangle, ellipse (filled/outline), rectangular + freeform + magic-wand select, move/cut/copy/paste of a selection, mirror/flip/rotate 90°. - Palette panel seeded from the asset's actual palette (the OKLab k-means palette baked in by `pixel_post`), with add/remove/reorder, palette swap across the whole sprite, and re-quantize-to-N after editing so a hand-edit can't silently blow the color count the art direction promised. - Per-pixel alpha, brush size, symmetry (x/y) toggle, dithering pattern fill. - Layers (at minimum: two — the sprite and a scratch/reference layer), plus onion skin against the neighbouring frame. - **Spritesheet awareness.** Sheets are a first-class output (`comfy_item` sheet path). The editor must read a frame grid (rows/cols or frame size), let you edit a single frame in isolation, and play the frames back as an animation at a settable fps so the edit is judged in motion. - Undo/redo stack, keyboard shortcuts matching common pixel-editor muscle memory (b/e/g/i, `[`/`]`, space-drag pan, ctrl+z/ctrl+shift+z). ## 2. Basic image editor (non-pixel art) For the HD/modern art direction of #24, and for any non-quantized image (backgrounds, UI, promo). Not Photoshop — the useful 10%: - Crop (free + fixed ratio), resize/scale (with resample choice), rotate/flip, canvas resize/pad. - Brightness/contrast, levels or curves (one is enough), saturation/hue shift, per-channel opacity. - Soft brush + eraser, alpha eraser / background knockout (flood-fill by color tolerance is enough — no ML matting). - Simple compositing: paste another repo asset in as a layer, move/scale it, flatten. - Export back as PNG (keep alpha) or WebP, with the byte size shown before saving. ## 3. Audio editor (basic) For `assets/{music,sfx,vocal}/*.wav` (`server/trog_lib/audio_item.py`). Waveform-first, WebAudio decode, edits applied to the PCM buffer and re-encoded as WAV on save: - Waveform view with zoom, scrub, playhead, selection range, play selection / play loop. - Trim to selection, delete selection, silence-trim head/tail, cut/copy/paste. - Gain on selection, normalize, fade in/out (and crossfade at a splice point). - Loop-point editing with an audible loop preview, wired to the existing seamless-loop helper (`audio_item.make_seamless_loop`) so a music track edited by hand still loops cleanly. - Speed/pitch (resample is acceptable; no need for a phase vocoder), mono/stereo convert, sample-rate display. - Undo/redo, and a hard cap on in-memory buffer length so a long music track doesn't wedge the tab. ## Choosing the right editor The files pane already sniffs by extension (`IMAGE_EXT` / `AUDIO_EXT`, `studio/src/panes/FilesPane.tsx:23`). Audio is unambiguous. Pixel-vs-not needs a signal, in priority order: 1. the game's art direction (once #24 lands, this is authoritative); 2. path convention (`assets/sprite/…` vs `assets/image/…`); 3. heuristic (low unique-color count + grid alignment); 4. a manual toggle in the editor header — the human always wins. ## Done when - [ ] Opening a pixel asset in the files pane offers "Edit", which opens the pixel editor in a pane; a hand edit saves and appears in the viewer without a reload. - [ ] Same for a non-pixel image (basic image tools) and for a wav (audio tools). - [ ] Every save produces exactly one commit in the game repo with a `studio: edit …` message, and survives a concurrent tweak (409 → refetch → re-apply or clearly refuse, never silently drop the edit). - [ ] Spritesheet frames can be edited individually and previewed as an animation. - [ ] Unit/typecheck coverage: `npm run typecheck --prefix studio` clean, plus tests for the non-UI cores (pixel ops, palette quantize, PCM trim/fade/normalize, WAV encode) so the math isn't only tested by eyeballing it. ## Out of scope - Vector/SVG authoring and tilemap/level editing (separate concerns; the map pane owns level structure). - Multi-track DAW features (mixing several sources, effects racks, MIDI). - Anything that regenerates via a seat — that's the tweak flow (#31), not the editor. Related: #24 (art-direction modes decide which editor is the default), #31 (tweak flow + viewer refresh bug), #28 (studio files pane follow-ups).

cmoriarty added this to the v1 - single user autonomous milestone

2026-07-27 12:19:35 -04:00

cmoriarty added the

enhancement

label

2026-07-27 12:19:35 -04:00