05.2 · Asset critics: analyze_music + critique_image judge the drain #15

New issue

Closed

opened 2026-07-24 09:24:58 -04:00 by cmoriarty · 2 comments

cmoriarty commented

2026-07-24 09:24:58 -04:00

Owner

Asset critics — the missing half of the pipeline (user-caught 2026-07-24: the mole drain's sfx were ACE-generated tiny songs and nothing failed them; a judge would have).

Scope — port both deferred critics from agentic-game-dev as drain-side judges:

analyze_music(url, intent) — backend is audio-server's /analyze, which the audio profile now already deploys (same image as the music/sfx seats). Judge each committed audio asset against its spec Brief; verdict on the trog ledger keyed to the asset's commit.
critique_image — the brain has vision; judge sprites/backgrounds/sheets against Brief + style block during (or right after) the drain, ledger-keyed the same way.

Semantics (match the gate pattern): critic verdict is a producer row; a deterministic check reads the ledger — failed items get bounded regeneration (new seed / adjusted prompt), then advance with the failure recorded, mirroring gate retry caps. Review page surfaces verdicts per asset.

Ordering note: image critique needs the brain, but the drain has the brain down — either critique in a post-drain pass (brain up, assets already committed — cheapest) or via audio-server's analyzer for audio only during the drain. Post-drain pass is the natural v1: conductor already runs the review-page hook there.

Evaluation bar unchanged: open source, self-hosted, ledger is truth.

Asset critics — the missing half of the pipeline (user-caught 2026-07-24: the mole drain's sfx were ACE-generated tiny songs and nothing failed them; a judge would have). **Scope — port both deferred critics from agentic-game-dev as drain-side judges:** - `analyze_music(url, intent)` — backend is audio-server's `/analyze`, which the audio profile now already deploys (same image as the music/sfx seats). Judge each committed audio asset against its spec Brief; verdict on the trog ledger keyed to the asset's commit. - `critique_image` — the brain has vision; judge sprites/backgrounds/sheets against Brief + style block during (or right after) the drain, ledger-keyed the same way. **Semantics (match the gate pattern):** critic verdict is a producer row; a deterministic check reads the ledger — failed items get bounded regeneration (new seed / adjusted prompt), then advance with the failure recorded, mirroring gate retry caps. Review page surfaces verdicts per asset. **Ordering note:** image critique needs the brain, but the drain has the brain down — either critique in a post-drain pass (brain up, assets already committed — cheapest) or via audio-server's analyzer for audio only during the drain. Post-drain pass is the natural v1: conductor already runs the review-page hook there. Evaluation bar unchanged: open source, self-hosted, ledger is truth.

cmoriarty added this to the v1 - single user autonomous milestone

2026-07-24 09:24:58 -04:00

cmoriarty referenced this issue

2026-07-24 09:57:10 -04:00

05 · Asset pipeline: spec document, generate_assets MCP tool, model-swap conductor #4

cmoriarty referenced this issue

2026-07-24 10:32:44 -04:00

05.1 · Asset file service on trogdor: intermediates, preview, cross-game library #14

cmoriarty referenced this issue from a commit

2026-07-24 10:39:26 -04:00

feat: asset critics — the drain gets judges (ticket 05.2, #15)

cmoriarty commented

2026-07-24 11:50:46 -04:00

Author

Owner

Implemented, live-validated — and the first cycle caught two of its own design flaws

Shipped: trog_lib/critics.py — vision critique via the brain (Brief + style block, gate-style VERDICT), audio critique via CLAP /analyze (≥0.2 threshold, proven upstream). Verdicts commit+target-keyed on the ledger. Bounded regeneration with the ledger as attempt-memory: first-time failures requeue once with the critique folded into the prompt; prior-fail assets advance with the failure recorded. Review pages render pass/fail chips + critic notes. 49 tests.

First live cycle (asset-bench): 8 judged → 4 visual failures → regen drain with critique-fed prompts → chest flipped fail→pass on retry. The critic's reasoning is genuinely good — it enforced the spec's 2px-outline rule and called out a highlight that "looks like a bruise"; the failures were legitimate style-contract violations, not confusion. All audio passed (the CLAP judge retroactively blessing the sfx-seat fix).

What the live run caught, fixed same-hour:

Verdict flip-flop — temp-0.7 re-judgment failed an unchanged asset it had passed an hour earlier, spawning a phantom requeue. Fix: verdicts are commit+target-keyed and reused — an unchanged asset is never re-judged (ledger.lookup gained a target param); critic temp is 0.2; cycle() drains to empty with a 3-pass cap. Regression test: an LLM that explodes on contact proves unchanged assets never reach the brain again.
Per-pass analyzer cost — container start + CLAP load every cycle. Now a standing CPU service in the compose (trog-analyzer, CLAP resident in RAM, user's call); conductor's post_drain shrank by the whole container dance. Verified healthy at http://analyzer:8092 from inside aegra.

Phantom job cancelled with reason on the queue. Final bench verdicts + chips: http://trogdor:3923/games/asset-bench/assets/review.html

Also this session: laptop base python 3.10 → 3.14 (was biting shell-embedded python), and #17 filed so the NEXT feature doesn't need a 90-minute full cycle to iterate.

## Implemented, live-validated — and the first cycle caught two of its own design flaws **Shipped:** `trog_lib/critics.py` — vision critique via the brain (Brief + style block, gate-style VERDICT), audio critique via CLAP `/analyze` (≥0.2 threshold, proven upstream). Verdicts commit+target-keyed on the ledger. Bounded regeneration with the ledger as attempt-memory: first-time failures requeue once with the critique folded into the prompt; prior-fail assets advance with the failure recorded. Review pages render pass/fail chips + critic notes. 49 tests. **First live cycle (asset-bench):** 8 judged → 4 visual failures → regen drain with critique-fed prompts → chest flipped fail→pass on retry. The critic's reasoning is genuinely good — it enforced the spec's 2px-outline rule and called out a highlight that "looks like a bruise"; the failures were legitimate style-contract violations, not confusion. All audio passed (the CLAP judge retroactively blessing the sfx-seat fix). **What the live run caught, fixed same-hour:** 1. **Verdict flip-flop** — temp-0.7 re-judgment failed an unchanged asset it had passed an hour earlier, spawning a phantom requeue. Fix: verdicts are commit+target-keyed and *reused* — an unchanged asset is never re-judged (`ledger.lookup` gained a target param); critic temp is 0.2; `cycle()` drains to empty with a 3-pass cap. Regression test: an LLM that explodes on contact proves unchanged assets never reach the brain again. 2. **Per-pass analyzer cost** — container start + CLAP load every cycle. Now a **standing CPU service in the compose** (`trog-analyzer`, CLAP resident in RAM, user's call); conductor's post_drain shrank by the whole container dance. Verified healthy at `http://analyzer:8092` from inside aegra. Phantom job cancelled with reason on the queue. Final bench verdicts + chips: http://trogdor:3923/games/asset-bench/assets/review.html Also this session: laptop base python 3.10 → 3.14 (was biting shell-embedded python), and #17 filed so the NEXT feature doesn't need a 90-minute full cycle to iterate.

cmoriarty referenced this issue

2026-07-24 12:06:12 -04:00

05.5 · Critic calibration: strictness, regen effectiveness, thresholds #18

cmoriarty commented

2026-07-24 12:06:31 -04:00

Author

Owner

Closing — the pipeline judges its own output now

The arc of this ticket, compressed: the sfx-as-songs failure that motivated it could never happen silently again — an asset now gets generated, committed, judged against its own spec, retried once with the critique in its prompt if it fails, and presented with its full verdict reasoning scrollable on the review page.

What shipped: trog_lib/critics.py (vision critique via the brain, CLAP audio critique via the standing CPU analyzer service — resident in RAM per design discussion), commit+target-keyed verdicts on the ledger, bounded regeneration with the ledger as attempt-memory, conductor cycle() draining to convergence under a hard cap, verdict chips + full scrollable critiques on review pages. 49 tests.

What the live cycles taught, fixed same-day: verdict flip-flop on unchanged pixels (→ verdicts are reused, never re-judged — proven live when a full re-run cost zero brain calls), the phantom requeue it spawned (→ cancelled with reason; cycle converges structurally), per-pass analyzer cost (→ standing service), truncated critiques (→ full text end-to-end).

Judge quality: genuinely good — enforced the spec's own 2px-outline rule, called an apple highlight "a bruise," and its critique-fed retry flipped the chest to a pass. Harsh, specific, and legible. The audio judge blessing all three corrected-seat sfx closed the loop this ticket opened.

Future work logged: #18 (critic calibration: strictness, regen A/B, judge-at-upscale, CLAP threshold, motion-clip critique), #16 (foley/audio investigations), #17 (testing tiers — this validation was the last feature that should ever NEED a full cycle to iterate).

Bench artifacts cleaned; tree clean; CI green; deployed container verified running the final code. The critic that would have caught the sfx failure exists — and it already caught two of its own.

## Closing — the pipeline judges its own output now The arc of this ticket, compressed: the sfx-as-songs failure that motivated it could never happen silently again — an asset now gets generated, committed, judged against its own spec, retried once with the critique in its prompt if it fails, and presented with its full verdict reasoning scrollable on the review page. **What shipped:** trog_lib/critics.py (vision critique via the brain, CLAP audio critique via the standing CPU analyzer service — resident in RAM per design discussion), commit+target-keyed verdicts on the ledger, bounded regeneration with the ledger as attempt-memory, conductor cycle() draining to convergence under a hard cap, verdict chips + full scrollable critiques on review pages. 49 tests. **What the live cycles taught, fixed same-day:** verdict flip-flop on unchanged pixels (→ verdicts are reused, never re-judged — proven live when a full re-run cost zero brain calls), the phantom requeue it spawned (→ cancelled with reason; cycle converges structurally), per-pass analyzer cost (→ standing service), truncated critiques (→ full text end-to-end). **Judge quality:** genuinely good — enforced the spec's own 2px-outline rule, called an apple highlight "a bruise," and its critique-fed retry flipped the chest to a pass. Harsh, specific, and legible. The audio judge blessing all three corrected-seat sfx closed the loop this ticket opened. **Future work logged:** #18 (critic calibration: strictness, regen A/B, judge-at-upscale, CLAP threshold, motion-clip critique), #16 (foley/audio investigations), #17 (testing tiers — this validation was the last feature that should ever NEED a full cycle to iterate). Bench artifacts cleaned; tree clean; CI green; deployed container verified running the final code. The critic that would have caught the sfx failure exists — and it already caught two of its own.

cmoriarty closed this issue

2026-07-24 12:06:32 -04:00