cmoriarty/trog
1
0
Fork 0
Code Issues 11 Pull requests Projects 1 Releases Packages Wiki Activity Actions

05.5 · Critic calibration: strictness, regen effectiveness, thresholds #18

New issue
Closed
opened 2026-07-24 12:06:12 -04:00 by cmoriarty · 1 comment
cmoriarty commented 2026-07-24 12:06:12 -04:00
Owner
Copy link

Follow-ups from the first live critic cycles (05.2 / #15) — the judges work; now calibrate them with data. Each item is investigate-then-decide.

  • Visual strictness calibration. First bench cycle failed 4/5 visual assets, largely on the spec's own 2px-outline rule — legitimate readings, but is the bar achievable? Decide per finding: teach the image seat to deliver it (outline enforcement in the pixel post-pass?), soften the style-block guidance the templates suggest, or accept high first-pass failure as intended behavior. Needs a few real games' worth of verdict data — track pass rates per kind on the ledger (the data already accumulates).
  • Regen effectiveness. Chest flipped fail→pass with critique-fed prompting; knight and apple did not. Measure: does folding the critique into the retry prompt actually move the pass rate vs a plain reroll (new seed only)? Cheap A/B once #17's mock tiers land.
  • Judge-at-upscale experiment. Critics currently see native-resolution pixel art (16-96px). The brain reads them fine (verdict reasoning is coherent), but test whether nearest-neighbor upscaling before judgment changes verdict quality/consistency — the review page upscales for humans for a reason.
  • CLAP threshold tuning. 0.2 inherited from agentic-game-dev; all bench audio passed. Once enough audio verdicts accumulate, check the threshold against human judgment (the review page's audio players make spot-checks cheap).
  • Motion-clip critique. Raw animate clips now land in trogdor-files intermediates but are unjudged — the sheet is judged as a static strip. Investigate judging the CLIP (frame sampling to the brain, or the HunyuanVideo-Foley-adjacent tooling from #16) for motion quality the strip can't show.

Decision output per item: adopt / reject / defer with trigger, recorded here and in the research docs.

Follow-ups from the first live critic cycles (05.2 / #15) — the judges work; now calibrate them with data. Each item is investigate-then-decide. - [x] **Visual strictness calibration.** First bench cycle failed 4/5 visual assets, largely on the spec's own 2px-outline rule — legitimate readings, but is the bar achievable? Decide per finding: teach the image seat to deliver it (outline enforcement in the pixel post-pass?), soften the style-block guidance the templates suggest, or accept high first-pass failure as intended behavior. Needs a few real games' worth of verdict data — track pass rates per kind on the ledger (the data already accumulates). - [x] **Regen effectiveness.** Chest flipped fail→pass with critique-fed prompting; knight and apple did not. Measure: does folding the critique into the retry prompt actually move the pass rate vs a plain reroll (new seed only)? Cheap A/B once #17's mock tiers land. - [x] **Judge-at-upscale experiment.** Critics currently see native-resolution pixel art (16-96px). The brain reads them fine (verdict reasoning is coherent), but test whether nearest-neighbor upscaling before judgment changes verdict quality/consistency — the review page upscales for humans for a reason. - [x] **CLAP threshold tuning.** 0.2 inherited from agentic-game-dev; all bench audio passed. Once enough audio verdicts accumulate, check the threshold against human judgment (the review page's audio players make spot-checks cheap). - [x] **Motion-clip critique.** Raw animate clips now land in trogdor-files intermediates but are unjudged — the sheet is judged as a static strip. Investigate judging the CLIP (frame sampling to the brain, or the HunyuanVideo-Foley-adjacent tooling from #16) for motion quality the strip can't show. **Decision output per item:** adopt / reject / defer with trigger, recorded here and in the research docs.
cmoriarty commented 2026-07-25 03:59:27 -04:00
Author
Owner
Copy link

Investigated all five items with ledger data + live experiments; decisions below, full evidence in docs/critic-calibration.md (fefc02e). Code shipped: b5359b2 (deployed).

Decisions

1 · Visual strictness — ADOPT current strictness. Latest bench: 3/5 pass, and both fails are specific + actionable (apple: brown leaf vs "green leaf"/"grass green"; chest: style drift). First-cycle 4/5-fail improved to 2/5 without touching the critics — generation moved, the bar didn't. High first-pass failure is intended; bounded regen exists for it. Corollary live-catch: borderline verdict instability comes from style text that lies about the pixels — the fixture's "no hard outlines" claim (its knights HAVE dark edges) made the judge flip pass/fail on identical input across fresh runs. Fixed the fixture style block (23350e12); baseline stable at 6 pass / 1 fail again. Seat-side outline enforcement: defer — trigger: outline complaints in ≥½ of image fails across ≥3 real games.

2 · Regen effectiveness — DEFER with measurement plan. n=1 today (chest flipped fail→pass critique-fed). Retries are already identifiable on the ledger (the "Previous attempt was rejected" prompt marker + per-target verdict sequences). Trigger: ≥10 first-fail requeues across real games → compute critique-fed flip rate from history, add a one-flag plain-reroll B-arm for the comparison batch.

3 · Judge-at-upscale — ADOPTED, shipped. Direct evidence from #22: the judge called a crisp 8px checkerboard "extremely blurry" (its vision input resamples softly) and failed an unfailable "hard edges" demand. critics.upscale_for_judge() now integer-NN-upscales to a 384px edge before judgment (drain critics + /critique; CRITIC_UPSCALE=0 disables). A/B on the fixture: 6/7 consistent, crisper pixel-specific notes, zero adverse flips (the one flip was a ≥384px sheet that bypasses upscale — the item-1 borderline case, not an upscale effect). Known limitation: wide sheets pass through, so frames still reach the judge soft — revisit with item 5.

4 · CLAP 0.2 — KEEP. All ledger scores: true matches 0.29–0.68 (n=8), mismatches ≤0.07 (n=3, sine-stub sentinels). The threshold sits in a wide empty gap. Trigger to revisit: any human-disputed audio verdict, or any score inside 0.1–0.3.

5 · Motion-clip critique — DEFER. Strip judging already produces motion verdicts (fixture card-flip failed on "animation logic" from the sheet alone), and raw clips are retained 14d in intermediates. Trigger: an animate asset that passes strip judging but fails human eyeballs, or #16's video tooling. First experiment then: 4–6 sampled frames as a multi-image judgment.

Also recorded: temp-0.2 narrows but doesn't eliminate borderline nondeterminism — the commit-keyed verdict cache remains the production stabilizer; ground-truth style text is the fix for chronic flip-floppers.

All checkboxes have their decision; leaving open for your read + close.

Investigated all five items with ledger data + live experiments; decisions below, full evidence in **docs/critic-calibration.md** (fefc02e). Code shipped: `b5359b2` (deployed). ### Decisions **1 · Visual strictness — ADOPT current strictness.** Latest bench: 3/5 pass, and both fails are specific + actionable (apple: brown leaf vs "green leaf"/"grass green"; chest: style drift). First-cycle 4/5-fail improved to 2/5 without touching the critics — generation moved, the bar didn't. High first-pass failure is intended; bounded regen exists for it. Corollary live-catch: borderline verdict instability comes from style text that lies about the pixels — the fixture's "no hard outlines" claim (its knights HAVE dark edges) made the judge flip pass/fail on identical input across fresh runs. Fixed the fixture style block (23350e12); baseline stable at 6 pass / 1 fail again. Seat-side outline enforcement: defer — trigger: outline complaints in ≥½ of image fails across ≥3 real games. **2 · Regen effectiveness — DEFER with measurement plan.** n=1 today (chest flipped fail→pass critique-fed). Retries are already identifiable on the ledger (the "Previous attempt was rejected" prompt marker + per-target verdict sequences). Trigger: ≥10 first-fail requeues across real games → compute critique-fed flip rate from history, add a one-flag plain-reroll B-arm for the comparison batch. **3 · Judge-at-upscale — ADOPTED, shipped.** Direct evidence from #22: the judge called a crisp 8px checkerboard "extremely blurry" (its vision input resamples softly) and failed an unfailable "hard edges" demand. `critics.upscale_for_judge()` now integer-NN-upscales to a 384px edge before judgment (drain critics + `/critique`; `CRITIC_UPSCALE=0` disables). A/B on the fixture: 6/7 consistent, crisper pixel-specific notes, zero adverse flips (the one flip was a ≥384px sheet that bypasses upscale — the item-1 borderline case, not an upscale effect). Known limitation: wide sheets pass through, so frames still reach the judge soft — revisit with item 5. **4 · CLAP 0.2 — KEEP.** All ledger scores: true matches 0.29–0.68 (n=8), mismatches ≤0.07 (n=3, sine-stub sentinels). The threshold sits in a wide empty gap. Trigger to revisit: any human-disputed audio verdict, or any score inside 0.1–0.3. **5 · Motion-clip critique — DEFER.** Strip judging already produces motion verdicts (fixture card-flip failed on "animation logic" from the sheet alone), and raw clips are retained 14d in intermediates. Trigger: an animate asset that passes strip judging but fails human eyeballs, or #16's video tooling. First experiment then: 4–6 sampled frames as a multi-image judgment. Also recorded: temp-0.2 narrows but doesn't eliminate borderline nondeterminism — the commit-keyed verdict cache remains the production stabilizer; ground-truth style text is the fix for chronic flip-floppers. All checkboxes have their decision; leaving open for your read + close.
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
feat/11-walking-skeleton
No results found.
Labels
Clear labels   
bug
Something is not working

  
duplicate
This issue or pull request already exists

  
enhancement
New feature

  
help wanted
Need some help

  
invalid
Something is wrong

  
question
More information is needed

  
wontfix
This won't be fixed

No labels
bug
duplicate
enhancement
help wanted
invalid
question
wontfix
Milestone
Clear milestone
No items
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
cmoriarty/trog#18
No description provided.