cmoriarty/trog
1
0
Fork 0
Code Issues 11 Pull requests Projects 1 Releases Packages Wiki Activity Actions

05.4 · Targeted service testing: mock cycle-test, seat CLI, fixture repo #17

New issue
Closed
opened 2026-07-24 11:00:03 -04:00 by cmoriarty · 6 comments
cmoriarty commented 2026-07-24 11:00:03 -04:00
Owner
Copy link

Full-cycle validation (brain down, ~30 min) has become the only way to test new pipeline functionality — reserve it as the FINAL regression (user decision, 2026-07-24) and build the intermediate tiers below. Root causes being attacked: the brain's GPU monopoly, items only flowing through queue→conductor, and seat lifecycle being trapped in conductor context (leading to bespoke docker commands).

Testing pyramid after this ticket: pytest → mock cycle-test → seat one-offs → full bench cycle (final regression only).

Tier 1 — mock seat + make cycle-test (the daily driver)

  • scripts/mock-seat.py: one stub process implementing ComfyUI's contract (/prompt, /history/{id}, /view, /upload/image, /system_stats) AND the audio-server contract (/music, /sfx, /health), returning canned PNG / short MP4 / WAV instantly
  • Profile hook: SEAT_MODE=mock makes _seat.sh start the stub instead of a container (no GPUs, brain untouched)
  • make cycle-test: queue the bench spec into a scratch repo → full conductor cycle against mocks → critics run against the REAL brain (it's up) → review page + mirror → assert queue all-done, verdicts recorded, review served. Target: under ~3 minutes
  • Honest scope note: mocks validate plumbing (queue, lanes, drivers, sheet assembly, commits, critics, review, mirror) — model-contract bugs (VAE pairing, duration floors) remain full-cycle territory by nature

Tier 2 — seat CLI (one-off services as a system feature)

  • scripts/seat.sh up <profile> [--cpu|--take-gpus] / run <profile> '<item-json>' [--repo <scratch>] / down — reuses _seat.sh + the real profiles verbatim so the test path IS the production path
  • Policy: GPU seats refuse to start while the brain is busy unless --take-gpus (which brackets brain down/up with the conductor's restore semantics); --cpu for plumbing-grade runs
  • Outputs land in a scratch repo or /drop on trogdor-files, never a real game
  • make seat-* conveniences

Tier 3 — standing fixture repo

  • trog-games/fixture: permanent, small committed asset set + spec doc — critics, review generation, and mirroring testable in seconds with zero generation
  • Never auto-deleted by cleanups; documented in the README testing section

Noted, not scoped: brain small-mode (reduced-footprint brain freeing 2 GPUs for a live seat beside it) — only if Tier 2's --take-gpus bracket proves too slow in practice.

Full-cycle validation (brain down, ~30 min) has become the only way to test new pipeline functionality — reserve it as the FINAL regression (user decision, 2026-07-24) and build the intermediate tiers below. Root causes being attacked: the brain's GPU monopoly, items only flowing through queue→conductor, and seat lifecycle being trapped in conductor context (leading to bespoke docker commands). **Testing pyramid after this ticket:** pytest → mock cycle-test → seat one-offs → full bench cycle (final regression only). ### Tier 1 — mock seat + `make cycle-test` (the daily driver) - [ ] `scripts/mock-seat.py`: one stub process implementing ComfyUI's contract (`/prompt`, `/history/{id}`, `/view`, `/upload/image`, `/system_stats`) AND the audio-server contract (`/music`, `/sfx`, `/health`), returning canned PNG / short MP4 / WAV instantly - [ ] Profile hook: `SEAT_MODE=mock` makes `_seat.sh` start the stub instead of a container (no GPUs, brain untouched) - [ ] `make cycle-test`: queue the bench spec into a scratch repo → full conductor cycle against mocks → critics run against the REAL brain (it's up) → review page + mirror → assert queue all-done, verdicts recorded, review served. Target: under ~3 minutes - Honest scope note: mocks validate plumbing (queue, lanes, drivers, sheet assembly, commits, critics, review, mirror) — model-contract bugs (VAE pairing, duration floors) remain full-cycle territory by nature ### Tier 2 — `seat` CLI (one-off services as a system feature) - [ ] `scripts/seat.sh up <profile> [--cpu|--take-gpus] / run <profile> '<item-json>' [--repo <scratch>] / down` — reuses `_seat.sh` + the real profiles verbatim so the test path IS the production path - [ ] Policy: GPU seats refuse to start while the brain is busy unless `--take-gpus` (which brackets brain down/up with the conductor's restore semantics); `--cpu` for plumbing-grade runs - [ ] Outputs land in a scratch repo or `/drop` on trogdor-files, never a real game - [ ] `make seat-*` conveniences ### Tier 3 — standing fixture repo - [ ] `trog-games/fixture`: permanent, small committed asset set + spec doc — critics, review generation, and mirroring testable in seconds with zero generation - [ ] Never auto-deleted by cleanups; documented in the README testing section **Noted, not scoped:** brain small-mode (reduced-footprint brain freeing 2 GPUs for a live seat beside it) — only if Tier 2's `--take-gpus` bracket proves too slow in practice.
cmoriarty commented 2026-07-24 17:01:44 -04:00
Author
Owner
Copy link

All three tiers implemented and green (commits 5d0d5fc, 1748eba). The mocks paid for themselves immediately: they exposed two latent production race bugs in the Forgejo commit path that slow real generation had been hiding — details below.

What landed:

  • Tier 1 — scripts/mock-seat.py (one stub, both seat contracts, canned PNG/MP4/WAV), SEAT_MODE=mock + SEAT_HOST indirection in the profiles, conductor leaves the brain alone in mock mode, make cycle-test with purge/queue/drain/critics/review/mirror + assertions. assetq grew a purge subcommand (deployed).
  • Tier 2 — scripts/seat.sh up/run/down reusing _seat.sh + profiles verbatim; GPU policy enforced (refusal message lists --take-gpus / --cpu / --mock); outputs default to trog-games/seat-scratch; SEAT_BASE_PATH lets animate one-offs skip the queue; make seat-up/seat-run/seat-down. This is the lifecycle the studio UI's one-off requests will ride (per the shelf-survey: no off-the-shelf component fits heterogeneous docker seats + brain GPU tenancy — Triton/Ray/KServe all assume dedicated GPUs and their own model formats).
  • Tier 3 — trog-games/fixture seeded with real generated assets (card-back, card-flip sheet, bg-title, card-snap, ambient-theme) + spec; make fixture-review.
  • README grew the testing-pyramid section.

The two bugs cycle-test caught on its first runs:

  1. Stale-HEAD reads — Forgejo's no-ref contents/raw endpoints resolve via HEAD, which lags fresh commits. Regen passes 404'd on files committed seconds earlier; the animate lane could miss a base sprite the image lane just landed. Every laptop-side read now pins ?ref=main.
  2. Branch-tip race — stage-B lanes commit to one repo in parallel; concurrent contents-API writes 500 when the tip moves mid-commit. Rare under real gen speeds, every-run under mocks. forgejo_commit now retries whole (fresh sha probe, backoff+jitter).

How to test:

make test              # 56 unit tests, includes new mock-seat contract tests (~2s)
make fixture-review    # critics + review + mirror vs trog-games/fixture (~6s cached, ~2.5min first run)
make cycle-test        # the daily driver: full conductor cycle vs mocks (~60-100s)
                       #   -> watch it purge, queue 5 items, drain 3 lanes, judge with the real
                       #      brain, render http://trogdor:3923/games/cycle-test/assets/review.html
make seat-up PROFILE=audio FLAGS=--mock
make seat-run PROFILE=audio ITEM='{"kind":"sfx","name":"test-snap","prompt":"a crisp snap","seconds":1}'
make seat-down PROFILE=audio
                       # one-off flow; asset lands in trog-games/seat-scratch
scripts/seat.sh up image   # (no flags) -> see the GPU-policy refusal with options

For a REAL one-off (brain bracket): make seat-up PROFILE=audio FLAGS=--take-gpus, run items, make seat-down PROFILE=audio restores the brain.

Full bench (make asset-bench + make assets-drain) remains the final regression, as scoped. Current state: 56 tests green, lint clean, cycle-test PASSED twice consecutively, brain + aegra healthy.

All three tiers implemented and green (commits 5d0d5fc, 1748eba). **The mocks paid for themselves immediately: they exposed two latent production race bugs** in the Forgejo commit path that slow real generation had been hiding — details below. **What landed:** - **Tier 1** — `scripts/mock-seat.py` (one stub, both seat contracts, canned PNG/MP4/WAV), `SEAT_MODE=mock` + `SEAT_HOST` indirection in the profiles, conductor leaves the brain alone in mock mode, `make cycle-test` with purge/queue/drain/critics/review/mirror + assertions. `assetq` grew a `purge` subcommand (deployed). - **Tier 2** — `scripts/seat.sh up/run/down` reusing `_seat.sh` + profiles verbatim; GPU policy enforced (refusal message lists `--take-gpus` / `--cpu` / `--mock`); outputs default to `trog-games/seat-scratch`; `SEAT_BASE_PATH` lets animate one-offs skip the queue; `make seat-up/seat-run/seat-down`. This is the lifecycle the studio UI's one-off requests will ride (per the shelf-survey: no off-the-shelf component fits heterogeneous docker seats + brain GPU tenancy — Triton/Ray/KServe all assume dedicated GPUs and their own model formats). - **Tier 3** — `trog-games/fixture` seeded with real generated assets (card-back, card-flip sheet, bg-title, card-snap, ambient-theme) + spec; `make fixture-review`. - README grew the testing-pyramid section. **The two bugs cycle-test caught on its first runs:** 1. **Stale-HEAD reads** — Forgejo's no-ref `contents`/`raw` endpoints resolve via HEAD, which lags fresh commits. Regen passes 404'd on files committed seconds earlier; the animate lane could miss a base sprite the image lane just landed. Every laptop-side read now pins `?ref=main`. 2. **Branch-tip race** — stage-B lanes commit to one repo in parallel; concurrent contents-API writes 500 when the tip moves mid-commit. Rare under real gen speeds, every-run under mocks. `forgejo_commit` now retries whole (fresh sha probe, backoff+jitter). **How to test:** ``` make test # 56 unit tests, includes new mock-seat contract tests (~2s) make fixture-review # critics + review + mirror vs trog-games/fixture (~6s cached, ~2.5min first run) make cycle-test # the daily driver: full conductor cycle vs mocks (~60-100s) # -> watch it purge, queue 5 items, drain 3 lanes, judge with the real # brain, render http://trogdor:3923/games/cycle-test/assets/review.html make seat-up PROFILE=audio FLAGS=--mock make seat-run PROFILE=audio ITEM='{"kind":"sfx","name":"test-snap","prompt":"a crisp snap","seconds":1}' make seat-down PROFILE=audio # one-off flow; asset lands in trog-games/seat-scratch scripts/seat.sh up image # (no flags) -> see the GPU-policy refusal with options ``` For a REAL one-off (brain bracket): `make seat-up PROFILE=audio FLAGS=--take-gpus`, run items, `make seat-down PROFILE=audio` restores the brain. Full bench (`make asset-bench` + `make assets-drain`) remains the final regression, as scoped. Current state: 56 tests green, lint clean, cycle-test PASSED twice consecutively, brain + aegra healthy.
cmoriarty commented 2026-07-24 17:23:07 -04:00
Author
Owner
Copy link

Good catch — that was a real tier-interaction bug, now fixed (commit 0277280, aegra redeployed):

What you saw: not intentional. The fixture is supposed to be REAL assets (mocks live only in trog-games/cycle-test). Chain of events: your make fixture-review ran critics → they failed 3 real assets → critics always requeued first-time failures → my later make cycle-test runs drained the whole queue with mock seats → mock output overwrote the permanent fixture. The "kaleidoscope" card-flip was the mock's testsrc2 clip assembled into a sheet; the two identical checkerboards were the 287-byte mock PNG.

Closures:

  1. critics run --no-requeue — fixture-review judges only, never queues regeneration (the fixture is permanent by definition).
  2. cycle-test now refuses to run if any other game has unfinished queued work (new assetq queued subcommand) — a mock drain can never eat real jobs again.
  3. Fixture reseeded with the real assets (card-back 71KB, bg-title 178KB, card-flip sheet 2.5MB back on the mirror).

Re-test: make fixture-review → http://trogdor:3923/games/fixture/assets/review.html now shows the real navy card-back, starfield bg-title, and the gold filigree card-flip sheet animating. 3 of 5 get honest FAIL verdicts from the critics (e.g. the sheet's frames don't really show a flip) — that's the fixture exercising both verdict paths, chips + critique notes included. Queue stays empty afterward (--no-requeue proof). cycle-test still PASSED (67s) with the guard in place.

Good catch — that was a real tier-interaction bug, now fixed (commit 0277280, aegra redeployed): **What you saw:** not intentional. The fixture is supposed to be REAL assets (mocks live only in `trog-games/cycle-test`). Chain of events: your `make fixture-review` ran critics → they failed 3 real assets → critics **always requeued** first-time failures → my later `make cycle-test` runs drained the whole queue **with mock seats** → mock output overwrote the permanent fixture. The "kaleidoscope" card-flip was the mock's testsrc2 clip assembled into a sheet; the two identical checkerboards were the 287-byte mock PNG. **Closures:** 1. `critics run --no-requeue` — fixture-review judges only, never queues regeneration (the fixture is permanent by definition). 2. `cycle-test` now **refuses to run** if any other game has unfinished queued work (new `assetq queued` subcommand) — a mock drain can never eat real jobs again. 3. Fixture reseeded with the real assets (card-back 71KB, bg-title 178KB, card-flip sheet 2.5MB back on the mirror). **Re-test:** `make fixture-review` → http://trogdor:3923/games/fixture/assets/review.html now shows the real navy card-back, starfield bg-title, and the gold filigree card-flip sheet animating. 3 of 5 get honest FAIL verdicts from the critics (e.g. the sheet's frames don't really show a flip) — that's the fixture exercising both verdict paths, chips + critique notes included. Queue stays empty afterward (`--no-requeue` proof). cycle-test still PASSED (67s) with the guard in place.
cmoriarty commented 2026-07-24 18:03:46 -04:00
Author
Owner
Copy link

Fixture feedback addressed (commit above, aegra redeployed):

  • Briefs were fiction — I wrote them before looking at the files. Spec now describes actual content (navy suit-row card-back, night-sky bg-title), and the header states the rule: fixture briefs are ground truth, not aspirations.
  • "card-flip" renamed card-shimmer — the sheet genuinely contains no flip, just the ornament shimmering; the entry now claims what the pixels do.
  • "Strangely large" — review pages applied the small-pixel-sprite 2× upscale to 704px frames (1408px wide!). Frames ≥128px now shrink to ~192px.
  • New critics run --fresh (FRESH=1 make fixture-review) — verdict cache is keyed to the asset commit, so brief/critic edits never re-judged unchanged pixels; fresh skips the cache for exactly these situations.

Fresh-judged with truthful briefs: 4 pass, 1 fail (card-shimmer — critic correctly flags the ivory/gold sheet against its navy base sprite). One standing honest fail is deliberate: the fixture exercises both verdict paths, chips + critique notes. Refresh http://trogdor:3923/games/fixture/assets/review.html to see it.

Fixture feedback addressed (commit above, aegra redeployed): - **Briefs were fiction** — I wrote them before looking at the files. Spec now describes actual content (navy suit-row card-back, night-sky bg-title), and the header states the rule: fixture briefs are ground truth, not aspirations. - **"card-flip" renamed `card-shimmer`** — the sheet genuinely contains no flip, just the ornament shimmering; the entry now claims what the pixels do. - **"Strangely large"** — review pages applied the small-pixel-sprite 2× upscale to 704px frames (1408px wide!). Frames ≥128px now shrink to ~192px. - **New `critics run --fresh`** (`FRESH=1 make fixture-review`) — verdict cache is keyed to the asset commit, so brief/critic edits never re-judged unchanged pixels; fresh skips the cache for exactly these situations. Fresh-judged with truthful briefs: **4 pass, 1 fail** (card-shimmer — critic correctly flags the ivory/gold sheet against its navy base sprite). One standing honest fail is deliberate: the fixture exercises both verdict paths, chips + critique notes. Refresh http://trogdor:3923/games/fixture/assets/review.html to see it.
cmoriarty commented 2026-07-24 18:10:13 -04:00
Author
Owner
Copy link

Renamed per feedback: make fixture-review → make critic-test (df07baa) — it judges the fixture's known assets, so the name now says what it tests. Usage unchanged: plain run = cached plumbing check (~10s), FRESH=1 make critic-test = real re-judgment after critic/brief edits (~3-5 min, baseline 4 pass / 1 fail). Earlier comments referencing fixture-review mean this command.

Renamed per feedback: `make fixture-review` → **`make critic-test`** (df07baa) — it judges the fixture's known assets, so the name now says what it tests. Usage unchanged: plain run = cached plumbing check (~10s), `FRESH=1 make critic-test` = real re-judgment after critic/brief edits (~3-5 min, baseline 4 pass / 1 fail). Earlier comments referencing fixture-review mean this command.
cmoriarty commented 2026-07-24 19:14:18 -04:00
Author
Owner
Copy link

Fixture reshaped per request — final baseline 6 pass / 1 fail, and the one fail is the deliberate sentinel:

  • orange — real FLUX-generated apple committed as orange.png with a brief describing an orange → critic FAILs it every time. Proves mismatch detection.
  • knight + knight-walk — new real generations (image seat + Wan animate via the tier-2 seat CLI with --take-gpus, its first real-GPU outing — worked cleanly, brain bracketed and restored both times). Walk cycle passes solidly: that is the pix3lwalk LoRA's home turf.
  • card-flip: your hunch was right. Two real attempts, two distinct critic-caught defects: attempt 1 revealed a franken-face (club indices, spade center pips); attempt 2 had a perspective break mid-flip ("the card fell over and then magically stood back up" — the critic's own words). The animate seat can't reliably do card physics, so card-flip is out of the fixture; noted as a seat capability ceiling.
  • knight brief + style Outline reworded to match the pixels (painterly shading, no hard outline, tan-gold plume) — knight passes now; the earlier fail was my brief overpromising.

Side benefit: the critics proved sharp — mixed suits on a card face and a mid-animation perspective break are exactly the catches we built them for.

Verify: make critic-test (~10s cached) → expect 7 judged (7 cached), 1 failed, page shows 6 green chips + orange's open FAIL critique: http://trogdor:3923/games/fixture/assets/review.html

Fixture reshaped per request — final baseline **6 pass / 1 fail**, and the one fail is the deliberate sentinel: - **orange** — real FLUX-generated apple committed as `orange.png` with a brief describing an orange → critic FAILs it every time. Proves mismatch detection. - **knight + knight-walk** — new real generations (image seat + Wan animate via the tier-2 seat CLI with `--take-gpus`, its first real-GPU outing — worked cleanly, brain bracketed and restored both times). Walk cycle passes solidly: that is the pix3lwalk LoRA's home turf. - **card-flip: your hunch was right.** Two real attempts, two distinct critic-caught defects: attempt 1 revealed a franken-face (club indices, spade center pips); attempt 2 had a perspective break mid-flip ("the card fell over and then magically stood back up" — the critic's own words). The animate seat can't reliably do card physics, so card-flip is out of the fixture; noted as a seat capability ceiling. - **knight brief + style Outline reworded to match the pixels** (painterly shading, no hard outline, tan-gold plume) — knight passes now; the earlier fail was my brief overpromising. Side benefit: the critics proved sharp — mixed suits on a card face and a mid-animation perspective break are exactly the catches we built them for. Verify: `make critic-test` (~10s cached) → expect `7 judged (7 cached), 1 failed`, page shows 6 green chips + orange's open FAIL critique: http://trogdor:3923/games/fixture/assets/review.html
cmoriarty commented 2026-07-24 22:52:38 -04:00
Author
Owner
Copy link

Final thoughts — 05.4 delivered its pyramid, and it earned its keep immediately.

What stands after this ticket:

  • make test (~2s) → make critic-test (~10s cached / ~9min fresh, baseline 6 pass + the orange sentinel) → make cycle-test (~60-100s, mocks, brain untouched) → seat one-offs (seat-up/run/down, mock/cpu/take-gpus) → full bench as the final regression. Each tier caught something real during its own build:
    • mocks exposed two latent production races (stale-HEAD reads, branch-tip commit collisions) — now fixed with ?ref=main pins + commit retries
    • the fixture exposed critic/brief coupling (--no-requeue, --fresh, truthful-briefs rule) and proved the critics catch real defects (franken card faces, mid-animation perspective breaks)
    • the seat CLI's --take-gpus bracket did four clean real-GPU sessions generating the fixture's assets
  • New queue hygiene: assetq purge/queued, and cycle-test refuses to mock-drain other games' work.

Honest debts this ticket leaves: the make-target surface is now large and stylistically mixed (bash-sourcing-bash, docker-exec'd python, heredocs), the seat CLI's GPU policy lives in shell flags, and one-offs still require knowing which profile owns which kind. That's not polish debt — it's the seam for the next redesign: #21 (06 · intuitive trog command layer) — one python CLI, central orchestrator owning seats/GPU tenancy, make retired to build duties. The plumbing built here (profiles, mock seats, fixture, queue, testing tiers) is what that layer will sit on.

Repo state: all commits pushed through f2239c7; aegra image live with critics --fresh/--no-requeue + assetq purge/queued; brain + aegra healthy; fixture at its documented baseline.

**Final thoughts — 05.4 delivered its pyramid, and it earned its keep immediately.** What stands after this ticket: - `make test` (~2s) → `make critic-test` (~10s cached / ~9min fresh, baseline 6 pass + the orange sentinel) → `make cycle-test` (~60-100s, mocks, brain untouched) → seat one-offs (`seat-up/run/down`, mock/cpu/take-gpus) → full bench as the final regression. Each tier caught something real during its own build: - mocks exposed two latent production races (stale-HEAD reads, branch-tip commit collisions) — now fixed with `?ref=main` pins + commit retries - the fixture exposed critic/brief coupling (`--no-requeue`, `--fresh`, truthful-briefs rule) and proved the critics catch real defects (franken card faces, mid-animation perspective breaks) - the seat CLI's `--take-gpus` bracket did four clean real-GPU sessions generating the fixture's assets - New queue hygiene: `assetq purge`/`queued`, and cycle-test refuses to mock-drain other games' work. Honest debts this ticket leaves: the make-target surface is now large and stylistically mixed (bash-sourcing-bash, docker-exec'd python, heredocs), the seat CLI's GPU policy lives in shell flags, and one-offs still require knowing which profile owns which kind. That's not polish debt — it's the seam for the next redesign: **#21 (06 · intuitive trog command layer)** — one python CLI, central orchestrator owning seats/GPU tenancy, make retired to build duties. The plumbing built here (profiles, mock seats, fixture, queue, testing tiers) is what that layer will sit on. Repo state: all commits pushed through f2239c7; aegra image live with critics `--fresh`/`--no-requeue` + assetq `purge`/`queued`; brain + aegra healthy; fixture at its documented baseline.
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
feat/11-walking-skeleton
No results found.
Labels
Clear labels   
bug
Something is not working

  
duplicate
This issue or pull request already exists

  
enhancement
New feature

  
help wanted
Need some help

  
invalid
Something is wrong

  
question
More information is needed

  
wontfix
This won't be fixed

No labels
bug
duplicate
enhancement
help wanted
invalid
question
wontfix
Milestone
Clear milestone
No items
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
cmoriarty/trog#17
No description provided.