cmoriarty/trog
1
0
Fork 0
Code Issues 11 Pull requests Projects 1 Releases Packages Wiki Activity Actions

06.1 · Drain absorption: trog drain/bench, conductor + bash profiles retire #22

New issue
Closed
opened 2026-07-25 01:04:32 -04:00 by cmoriarty · 5 comments
cmoriarty commented 2026-07-25 01:04:32 -04:00
Owner
Copy link

Phase 2 of the command layer (#21). The orchestrator already owns everything a drain needs — seat lifecycle, GPU-conflict parallelism (audio ∥ animate = the conductor's stage-B), brain bracketing with always-restore, and the queue — so batch drains move in and the bash stack retires.

  • trog drain — consume origin='drain' jobs through the orchestrator: image exclusive first, then audio ∥ animate (the existing runnable() matrix already encodes this), post-drain critics → review page → mirror (port post_drain from conductor.sh into python)
  • trog bench — queue the canned big-and-obvious spec (asset-bench.sh becomes a spec constant + submit call)
  • Regen loop — critics requeue → second drain pass → convergence caps (cycle() semantics move over: commit-keyed verdict cache + prior-fail-never-requeues already live server-side)
  • Retire conductor.sh, scripts/asset-profiles/*.sh, relaunch/window scripts — the profiles' seat truth already lives in trog_lib.seats; delete the shims when nothing sources them
  • trog update real story — wrap check-updates/try-updates + image build/redeploy reporting; decide what stays laptop-side (builds stay make)
  • Audio --from — cover/repaint routes on the audio seat (audio-server image work in agentic-game-dev: expose ACE src_audio/cover + SAO-style restyle if feasible), then the orchestrator route + spec field (the 422 guard drops)

Exit: make assets-drain/make asset-bench deleted; a full drain runs trog drain end-to-end with the same guarantees (restore trap, per-item truth in the queue, review + mirror).

Phase 2 of the command layer (#21). The orchestrator already owns everything a drain needs — seat lifecycle, GPU-conflict parallelism (audio ∥ animate = the conductor's stage-B), brain bracketing with always-restore, and the queue — so batch drains move in and the bash stack retires. - [ ] **`trog drain`** — consume `origin='drain'` jobs through the orchestrator: image exclusive first, then audio ∥ animate (the existing runnable() matrix already encodes this), post-drain critics → review page → mirror (port `post_drain` from conductor.sh into python) - [ ] **`trog bench`** — queue the canned big-and-obvious spec (asset-bench.sh becomes a spec constant + submit call) - [ ] **Regen loop** — critics requeue → second drain pass → convergence caps (cycle() semantics move over: commit-keyed verdict cache + prior-fail-never-requeues already live server-side) - [ ] **Retire** conductor.sh, scripts/asset-profiles/*.sh, relaunch/window scripts — the profiles' seat truth already lives in trog_lib.seats; delete the shims when nothing sources them - [ ] **`trog update` real story** — wrap check-updates/try-updates + image build/redeploy reporting; decide what stays laptop-side (builds stay make) - [ ] **Audio `--from`** — cover/repaint routes on the audio seat (audio-server image work in agentic-game-dev: expose ACE src_audio/cover + SAO-style restyle if feasible), then the orchestrator route + spec field (the 422 guard drops) Exit: `make assets-drain`/`make asset-bench` deleted; a full drain runs `trog drain` end-to-end with the same guarantees (restore trap, per-item truth in the queue, review + mirror).
cmoriarty commented 2026-07-25 01:20:34 -04:00
Author
Owner
Copy link

Implemented in 2435df4 (main).

Done

  • trog drain — drain engine moved into the orchestrator (trog_lib.drain): stage A image exclusive, stage B audio ∥ animate via the existing runnable() GPU matrix; POST /drain + GET /drain event feed, CLI streams the same spinner UX as one-offs. Per-item truth stays in the queue; a lane's seat failure fails its job and the rest continues.
  • Post-drain — critics run in-process (requeue semantics untouched) → review page (trog_lib.review, async port of asset-review.py) → mirror via the Forgejo archive API into /mnt/output/games/ (no git binary in the container).
  • Regen loop — cycle() ported: 3-pass cap; commit-keyed verdict cache + prior-fail-never-requeues already live server-side.
  • trog bench — spec constant + POST /bench; trog bench --drain chains into a drain.
  • trog update — CLI self-update hint + runs check-updates.py from a checkout (cwd or TROG_SRC); --try runs try-updates. Builds/deploys stay make deploy.
  • Retired — conductor.sh, asset-bench.sh, seat.sh, cycle-test.sh, asset-review.py, scripts/asset-profiles/, and the assets-drain asset-bench cycle-test seat-* make targets. critic-test.sh now renders the review page in-container via python -m trog_lib.review.
  • Restore guarantee: seats release through the SeatManager (brain always restored); boot-time reconcile() covers a crash mid-drain.

Not done (blocked)

  • Audio --from (cover/repaint) — needs ACE src_audio/cover routes in the audio-server image (agentic-game-dev). The 422 guard stays and now points here.

72 tests pass (5 new in tests/test_drain.py), ruff clean. Ship: make deploy + uv tool upgrade trog-cli. Mock cycle = trog drain against an ORCH_SEAT_MODE=mock orchestrator until phase 3's trog test cycle.

Implemented in 2435df4 (main). **Done** - **`trog drain`** — drain engine moved into the orchestrator (`trog_lib.drain`): stage A image exclusive, stage B audio ∥ animate via the existing `runnable()` GPU matrix; `POST /drain` + `GET /drain` event feed, CLI streams the same spinner UX as one-offs. Per-item truth stays in the queue; a lane's seat failure fails its job and the rest continues. - **Post-drain** — critics run in-process (requeue semantics untouched) → review page (`trog_lib.review`, async port of asset-review.py) → mirror via the Forgejo **archive API** into `/mnt/output/games/` (no git binary in the container). - **Regen loop** — `cycle()` ported: 3-pass cap; commit-keyed verdict cache + prior-fail-never-requeues already live server-side. - **`trog bench`** — spec constant + `POST /bench`; `trog bench --drain` chains into a drain. - **`trog update`** — CLI self-update hint + runs `check-updates.py` from a checkout (cwd or `TROG_SRC`); `--try` runs try-updates. Builds/deploys stay `make deploy`. - **Retired** — conductor.sh, asset-bench.sh, seat.sh, cycle-test.sh, asset-review.py, scripts/asset-profiles/, and the `assets-drain asset-bench cycle-test seat-*` make targets. critic-test.sh now renders the review page in-container via `python -m trog_lib.review`. - Restore guarantee: seats release through the SeatManager (brain always restored); boot-time `reconcile()` covers a crash mid-drain. **Not done (blocked)** - Audio `--from` (cover/repaint) — needs ACE src_audio/cover routes in the audio-server image (agentic-game-dev). The 422 guard stays and now points here. 72 tests pass (5 new in tests/test_drain.py), ruff clean. Ship: `make deploy` + `uv tool upgrade trog-cli`. Mock cycle = `trog drain` against an `ORCH_SEAT_MODE=mock` orchestrator until phase 3's `trog test cycle`.
cmoriarty commented 2026-07-25 01:25:56 -04:00
Author
Owner
Copy link

Test instructions (2435df4)

0. Unit tier (laptop, no deploy)

make test lint

Expect: 72 pass (5 new in tests/test_drain.py), ruff clean.

1. Ship it

make deploy                # new orchestrator image (drain engine rides in it)
uv tool upgrade trog-cli   # CLI 0.2.0: drain / bench / update
trog status                # orchestrator ok, brain up, queue empty

2. Mock cycle (plumbing, no GPUs, brain untouched)
Set ORCH_SEAT_MODE=mock in the Portainer stack env, redeploy, then:

trog bench
trog drain

Expect: stage A then "stage B: audio + animate in parallel" in the event stream; per-job done lines; critics summary line per game; "review page updated"; "mirrored -> http://trogdor:3923/games/asset-bench/assets/review.html". Open the review URL — cards render, mock checkerboards fine here. Queue empty after (trog status).
Unset ORCH_SEAT_MODE and redeploy before step 3.

3. Real bench (the actual quality regression, ~30 min)

trog bench --drain

Watch for: brain goes down once for stage A; audio and animate genuinely overlap in stage B; brain restored at the end (trog status shows brain up). Judge the review page by eye — big knight/apple/chest are the point.
If critics fail anything first-time, expect a second pass in the same run ("critics requeued work — regeneration pass next"), capped at 3.

4. Guarantees to poke

  • Ctrl-C the CLI mid-drain: drain keeps running server-side; trog drain re-attaches (409 → "already running — attaching").
  • Kill the orchestrator container mid-drain: on restart, reconcile() removes orphan seats and restores the brain.
  • Queue a trog sfx "finger snap" one-off during stage A: it must NOT be consumed by the drain (origin split) and runs when GPUs free up.

5. Leftovers still on make

make critic-test           # now renders review via python -m trog_lib.review in-container
trog update                # drift report; --try for scratch-venv pin test

Regression watch: make assets-drain / make asset-bench are gone — anything scripted against them needs the trog commands.

## Test instructions (2435df4) **0. Unit tier (laptop, no deploy)** ``` make test lint ``` Expect: 72 pass (5 new in `tests/test_drain.py`), ruff clean. **1. Ship it** ``` make deploy # new orchestrator image (drain engine rides in it) uv tool upgrade trog-cli # CLI 0.2.0: drain / bench / update trog status # orchestrator ok, brain up, queue empty ``` **2. Mock cycle (plumbing, no GPUs, brain untouched)** Set `ORCH_SEAT_MODE=mock` in the Portainer stack env, redeploy, then: ``` trog bench trog drain ``` Expect: stage A then "stage B: audio + animate in parallel" in the event stream; per-job done lines; critics summary line per game; "review page updated"; "mirrored -> http://trogdor:3923/games/asset-bench/assets/review.html". Open the review URL — cards render, mock checkerboards fine here. Queue empty after (`trog status`). Unset `ORCH_SEAT_MODE` and redeploy before step 3. **3. Real bench (the actual quality regression, ~30 min)** ``` trog bench --drain ``` Watch for: brain goes down once for stage A; audio and animate genuinely overlap in stage B; brain restored at the end (`trog status` shows brain up). Judge the review page by eye — big knight/apple/chest are the point. If critics fail anything first-time, expect a second pass in the same run ("critics requeued work — regeneration pass next"), capped at 3. **4. Guarantees to poke** - Ctrl-C the CLI mid-drain: drain keeps running server-side; `trog drain` re-attaches (409 → "already running — attaching"). - Kill the orchestrator container mid-drain: on restart, `reconcile()` removes orphan seats and restores the brain. - Queue a `trog sfx "finger snap"` one-off during stage A: it must NOT be consumed by the drain (origin split) and runs when GPUs free up. **5. Leftovers still on make** ``` make critic-test # now renders review via python -m trog_lib.review in-container trog update # drift report; --try for scratch-venv pin test ``` Regression watch: `make assets-drain` / `make asset-bench` are gone — anything scripted against them needs the trog commands.
cmoriarty commented 2026-07-25 01:32:13 -04:00
Author
Owner
Copy link

Update (5537f21): no stack-env fiddling needed for the mock tier anymore — --mock is a per-run flag.

Step 2 becomes:

trog bench --mock    # queues the small cycle spec into trog-games/cycle-test (purged first)
trog drain --mock    # in-process stub seats for this run: no GPUs, brain untouched, no redeploy

or in one line: trog bench --mock --drain.

Expect the same checks as before (stage A → "stage B: audio + animate in parallel", critics line, review page, mirror at :3923/games/cycle-test/…) plus a first event line "mock drain: local stub seats, brain untouched". Critics/review/mirror stay real — only the seats are stubs.

Guard ported from cycle-test.sh: trog drain --mock refuses (409, with the game list) while any real game has queued/claimed/running drain work — mock output overwriting real assets was the live-caught failure this guards against. Extra poke for step 4: queue trog bench (real) then try trog drain --mock — expect the refusal naming trog-games/asset-bench.

ORCH_SEAT_MODE=mock still exists for a fully GPU-less orchestrator (mocks one-offs too); the flag doesn't replace it, it just makes the plumbing tier one command.

Update (5537f21): no stack-env fiddling needed for the mock tier anymore — `--mock` is a per-run flag. **Step 2 becomes:** ``` trog bench --mock # queues the small cycle spec into trog-games/cycle-test (purged first) trog drain --mock # in-process stub seats for this run: no GPUs, brain untouched, no redeploy ``` or in one line: `trog bench --mock --drain`. Expect the same checks as before (stage A → "stage B: audio + animate in parallel", critics line, review page, mirror at `:3923/games/cycle-test/…`) plus a first event line "mock drain: local stub seats, brain untouched". Critics/review/mirror stay real — only the seats are stubs. Guard ported from cycle-test.sh: `trog drain --mock` refuses (409, with the game list) while any real game has queued/claimed/running drain work — mock output overwriting real assets was the live-caught failure this guards against. Extra poke for step 4: queue `trog bench` (real) then try `trog drain --mock` — expect the refusal naming trog-games/asset-bench. `ORCH_SEAT_MODE=mock` still exists for a fully GPU-less orchestrator (mocks one-offs too); the flag doesn't replace it, it just makes the plumbing tier one command.
cmoriarty commented 2026-07-25 01:48:20 -04:00
Author
Owner
Copy link

Naming pass (7e9ced9) — drain/bench described queue plumbing, not intent, and bench didn't run what it queued. New CLI surface:

old new what it does
trog drain trog process-asset-queue run everything queued: generate (image exclusive → audio ∥ animate) → critics → review page → mirror, regen passes until converged
trog bench --mock --drain trog asset-mock-test plumbing smoke test, end-to-end by default: small canned spec + in-process stub seats; minutes, zero GPUs, brain untouched. Refuses while real games have queued work
trog bench --drain trog asset-test fixed-input quality check, end-to-end by default: canned big-and-obvious spec on the REAL seats (~30 min). Same spec every run so quality drift shows on the review page

Both test commands take --queue-only to just enqueue (then trog process-asset-queue later). The --drain combo flag is gone — tests run what they queue.

Revised test instructions:

make test lint                # 74 pass, ruff clean
make deploy && uv tool upgrade trog-cli   # CLI 0.3.0
trog asset-mock-test          # step 2: smoke test — no env fiddling, no redeploy
trog asset-test               # step 3: real quality check, eyeball the review page

Step 4 guard poke: trog asset-test --queue-only then trog asset-mock-test — expect the 409 naming trog-games/asset-bench.

Server API (POST /drain, POST /bench) and trog_lib.drain keep their internal names — only the user-facing layer renamed.

Naming pass (7e9ced9) — `drain`/`bench` described queue plumbing, not intent, and `bench` didn't run what it queued. New CLI surface: | old | new | what it does | |---|---|---| | `trog drain` | `trog process-asset-queue` | run everything queued: generate (image exclusive → audio ∥ animate) → critics → review page → mirror, regen passes until converged | | `trog bench --mock --drain` | `trog asset-mock-test` | plumbing smoke test, end-to-end by default: small canned spec + in-process stub seats; minutes, zero GPUs, brain untouched. Refuses while real games have queued work | | `trog bench --drain` | `trog asset-test` | fixed-input quality check, end-to-end by default: canned big-and-obvious spec on the REAL seats (~30 min). Same spec every run so quality drift shows on the review page | Both test commands take `--queue-only` to just enqueue (then `trog process-asset-queue` later). The `--drain` combo flag is gone — tests run what they queue. Revised test instructions: ``` make test lint # 74 pass, ruff clean make deploy && uv tool upgrade trog-cli # CLI 0.3.0 trog asset-mock-test # step 2: smoke test — no env fiddling, no redeploy trog asset-test # step 3: real quality check, eyeball the review page ``` Step 4 guard poke: `trog asset-test --queue-only` then `trog asset-mock-test` — expect the 409 naming trog-games/asset-bench. Server API (`POST /drain`, `POST /bench`) and `trog_lib.drain` keep their internal names — only the user-facing layer renamed.
cmoriarty commented 2026-07-25 03:38:19 -04:00
Author
Owner
Copy link

Final report — closing

Everything on this ticket is implemented, deployed (stack + image current, CLI 0.4.2), and verified end-to-end on the real system.

Delivered

  • trog process-asset-queue — drain engine in the orchestrator (trog_lib.drain): image exclusive → audio ∥ animate, post-drain critics → review page (trog_lib.review) → mirror via Forgejo archive API, 3-pass regen cycle. Present-tense active map drives the CLI spinner; events are dim history.
  • trog asset-mock-test — end-to-end plumbing smoke test: canned spec with declared expectations (pass briefs describe the stub media, fail briefs are mismatch sentinels), per-run in-process stub seats, fresh judging, PLUMBING OK/FAILED verdict, expected verdicts: N/M before the review URL. Review page chips each declared asset expected pass|fail + ✓ as expected/✗ unexpected with a legend.
  • trog asset-test — real-seat fixed-input quality check, end-to-end by default (--queue-only to just enqueue).
  • trog update — CLI hint + check-updates/try-updates drift report (currently: brain image + docker pin behind).
  • Retired: conductor.sh, asset-bench.sh, seat.sh, cycle-test.sh, asset-review.py, asset-profiles/, their make targets. critic-test.sh now renders reviews via python -m trog_lib.review in-container (verified: 7 judged, orange sentinel fails, review served).

Verified (final round)

  • asset-mock-test: PLUMBING OK, 7/7 expected verdicts, 193s, brain untouched.
  • Guard both ways: real work queued → mock test refuses before queueing (first version queued then refused — fixed).
  • asset-test: full 34-min real run — brain swapped out once and restored, FLUX/Wan/audio all delivered, critics 8 judged / 2 failed (apple leaf color, chest style — legit #18 calibration data), review + mirror verified visually.
  • process-asset-queue on empty queue: clean no-op. trog status, trog update: working.

Live-caught bugs fixed along the way

  1. Idle sweep reaped a seat mid-render — FLUX outlives the 180s keep-warm TTL and last_used only moves on item completion; the sweep stopped the image seat under it. SeatManager.busy() hold now marks in-flight generations; sweep skips held seats. (The mock tier structurally can't catch this — stub items finish in ms.)
  2. Forgejo contents probe served a stale per-path 404 on the slash form of path?ref=main (file in tree, raw 200) → every overwrite became a doomed create (422). Contents paths now %2F-encoded.
  3. Empty-diff commits 422 — regenerated identical media now no-ops instead of committing.
  4. Judge resolution: the brain's vision input upscales 96px art softly, so briefs demanding "hard edges" are unfailable — recorded as direct evidence for #18's judge-at-upscale item.
  5. Rich markup ate [game]/[1/3] in CLI output; verdict cache pinned stale verdicts for never-changing stub bytes (mock runs judge fresh now).

Left open (tracked elsewhere)

  • Audio --from (cover/repaint): blocked on ACE src_audio/cover routes in the audio-server image (agentic-game-dev). The 422 guard stays.
  • Critic strictness/threshold work → #18 (this ticket's runs already produced fresh verdict data).

Exit criteria met: make assets-drain/make asset-bench deleted; a full drain runs through trog end-to-end with the same guarantees (restore trap via SeatManager + boot reconcile, per-item truth in the queue, review + mirror).

## Final report — closing Everything on this ticket is implemented, deployed (stack + image current, CLI 0.4.2), and verified end-to-end on the real system. ### Delivered - **`trog process-asset-queue`** — drain engine in the orchestrator (`trog_lib.drain`): image exclusive → audio ∥ animate, post-drain critics → review page (`trog_lib.review`) → mirror via Forgejo archive API, 3-pass regen cycle. Present-tense `active` map drives the CLI spinner; events are dim history. - **`trog asset-mock-test`** — end-to-end plumbing smoke test: canned spec with **declared expectations** (pass briefs describe the stub media, fail briefs are mismatch sentinels), per-run in-process stub seats, fresh judging, `PLUMBING OK/FAILED` verdict, `expected verdicts: N/M` before the review URL. Review page chips each declared asset `expected pass|fail` + `✓ as expected`/`✗ unexpected` with a legend. - **`trog asset-test`** — real-seat fixed-input quality check, end-to-end by default (`--queue-only` to just enqueue). - **`trog update`** — CLI hint + check-updates/try-updates drift report (currently: brain image + docker pin behind). - **Retired**: conductor.sh, asset-bench.sh, seat.sh, cycle-test.sh, asset-review.py, asset-profiles/, their make targets. critic-test.sh now renders reviews via `python -m trog_lib.review` in-container (verified: 7 judged, orange sentinel fails, review served). ### Verified (final round) - `asset-mock-test`: **PLUMBING OK, 7/7 expected verdicts**, 193s, brain untouched. - Guard both ways: real work queued → mock test refuses **before queueing** (first version queued then refused — fixed). - `asset-test`: full 34-min real run — brain swapped out once and restored, FLUX/Wan/audio all delivered, critics 8 judged / 2 failed (apple leaf color, chest style — legit #18 calibration data), review + mirror verified visually. - `process-asset-queue` on empty queue: clean no-op. `trog status`, `trog update`: working. ### Live-caught bugs fixed along the way 1. **Idle sweep reaped a seat mid-render** — FLUX outlives the 180s keep-warm TTL and `last_used` only moves on item completion; the sweep stopped the image seat under it. `SeatManager.busy()` hold now marks in-flight generations; sweep skips held seats. (The mock tier structurally can't catch this — stub items finish in ms.) 2. **Forgejo contents probe served a stale per-path 404** on the slash form of `path?ref=main` (file in tree, raw 200) → every overwrite became a doomed create (422). Contents paths now %2F-encoded. 3. **Empty-diff commits 422** — regenerated identical media now no-ops instead of committing. 4. **Judge resolution**: the brain's vision input upscales 96px art softly, so briefs demanding "hard edges" are unfailable — recorded as direct evidence for #18's judge-at-upscale item. 5. Rich markup ate `[game]`/`[1/3]` in CLI output; verdict cache pinned stale verdicts for never-changing stub bytes (mock runs judge fresh now). ### Left open (tracked elsewhere) - **Audio `--from`** (cover/repaint): blocked on ACE src_audio/cover routes in the audio-server image (agentic-game-dev). The 422 guard stays. - Critic strictness/threshold work → #18 (this ticket's runs already produced fresh verdict data). Exit criteria met: `make assets-drain`/`make asset-bench` deleted; a full drain runs through `trog` end-to-end with the same guarantees (restore trap via SeatManager + boot reconcile, per-item truth in the queue, review + mirror).
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
feat/11-walking-skeleton
No results found.
Labels
Clear labels   
bug
Something is not working

  
duplicate
This issue or pull request already exists

  
enhancement
New feature

  
help wanted
Need some help

  
invalid
Something is wrong

  
question
More information is needed

  
wontfix
This won't be fixed

No labels
bug
duplicate
enhancement
help wanted
invalid
question
wontfix
Milestone
Clear milestone
No items
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
cmoriarty/trog#22
No description provided.