cmoriarty/trog
1
0
Fork 0
Code Issues 11 Pull requests Projects 1 Releases Packages Wiki Activity Actions

06.2 · Testing absorption + make retirement: trog test, seat.sh retires, SSE polish #23

New issue
Closed
opened 2026-07-25 01:04:33 -04:00 by cmoriarty · 4 comments
cmoriarty commented 2026-07-25 01:04:33 -04:00
Owner
Copy link

Phase 3 of the command layer (#21), after 06.1 lands.

  • trog test cycle — the mock cycle-test as a trog command (orchestrator mock mode + drain path from 06.1; assertions unchanged: queue clean, critics reported, review served, other-games guard)
  • trog test critics — critic-test (fixture judge, --fresh passthrough)
  • Retire scripts/seat.sh + seat-* make targets (the orchestrator replaced the lifecycle), cycle-test.sh / critic-test.sh once wrapped
  • make shrinks to what make is good at: build, push, redeploy, test (pytest), lint, check-updates — everything else is trog
  • SSE job event streaming (replace the CLI's 2s polling; orchestrator already appends events)
  • --out — also download the artifact locally after done (design-doc leftover)
  • README/docs sweep: testing pyramid + CLI sections reflect the trog-only surface

Exit: the README's command surface is trog … plus a five-target Makefile.

Phase 3 of the command layer (#21), after 06.1 lands. - [x] **`trog test cycle`** — the mock cycle-test as a trog command (orchestrator mock mode + drain path from 06.1; assertions unchanged: queue clean, critics reported, review served, other-games guard) - [x] **`trog test critics`** — critic-test (fixture judge, `--fresh` passthrough) - [x] **Retire** scripts/seat.sh + seat-* make targets (the orchestrator replaced the lifecycle), cycle-test.sh / critic-test.sh once wrapped - [x] **make shrinks to what make is good at**: build, push, redeploy, test (pytest), lint, check-updates — everything else is `trog` - [x] **SSE** job event streaming (replace the CLI's 2s polling; orchestrator already appends events) - [x] **`--out`** — also download the artifact locally after done (design-doc leftover) - [x] README/docs sweep: testing pyramid + CLI sections reflect the trog-only surface Exit: the README's command surface is `trog …` plus a five-target Makefile.
cmoriarty commented 2026-07-25 09:28:47 -04:00
Author
Owner
Copy link

Status after 06.1 (#22, closed 2026-07-25) — part of this landed early:

Already done in #22:

  • Retire scripts/seat.sh + seat-* make targets — deleted, along with cycle-test.sh and the cycle-test/assets-drain/asset-bench targets
  • trog test cycle exists in substance as trog asset-mock-test: per-run in-process stub seats (no stack mock mode needed), cycle-test's other-games guard (refuses before queueing), plus stronger assertions than the original — declared per-asset verdict expectations (pass briefs describe stub media, fail sentinels) with a PLUMBING OK/FAILED exit verdict. Open question for this ticket: keep the asset-* names or fold under a trog test … subcommand family.

Still open here:

  • trog test critics — critic-test.sh is the last bash test script (it already renders the review via python -m trog_lib.review in-container; wrapping = fixture judge + --fresh passthrough + baseline assertion, 6 pass / 1 fail orange sentinel)
  • SSE job event streaming (CLI polls every 2-3s today; drain adds the active map worth streaming too)
  • --out artifact download
  • README/docs/Makefile final sweep (Makefile is already close: build/push/redeploy/deploy/ps/logs/health/hello/test/lint/check-updates/try-updates + critic-test)
Status after 06.1 (#22, closed 2026-07-25) — part of this landed early: **Already done in #22:** - ~~Retire scripts/seat.sh + seat-* make targets~~ — deleted, along with cycle-test.sh and the `cycle-test`/`assets-drain`/`asset-bench` targets - `trog test cycle` exists in substance as **`trog asset-mock-test`**: per-run in-process stub seats (no stack mock mode needed), cycle-test's other-games guard (refuses before queueing), plus stronger assertions than the original — declared per-asset verdict expectations (pass briefs describe stub media, fail sentinels) with a `PLUMBING OK/FAILED` exit verdict. Open question for this ticket: keep the `asset-*` names or fold under a `trog test …` subcommand family. **Still open here:** - `trog test critics` — critic-test.sh is the last bash test script (it already renders the review via `python -m trog_lib.review` in-container; wrapping = fixture judge + `--fresh` passthrough + baseline assertion, 6 pass / 1 fail orange sentinel) - SSE job event streaming (CLI polls every 2-3s today; drain adds the `active` map worth streaming too) - `--out` artifact download - README/docs/Makefile final sweep (Makefile is already close: build/push/redeploy/deploy/ps/logs/health/hello/test/lint/check-updates/try-updates + critic-test)
cmoriarty commented 2026-07-25 10:56:55 -04:00
Author
Owner
Copy link

Implemented — the trog test family (naming per user direction 2026-07-25)

trog test                   # quick suite: unit + critics + asset --mock (no GPU swaps)
trog test asset [--mock]    # canned asset run (replaces asset-test / asset-mock-test)
trog test critics [--fresh] # fixture judge + baseline assertion, server-side
trog test regression        # everything incl. the real-GPU run → PASS/FAIL report

Checkbox status:

  • ✅ trog test cycle → shipped as trog test asset --mock (assertions upgraded: queue clean, critics reported, review served, other-games guard, PLUS declared per-asset verdict expectations and a PLUMBING OK/FAILED exit verdict)
  • ✅ trog test critics → POST /test/critics runs fixture judge (--fresh passthrough) + baseline assertion (orange sentinel fails, rest pass) + review page + mirror, all server-side; critic-test.sh deleted
  • ✅ Retirements: critic-test.sh, hello-test.sh, _stream_filter.py gone (seat.sh/cycle-test.sh went in #22); make health/hello retired — trog status now probes aegra too
  • ✅ make shrinks to build-shaped work: login/build/push/redeploy/deploy/ps/logs/test/lint/check-updates/try-updates
  • ✅ SSE: /jobs/{id}/stream + /drain/stream; CLI streams events live and falls back to polling against older orchestrators
  • ✅ --out on image/music/sfx/animate — downloads the finished asset (git deliveries via the raw view). Verified: trog sfx "water drop" --out drop.wav → 176KB local file
  • ✅ README/docs sweep: testing pyramid + CLI sections are trog-only; command-layer.md phase 3 marked done

Verification: trog test → overall PASS (unit 79 / critics baseline ok / mock 7/7). trog test regression → overall PASS, exit 0 (real tier: 29 min, all 8 items generated and delivered).

The suite already paid for itself: the FIRST regression run failed its mock tier legitimately — a prior one-off's keep-warm seat had left the brain down when the mock drain's critics started (the mock path skipped brain restore on the assumption mock never touches the brain). Fixed: post-drain always releases idle seats and waits for brain health before judging. The failing report → fix → clean rerun cycle is exactly what this command is for.

CLI 0.5.0, stack deployed. Exit criteria met: README's command surface is trog … + a build-shaped Makefile.

## Implemented — the `trog test` family (naming per user direction 2026-07-25) ``` trog test # quick suite: unit + critics + asset --mock (no GPU swaps) trog test asset [--mock] # canned asset run (replaces asset-test / asset-mock-test) trog test critics [--fresh] # fixture judge + baseline assertion, server-side trog test regression # everything incl. the real-GPU run → PASS/FAIL report ``` **Checkbox status:** - ✅ `trog test cycle` → shipped as `trog test asset --mock` (assertions upgraded: queue clean, critics reported, review served, other-games guard, PLUS declared per-asset verdict expectations and a PLUMBING OK/FAILED exit verdict) - ✅ `trog test critics` → `POST /test/critics` runs fixture judge (--fresh passthrough) + baseline assertion (orange sentinel fails, rest pass) + review page + mirror, all server-side; critic-test.sh deleted - ✅ Retirements: critic-test.sh, hello-test.sh, _stream_filter.py gone (seat.sh/cycle-test.sh went in #22); `make health`/`hello` retired — `trog status` now probes aegra too - ✅ make shrinks to build-shaped work: login/build/push/redeploy/deploy/ps/logs/test/lint/check-updates/try-updates - ✅ SSE: `/jobs/{id}/stream` + `/drain/stream`; CLI streams events live and falls back to polling against older orchestrators - ✅ `--out` on image/music/sfx/animate — downloads the finished asset (git deliveries via the raw view). Verified: `trog sfx "water drop" --out drop.wav` → 176KB local file - ✅ README/docs sweep: testing pyramid + CLI sections are trog-only; command-layer.md phase 3 marked done **Verification:** `trog test` → overall PASS (unit 79 / critics baseline ok / mock 7/7). `trog test regression` → **overall PASS, exit 0** (real tier: 29 min, all 8 items generated and delivered). **The suite already paid for itself:** the FIRST regression run failed its mock tier legitimately — a prior one-off's keep-warm seat had left the brain down when the mock drain's critics started (the mock path skipped brain restore on the assumption mock never touches the brain). Fixed: post-drain always releases idle seats and waits for brain health before judging. The failing report → fix → clean rerun cycle is exactly what this command is for. CLI 0.5.0, stack deployed. Exit criteria met: README's command surface is `trog …` + a build-shaped Makefile.
cmoriarty commented 2026-07-25 11:06:59 -04:00
Author
Owner
Copy link

Final thoughts — closing

Command layer is complete: all three phases (#21 one-offs, #22 drain absorption, #23 testing absorption) landed within two days, and the exit criterion holds — the README's command surface is trog … plus a build-shaped Makefile. Both trog test (PASS) and trog test regression (PASS, exit 0) verified on the deployed stack; final state: queue empty, brain up, aegra up, no stray seats, git and image current.

Things worth remembering from this ticket:

  1. The regression suite caught a real cross-feature bug on its very first run — a one-off's keep-warm seat left the brain down under a mock drain's critics. No single-tier test could have seen it: it needed a one-off followed by a mock cycle within the TTL window. That interaction class (keep-warm × consumer handoff) is where future bugs will live too.
  2. Testing UX principle that emerged (now also in the CLI design memory): a test command owns its whole story — it queues, runs, asserts, and prints its own verdict; the review page marks what SHOULD fail. Nobody should have to know baseline lore to read a result.
  3. Deploy loop friction is real: five build→push→webhook→wait cycles in this ticket. The webhook-needs-a-git-change trap is documented, but a make deploy that pushes first (or fails loudly on an unpushed tree) would remove the whole class — cheap future QoL.
  4. Deferred small items: SSE for one-off submission is done, but the studio UI will want a multiplexed stream (all jobs + drain in one feed) — that's a v2/UI concern. trog test critics --fresh exists for post-critic-edit runs; the fixture style-block ground-truth rule (#18) is what keeps that baseline stable.

Phase 4 (studio UI rides the same API) is the only remaining command-layer story, and it belongs to the studio tickets (#7–#10).

## Final thoughts — closing Command layer is complete: all three phases (#21 one-offs, #22 drain absorption, #23 testing absorption) landed within two days, and the exit criterion holds — the README's command surface is `trog …` plus a build-shaped Makefile. Both `trog test` (PASS) and `trog test regression` (PASS, exit 0) verified on the deployed stack; final state: queue empty, brain up, aegra up, no stray seats, git and image current. Things worth remembering from this ticket: 1. **The regression suite caught a real cross-feature bug on its very first run** — a one-off's keep-warm seat left the brain down under a mock drain's critics. No single-tier test could have seen it: it needed a one-off followed by a mock cycle within the TTL window. That interaction class (keep-warm × consumer handoff) is where future bugs will live too. 2. **Testing UX principle that emerged** (now also in the CLI design memory): a test command owns its whole story — it queues, runs, asserts, and prints its own verdict; the review page marks what SHOULD fail. Nobody should have to know baseline lore to read a result. 3. **Deploy loop friction is real**: five build→push→webhook→wait cycles in this ticket. The webhook-needs-a-git-change trap is documented, but a `make deploy` that pushes first (or fails loudly on an unpushed tree) would remove the whole class — cheap future QoL. 4. Deferred small items: SSE for one-off submission is done, but the studio UI will want a multiplexed stream (all jobs + drain in one feed) — that's a v2/UI concern. `trog test critics --fresh` exists for post-critic-edit runs; the fixture style-block ground-truth rule (#18) is what keeps that baseline stable. Phase 4 (studio UI rides the same API) is the only remaining command-layer story, and it belongs to the studio tickets (#7–#10).
cmoriarty commented 2026-07-25 11:10:38 -04:00
Author
Owner
Copy link

Post-close QoL from the final thoughts, landed as 5247e80 — make redeploy now guards the git-connected-stack trap: fails loudly on a dirty tree (image would build from your tree while the stack deploys from git) or unpushed commits (the webhook only redeploys when Forgejo main moves), each with the exact fix in the message; FORCE=1 bypasses. All three paths verified live.

Post-close QoL from the final thoughts, landed as 5247e80 — `make redeploy` now guards the git-connected-stack trap: fails loudly on a dirty tree (image would build from your tree while the stack deploys from git) or unpushed commits (the webhook only redeploys when Forgejo main moves), each with the exact fix in the message; `FORCE=1` bypasses. All three paths verified live.
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
feat/11-walking-skeleton
No results found.
Labels
Clear labels   
bug
Something is not working

  
duplicate
This issue or pull request already exists

  
enhancement
New feature

  
help wanted
Need some help

  
invalid
Something is wrong

  
question
More information is needed

  
wontfix
This won't be fixed

No labels
bug
duplicate
enhancement
help wanted
invalid
question
wontfix
Milestone
Clear milestone
No items
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
cmoriarty/trog#23
No description provided.