cmoriarty/trog
1
0
Fork 0
Code Issues 11 Pull requests Projects 1 Releases Packages Wiki Activity Actions

10 · E2E walking skeleton: brief → finished game, fully autonomous #11

New issue
Open
opened 2026-07-23 00:07:15 -04:00 by cmoriarty · 4 comments
cmoriarty commented 2026-07-23 00:07:15 -04:00
Owner
Copy link

The milestone-closing integration ticket. trog run "a cozy fishing game with a day/night cycle" produces a tiny finished Phaser game with nobody watching:

  • Every gate green, work sent backward when gates fail
  • Assets generated through the swap pipeline
  • Playtested headlessly; results in the ledger
  • Committed to its own Forgejo repo
  • The whole run observable live from the studio

Done when: one command, zero intervention, playable URL at the end.

The milestone-closing integration ticket. `trog run "a cozy fishing game with a day/night cycle"` produces a tiny finished Phaser game with nobody watching: - Every gate green, work sent backward when gates fail - Assets generated through the swap pipeline - Playtested headlessly; results in the ledger - Committed to its own Forgejo repo - The whole run observable live from the studio **Done when:** one command, zero intervention, playable URL at the end.
cmoriarty changed title from E2E walking skeleton: brief → finished game, fully autonomous to 10 · E2E walking skeleton: brief → finished game, fully autonomous 2026-07-23 00:10:12 -04:00
cmoriarty commented 2026-07-23 17:56:30 -04:00
Author
Owner
Copy link

Design note (from 03.1/03.2 work): where OpenCode fits

Two harnesses, one brain. The graph and OpenCode are both host applications around the same llama.cpp endpoint — the model never speaks MCP or LangChain; each host does its own schema-binding and tool loop.

tier harness owns shape of work
process spine Aegra graph + ChatOpenAI phases, gates, ledger, git truth doc writing, bounded research loops (03.2's _research_invoke), critique
workspace sessions OpenCode edit → build → test inside a game checkout long agentic loops with fs/bash tools, own tool executor

Division: ideation / preproduction / slice spec / alpha-beta reviews stay direct ChatOpenAI (document work — current code already right). Production / alpha / beta implementation nodes change character in this ticket: instead of llm.ainvoke → commit doc, a node does

clone game repo → OpenCode session("build milestone N per docs/03-milestones.md",
                                   workspace, brain endpoint, Phaser skills/)
→ session commits code → push → trog_lib.check_phase verifies at the commit
→ gate critiques the build

Node = producer; OpenCode = the developer seat hired for the task. Git + ledger remain the only truth crossing the boundary — LangGraph can't checkpoint inside an OpenCode session and doesn't need to; the workspace result lands in git, which is what the git-is-truth design already assumes.

Session mechanics — decide in this ticket:

  1. opencode run "<task>" — headless one-shot in a workspace container; simple, transcript on exit
  2. opencode serve + HTTP API/SDK — long-lived service, per-task sessions, streamable events

Lean (2): ticket 08's "opencode-style live feed" then becomes literally OpenCode's event stream surfaced in the studio, not a reimplementation.

Wiring already decided elsewhere: OpenCode confirmed as the code agent (03.1 audit — revisit trigger: persistent tool-call failures → cheap A/B vs Qwen Code, same endpoint config). Point sessions at Phaser 4's in-repo skills/; agent-facing phase prompts ship as SKILL.md in the game repo. Websearch MCP (common-mcps, 03.2) can mount into OpenCode config too — same endpoint the graph's toolbelt uses.

Budget note from the firefly run (issue #5): gate loops already cost ~40 min; OpenCode build sessions add real wall-clock on top — plan run budgets accordingly.

## Design note (from 03.1/03.2 work): where OpenCode fits Two harnesses, one brain. The graph and OpenCode are both *host applications* around the same llama.cpp endpoint — the model never speaks MCP or LangChain; each host does its own schema-binding and tool loop. | tier | harness | owns | shape of work | |---|---|---|---| | process spine | Aegra graph + ChatOpenAI | phases, gates, ledger, git truth | doc writing, bounded research loops (03.2's `_research_invoke`), critique | | workspace sessions | OpenCode | edit → build → test inside a game checkout | long agentic loops with fs/bash tools, own tool executor | **Division:** ideation / preproduction / slice spec / alpha-beta *reviews* stay direct ChatOpenAI (document work — current code already right). Production / alpha / beta *implementation* nodes change character in this ticket: instead of `llm.ainvoke → commit doc`, a node does ``` clone game repo → OpenCode session("build milestone N per docs/03-milestones.md", workspace, brain endpoint, Phaser skills/) → session commits code → push → trog_lib.check_phase verifies at the commit → gate critiques the build ``` Node = producer; OpenCode = the developer seat hired for the task. Git + ledger remain the only truth crossing the boundary — LangGraph can't checkpoint inside an OpenCode session and doesn't need to; the workspace result lands in git, which is what the git-is-truth design already assumes. **Session mechanics — decide in this ticket:** 1. `opencode run "<task>"` — headless one-shot in a workspace container; simple, transcript on exit 2. `opencode serve` + HTTP API/SDK — long-lived service, per-task sessions, **streamable events** Lean (2): ticket 08's "opencode-style live feed" then becomes literally OpenCode's event stream surfaced in the studio, not a reimplementation. **Wiring already decided elsewhere:** OpenCode confirmed as the code agent (03.1 audit — revisit trigger: persistent tool-call failures → cheap A/B vs Qwen Code, same endpoint config). Point sessions at Phaser 4's in-repo `skills/`; agent-facing phase prompts ship as SKILL.md in the game repo. Websearch MCP (common-mcps, 03.2) can mount into OpenCode config too — same endpoint the graph's toolbelt uses. **Budget note from the firefly run (issue #5):** gate loops already cost ~40 min; OpenCode build sessions add real wall-clock on top — plan run budgets accordingly.
cmoriarty commented 2026-07-24 00:12:26 -04:00
Author
Owner
Copy link

Design note: llama-swap belongs here (if anywhere), not in the conductor

Evaluated (2026-07-24) whether llama-swap should replace the asset conductor's brain up/down — verdict: no. It's demand-driven OpenAI-endpoint multiplexing; the conductor's swap is a scheduled drain window against non-LLM ComfyUI seats, and demand-driven semantics are exactly wrong there (a brain request mid-drain would evict a generator mid-render; trog wants runs to PARK instead — the model_swap_interrupt handshake).

Where it fits: this ticket. If the OpenCode seat wants a coder-tuned model while the graph brain runs Qwen-general, that's LLM-vs-LLM arbitration on the same GPUs with the same OpenAI protocol — llama-swap's home turf:

  • one llama-swap in front of llama.cpp configs (brain model, coder model), swap keyed on the model field
  • aegra's BRAIN_URL and OpenCode's endpoint both point at the proxy; each names its model
  • TTL/group config handles "coder model resident during build sessions, brain back for gates"
  • conductor keeps owning the LLM-vs-generator boundary; llama-swap owns the LLM-vs-LLM one — the two never overlap because generators only run when ALL LLMs are drained

Decide during implementation whether OpenCode actually benefits from a different model than the brain (the 03.1 audit found Qwen3.6-27B strong for both roles — a second model must earn its VRAM swap cost with a measurable coding win).

## Design note: llama-swap belongs here (if anywhere), not in the conductor Evaluated (2026-07-24) whether [llama-swap](https://github.com/mostlygeek/llama-swap) should replace the asset conductor's brain up/down — verdict: no. It's demand-driven OpenAI-endpoint multiplexing; the conductor's swap is a scheduled drain window against non-LLM ComfyUI seats, and demand-driven semantics are exactly wrong there (a brain request mid-drain would evict a generator mid-render; trog wants runs to PARK instead — the `model_swap_interrupt` handshake). **Where it fits: this ticket.** If the OpenCode seat wants a coder-tuned model while the graph brain runs Qwen-general, that's LLM-vs-LLM arbitration on the same GPUs with the same OpenAI protocol — llama-swap's home turf: - one llama-swap in front of llama.cpp configs (brain model, coder model), swap keyed on the `model` field - aegra's `BRAIN_URL` and OpenCode's endpoint both point at the proxy; each names its model - TTL/group config handles "coder model resident during build sessions, brain back for gates" - conductor keeps owning the LLM-vs-generator boundary; llama-swap owns the LLM-vs-LLM one — the two never overlap because generators only run when ALL LLMs are drained Decide during implementation whether OpenCode actually benefits from a different model than the brain (the 03.1 audit found Qwen3.6-27B strong for both roles — a second model must earn its VRAM swap cost with a measurable coding win).
cmoriarty commented 2026-07-26 10:46:13 -04:00
Author
Owner
Copy link

Moved from #8 (studio feed): production-graph adoption of the feed contracts — (1) maintain the todos state key ([{text, status: pending|in_progress|done}], see phasewalk for the reference implementation) so the studio todo strip tracks real runs; (2) interrupt-node steering — the feed currently steers by stop-and-fold (halt run, append guidance, rerun on checkpointed state, verified working); graph-level interrupt() nodes would fold guidance mid-node without killing the run. Both belong with the walking-skeleton work.

Moved from #8 (studio feed): **production-graph adoption of the feed contracts** — (1) maintain the `todos` state key (`[{text, status: pending|in_progress|done}]`, see phasewalk for the reference implementation) so the studio todo strip tracks real runs; (2) interrupt-node steering — the feed currently steers by stop-and-fold (halt run, append guidance, rerun on checkpointed state, verified working); graph-level `interrupt()` nodes would fold guidance mid-node without killing the run. Both belong with the walking-skeleton work.
cmoriarty commented 2026-07-27 20:57:47 -04:00
Author
Owner
Copy link

First end-to-end run: failed, deliberately stopped

The walking skeleton is built and deployed; the first real trog run "a cozy fishing game with a day/night cycle" was abandoned rather than finished, because the art it generated is unusable. Writing down what worked, what broke, and what that cost.

What is in place

  • opencode/ — the code agent seat, pinned opencode-ai@1.18.7. One container per session (opencode serve binds a server to one project directory, so a shared server could only ever build one game), raised and torn down by the orchestrator like a generator seat.
  • trog_lib/codeseat.py — workspace unpacked from the repo archive, session driven over HTTP, the diff committed back as ONE commit via a new forgejo.commit_files. No Forgejo credential enters the container: the agent gets a directory, not a push token. Per-session OPENCODE_SERVER_PASSWORD (verified: no auth 401, wrong password 401, correct 200).
  • The graph — assets → drain → build ⇄ gate_playtest → alpha → build_alpha → beta → build_beta → ship. Milestone walk uncapped (the plan lists what it lists); repair capped, which is what makes an unattended run terminate. Nodes maintain todos and narrate to messages, so a real run drives the studio.
  • check_phase — build phases judged by what runs: entry committed, every referenced asset present at that commit, a passing playtest for that commit.
  • trog run "<brief>" — one command, streams phase/todo transitions, resumes from the checkpoint if a run is interrupted.

Proven on real hardware

The seat path works. A build session against a scratch repo:

workspace ready (12 files)
code agent up (trog-opencode:latest, qwen3.6-27b)
write: index.html
committed 1 file(s) as 12f0c9f in 155s

...and the headless browser on that exact commit: PASS (60.1 fps; 1 canvas; 36 colors; motion 0.42). Build → commit → playtest gate is real.

The run itself got through ideation, preproduction (fun gate: pass), vertical slice (bar gate: pass), a 4-milestone production plan parsed cleanly from the brain's own table, a 27-asset spec, and a full drain: 14 images, 14 sounds, 3 animations, then critics and a regeneration pass.

Why it was stopped

Every sprite and UI asset is RGB with no alpha channel — the subject painted onto an opaque background. In Phaser each would composite as a solid rectangle. The build agent would have loaded them and produced a game of dark boxes.

Worse, the visual critic passed one of them. Its reasoning about bird.png:

"The dark, cool-toned body of the bird provides clean separation against the lighter, warm-toned background bands, adhering to the 'clean separation' requirement."

The brief asked for separation FROM the sky — a cutout. The critic read it as "contrasts nicely with the sky painted into the picture."

Root cause is upstream of the critic and predates this ticket: pixel_post.py flattens to RGB at every stage and no workflow removes a background. The pipeline has never been able to produce a transparent sprite. #11 is simply the first thing that would have loaded one into a game engine. Filed as #34.

Fixed along the way (all found by running it, not by testing it)

  1. Aegra cancels a run at 3600s — execution_seconds=3600.01 Worker job cancelled. The run died mid-drain with four milestones to go. BG_JOB_TIMEOUT_SECS=86400, plus trog run now resumes from the checkpoint, because a cap is not a guarantee — a redeploy or a crash ends a run just as dead.
  2. A stream that ends is not a run that ended — the CLI printed "no playable URL" while the GPUs were still working.
  3. A stale terminal status was read as the current one — a build node would have seen the previous milestone's done and skipped an hour of work.
  4. The drain reported no progress at all — jobprogress was wired into the one-off worker only, so every drain job came back progress: {}. Image and audio hid it by finishing an item a minute; animate at ten minutes an item looked hung.
  5. The audio critic waited for the GPUs it never wanted — it is CPU-only, so it now judges each sound as it commits, beside the lanes still running.
  6. Sprites that aren't cutouts are now rejected deterministically, before a vision call is spent on them.

Where this leaves the ticket

Not done. The done-condition is "one command, zero intervention, playable URL at the end", and no run has reached it. The process spine, the code agent, the drain integration and the playtest gate are all proven individually on real hardware; the blocker is #34, because a run that ships a game of opaque rectangles has not met the bar even if it produces a URL.

Next: fix #34, then re-run end to end.

## First end-to-end run: failed, deliberately stopped The walking skeleton is built and deployed; the first real `trog run "a cozy fishing game with a day/night cycle"` was **abandoned rather than finished**, because the art it generated is unusable. Writing down what worked, what broke, and what that cost. ### What is in place - **`opencode/`** — the code agent seat, pinned `opencode-ai@1.18.7`. One container per session (`opencode serve` binds a server to one project directory, so a shared server could only ever build one game), raised and torn down by the orchestrator like a generator seat. - **`trog_lib/codeseat.py`** — workspace unpacked from the repo archive, session driven over HTTP, the diff committed back as ONE commit via a new `forgejo.commit_files`. No Forgejo credential enters the container: the agent gets a directory, not a push token. Per-session `OPENCODE_SERVER_PASSWORD` (verified: no auth 401, wrong password 401, correct 200). - **The graph** — `assets → drain → build ⇄ gate_playtest → alpha → build_alpha → beta → build_beta → ship`. Milestone walk uncapped (the plan lists what it lists); *repair* capped, which is what makes an unattended run terminate. Nodes maintain `todos` and narrate to `messages`, so a real run drives the studio. - **`check_phase`** — build phases judged by what runs: entry committed, every referenced asset present at that commit, a passing playtest for that commit. - **`trog run "<brief>"`** — one command, streams phase/todo transitions, resumes from the checkpoint if a run is interrupted. ### Proven on real hardware The seat path works. A build session against a scratch repo: ``` workspace ready (12 files) code agent up (trog-opencode:latest, qwen3.6-27b) write: index.html committed 1 file(s) as 12f0c9f in 155s ``` ...and the headless browser on that exact commit: `PASS (60.1 fps; 1 canvas; 36 colors; motion 0.42)`. Build → commit → playtest gate is real. The run itself got through ideation, preproduction (fun gate: pass), vertical slice (bar gate: pass), a 4-milestone production plan parsed cleanly from the brain's own table, a 27-asset spec, and a full drain: 14 images, 14 sounds, 3 animations, then critics and a regeneration pass. ### Why it was stopped **Every sprite and UI asset is RGB with no alpha channel** — the subject painted onto an opaque background. In Phaser each would composite as a solid rectangle. The build agent would have loaded them and produced a game of dark boxes. Worse, the visual critic *passed* one of them. Its reasoning about `bird.png`: > "The dark, cool-toned body of the bird provides clean separation against the lighter, warm-toned background bands, adhering to the 'clean separation' requirement." The brief asked for separation FROM the sky — a cutout. The critic read it as "contrasts nicely with the sky painted into the picture." Root cause is upstream of the critic and predates this ticket: `pixel_post.py` flattens to RGB at every stage and no workflow removes a background. **The pipeline has never been able to produce a transparent sprite.** #11 is simply the first thing that would have loaded one into a game engine. Filed as #34. ### Fixed along the way (all found by running it, not by testing it) 1. **Aegra cancels a run at 3600s** — `execution_seconds=3600.01 Worker job cancelled`. The run died mid-drain with four milestones to go. `BG_JOB_TIMEOUT_SECS=86400`, plus `trog run` now resumes from the checkpoint, because a cap is not a guarantee — a redeploy or a crash ends a run just as dead. 2. **A stream that ends is not a run that ended** — the CLI printed "no playable URL" while the GPUs were still working. 3. **A stale terminal status was read as the current one** — a build node would have seen the previous milestone's `done` and skipped an hour of work. 4. **The drain reported no progress at all** — `jobprogress` was wired into the one-off worker only, so every drain job came back `progress: {}`. Image and audio hid it by finishing an item a minute; animate at ten minutes an item looked hung. 5. **The audio critic waited for the GPUs it never wanted** — it is CPU-only, so it now judges each sound as it commits, beside the lanes still running. 6. **Sprites that aren't cutouts** are now rejected deterministically, before a vision call is spent on them. ### Where this leaves the ticket **Not done.** The done-condition is "one command, zero intervention, playable URL at the end", and no run has reached it. The process spine, the code agent, the drain integration and the playtest gate are all proven individually on real hardware; the blocker is #34, because a run that ships a game of opaque rectangles has not met the bar even if it produces a URL. Next: fix #34, then re-run end to end.
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
feat/11-walking-skeleton
No results found.
Labels
Clear labels   
bug
Something is not working

  
duplicate
This issue or pull request already exists

  
enhancement
New feature

  
help wanted
Need some help

  
invalid
Something is wrong

  
question
More information is needed

  
wontfix
This won't be fixed

No labels
bug
duplicate
enhancement
help wanted
invalid
question
wontfix
Milestone
Clear milestone
No items
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
cmoriarty/trog#11
No description provided.