cmoriarty/trog
1
0
Fork 0
Code Issues 11 Pull requests Projects 1 Releases Packages Wiki Activity Actions

Tweak flow: real progress bars, stale viewer/top-bar after image gen, audio tweak #31

New issue
Closed
opened 2026-07-27 12:15:36 -04:00 by cmoriarty · 4 comments
cmoriarty commented 2026-07-27 12:15:36 -04:00
Owner
Copy link

Found while tweaking an image in the studio files pane. Four parts: two are honest-feedback work (real progress bars), one is a bug (the tweak lands but the UI lies about it), one extends tweak to audio.

Today the tweak flow is: TweakBar POSTs /orch/jobs, then follows /orch/jobs/{id}/stream (SSE) and renders whatever msg the orchestrator narrates next to three blinking pixels (<Pulse />) — studio/src/panes/FilesPane.tsx:158-275. The narration is phase-level only (Swapping brain for image stack → Generating sprite → committed …), because that's all the backend emits (server/trog_lib/orchestrator.py:202-258).

1. Real progress bar + time estimate for image generation

Replace the blinking lights during a tweak with a determinate bar and a remaining-time estimate.

Backend is the blocker, not the UI. comfy_item.run_workflow (server/trog_lib/comfy_item.py:234-251) submits /prompt and then polls GET /history/{prompt_id} every 5s until outputs appear — it never sees step progress. ComfyUI's WebSocket (/ws?clientId=…) carries progress messages (value / max = sampler steps) plus executing node transitions; subscribing there is what turns this into a real percentage. Feed that into assetq (either frequent progress events or a numeric progress column on the job) so /orch/jobs/{id}/stream can carry it to the studio.

The bar must cover the whole job, not just sampling, because the phases have very different costs:

phase cost estimable from
brain swap + seat boot large, near-fixed; skipped when the seat is already warm (Image seat already warm) measured per profile; the job knows which case it's in
generation step-proportional ComfyUI progress value/max
CPU pixel post-pass (PixelOE + OKLab k-means, pixel_post) + git commit short measured

For the estimate, prefer measured history over a hardcoded guess: jobs already record wall time (the completion event ends with (Ns)), so a rolling median per profile+kind, split by warm vs cold seat, gives a defensible ETA. Show a range or a "~" prefix rather than false precision, and let the ETA correct itself as steps land.

2. Bug: the tweak commits, but the viewer and the top bar both lie

Reproduced by the user: the job reported done and said the viewer refreshed, but the displayed image never changed, and the top bar kept showing image gen with the pulse animating long after.

2a — viewer shows the old image. On done, TweakBar calls onDone() → setAssetV(v => v + 1) → the <img> src becomes /orch/repo/{game}/raw/{path}?v=N (FilesPane.tsx:405-406, 469-475), which should defeat any cache. Prime suspect is not caching but path mismatch: the studio sends kind: path.includes("background") ? "background" : "image" (FilesPane.tsx:230), the orchestrator maps that to spec kind sprite/background (orchestrator.py:120-123), and delivery hardcodes the destination as assets/{kind}/{name}.{ext} (comfy_item.build_artifacts:324, deliver_git:327-335). So a tweak of any image not already living at assets/sprite/<name>.png — an animated-sprite sheet, a tile, a differently-named folder, or a .jpg — is committed to a different path than the one the viewer is showing. The commit succeeds, the message is truthful, and the open file genuinely didn't change.

Fix direction: the tweak should recommit the path it was invoked on (pass the source path through the job spec and have delivery honour it), and the "recommitted — viewer refreshed" note should name the path it wrote. If a tweak cannot write back to the same path, say so instead of claiming a refresh. Worth confirming the commit path in a repro before building the fix — if some other cause is at play, the same repro will show it.

2b — top bar keeps flashing image gen. Not a display bug in the top bar: /orch/health reports warm_seats: manager.active (orchestrator.py:580-590) and a finished job's seat deliberately stays warm for SEAT_TTL (default 180s, orchestrator.py:54-56) waiting for follow-up work. App.tsx maps every warm seat to an activity label and animates the pulse, so for three minutes after a job ends the bar claims generation is running. Keep-warm is correct behaviour; the readout conflating warm with working is not. Health should distinguish running jobs from idle-but-warm seats, and the top bar should only animate for actual work (a warm idle seat, if shown at all, should read as a static image seat warm).

3. Audio tweak, with the same progress bar

TweakBar is only rendered on the image branch of the viewer (FilesPane.tsx:469-475); the audio branch is a bare <audio controls>. The backend refuses audio tweaks outright today:

--from is image-only for now: audio tweak needs the seat's cover/repaint routes wired (tracked on #22; blocked on audio-server image work) — orchestrator.py:130-134

So this task is: check whether the audio seat can now do a source-conditioned regeneration (cover / repaint / audio-to-audio); if it can, wire the route, drop the 422, and render TweakBar for AUDIO_EXT files with the same determinate progress bar and ETA. If the seat still can't, say so in this ticket and leave the audio tweak box out entirely rather than shipping a button that 422s — and keep #22 as the blocker.

Note that the audio path has no step-progress source at all: audio_item.produce is a single blocking HTTP call with a 900s timeout (server/trog_lib/audio_item.py:56-80). Audio progress therefore needs either a progress endpoint on the audio seat or a duration-model estimate — decide which before promising a percentage.

4. Progress bars must finish at 100%

Design rule for every bar this ticket adds: 100% means done. A bar that reaches 100% and then sits there spinning is worse than no bar — it converts a progress indicator into a liar. Concretely:

  • the bar reaches 100% only when the terminal event has landed and the UI has the new asset in hand
  • work that isn't step-measurable (seat boot, commit, git round-trip) gets its own budgeted slice of the bar, not a silent stall at the end
  • when the remaining time is genuinely unknown, hold below 100% and show it as unknown — never round up to 100 and wait
  • overrun (the job takes longer than estimated) stretches the estimate, it does not pin the bar at 100

Acceptance

  • tweaking an image shows a moving bar with a remaining-time estimate covering seat boot → generation → commit, and the bar hits 100% at the moment the new asset is on screen
  • the tweaked image visibly replaces the old one in the viewer, or the UI says plainly that it wrote somewhere else
  • within one poll tick of a job finishing, the top bar stops animating; a warm idle seat never reads as active generation
  • audio files either get a working tweak box with the same bar, or none at all, with the reason recorded here
  • no bar parks at 100%

Testable GPU-free: SEAT_MODE=mock / trog test asset --mock drives the whole path through the mock seat (server/trog_lib/mock_seat.py), which is also where a fake step-progress stream belongs so the bar can be exercised in CI.

Found while tweaking an image in the studio files pane. Four parts: two are honest-feedback work (real progress bars), one is a bug (the tweak lands but the UI lies about it), one extends tweak to audio. Today the tweak flow is: `TweakBar` POSTs `/orch/jobs`, then follows `/orch/jobs/{id}/stream` (SSE) and renders whatever `msg` the orchestrator narrates next to three blinking pixels (`<Pulse />`) — `studio/src/panes/FilesPane.tsx:158-275`. The narration is phase-level only (`Swapping brain for image stack` → `Generating sprite` → `committed …`), because that's all the backend emits (`server/trog_lib/orchestrator.py:202-258`). ## 1. Real progress bar + time estimate for image generation Replace the blinking lights during a tweak with a determinate bar and a remaining-time estimate. **Backend is the blocker, not the UI.** `comfy_item.run_workflow` (`server/trog_lib/comfy_item.py:234-251`) submits `/prompt` and then polls `GET /history/{prompt_id}` every 5s until outputs appear — it never sees step progress. ComfyUI's WebSocket (`/ws?clientId=…`) carries `progress` messages (`value` / `max` = sampler steps) plus `executing` node transitions; subscribing there is what turns this into a real percentage. Feed that into `assetq` (either frequent progress events or a numeric progress column on the job) so `/orch/jobs/{id}/stream` can carry it to the studio. The bar must cover the *whole* job, not just sampling, because the phases have very different costs: | phase | cost | estimable from | |---|---|---| | brain swap + seat boot | large, near-fixed; skipped when the seat is already warm (`Image seat already warm`) | measured per profile; the job knows which case it's in | | generation | step-proportional | ComfyUI `progress` value/max | | CPU pixel post-pass (PixelOE + OKLab k-means, `pixel_post`) + git commit | short | measured | For the estimate, prefer measured history over a hardcoded guess: jobs already record wall time (the completion event ends with `(Ns)`), so a rolling median per profile+kind, split by warm vs cold seat, gives a defensible ETA. Show a range or a "~" prefix rather than false precision, and let the ETA correct itself as steps land. ## 2. Bug: the tweak commits, but the viewer and the top bar both lie Reproduced by the user: the job reported done and said the viewer refreshed, but the displayed image never changed, and the top bar kept showing `image gen` with the pulse animating long after. **2a — viewer shows the old image.** On `done`, `TweakBar` calls `onDone()` → `setAssetV(v => v + 1)` → the `<img>` src becomes `/orch/repo/{game}/raw/{path}?v=N` (`FilesPane.tsx:405-406`, `469-475`), which should defeat any cache. Prime suspect is not caching but **path mismatch**: the studio sends `kind: path.includes("background") ? "background" : "image"` (`FilesPane.tsx:230`), the orchestrator maps that to spec kind `sprite`/`background` (`orchestrator.py:120-123`), and delivery hardcodes the destination as `assets/{kind}/{name}.{ext}` (`comfy_item.build_artifacts:324`, `deliver_git:327-335`). So a tweak of any image *not* already living at `assets/sprite/<name>.png` — an animated-sprite sheet, a tile, a differently-named folder, or a `.jpg` — is committed to a different path than the one the viewer is showing. The commit succeeds, the message is truthful, and the open file genuinely didn't change. Fix direction: the tweak should recommit **the path it was invoked on** (pass the source path through the job spec and have delivery honour it), and the "recommitted — viewer refreshed" note should name the path it wrote. If a tweak cannot write back to the same path, say so instead of claiming a refresh. Worth confirming the commit path in a repro before building the fix — if some other cause is at play, the same repro will show it. **2b — top bar keeps flashing `image gen`.** Not a display bug in the top bar: `/orch/health` reports `warm_seats: manager.active` (`orchestrator.py:580-590`) and a finished job's seat deliberately stays warm for `SEAT_TTL` (default 180s, `orchestrator.py:54-56`) waiting for follow-up work. `App.tsx` maps every warm seat to an activity label and animates the pulse, so for three minutes after a job ends the bar claims generation is running. Keep-warm is correct behaviour; the readout conflating *warm* with *working* is not. Health should distinguish running jobs from idle-but-warm seats, and the top bar should only animate for actual work (a warm idle seat, if shown at all, should read as a static `image seat warm`). ## 3. Audio tweak, with the same progress bar `TweakBar` is only rendered on the image branch of the viewer (`FilesPane.tsx:469-475`); the audio branch is a bare `<audio controls>`. The backend refuses audio tweaks outright today: > `--from is image-only for now: audio tweak needs the seat's cover/repaint routes wired (tracked on #22; blocked on audio-server image work)` — `orchestrator.py:130-134` So this task is: check whether the audio seat can now do a source-conditioned regeneration (cover / repaint / audio-to-audio); if it can, wire the route, drop the 422, and render `TweakBar` for `AUDIO_EXT` files with the same determinate progress bar and ETA. If the seat still can't, say so in this ticket and leave the audio tweak box out entirely rather than shipping a button that 422s — and keep #22 as the blocker. Note that the audio path has no step-progress source at all: `audio_item.produce` is a single blocking HTTP call with a 900s timeout (`server/trog_lib/audio_item.py:56-80`). Audio progress therefore needs either a progress endpoint on the audio seat or a duration-model estimate — decide which before promising a percentage. ## 4. Progress bars must finish at 100% Design rule for every bar this ticket adds: **100% means done.** A bar that reaches 100% and then sits there spinning is worse than no bar — it converts a progress indicator into a liar. Concretely: - the bar reaches 100% only when the terminal event has landed and the UI has the new asset in hand - work that isn't step-measurable (seat boot, commit, git round-trip) gets its own budgeted slice of the bar, not a silent stall at the end - when the remaining time is genuinely unknown, hold below 100% and show it as unknown — never round up to 100 and wait - overrun (the job takes longer than estimated) stretches the estimate, it does not pin the bar at 100 ## Acceptance - tweaking an image shows a moving bar with a remaining-time estimate covering seat boot → generation → commit, and the bar hits 100% at the moment the new asset is on screen - the tweaked image visibly replaces the old one in the viewer, or the UI says plainly that it wrote somewhere else - within one poll tick of a job finishing, the top bar stops animating; a warm idle seat never reads as active generation - audio files either get a working tweak box with the same bar, or none at all, with the reason recorded here - no bar parks at 100% Testable GPU-free: `SEAT_MODE=mock` / `trog test asset --mock` drives the whole path through the mock seat (`server/trog_lib/mock_seat.py`), which is also where a fake step-progress stream belongs so the bar can be exercised in CI.
cmoriarty commented 2026-07-27 13:21:18 -04:00
Author
Owner
Copy link

Done and deployed. Seven live tweaks against trog-games/cycle-test drove the verification, and most of what follows was found by those runs rather than by reading code.

The stale viewer had a different cause than this ticket guessed

The ticket blamed the kind-derived destination path. That bug was real (and is fixed), but it was not what the user hit. Forgejo's raw API serves a stale blob when the ref is a branch name. Measured on the live instance, minutes after a tweak committed:

request bytes
raw ?ref=main 3233 — the previous asset
raw ?ref=<head sha> 4008 — the new one
raw (no ref) 4008
contents ?ref=main 4008 (this endpoint is fine)

Still stale twelve minutes later, so it is a branch-ref cache, not a propagation window. forgejo.read_binary always passed ref="main", so the studio's viewer, the tweak's own source fetch, and the critics were all reading yesterday's bytes. It now resolves the ref to a commit sha and reads that.

What shipped

1. Progress bar with a real estimate. The orchestrator keeps a per-job estimator (server/trog_lib/jobprogress.py), budgets seat boot / generation / delivery from the measured medians of comparable past runs (trog_asset_jobs.timings), and publishes a snapshot once a second that the SSE feed relays to the studio. The CLI spinner shows the same percentage.

Worth recording that the obvious implementation does not work: sampler steps make a bad bar. The pixel graphs report per-node counters, not one sampler counting up — a single render walked 7/8 → 8/9 → 13/14 → 1/11 → 26/26. A fraction built on that either walks backwards or, clamped monotone, pins near its cap. A pass-counting rescue attempt recorded "39 passes" for that graph. Wall time against a corpus of identical runs is the honest signal here: five consecutive renders landed within 2% of each other. Steps remain in the readout as narration (step 7/8 tells you the seat is alive); the bar is time. ComfyUI's websocket is still consumed — that is where the steps come from.

2. The stale viewer (item 2a), above, plus the destination-path fix: a tweak now carries dest_path, sanitized server-side, so it writes back over the file it was invoked on instead of assets/<kind>/<name>.<ext>. The studio re-reads the committed bytes and only claims "viewer refreshed" when they actually changed — otherwise it says the commit landed elsewhere, or that the bytes are identical.

3. The top bar that kept flashing (item 2b) had two causes. Keep-warm was reported as work, so /orch/health now separates working_seats from warm_seats and the readout shows idle · image gen warm unanimated. The other cause was worse: a restart orphaned in-flight jobs, leaving the row running forever with its SSE feed never terminating. Job 133 was sitting in exactly that state, orphaned by a redeploy mid-tweak — very likely what was actually seen. Boot now fails those rows, and job 133 duly reported FAILED: orchestrator restarted while this job was running on the first deploy of the fix.

4. Bars end at 100%. The fraction is linear to 95% of a phase's measured budget and asymptotic past it; an overrun stretches the estimate rather than pinning the bar; and the studio holds below 100% until the new bytes are in hand.

Audio tweak: still not possible, and now says so. Re-checked the seat — /music, /sfx and /vocal generate from text alone; there is no cover, repaint, or audio-to-audio route to condition on a source, and audio_item.produce is one blocking call with no progress feed. Rather than ship a button that 422s, the audio viewer states the reason and points at #22. The orchestrator's 422 was updated to say the same thing precisely.

Live proof (final run, orchestrator image digest confirmed)

     6s frac=0.030 eta=188s phase=boot
    48s frac=0.239 eta=152s phase=generate step=7/8
   106s frac=0.524 eta=96s  phase=generate step=3/11
   164s frac=0.808 eta=39s  phase=generate step=8/11
   189s frac=0.933 eta=14s  phase=generate step=26/26
   192s frac=1.000 eta=0s   phase=deliver  done

Monotone throughout, no pin, 100% only at completion; predicted 188s at second six against an actual 192s. Afterwards: note ✓ 'checker' recommitted — viewer refreshed (the re-read confirmed changed bytes), viewer showing the new image, top bar idle · image gen warm with the pulse animation off.

147 unit tests pass, including regression tests for each live-caught failure. Commits: 6cacef5 → 55fab9e.

One thing this did not fix: /orch/repo/{slug}/raw reads two Forgejo endpoints per request now (resolve sha, then fetch). If that shows up in the files pane's latency, caching the head sha per repo for a second or two is the obvious next step.

Done and deployed. Seven live tweaks against `trog-games/cycle-test` drove the verification, and most of what follows was found by those runs rather than by reading code. ## The stale viewer had a different cause than this ticket guessed The ticket blamed the kind-derived destination path. That bug was real (and is fixed), but it was not what the user hit. Forgejo's **raw API serves a stale blob when the ref is a branch name**. Measured on the live instance, minutes after a tweak committed: | request | bytes | |---|---| | `raw ?ref=main` | 3233 — the *previous* asset | | `raw ?ref=<head sha>` | 4008 — the new one | | `raw` (no ref) | 4008 | | `contents ?ref=main` | 4008 (this endpoint is fine) | Still stale twelve minutes later, so it is a branch-ref cache, not a propagation window. `forgejo.read_binary` always passed `ref="main"`, so the studio's viewer, the tweak's own source fetch, and the critics were all reading yesterday's bytes. It now resolves the ref to a commit sha and reads that. ## What shipped **1. Progress bar with a real estimate.** The orchestrator keeps a per-job estimator (`server/trog_lib/jobprogress.py`), budgets seat boot / generation / delivery from the measured medians of comparable past runs (`trog_asset_jobs.timings`), and publishes a snapshot once a second that the SSE feed relays to the studio. The CLI spinner shows the same percentage. Worth recording that the obvious implementation does not work: **sampler steps make a bad bar.** The pixel graphs report per-*node* counters, not one sampler counting up — a single render walked 7/8 → 8/9 → 13/14 → 1/11 → 26/26. A fraction built on that either walks backwards or, clamped monotone, pins near its cap. A pass-counting rescue attempt recorded "39 passes" for that graph. Wall time against a corpus of identical runs is the honest signal here: five consecutive renders landed within 2% of each other. Steps remain in the readout as narration (`step 7/8` tells you the seat is alive); the bar is time. ComfyUI's websocket is still consumed — that is where the steps come from. **2. The stale viewer (item 2a)**, above, plus the destination-path fix: a tweak now carries `dest_path`, sanitized server-side, so it writes back over the file it was invoked on instead of `assets/<kind>/<name>.<ext>`. The studio re-reads the committed bytes and only claims "viewer refreshed" when they actually changed — otherwise it says the commit landed elsewhere, or that the bytes are identical. **3. The top bar that kept flashing (item 2b)** had *two* causes. Keep-warm was reported as work, so `/orch/health` now separates `working_seats` from `warm_seats` and the readout shows `idle · image gen warm` unanimated. The other cause was worse: a restart orphaned in-flight jobs, leaving the row `running` forever with its SSE feed never terminating. Job 133 was sitting in exactly that state, orphaned by a redeploy mid-tweak — very likely what was actually seen. Boot now fails those rows, and job 133 duly reported `FAILED: orchestrator restarted while this job was running` on the first deploy of the fix. **4. Bars end at 100%.** The fraction is linear to 95% of a phase's measured budget and asymptotic past it; an overrun stretches the estimate rather than pinning the bar; and the studio holds below 100% until the new bytes are in hand. **Audio tweak: still not possible, and now says so.** Re-checked the seat — `/music`, `/sfx` and `/vocal` generate from text alone; there is no cover, repaint, or audio-to-audio route to condition on a source, and `audio_item.produce` is one blocking call with no progress feed. Rather than ship a button that 422s, the audio viewer states the reason and points at #22. The orchestrator's 422 was updated to say the same thing precisely. ## Live proof (final run, orchestrator image digest confirmed) ``` 6s frac=0.030 eta=188s phase=boot 48s frac=0.239 eta=152s phase=generate step=7/8 106s frac=0.524 eta=96s phase=generate step=3/11 164s frac=0.808 eta=39s phase=generate step=8/11 189s frac=0.933 eta=14s phase=generate step=26/26 192s frac=1.000 eta=0s phase=deliver done ``` Monotone throughout, no pin, 100% only at completion; predicted 188s at second six against an actual 192s. Afterwards: note `✓ 'checker' recommitted — viewer refreshed` (the re-read confirmed changed bytes), viewer showing the new image, top bar `idle · image gen warm` with the pulse animation off. 147 unit tests pass, including regression tests for each live-caught failure. Commits: 6cacef5 → 55fab9e. One thing this did not fix: `/orch/repo/{slug}/raw` reads two Forgejo endpoints per request now (resolve sha, then fetch). If that shows up in the files pane's latency, caching the head sha per repo for a second or two is the obvious next step.
cmoriarty commented 2026-07-27 13:53:47 -04:00
Author
Owner
Copy link

Two more UI fixes on this ticket, both from the same root observation: asset generation stops the brain to take its GPUs, and two places treated that designed behaviour as breakage.

5. The feed queues instead of erroring while the brain is away

Before: the model picker was removed outright (taking the explanation with it), the input still accepted sends, and the send reached a stopped brain — API ConnectionError: execution failed, with the typed line lost.

Now, while the brain is down:

  • the picker greys out with its real contents instead of vanishing. The roster is cached in localStorage, so a pane mounted during a swap still has something to grey rather than rendering nothing.
  • a passive notice sits above the input: "the brain is down — its GPUs are running asset generation. Anything you send waits here and goes out as soon as it's back." It names the reason it can tell: generation running, a warm seat still holding the cards, or just down.
  • the send button becomes queue. The line is held, shown in the feed dimmed and dashed as you · queued, and the brain-bar note counts what is waiting.
  • when the brain returns, queued lines are submitted in order, one per free thread — no user action, no error.

Nothing is queued server-side; this is the studio holding text it could not deliver. A reload loses the queue, which is why the lines stay visible in the feed rather than disappearing into a background buffer.

6. The system light no longer counts the brain

Green now means aegra and the orchestrator. Gating on the brain painted the light grey for every image job — the system working exactly as designed, reported as the system being down. The brain still rides in the tooltip: trog up · aegra · orchestrator · brain swapped out for the seats.

Verified against a real swap

Ran image tweaks on cycle-test and watched the studio through the whole cycle rather than simulating it:

moment observed
seat boots, brain stopped light stays green, tooltip brain swapped out for the seats; notice shown; picker disabled=true holding qwen3.6-27b; button reads queue; placeholder brain's away — your line queues until it's back…
line typed and queued rendered as you · queued, input cleared, no .feed-error — the exact case that used to throw ConnectionError
job finishes, brain returns queue empties itself, the line becomes a real human message, the hello graph streams a reply back (thinking + answer), still no error
cold mount mid-swap picker greyed with the full cached roster [gemma-4-31b, qwen3-coder-next, qwen3.6-27b, qwen3.6-35b-a3b]

tsc --noEmit clean, 147 tests pass, deployed and confirmed on the live bundle (index-y01O9E_a.js). Commit bda3000.

Two more UI fixes on this ticket, both from the same root observation: **asset generation stops the brain to take its GPUs**, and two places treated that designed behaviour as breakage. ## 5. The feed queues instead of erroring while the brain is away Before: the model picker was removed outright (taking the explanation with it), the input still accepted sends, and the send reached a stopped brain — `API ConnectionError: execution failed`, with the typed line lost. Now, while the brain is down: - the picker **greys out with its real contents** instead of vanishing. The roster is cached in localStorage, so a pane mounted *during* a swap still has something to grey rather than rendering nothing. - a passive notice sits above the input: *"the brain is down — its GPUs are running asset generation. Anything you send waits here and goes out as soon as it's back."* It names the reason it can tell: generation running, a warm seat still holding the cards, or just down. - the send button becomes **queue**. The line is held, shown in the feed dimmed and dashed as `you · queued`, and the brain-bar note counts what is waiting. - when the brain returns, queued lines are submitted in order, one per free thread — no user action, no error. Nothing is queued server-side; this is the studio holding text it could not deliver. A reload loses the queue, which is why the lines stay visible in the feed rather than disappearing into a background buffer. ## 6. The system light no longer counts the brain Green now means aegra **and** the orchestrator. Gating on the brain painted the light grey for every image job — the system working exactly as designed, reported as the system being down. The brain still rides in the tooltip: `trog up · aegra · orchestrator · brain swapped out for the seats`. ## Verified against a real swap Ran image tweaks on `cycle-test` and watched the studio through the whole cycle rather than simulating it: | moment | observed | |---|---| | seat boots, brain stopped | light **stays green**, tooltip `brain swapped out for the seats`; notice shown; picker `disabled=true` holding `qwen3.6-27b`; button reads `queue`; placeholder `brain's away — your line queues until it's back…` | | line typed and queued | rendered as `you · queued`, input cleared, **no `.feed-error`** — the exact case that used to throw ConnectionError | | job finishes, brain returns | queue empties itself, the line becomes a real human message, the hello graph streams a reply back (thinking + answer), still no error | | cold mount mid-swap | picker greyed with the full cached roster `[gemma-4-31b, qwen3-coder-next, qwen3.6-27b, qwen3.6-35b-a3b]` | `tsc --noEmit` clean, 147 tests pass, deployed and confirmed on the live bundle (`index-y01O9E_a.js`). Commit bda3000.
cmoriarty commented 2026-07-27 14:15:20 -04:00
Author
Owner
Copy link

Follow-up on the queueing work: after a tweak finished, the brain stayed away for two to three minutes with nothing said about it, the model picker couldn't be touched, and a queued line sat there until the brain "randomly" started swapping.

Nothing random. Keep-warm holds the seat for ORCH_SEAT_TTL (180s) after a job so a follow-up asset job skips the boot, and the brain only comes back when the worker's idle sweep releases the last seat. The swap at the end was the queued line going out and loading its model. Correct behaviour — and completely invisible, which is the actual defect: the studio said "waits until it's back" and left it at that for three minutes.

Shipped

  • /orch/health now reports seats (per-seat working + idle_s), seat_ttl, and brain_back_in.
  • The feed's notice counts it down: "generation finished — the seat keeps the GPUs for another 178s in case another asset job follows, then the brain comes back."
  • POST /orch/seats/release hands the GPUs back immediately, with a bring it back now button in that notice. It refuses (409) while a generation is actually running — that is the one case where the seats aren't the caller's to take.
  • The picker is muted, not disabled. Which model to load next is a local preference, and setting it before the queue drains is exactly what you'd want to do while waiting. Greying it out was over-correcting the original "it vanishes" complaint.

Verified on the live stack

Ran a tweak, then watched the window that used to be silent:

job done
brain down | seats [{'profile':'image','working':False,'idle_s':9.5}]  | back_in 170.5
brain down | seats [{'profile':'image','working':False,'idle_s':17.8}] | back_in 162.2
brain down | seats [{'profile':'image','working':False,'idle_s':26.1}] | back_in 153.9

In the studio at that moment: notice reading "...another 178s..." with the button, brain-note showing away · back in ~178s, picker enabled at gemma-4-31b. Changed it to qwen3.6-27b while the brain was away — the change stuck. Pressed bring it back now: seats emptied, the brain was serving models again ~40 seconds later instead of the remaining ~three minutes, the notice cleared, and the send button returned. The system light stayed green throughout.

147 tests pass (new one covers seat_state's idle reporting). Commit 1a910f6, deployed and confirmed on both new images.

Worth knowing: releasing the seat costs the next asset job its warm start (~20s of boot). The button is there for "I want the brain now", not as a default.

Follow-up on the queueing work: after a tweak finished, the brain stayed away for two to three minutes with nothing said about it, the model picker couldn't be touched, and a queued line sat there until the brain "randomly" started swapping. **Nothing random.** Keep-warm holds the seat for `ORCH_SEAT_TTL` (180s) after a job so a follow-up asset job skips the boot, and the brain only comes back when the worker's idle sweep releases the last seat. The swap at the end was the queued line going out and loading its model. Correct behaviour — and completely invisible, which is the actual defect: the studio said "waits until it's back" and left it at that for three minutes. ## Shipped - `/orch/health` now reports `seats` (per-seat `working` + `idle_s`), `seat_ttl`, and `brain_back_in`. - The feed's notice counts it down: *"generation finished — the seat keeps the GPUs for another 178s in case another asset job follows, then the brain comes back."* - **`POST /orch/seats/release`** hands the GPUs back immediately, with a *bring it back now* button in that notice. It refuses (409) while a generation is actually running — that is the one case where the seats aren't the caller's to take. - The picker is **muted, not disabled**. Which model to load next is a local preference, and setting it before the queue drains is exactly what you'd want to do while waiting. Greying it out was over-correcting the original "it vanishes" complaint. ## Verified on the live stack Ran a tweak, then watched the window that used to be silent: ``` job done brain down | seats [{'profile':'image','working':False,'idle_s':9.5}] | back_in 170.5 brain down | seats [{'profile':'image','working':False,'idle_s':17.8}] | back_in 162.2 brain down | seats [{'profile':'image','working':False,'idle_s':26.1}] | back_in 153.9 ``` In the studio at that moment: notice reading *"...another 178s..."* with the button, `brain-note` showing `away · back in ~178s`, picker **enabled** at `gemma-4-31b`. Changed it to `qwen3.6-27b` while the brain was away — the change stuck. Pressed *bring it back now*: seats emptied, the brain was serving models again ~40 seconds later instead of the remaining ~three minutes, the notice cleared, and the send button returned. The system light stayed green throughout. 147 tests pass (new one covers `seat_state`'s idle reporting). Commit 1a910f6, deployed and confirmed on both new images. Worth knowing: releasing the seat costs the next asset job its warm start (~20s of boot). The button is there for "I want the brain now", not as a default.
cmoriarty commented 2026-07-27 15:10:13 -04:00
Author
Owner
Copy link

Closing — all four items shipped and verified on the live stack, plus two follow-ups reported during testing.

What was asked, and where it landed

  1. Progress bar with a time estimate — determinate bar over the whole job (seat boot → generation → commit), fed by a per-job estimator whose phase budgets come from measured medians of comparable past runs. Final live arc: 3% at 6s, 52% at 106s, 93% at 189s, 100% at 192s, having predicted 188s at second six.
  2. Tweak said "refreshed" over an unchanged image — two independent causes, both fixed: Forgejo's raw API serving stale bytes for branch-name refs (the real one), and delivery deriving the destination from the item kind instead of the file you opened. The studio now re-reads the committed bytes and says what actually happened.
  3. Top bar stuck "generating" — keep-warm was reported as work, and a restart left jobs running forever with their SSE feed never terminating. Health separates working_seats from warm_seats; boot fails orphans.
  4. Bars end at 100% — the fraction is linear to 95% of a phase's measured budget and asymptotic past it, an overrun stretches the estimate rather than pinning the bar, and the studio holds below 100% until the new bytes are in hand.

Plus, from testing this in the studio:

  1. The feed queues instead of erroring while the brain is away, with a notice explaining why, and the keep-warm countdown is now visible and skippable (POST /orch/seats/release).
  2. The system light ignores the brain — asset generation stops the brain by design, so gating on it painted the light grey for every image job.

What the live runs taught that reading the code did not

  • Sampler steps make a bad progress bar. The graphs report per-node counters, not one sampler counting up (7/8 → 8/9 → 13/14 → 1/11 → 26/26). Step-driven fractions walk backwards, and clamping them monotone pins the bar near its cap for minutes. Wall time against a corpus of identical runs is the honest signal here — five renders landed within 2% of each other. Steps stayed as narration.
  • raw ?ref=main is stale, raw ?ref=<sha> is not, and it does not self-correct — still stale twelve minutes after the commit. Every asset read now resolves to a sha first.
  • Warm/cold only matters for boot. Splitting the whole timing corpus on it left the first warm job estimating generation from a cold-start default while measured runs sat unused.

Not done, deliberately

  • Audio tweak. The seat has no source-conditioned route — /music, /sfx and /vocal generate from text alone, and generation is one blocking call with no progress feed. The studio says so where the box would be rather than shipping a button that 422s. Stays blocked on #22.
  • A brain that dies mid-run still fails the old way; the queue only catches sends made while it is already known to be down.
  • /orch/repo/{slug}/raw now costs two Forgejo calls (resolve sha, then fetch). If the files pane feels slower, a short-lived head-sha cache per repo is the fix.

State

147 unit tests pass, with regression coverage for each live-caught failure; ruff and tsc --noEmit clean; studio smoke suite green against the deployed :2027. Eleven commits, c05f1a1 → 8da8c49, all deployed and confirmed running by image digest. Queue is healthy: 37 done, 2 failed (the orphan sweep's own catch), nothing stuck.

Closing — all four items shipped and verified on the live stack, plus two follow-ups reported during testing. ## What was asked, and where it landed 1. **Progress bar with a time estimate** — determinate bar over the whole job (seat boot → generation → commit), fed by a per-job estimator whose phase budgets come from measured medians of comparable past runs. Final live arc: 3% at 6s, 52% at 106s, 93% at 189s, 100% at 192s, having predicted 188s at second six. 2. **Tweak said "refreshed" over an unchanged image** — two independent causes, both fixed: Forgejo's raw API serving stale bytes for branch-name refs (the real one), and delivery deriving the destination from the item kind instead of the file you opened. The studio now re-reads the committed bytes and says what actually happened. 3. **Top bar stuck "generating"** — keep-warm was reported as work, and a restart left jobs `running` forever with their SSE feed never terminating. Health separates `working_seats` from `warm_seats`; boot fails orphans. 4. **Bars end at 100%** — the fraction is linear to 95% of a phase's measured budget and asymptotic past it, an overrun stretches the estimate rather than pinning the bar, and the studio holds below 100% until the new bytes are in hand. Plus, from testing this in the studio: 5. **The feed queues instead of erroring** while the brain is away, with a notice explaining why, and the keep-warm countdown is now visible and skippable (`POST /orch/seats/release`). 6. **The system light ignores the brain** — asset generation stops the brain by design, so gating on it painted the light grey for every image job. ## What the live runs taught that reading the code did not - **Sampler steps make a bad progress bar.** The graphs report per-node counters, not one sampler counting up (7/8 → 8/9 → 13/14 → 1/11 → 26/26). Step-driven fractions walk backwards, and clamping them monotone pins the bar near its cap for minutes. Wall time against a corpus of identical runs is the honest signal here — five renders landed within 2% of each other. Steps stayed as narration. - **`raw ?ref=main` is stale, `raw ?ref=<sha>` is not**, and it does not self-correct — still stale twelve minutes after the commit. Every asset read now resolves to a sha first. - **Warm/cold only matters for boot.** Splitting the whole timing corpus on it left the first warm job estimating *generation* from a cold-start default while measured runs sat unused. ## Not done, deliberately - **Audio tweak.** The seat has no source-conditioned route — `/music`, `/sfx` and `/vocal` generate from text alone, and generation is one blocking call with no progress feed. The studio says so where the box would be rather than shipping a button that 422s. Stays blocked on #22. - **A brain that dies mid-run** still fails the old way; the queue only catches sends made while it is already known to be down. - **`/orch/repo/{slug}/raw` now costs two Forgejo calls** (resolve sha, then fetch). If the files pane feels slower, a short-lived head-sha cache per repo is the fix. ## State 147 unit tests pass, with regression coverage for each live-caught failure; ruff and `tsc --noEmit` clean; studio smoke suite green against the deployed :2027. Eleven commits, c05f1a1 → 8da8c49, all deployed and confirmed running by image digest. Queue is healthy: 37 done, 2 failed (the orphan sweep's own catch), nothing stuck.
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
feat/11-walking-skeleton
No results found.
Labels
Clear labels   
bug
Something is not working

  
duplicate
This issue or pull request already exists

  
enhancement
New feature

  
help wanted
Need some help

  
invalid
Something is wrong

  
question
More information is needed

  
wontfix
This won't be fixed

No labels
bug
duplicate
enhancement
help wanted
invalid
question
wontfix
Milestone
Clear milestone
No items
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
cmoriarty/trog#31
No description provided.