cmoriarty/trog
1
0
Fork 0
Code Issues 11 Pull requests Projects 1 Releases Packages Wiki Activity Actions

05.6 · HeartMuLa vocals: implement vocal themes, settle the ACE/HeartMuLa seat split #19

New issue
Closed
opened 2026-07-24 16:16:06 -04:00 by cmoriarty · 1 comment
cmoriarty commented 2026-07-24 16:16:06 -04:00
Owner
Copy link

Follow-up to #16 (05.3 audio bench), which decided adopt on-demand for HeartMuLa-oss-3B: Apache-2.0, fp16-stable on Turing, 12.7GiB peak, RTF≈1.05 — and the only model on this hardware that can sing (ACE-Step's vocal path NaNs at every length on Turing fp16; instrumental untouched). This ticket is the "demand" side: wire vocals for real, and answer the seat-split question the bench opened.

Questions to answer before wiring (bench first, decide second):

  • Can HeartMuLa do instrumental? It's lyrics-conditioned song generation — test empty/[inst] lyrics and instrumental-style tags. If it makes good instrumental BGM, it's a candidate for the whole music seat, not just vocals.
  • Head-to-head instrumental A/B vs ACE-Step (same briefs through both, review page like the 05.3 bench): does HeartMuLa beat the incumbent on ordinary game BGM? ACE currently wins on ops (offload streams on/off the card per call, ~6GB working set, turbo 8-step is fast); HeartMuLa is a 21GB checkpoint set at RTF≈1.0 with no offload story yet — quality would have to clearly win to justify the seat swap.
  • Tag adherence retest at cfg_scale 2.5–3 — 05.3 saw folk-rock/male-vocal tags produce pop/female at the default cfg 1.5 (tags files verified correct). If adherence doesn't improve with cfg, vocal briefs need prompt-engineering guidance in the spec template instead.
  • Seat split decision: (a) ACE keeps music, HeartMuLa is a swap-in vocal seat (new vocal kind in the audio profile, kind-routing like music/sfx today); (b) HeartMuLa takes the whole music seat and ACE keeps only sfx-adjacent task modes (lego intensity mixes for ticket 10 layering — those are proven and ACE-only); or (c) HeartMuLa proves out badly and vocals stay a documented manual recipe. Decide from the A/B evidence.
  • Seat shape: VRAM/load-time for a HeartMuLa seat container (bench ran in a host venv; production wants the audio-image pattern). Codec fp32 + mula fp16 placement, lazy_load behavior, and whether it fits the one-GPU audio lane alongside ACE/SAO swaps.
  • Spec/template wiring: vocal (or song) asset kind in docs/08-assets.md parsing, lyrics block format (the audio Style block should carry vocal language/voice), variations cap, and critic coverage (does the 05.2 sound judge need a vocal mode?).

Working material: recipes + drivers in trog-games/audio-model-bench bench/ (README covers ckpt symlink layout, the torchaudio→soundfile save shim, and GPU-window discipline); weights + venvs still live at /mnt/models/audio-bench on trogdor; 05.3 evidence at http://trogdor:3923/games/audio-model-bench/assets/review.html.

Decision output: seat split chosen and wired (or explicitly rejected with reason in docs/audio-gen-research.md), vocal kind end-to-end through a drain if adopted.

Follow-up to #16 (05.3 audio bench), which decided **adopt on-demand** for HeartMuLa-oss-3B: Apache-2.0, fp16-stable on Turing, 12.7GiB peak, RTF≈1.05 — and the only model on this hardware that can sing (**ACE-Step's vocal path NaNs at every length on Turing fp16**; instrumental untouched). This ticket is the "demand" side: wire vocals for real, and answer the seat-split question the bench opened. **Questions to answer before wiring (bench first, decide second):** - [ ] **Can HeartMuLa do instrumental?** It's lyrics-conditioned song generation — test empty/`[inst]` lyrics and instrumental-style tags. If it makes good instrumental BGM, it's a candidate for the whole music seat, not just vocals. - [ ] **Head-to-head instrumental A/B vs ACE-Step** (same briefs through both, review page like the 05.3 bench): does HeartMuLa beat the incumbent on ordinary game BGM? ACE currently wins on ops (offload streams on/off the card per call, ~6GB working set, turbo 8-step is fast); HeartMuLa is a 21GB checkpoint set at RTF≈1.0 with no offload story yet — quality would have to clearly win to justify the seat swap. - [ ] **Tag adherence retest at cfg_scale 2.5–3** — 05.3 saw folk-rock/male-vocal tags produce pop/female at the default cfg 1.5 (tags files verified correct). If adherence doesn't improve with cfg, vocal briefs need prompt-engineering guidance in the spec template instead. - [ ] **Seat split decision**: (a) ACE keeps music, HeartMuLa is a swap-in vocal seat (new `vocal` kind in the audio profile, kind-routing like music/sfx today); (b) HeartMuLa takes the whole music seat and ACE keeps only sfx-adjacent task modes (lego intensity mixes for ticket 10 layering — those are proven and ACE-only); or (c) HeartMuLa proves out badly and vocals stay a documented manual recipe. Decide from the A/B evidence. - [ ] **Seat shape**: VRAM/load-time for a HeartMuLa seat container (bench ran in a host venv; production wants the audio-image pattern). Codec fp32 + mula fp16 placement, lazy_load behavior, and whether it fits the one-GPU audio lane alongside ACE/SAO swaps. - [ ] **Spec/template wiring**: `vocal` (or `song`) asset kind in docs/08-assets.md parsing, lyrics block format (the audio Style block should carry vocal language/voice), variations cap, and critic coverage (does the 05.2 sound judge need a vocal mode?). **Working material:** recipes + drivers in `trog-games/audio-model-bench` `bench/` (README covers ckpt symlink layout, the torchaudio→soundfile save shim, and GPU-window discipline); weights + venvs still live at `/mnt/models/audio-bench` on trogdor; 05.3 evidence at http://trogdor:3923/games/audio-model-bench/assets/review.html. **Decision output:** seat split chosen and wired (or explicitly rejected with reason in docs/audio-gen-research.md), vocal kind end-to-end through a drain if adopted.
cmoriarty commented 2026-07-25 18:08:35 -04:00
Author
Owner
Copy link

05.6 complete — vocal seat live, seat split settled

Bench: trog-games/vocal-bench · ear test http://trogdor:3923/games/vocal-bench/assets/pick.html · decisions in docs/audio-gen-research.md 05.6.

Two-lane GPU window (HeartMuLa GPU0 ∥ ACE-Step GPU1, both fit 16GB) ran the whole matrix in 6 minutes wall — ACE finished its three clips in ~30s, HeartMuLa was the critical path.

Open questions, answered

  • Can HeartMuLa do instrumental? YES — the empty lyrics mode worked on the first probe ([inst] never needed).
  • Is it worth the music seat? NO. Ear test went 2-1 to ACE (HeartMuLa won the ambient explore brief only), against 6.5× generation time (50s vs 7.6s per 30s clip), 2.2× VRAM (12.7 vs 5.8 GiB), and a 1-in-3 early-EOS failure (a 30s request returned 1.4s) that ACE never showed.
  • Does cfg fix tag adherence? NO — the full 1.5/2.5/3.0 ladder was ear-tested on identical "folk rock, male vocal" tags and none delivered it. Voice/genre steering is a model property, not a knob, so the fix moved to the template.
  • cfg default: 2.5 — the ear preferred it over 1.5 on the take that already worked. Quality, not adherence.

Decisions

Seat split (a): ACE-Step keeps music; HeartMuLa is a vocal-only seat. It earns a seat for the one thing nothing else here can do — sing (ACE's vocal path NaNs on Turing).

Spec guidance is the adherence workaround: the vocal template entry now tells the brain not to promise a singer — describe the song, accept the voice, ask for takes if a specific voice matters. An unachievable brief is worse than a vague one.

Deferred with trigger: HeartMuLa instrumental for ambient/pad-heavy games (it won that one brief) — A/B a real game's briefs before touching the split.

Wired and live

  • The audio image is now vendored into trog (audio-server/, per your call that agentic-game-dev is deprecated): make build push builds both trog images, the build overlay hangs the audio context on the analyzer service (one image, two roles — standing CPU analyzer + GPU seat).
  • HeartMuLaBackend + POST /vocal: lyrics required and non-empty (empty = instrumental mode, so a lyricless call would silently return not-a-song), cfg 2.5, ≤180s, early-EOS output fails loudly instead of shipping a 1.4s "song". Dockerfile installs heartlib with its deps and asserts torch stays 2.10.0+cu128 — a future rebind fails the build, not the seat at 3am.
  • vocal asset kind end-to-end: fenced-lyrics parsing (multi-line, dedented), seconds clamped to the seat range, audio_item.route_for() so drain and orchestrator can't disagree on routes, lyricless guards at parse/produce/one-off, trog vocal --lyrics, CLAP critic coverage.
  • Weights moved to /mnt/models/audio/heartmula with relative ckpt symlinks (absolute ones can't resolve in-container) and a compat symlink at the bench path.

Proof

Real one-off through the deployed stack: trog vocal → brain swap → seat → 45.0s sung theme, 48kHz stereo, delivered in 84s, critic PASS (clap 0.478). http://trogdor:3923/oneoffs/2026-07-25/smoke-vocal-220700.wav

CLAP regression checked too — heartlib pins numpy 2.0.2, and the analyzer still scores 0.523 on a matching brief. 97 tests green.

Ticket complete from my side — close when satisfied.

## 05.6 complete — vocal seat live, seat split settled **Bench:** trog-games/vocal-bench · ear test http://trogdor:3923/games/vocal-bench/assets/pick.html · decisions in `docs/audio-gen-research.md` 05.6. Two-lane GPU window (HeartMuLa GPU0 ∥ ACE-Step GPU1, both fit 16GB) ran the whole matrix in **6 minutes** wall — ACE finished its three clips in ~30s, HeartMuLa was the critical path. ### Open questions, answered - **Can HeartMuLa do instrumental? YES** — the `empty` lyrics mode worked on the first probe (`[inst]` never needed). - **Is it worth the music seat? NO.** Ear test went 2-1 to ACE (HeartMuLa won the ambient `explore` brief only), against 6.5× generation time (50s vs 7.6s per 30s clip), 2.2× VRAM (12.7 vs 5.8 GiB), and a **1-in-3 early-EOS failure** (a 30s request returned 1.4s) that ACE never showed. - **Does cfg fix tag adherence? NO** — the full 1.5/2.5/3.0 ladder was ear-tested on identical "folk rock, male vocal" tags and **none** delivered it. Voice/genre steering is a model property, not a knob, so the fix moved to the template. - **cfg default: 2.5** — the ear preferred it over 1.5 on the take that already worked. Quality, not adherence. ### Decisions **Seat split (a): ACE-Step keeps music; HeartMuLa is a vocal-only seat.** It earns a seat for the one thing nothing else here can do — sing (ACE's vocal path NaNs on Turing). **Spec guidance is the adherence workaround:** the vocal template entry now tells the brain *not to promise a singer* — describe the song, accept the voice, ask for takes if a specific voice matters. An unachievable brief is worse than a vague one. **Deferred with trigger:** HeartMuLa instrumental for ambient/pad-heavy games (it won that one brief) — A/B a real game's briefs before touching the split. ### Wired and live - **The audio image is now vendored into trog** (`audio-server/`, per your call that agentic-game-dev is deprecated): `make build push` builds both trog images, the build overlay hangs the audio context on the analyzer service (one image, two roles — standing CPU analyzer + GPU seat). - `HeartMuLaBackend` + `POST /vocal`: lyrics required and non-empty (empty = instrumental mode, so a lyricless call would silently return not-a-song), cfg 2.5, ≤180s, early-EOS output **fails loudly** instead of shipping a 1.4s "song". Dockerfile installs heartlib with its deps and **asserts** torch stays 2.10.0+cu128 — a future rebind fails the build, not the seat at 3am. - `vocal` asset kind end-to-end: fenced-lyrics parsing (multi-line, dedented), seconds clamped to the seat range, `audio_item.route_for()` so drain and orchestrator can't disagree on routes, lyricless guards at parse/produce/one-off, `trog vocal --lyrics`, CLAP critic coverage. - Weights moved to `/mnt/models/audio/heartmula` with **relative** ckpt symlinks (absolute ones can't resolve in-container) and a compat symlink at the bench path. ### Proof Real one-off through the deployed stack: `trog vocal` → brain swap → seat → **45.0s sung theme, 48kHz stereo, delivered in 84s**, critic PASS (clap 0.478). http://trogdor:3923/oneoffs/2026-07-25/smoke-vocal-220700.wav CLAP regression checked too — heartlib pins numpy 2.0.2, and the analyzer still scores 0.523 on a matching brief. 97 tests green. Ticket complete from my side — close when satisfied.
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
feat/11-walking-skeleton
No results found.
Labels
Clear labels   
bug
Something is not working

  
duplicate
This issue or pull request already exists

  
enhancement
New feature

  
help wanted
Need some help

  
invalid
Something is wrong

  
question
More information is needed

  
wontfix
This won't be fixed

No labels
bug
duplicate
enhancement
help wanted
invalid
question
wontfix
Milestone
Clear milestone
No items
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
cmoriarty/trog#19
No description provided.