cmoriarty/trog
1
0
Fork 0
Code Issues 11 Pull requests Projects 1 Releases Packages Wiki Activity Actions

05.3 · Audio-gen follow-ups: video-synced foley, vocal themes, adaptive layers #16

New issue
Closed
opened 2026-07-24 09:54:18 -04:00 by cmoriarty · 4 comments
cmoriarty commented 2026-07-24 09:54:18 -04:00
Owner
Copy link

Follow-ups from docs/audio-gen-research.md (2026-07-24) — each is an investigate-then-decide, not a commitment. Evaluation bar as ever: open weights, commercially usable license, self-hosted, swap-friendly, and it must beat the incumbent on something measurable.

  • HunyuanVideo-Foley — foley synced to animation clips. The animate seat already produces a motion VIDEO before sheet assembly; this model generates foley time-aligned to video (SOTA on open foley benchmarks). Investigate: license (Tencent community terms — commercial OK?), VRAM/seat shape, and whether per-animation foley (footsteps matching the actual walk frames) beats prompt-only sfx enough to earn a seat. Natural pairing with the 05.2 critics pass.
  • HeartMuLa-oss-3B — vocal themes. Apache-2.0, lyrics-conditioned song generation, "best open model of 2026" per its repo. Wrong shape for instrumental BGM (ACE-Step holds), but the day a game wants a title song with vocals, this is the candidate. Investigate only when a spec asks for one — don't pre-adopt a seat with no consumer.
  • Adaptive audio layering. Best practice from the research: base ambient layer + intensity layers (combat adds on top), engine-side crossfading. ACE-Step's task modes (layering, track extraction) can generate the stems. Needs Phaser-side support in the game scaffolds — belongs with ticket 10's build sessions. Investigate: can ACE reliably produce aligned stems (same tempo/key/length) from one prompt?
  • TangoFlux relicense watch. Outbenchmarks stable-audio-open (CLAP 0.480 vs lower, faster) but checkpoints are non-commercial (WavCaps academic-only). No action possible — just re-check the license on occasion; if it or a successor ships commercially usable weights, bench it against SAO for the sfx seat.

Decision output per item: adopt (with seat/wiring plan), reject (with reason in the research doc), or defer (with the trigger that reopens it).

Follow-ups from [docs/audio-gen-research.md](https://forgejo.underthere.xyz/cmoriarty/trog/src/branch/main/docs/audio-gen-research.md) (2026-07-24) — each is an investigate-then-decide, not a commitment. Evaluation bar as ever: open weights, commercially usable license, self-hosted, swap-friendly, and it must beat the incumbent on something measurable. - [ ] **HunyuanVideo-Foley — foley synced to animation clips.** The animate seat already produces a motion VIDEO before sheet assembly; this model generates foley time-aligned to video (SOTA on open foley benchmarks). Investigate: license (Tencent community terms — commercial OK?), VRAM/seat shape, and whether per-animation foley (footsteps matching the actual walk frames) beats prompt-only sfx enough to earn a seat. Natural pairing with the 05.2 critics pass. - [ ] **HeartMuLa-oss-3B — vocal themes.** Apache-2.0, lyrics-conditioned song generation, "best open model of 2026" per its repo. Wrong shape for instrumental BGM (ACE-Step holds), but the day a game wants a title song with vocals, this is the candidate. Investigate only when a spec asks for one — don't pre-adopt a seat with no consumer. - [ ] **Adaptive audio layering.** Best practice from the research: base ambient layer + intensity layers (combat adds on top), engine-side crossfading. ACE-Step's task modes (layering, track extraction) can generate the stems. Needs Phaser-side support in the game scaffolds — belongs with ticket 10's build sessions. Investigate: can ACE reliably produce aligned stems (same tempo/key/length) from one prompt? - [ ] **TangoFlux relicense watch.** Outbenchmarks stable-audio-open (CLAP 0.480 vs lower, faster) but checkpoints are non-commercial (WavCaps academic-only). No action possible — just re-check the license on occasion; if it or a successor ships commercially usable weights, bench it against SAO for the sfx seat. **Decision output per item:** adopt (with seat/wiring plan), reject (with reason in the research doc), or defer (with the trigger that reopens it).
cmoriarty commented 2026-07-24 15:25:43 -04:00
Author
Owner
Copy link

Bench complete — every candidate generated real assets on trogdor; judge by ear on the bench review page (repo trog-games/audio-model-bench, asset-review layout: foley A/B videos, vocal players, stem mixer with play-together, licenses).

Decisions (detail + mechanics in docs/audio-gen-research.md):

  • HunyuanVideo-Foley — defer, ear test may promote. Runs on our hardware: XL ckpt, ~11GiB peak, ~20s per 2.5s clip (bf16 emulated on sm_75). The drag is the license: commercial OK <100M MAU but the agreement does not apply in EU/UK/South Korea, and outputs must be labeled machine-generated. Page shows it head-to-head against prompt-only stable-audio on the same animate clips (card flip, marching crowd, tumbling cards).
  • HeartMuLa-oss-3B — adopt on-demand, no standing seat. Apache-2.0, fp16-stable, 12.7GiB, RTF≈1.05 (90s song in 94s). Bench surprise: ACE-Step's vocal path NaNs on our stack at every length (90/60/24s) — Turing fp16 numerics; instrumental untouched. A vocal title theme has exactly one working model here. Recipe kept in /mnt/models/audio-bench.
  • Adaptive layering — generation side proven, still gated on ticket 10. ACE lego adds a combat layer on top of the base with duration locked by construction (24.000s across all stems, pinned bpm/key); extract pulls drums back out. Mixer on the page answers the groove question.
  • TangoFlux — watch continues. Still non-commercial as of 2026-07-24.

Scratch envs + weights: /mnt/models/audio-bench on trogdor (~45GB, rm -rf to reclaim; setup scripts re-create it). Brain was down only for the gen window and is back healthy.

Bench complete — every candidate generated real assets on trogdor; judge by ear on the **[bench review page](http://trogdor:3923/games/audio-model-bench/assets/review.html)** (repo `trog-games/audio-model-bench`, asset-review layout: foley A/B videos, vocal players, stem mixer with play-together, licenses). **Decisions** (detail + mechanics in [docs/audio-gen-research.md](https://forgejo.underthere.xyz/cmoriarty/trog/src/branch/main/docs/audio-gen-research.md)): - **HunyuanVideo-Foley — defer, ear test may promote.** Runs on our hardware: XL ckpt, ~11GiB peak, ~20s per 2.5s clip (bf16 emulated on sm_75). The drag is the license: commercial OK <100M MAU but the agreement **does not apply in EU/UK/South Korea**, and outputs must be labeled machine-generated. Page shows it head-to-head against prompt-only stable-audio on the same animate clips (card flip, marching crowd, tumbling cards). - **HeartMuLa-oss-3B — adopt on-demand, no standing seat.** Apache-2.0, fp16-stable, 12.7GiB, RTF≈1.05 (90s song in 94s). Bench surprise: **ACE-Step's vocal path NaNs on our stack at every length (90/60/24s)** — Turing fp16 numerics; instrumental untouched. A vocal title theme has exactly one working model here. Recipe kept in `/mnt/models/audio-bench`. - **Adaptive layering — generation side proven, still gated on ticket 10.** ACE `lego` adds a combat layer on top of the base with duration locked by construction (24.000s across all stems, pinned bpm/key); `extract` pulls drums back out. Mixer on the page answers the groove question. - **TangoFlux — watch continues.** Still non-commercial as of 2026-07-24. Scratch envs + weights: `/mnt/models/audio-bench` on trogdor (~45GB, `rm -rf` to reclaim; setup scripts re-create it). Brain was down only for the gen window and is back healthy.
cmoriarty commented 2026-07-24 16:03:05 -04:00
Author
Owner
Copy link

Ear verdicts recorded (page now shows verdict chips + notes; doc updated in b245f1d). Follow-up diagnosis on the three oddities:

  • cards-tumble still frame: confirmed — the animate source clip (title-video-214603.mp4, 5 frames) is essentially static; frames 0 and 4 are near-identical. Invalid input, not a mux bug. Case marked VOID on the page.
  • vocal-theme-b wrong voice/genre: not a harness mixup — tags files and meta both verified correct (folk rock, male vocal, warm acoustic guitar…). HeartMuLa ignored genre/gender at its default cfg_scale 1.5; retest at 2.5–3 when a real vocal-theme consumer exists. Noted as an adherence caveat on the adopt-on-demand decision.
  • layering distortion: your ear was exactly right. Waveform correlation shows lego returns the full mix, base included — corr(base, layer1)=0.980, corr(base, layer2)=0.962 — so play-together double-plays the base (louder + phasey). And extract never isolated drums: corr(base, extract)=0.938, it's ~the whole mix again. Working adaptive pattern with these exact files: engine crossfades between duration-locked full mixes (base ↔ layer1), no stacking. Doc updated accordingly; stem isolation marked unproven.

Final decisions standing: foley — reject for the sfx seat (SAO won the transient case decisively; HV's scene-ambience strength deferred as a future idea given the EU/UK/KR license carve-out) · HeartMuLa — adopt on-demand with the cfg adherence caveat · layering — full-mix crossfade proven, gated on ticket 10 · TangoFlux — watch.

Ear verdicts recorded (page now shows verdict chips + notes; doc updated in b245f1d). Follow-up diagnosis on the three oddities: - **cards-tumble still frame**: confirmed — the animate source clip (`title-video-214603.mp4`, 5 frames) is essentially static; frames 0 and 4 are near-identical. Invalid input, not a mux bug. Case marked VOID on the page. - **vocal-theme-b wrong voice/genre**: not a harness mixup — tags files and meta both verified correct (`folk rock, male vocal, warm acoustic guitar…`). HeartMuLa ignored genre/gender at its default cfg_scale 1.5; retest at 2.5–3 when a real vocal-theme consumer exists. Noted as an adherence caveat on the adopt-on-demand decision. - **layering distortion**: your ear was exactly right. Waveform correlation shows `lego` returns the **full mix, base included** — corr(base, layer1)=0.980, corr(base, layer2)=0.962 — so play-together double-plays the base (louder + phasey). And `extract` never isolated drums: corr(base, extract)=0.938, it's ~the whole mix again. Working adaptive pattern with these exact files: **engine crossfades between duration-locked full mixes** (base ↔ layer1), no stacking. Doc updated accordingly; stem isolation marked unproven. Final decisions standing: **foley — reject for the sfx seat** (SAO won the transient case decisively; HV's scene-ambience strength deferred as a future idea given the EU/UK/KR license carve-out) · **HeartMuLa — adopt on-demand** with the cfg adherence caveat · **layering — full-mix crossfade proven, gated on ticket 10** · **TangoFlux — watch**.
cmoriarty commented 2026-07-24 16:09:20 -04:00
Author
Owner
Copy link

HV-Foley verdict amended per license review: cleared for scene-ambience use when relevant (US-hosted use licensed; label outputs machine-generated in credits; EU/UK/KR clause binds model use, not shipped audio — untested reading, negligible at hobby scale). Still no sfx seat — that stands on quality. Doc updated.

HV-Foley verdict amended per license review: **cleared for scene-ambience use when relevant** (US-hosted use licensed; label outputs machine-generated in credits; EU/UK/KR clause binds model use, not shipped audio — untested reading, negligible at hobby scale). Still no sfx seat — that stands on quality. Doc updated.
cmoriarty commented 2026-07-24 16:12:41 -04:00
Author
Owner
Copy link

Closing out 05.3. All four items landed a decision, every decision is backed by generated assets and an ear test, and the evidence + recipes are preserved.

Final state:

  • HunyuanVideo-Foley — no sfx seat (SAO won the transient A/B decisively); cleared for scene-ambience use when relevant (US-hosted use licensed; label outputs machine-generated in credits).
  • HeartMuLa-oss-3B — adopt on-demand, no standing seat. Only working vocal model on this hardware (ACE-Step vocals NaN on Turing fp16 at every length — a real bench discovery). Caveat: weak tag adherence at cfg 1.5; try 2.5–3 when a consumer exists.
  • Adaptive layering — lego produces duration-locked full mixes (corr≈0.98 with base): the engine pattern is crossfade between intensity mixes, never stacking; extract doesn't isolate. Generation side ready for ticket 10.
  • TangoFlux — still non-commercial; watch continues.

Artifacts:

Cleanup: trogdor /mnt/models/audio-bench trimmed of outputs/logs/caches; kept weights + venvs (55G) as the working recipe the research doc points to — rm -rf the dir if the disk is ever needed; bench/ in git re-creates it. Brain + aegra healthy; docs-only changes, nothing to redeploy.

**Closing out 05.3.** All four items landed a decision, every decision is backed by generated assets and an ear test, and the evidence + recipes are preserved. **Final state:** - **HunyuanVideo-Foley** — no sfx seat (SAO won the transient A/B decisively); **cleared for scene-ambience use when relevant** (US-hosted use licensed; label outputs machine-generated in credits). - **HeartMuLa-oss-3B** — **adopt on-demand**, no standing seat. Only working vocal model on this hardware (ACE-Step vocals NaN on Turing fp16 at every length — a real bench discovery). Caveat: weak tag adherence at cfg 1.5; try 2.5–3 when a consumer exists. - **Adaptive layering** — `lego` produces duration-locked **full mixes** (corr≈0.98 with base): the engine pattern is crossfade between intensity mixes, never stacking; `extract` doesn't isolate. Generation side ready for ticket 10. - **TangoFlux** — still non-commercial; watch continues. **Artifacts:** - Review page (with ear-verdict chips): http://trogdor:3923/games/audio-model-bench/assets/review.html - Bench recipes now in git: `trog-games/audio-model-bench` `bench/` (setup, drivers, GPU-window orchestration, page builder, README with the sm_75 dtype patch + ckpt layout) - Decisions table: [docs/audio-gen-research.md](https://forgejo.underthere.xyz/cmoriarty/trog/src/branch/main/docs/audio-gen-research.md) (commits 342865f, b245f1d, 293d027) **Cleanup:** trogdor `/mnt/models/audio-bench` trimmed of outputs/logs/caches; kept weights + venvs (55G) as the working recipe the research doc points to — `rm -rf` the dir if the disk is ever needed; `bench/` in git re-creates it. Brain + aegra healthy; docs-only changes, nothing to redeploy.
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
feat/11-walking-skeleton
No results found.
Labels
Clear labels   
bug
Something is not working

  
duplicate
This issue or pull request already exists

  
enhancement
New feature

  
help wanted
Need some help

  
invalid
Something is wrong

  
question
More information is needed

  
wontfix
This won't be fixed

No labels
bug
duplicate
enhancement
help wanted
invalid
question
wontfix
Milestone
Clear milestone
No items
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
cmoriarty/trog#16
No description provided.