Test a fix in minutes: capture a run's step and replay it locally on Free #340

Closed
opened 2026-10-30 12:51:39 -04:00 by cmoriarty · 1 comment
Owner

What happens
Verifying a fix means waiting on a production run that takes hours or days: #336 waited for a resume, #338 for one Visual check, #306 for one spec-test fan-out. The sims are fast but use a scripted model, so they can't show how a real model behaves. Today we stopped, resumed and reran whole projects to watch one step.

Fix

  1. Capture: export one step of a production run as a capsule: the worktree at its entry checkpoint (refs/osf/ckpt/…/entry, as a git bundle), the run's frozen settings, its recorded graph and brief, and the attempt's transcript, including which models served it.
  2. Replay on Free, locally: a local osfd (tools/demo.sh free-up, the real pinned router) starts a run at that step from the capsule and runs it, either pinned to the models that served the incident or on today's chain. It honours the run's privacy choices, and the fake forge takes any push.
  3. Scripted replay: the capsule becomes a sim scenario, with its worktree as the fixture and its transcript as the play: a regression test of exactly what happened.

Done when
A production step is captured and replayed on the laptop in minutes, on Free and scripted, and an issue can close on a test or sim plus a local Free replay, without waiting on a production run.

**What happens** Verifying a fix means waiting on a production run that takes hours or days: #336 waited for a resume, #338 for one Visual check, #306 for one spec-test fan-out. The sims are fast but use a scripted model, so they can't show how a real model behaves. Today we stopped, resumed and reran whole projects to watch one step. **Fix** 1. **Capture:** export one step of a production run as a capsule: the worktree at its entry checkpoint (`refs/osf/ckpt/…/entry`, as a git bundle), the run's frozen settings, its recorded graph and brief, and the attempt's transcript, including which models served it. 2. **Replay on Free, locally:** a local osfd (`tools/demo.sh free-up`, the real pinned router) starts a run at that step from the capsule and runs it, either pinned to the models that served the incident or on today's chain. It honours the run's privacy choices, and the fake forge takes any push. 3. **Scripted replay:** the capsule becomes a sim scenario, with its worktree as the fixture and its transcript as the play: a regression test of exactly what happened. **Done when** A production step is captured and replayed on the laptop in minutes, on Free and scripted, and an issue can close on a test or sim plus a local Free replay, without waiting on a production run.
Author
Owner

Shipped in #343 (deployed, production on c3407b8) and #348 (tooling only, merged with [skip ci]). Change replay-a-step is archived, with the new spec step-replay.

Verified on run 159's Visual check (attempt 1), the incident behind #338:

  • ./tools/replay.sh capture '<step address>' --attempt 1: 0.9 s from production.
  • scripted: 24 s. The captured turns ran on today's code, and the judge's 7 browser subagent calls were refused.
  • free: 47 min on real Free models, in Braid's image. Production's attempt ran 66 min with no report; the replay wrote one.

Fixed while verifying (#348):

  • A scripted replay's retry had started on Zen's deepseek, a real model with its shell on the laptop. A replay is now one attempt, and its fake Free profile has no Zen.
  • A skipped replay prints why, with the failed step's output. Docker's disk had filled with 2 GB image rebuilds; the tool now prunes its old image.
  • Ctrl-C no longer leaves a replay's stack running.

How to use it: docs/testing.md, "Replaying a step" and "Closing an issue".

Shipped in #343 (deployed, production on c3407b8) and #348 (tooling only, merged with `[skip ci]`). Change `replay-a-step` is archived, with the new spec `step-replay`. **Verified on run 159's Visual check (attempt 1), the incident behind #338:** - `./tools/replay.sh capture '<step address>' --attempt 1`: **0.9 s** from production. - `scripted`: **24 s**. The captured turns ran on today's code, and the judge's 7 `browser` subagent calls were refused. - `free`: **47 min** on real Free models, in Braid's image. Production's attempt ran 66 min with no report; the replay wrote one. **Fixed while verifying (#348):** - A scripted replay's retry had started on Zen's deepseek, a real model with its shell on the laptop. A replay is now one attempt, and its fake Free profile has no Zen. - A skipped replay prints why, with the failed step's output. Docker's disk had filled with 2 GB image rebuilds; the tool now prunes its old image. - Ctrl-C no longer leaves a replay's stack running. How to use it: `docs/testing.md`, "Replaying a step" and "Closing an issue".
Sign in to join this conversation.
No labels
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
cmoriarty/braid#340
No description provided.