Re-run the pinball comparison after these fixes #260

Closed
opened 2026-10-27 15:30:59 -04:00 by cmoriarty · 4 comments
Owner

So far (same brief, Claude Sonnet 5.5, time not spent waiting on a person)

  • Claude Code: 19.8 min. Looks great; the ball drains too easily.
  • Braid, planned: 64.6 min for the same scope. Placeholder look.
  • Braid, one-shot: 22.0 min. Looks good, comparable to Claude Code's; the plunger stops working after a drain.

Next, once the fixes land

  • The same brief, planned.
  • The same brief as a one-shot on solo and on minimalist, to see what thorough's extra steps buy.
  • Claude Code in a clean folder (claude -p --model sonnet, no CLAUDE.md). Last time it saw Braid's CLAUDE.md.

Compare
Time, tokens, and a hands-on play of each game: look, physics, bugs.

Waits on: #244, #245, #246, #247, #248, #249, #250, #251, #252, #253, #254, #255, #256, #257, #258, #259.

From the Braid vs Claude Code pinball test, 2026-10-27 (runs 110–120).

**So far** (same brief, Claude Sonnet 5.5, time not spent waiting on a person) - Claude Code: 19.8 min. Looks great; the ball drains too easily. - Braid, planned: 64.6 min for the same scope. Placeholder look. - Braid, one-shot: 22.0 min. Looks good, comparable to Claude Code's; the plunger stops working after a drain. **Next, once the fixes land** - The same brief, planned. - The same brief as a one-shot on `solo` and on `minimalist`, to see what `thorough`'s extra steps buy. - Claude Code in a clean folder (`claude -p --model sonnet`, no CLAUDE.md). Last time it saw Braid's CLAUDE.md. **Compare** Time, tokens, and a hands-on play of each game: look, physics, bugs. Waits on: #244, #245, #246, #247, #248, #249, #250, #251, #252, #253, #254, #255, #256, #257, #258, #259. _From the Braid vs Claude Code pinball test, 2026-10-27 (runs 110–120)._
Author
Owner

Baseline from the first test (2026-10-27, Claude Sonnet 5.5 on both sides)

Braid one-shot is run 120. Braid planned is runs 110–116 for time (the same scope as Claude Code's game), and runs 110–119 for the rest.

Time (minutes)

Claude Code Braid one-shot Braid planned
Wall clock 19.8 39.9 116.9
Waiting on a person 0 17.9 52.3
Unattended 19.8 22.0 64.6

Where Braid's time went (minutes, gate waits left out)

Phase One-shot Planned (runs 110–119)
Building (apply, spec tests) 12.9 31.1
Tests and project CI 5.4 35.9 (local tests: 3.7)
Merge, git, archive 0.2 20.6 (18.6 of it false-conflict retries)
Judge, fix, code review 0.7 11.8
Propose 1.2 9.4
Proposal review 0.5 4.9
Explore, summary, plan upkeep 0.5 9.2

Tokens

Claude Code Braid one-shot Braid planned
Turns 55 94 676
Tool calls 57 81 647
Output 137k 105k 377k
Cache read 7.6M 3.9M 14.8M
Cache write 0.22M 0.36M 3.08M

Code and tests (each repo's main branch)

Claude Code Braid one-shot Braid planned
Source lines 2,303 2,568 1,819
Test lines 921 1,374 4,263
Test lines per source line 0.4 0.5 2.3
Unit tests (files) 66 (9) 82 (9) 252 (41)
Browser test files 0 1 8

The one-shot kept its tests lean. The planned project's bloat came with one browser spec per change.

**Baseline from the first test** (2026-10-27, Claude Sonnet 5.5 on both sides) Braid one-shot is run 120. Braid planned is runs 110–116 for time (the same scope as Claude Code's game), and runs 110–119 for the rest. **Time** (minutes) | | Claude Code | Braid one-shot | Braid planned | |---|---|---|---| | Wall clock | 19.8 | 39.9 | 116.9 | | Waiting on a person | 0 | 17.9 | 52.3 | | **Unattended** | **19.8** | **22.0** | **64.6** | **Where Braid's time went** (minutes, gate waits left out) | Phase | One-shot | Planned (runs 110–119) | |---|---|---| | Building (apply, spec tests) | 12.9 | 31.1 | | Tests and project CI | 5.4 | 35.9 (local tests: 3.7) | | Merge, git, archive | 0.2 | 20.6 (18.6 of it false-conflict retries) | | Judge, fix, code review | 0.7 | 11.8 | | Propose | 1.2 | 9.4 | | Proposal review | 0.5 | 4.9 | | Explore, summary, plan upkeep | 0.5 | 9.2 | **Tokens** | | Claude Code | Braid one-shot | Braid planned | |---|---|---|---| | Turns | 55 | 94 | 676 | | Tool calls | 57 | 81 | 647 | | Output | 137k | 105k | 377k | | Cache read | 7.6M | 3.9M | 14.8M | | Cache write | 0.22M | 0.36M | 3.08M | **Code and tests** (each repo's main branch) | | Claude Code | Braid one-shot | Braid planned | |---|---|---|---| | Source lines | 2,303 | 2,568 | 1,819 | | Test lines | 921 | 1,374 | 4,263 | | Test lines per source line | 0.4 | 0.5 | 2.3 | | Unit tests (files) | 66 (9) | 82 (9) | 252 (41) | | Browser test files | 0 | 1 | 8 | The one-shot kept its tests lean. The planned project's bloat came with one browser spec per change.
Author
Owner

Findings from the re-run (final, after a full read of the transcripts of runs 121–132 and the hands-on play)

Bugs, filed

  • #264 The shell guard refuses file writes that hold a /** comment: 15 times in 9 of 12 runs; in run 125 it forced a step to run again.
  • #271 Build steps write tests without spec tags, so round 1 of the fix loop only adds them (6 of 11 build runs).
  • #272 solo never asked its new web game for window.__app, and the judge's "unclear" play-through passed (run 122, the table with no drain).
  • #269 The judge passes a backwards flipper, no centre drain and glaring bloom (hands-on play).
  • #266 A visual fix loop goes in circles: the fixer never looks, and the judge's browser keeps old state (runs 123, 131).
  • #265 Failed judge rounds show as green checks.
  • #267 The fix loop asks a person after one round without progress (run 126).
  • #270 Re-planning drops runs' Not done gaps from run 127 on, and the project finishes with them untracked.
  • #268 A planned project still adds a browser spec per change (6 specs, 1.4 test lines per source line).
  • #273 A Braid-built Node project won't start on a Mac: its lockfile lacks the Mac binaries.
  • #263 The project page and Home row are a wall of text; "0 of 2 items" counts only the current milestone.
  • #276 Compacting a forked session sends only the forked-from history and the brief (found in the simulated runs while building #275).
  • #278 Lint rule B7 judges a test by its file name; run 133's build renamed a test twice to please it.
  • Fixed since: #271 (build tests untagged) in #275, verified on run 133.

Smaller, not filed

  • Builds spend about 26 of their 66 minutes re-running unit tests (115 runs) and a full tsc (58 runs, ~10 s each); tsc --incremental would cut most of that.
  • check-replan's pass message ("PLAN.md keeps the plan's rules…") reads like a problem; agents second-guessed it four times.
  • The summary step goes over its 8-bullet limit and asks "tell me if you want it cut", which nobody answers.
  • Judges read window.__app.state without calling it (3 times), and a few judge browser calls lost their page.

Checked and fine

  • The proposal reviews earn their cost: 134 findings in 31 reports (87 should-fix, 2 blocking), acted on by the new revision step at about 2 min a run.
  • The judge, summary and code review don't reinstall or re-run the suite; an earlier count of mine matched text inside their reports.

Now working as intended

  • No review waited for a person (#244); no false merge conflicts (#245); tests took minutes with Use CI off (#248).
  • The project carried on after every run and finished its plan (#246); its first item was the whole game (#251).
  • A person's words at a stopped fix loop reached the fixer (#257); the judge's play-through caught a real bug (#254).
**Findings from the re-run** (final, after a full read of the transcripts of runs 121–132 and the hands-on play) **Bugs, filed** - #264 The shell guard refuses file writes that hold a `/**` comment: 15 times in 9 of 12 runs; in run 125 it forced a step to run again. - #271 Build steps write tests without spec tags, so round 1 of the fix loop only adds them (6 of 11 build runs). - #272 `solo` never asked its new web game for `window.__app`, and the judge's "unclear" play-through passed (run 122, the table with no drain). - #269 The judge passes a backwards flipper, no centre drain and glaring bloom (hands-on play). - #266 A visual fix loop goes in circles: the fixer never looks, and the judge's browser keeps old state (runs 123, 131). - #265 Failed judge rounds show as green checks. - #267 The fix loop asks a person after one round without progress (run 126). - #270 Re-planning drops runs' Not done gaps from run 127 on, and the project finishes with them untracked. - #268 A planned project still adds a browser spec per change (6 specs, 1.4 test lines per source line). - #273 A Braid-built Node project won't start on a Mac: its lockfile lacks the Mac binaries. - #263 The project page and Home row are a wall of text; "0 of 2 items" counts only the current milestone. - #276 Compacting a forked session sends only the forked-from history and the brief (found in the simulated runs while building #275). - #278 Lint rule B7 judges a test by its file name; run 133's build renamed a test twice to please it. - Fixed since: #271 (build tests untagged) in #275, verified on run 133. **Smaller, not filed** - Builds spend about 26 of their 66 minutes re-running unit tests (115 runs) and a full `tsc` (58 runs, ~10 s each); `tsc --incremental` would cut most of that. - `check-replan`'s pass message ("PLAN.md keeps the plan's rules…") reads like a problem; agents second-guessed it four times. - The summary step goes over its 8-bullet limit and asks "tell me if you want it cut", which nobody answers. - Judges read `window.__app.state` without calling it (3 times), and a few judge browser calls lost their page. **Checked and fine** - The proposal reviews earn their cost: 134 findings in 31 reports (87 should-fix, 2 blocking), acted on by the new revision step at about 2 min a run. - The judge, summary and code review don't reinstall or re-run the suite; an earlier count of mine matched text inside their reports. **Now working as intended** - No review waited for a person (#244); no false merge conflicts (#245); tests took minutes with Use CI off (#248). - The project carried on after every run and finished its plan (#246); its first item was the whole game (#251). - A person's words at a stopped fix loop reached the fixer (#257); the judge's play-through caught a real bug (#254).
Author
Owner

Re-run after the fixes (#261, live as 5598ddf). Same brief, Claude Sonnet 5.5, auto-approve on, Use CI off (the new default). The three Braid runs ran side by side. Claude Code was not re-run.

The same scope as Claude Code's game (minutes, not counting waits on a person)

First test Now
Braid planned 64.6 (runs 110–116) 24.7 (run 121's plan, run 124's whole game)
Braid one-shot 22.0 (thorough, run 120) 15.3 (solo, run 122) · 37.5 (minimalist, run 123)
Claude Code 19.8 not re-run

minimalist is slow for one reason: 8 judge rounds over the ball's look (#266).

The whole planned project

First test (runs 110–119) Now (runs 121, 124–132)
How it ended a silent stop after 10 runs finished its whole plan by itself
What it built up to multiball 9 items: the game, missions and multiball, tilt and high scores, settings, a second table, achievements, sounds, gamepad and touch, an online score service
Braid's own time about 105 min 131 min
Waiting on a person 52.3 min (runs 110–116 alone) 0.8 min

Where the time went now (minutes)

Phase solo minimalist Planned, whole project
Building (apply, spec tests) 11.0 18.5 57.9
Judge, fix, code review 2.2 14.8 26.4
Tests 0.2 2.0 13.2
Propose and its review 0.8 1.4 21.2
Merge, git, archive 0.2 0.2 1.7
Explore, summary, plan 0.6 0.2 9.6

Tokens

solo minimalist Planned, the whole game Planned, whole project
Turns 64 252 147 1,034
Tool calls 80 320 173 1,073
Output 70k 177k 129k 662k
Cache read 3.5M 11.7M 7.8M 36.4M
Cache write 0.18M 0.89M 0.47M 4.95M

Code and tests (each repo's main branch, all counted the same way)

Claude Code First one-shot First planned solo minimalist Planned now
Source lines 2,372 2,599 1,853 1,928 2,783 4,771
Test lines 929 1,374 4,263 498 1,195 6,727
Tests per source line 0.4 0.5 2.3 0.3 0.4 1.4
Unit tests (files) 66 (9) 82 (9) 248 (41) 26 (4) 81 (12) 342 (56)
Browser test files 0 1 8 0 1 6

What changed

  • No review waited for a person: every gate auto-approve answered took under a second (#244).
  • No false merge conflicts: merge, git and archive took 1.7 min over ten runs, against 20.6 (#245).
  • Tests took 13.2 min over ten runs with Use CI off, against 35.9 with CI (#248).
  • The first item was the whole game, and the project re-planned and carried on after every run (#246, #251).
  • The judge's play-through caught a real bug: every launch re-armed the ball save, so a game never ended (#254).
  • A person's words at a stopped fix loop reached the fixer, and its next round passed (#257).

What didn't (bugs filed: see the findings comment)

  • The shell guard refused 15 file writes in 9 of 12 runs (#264).
  • A visual fix loop went in circles for 8 rounds (#265, #266), and a loop stopped once too early (#267).
  • The planned project's tests still grow with every change (#268).

Still to do
A hands-on play of each game: cmoriarty/pinball-braid-2 (planned), cmoriarty/pinball-braid-solo, cmoriarty/pinball-braid-minimalist.

**Re-run after the fixes** (#261, live as 5598ddf). Same brief, Claude Sonnet 5.5, auto-approve on, Use CI off (the new default). The three Braid runs ran side by side. Claude Code was not re-run. **The same scope as Claude Code's game** (minutes, not counting waits on a person) | | First test | Now | |---|---|---| | Braid planned | 64.6 (runs 110–116) | **24.7** (run 121's plan, run 124's whole game) | | Braid one-shot | 22.0 (`thorough`, run 120) | **15.3** (`solo`, run 122) · **37.5** (`minimalist`, run 123) | | Claude Code | 19.8 | not re-run | `minimalist` is slow for one reason: 8 judge rounds over the ball's look (#266). **The whole planned project** | | First test (runs 110–119) | Now (runs 121, 124–132) | |---|---|---| | How it ended | a silent stop after 10 runs | **finished its whole plan by itself** | | What it built | up to multiball | 9 items: the game, missions and multiball, tilt and high scores, settings, a second table, achievements, sounds, gamepad and touch, an online score service | | Braid's own time | about 105 min | 131 min | | Waiting on a person | 52.3 min (runs 110–116 alone) | **0.8 min** | **Where the time went now** (minutes) | Phase | `solo` | `minimalist` | Planned, whole project | |---|---|---|---| | Building (apply, spec tests) | 11.0 | 18.5 | 57.9 | | Judge, fix, code review | 2.2 | 14.8 | 26.4 | | Tests | 0.2 | 2.0 | 13.2 | | Propose and its review | 0.8 | 1.4 | 21.2 | | Merge, git, archive | 0.2 | 0.2 | 1.7 | | Explore, summary, plan | 0.6 | 0.2 | 9.6 | **Tokens** | | `solo` | `minimalist` | Planned, the whole game | Planned, whole project | |---|---|---|---|---| | Turns | 64 | 252 | 147 | 1,034 | | Tool calls | 80 | 320 | 173 | 1,073 | | Output | 70k | 177k | 129k | 662k | | Cache read | 3.5M | 11.7M | 7.8M | 36.4M | | Cache write | 0.18M | 0.89M | 0.47M | 4.95M | **Code and tests** (each repo's main branch, all counted the same way) | | Claude Code | First one-shot | First planned | `solo` | `minimalist` | Planned now | |---|---|---|---|---|---|---| | Source lines | 2,372 | 2,599 | 1,853 | 1,928 | 2,783 | 4,771 | | Test lines | 929 | 1,374 | 4,263 | 498 | 1,195 | 6,727 | | Tests per source line | 0.4 | 0.5 | 2.3 | 0.3 | 0.4 | 1.4 | | Unit tests (files) | 66 (9) | 82 (9) | 248 (41) | 26 (4) | 81 (12) | 342 (56) | | Browser test files | 0 | 1 | 8 | 0 | 1 | 6 | **What changed** - No review waited for a person: every gate auto-approve answered took under a second (#244). - No false merge conflicts: merge, git and archive took 1.7 min over ten runs, against 20.6 (#245). - Tests took 13.2 min over ten runs with Use CI off, against 35.9 with CI (#248). - The first item was the whole game, and the project re-planned and carried on after every run (#246, #251). - The judge's play-through caught a real bug: every launch re-armed the ball save, so a game never ended (#254). - A person's words at a stopped fix loop reached the fixer, and its next round passed (#257). **What didn't** (bugs filed: see the findings comment) - The shell guard refused 15 file writes in 9 of 12 runs (#264). - A visual fix loop went in circles for 8 rounds (#265, #266), and a loop stopped once too early (#267). - The planned project's tests still grow with every change (#268). **Still to do** A hands-on play of each game: `cmoriarty/pinball-braid-2` (planned), `cmoriarty/pinball-braid-solo`, `cmoriarty/pinball-braid-minimalist`.
Author
Owner

Hands-on play (the person's verdict)

  • All three look good. The planned game and solo turned the glow up too high: glaring and fuzzy.
  • The planned game plays best.
  • solo left no drain between the flippers, so it's hard to lose a ball.
  • minimalist has badly set-up flippers; the right one faces the wrong way.

The judge passed all three, so it missed each of these: #269.

That completes the re-run: results above, findings in the list.

**Hands-on play** (the person's verdict) - **All three look good.** The planned game and `solo` turned the glow up too high: glaring and fuzzy. - **The planned game plays best.** - **`solo`** left no drain between the flippers, so it's hard to lose a ball. - **`minimalist`** has badly set-up flippers; the right one faces the wrong way. The judge passed all three, so it missed each of these: #269. That completes the re-run: results above, findings in the list.
Sign in to join this conversation.
No labels
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
cmoriarty/braid#260
No description provided.