Findings: a whole Free run on a new project (Scratch) #365
Loading…
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
The findings log for a whole Free run on production:
thorough, a new multiplayer card game (scratch), with auto-approve on, started 2026-11-03 to see where Free stands. Findings arrive as comments and get their own issues where they need a fix.So far (through the proposal's review)
agent.review.combined: Cloudflare'sgemma-4-26b-a4b-itwas cut at 300 s ("provider stream ended before completion: unexpected EOF"). Braid retried once and benched the model for 30 minutes, and the step succeeded on Qwen. The cut at almost exactly 300 s looks like a limit on the provider's side, not confirmed.north-mini-code, then Gemma, then back to Qwen. Why it left Qwen isn't known yet.provider.unknownline for something Braid dealt with itself.Fine:
agent.revisetook 0.4 min onreflection/beam-free, and made the review's one finding (Depends on:lines) correctly. The proposal gate was auto-approved as set.More on finding 1, the 300 s cut. The request wasn't silent: its reasoning part arrived at the cut with about 50 KB in it, and no tool call or answer came. So Gemma spent the whole request thinking, and all of it was lost. A timer on silence wouldn't have caught this one.
Braid only recovers afterwards, by retrying and benching the model for 30 minutes. Nothing limits a request short of the router's 3,600 s cap. One sample, so the 300 s limit on Cloudflare's side is still unconfirmed.
Finding 1 now has its own ticket with the research: #366. The connection closed from the far side, not on Braid's 300 s timer, and the model was stuck in a thinking loop.
Build step, first 10 minutes: Kilo ran out, and the step kept going. 351 tool events so far.
beam-free61, Qwen 397B 4, Gemma 4deepseek-v4.1-flash6, none 3deepseek-v4.1-flash4 (still going)catalog.pysays, or something else counted against this IP. So Braid's own count thinks Kilo has room when it doesn't. That's probably why every later request tried Kilo first and hit its circuit breaker (39 times in this step, no time lost on each, but noise).free quota spentfor about 2 s. The quota wasn't spent; the provider's breaker was open. It cost seconds, but the label is wrong.context canceled(client_disconnected) on Qwen 1-2 s after a failover succeeded. Unknown yet whether opencode dropped them on purpose.Finding 4 (Kilo) now has its own ticket: #367.
openspec.apply[all]went from Qwen 397B to Zen'sdeepseek-v4.1-flash-freeand was compacted from 64,309 to 20,519 tokens, reasonprovider_switch, with 10 tasks still to do. This is finding 6's cost: the next model starts from a summary, not from what the last one had read. Watching whether the work after it repeats or undoes anything.The build step finished: 36.5 minutes, all 25 tasks ticked, no cuts. That's about what a similar build took on vLLM (36 minutes, run 147), though on a different project, so it's only a rough comparison.
It ran on six models: Zen's
deepseek-v4.1-flash-free184 requests,beam-free61, Gemma 26, Qwen 397B 16,gpt-oss-120b2,glm-4.5-flash2. Zen carried most of it once Kilo refused, so finding 4 cost less time than feared. Whether the code holds together across those changes is for the tests and the Visual check to say.Run 170 failed after 64.5 minutes, at the first spec-test step. Its proposal, review and build all finished. Skipped: 45 steps, including three of the four spec-test steps, the Visual check and the merge. So this run says nothing yet about whether the code holds together.
exportended the run. Not a Free problem: #368.free quota spent, each up to 12 s. Only 2 or 3 requests a minute were served, then 10 in the minute Kilo came back. In the build step the same breaker failed over to other models, but here most of those requests ended on Kilo alone, and Zen served none of this step. Why is unknown: #367.Free's scorecard for this run, to the point it stopped
export.Correction to finding 11: the slow step ran out of capacity, rather than failing to fail over. OVH's 2 a minute was the only strong model left (details on #367). The sweep of free plans and what could add capacity is on #369.
Run 171 started: the same brief and settings as run 170, on the same repository, at
d53f9e9. New since run 170: Kilo first and Hetzner second (#369), Kilo's refusals counted (#367), and a spec-test step's edit put back before its retry (#368). Findings from it continue here.#and stops mid-word:# Build "Scratch" — a multiplayer online card ga. It's the first line's first 48 characters, and it shows in the top bar and on the Inference page.Run 171, through the proposal: 2.7 minutes, all on
beam-free(explore 0.9, propose 1.8 with one retry). In run 170 these two steps were also fine, so this says nothing yet about Kilo running out.beam-freewrote the proposal to a file literally namedspecs/**/*.md.openspec instructions proposalhad given it the right path.openspec validate --strictthen passed with noproposal.mdat all, and Braid's check on the step caught it: "proposal: ready — 0 non-empty path(s)". The retry wrote it to the right place and removed the stray file, in 37 s. Braid's check did its job here, so nothing to fix; one model slip, worth watching if it repeats.Run 171, the proposal's review took 9.8 minutes, and one request was 7 of them. The build has now started, about 13 minutes into the run, with the proposal gate auto-approved as set.
Qwen3.8-27Bspent 424 s on one request of the review (14,753 tokens out, about 35 tok/s), where the step's other 11 requests took 2.5 minutes between them. It did finish and answer, so nothing was lost this time, but it's #366's question again: nothing limits a request short of the router's cap, and run 170's same step lost 5 minutes the same way on another model.Not a bug, checked: the review skipped Gemini 3.8 Flash, first in the judge tier, because its 20 a day are kept for the code review and the Visual check. They're still untouched for when this run gets there.
Run 171 is at the code review after 31 minutes: 245 requests, none refused, nothing waited for. Run 170 had taken 64.5 minutes to reach its first spec-test step and ended there.
beam-freebeam-free233 of the 245 went to
beam-free; the other 12 were the proposal's review. So the speed comes from Kilo leading, as the replay said it would (#369), and Kilo hasn't reached its count of about 300 yet.The speed says nothing yet about the code: the tests and the Visual check have that to say. A spec-test step hasn't needed #368's put-back so far, so that fix is still unproven on this run.
Run 171's code review: 15.2 minutes, and it found real gaps in the fast build. It reported fixing these, and its tests pass, 43 in all:
127.0.0.1, so Docker's port forwarding wouldn't reach it.That's the point of the step, and it means the 5.7-minute build wasn't complete on its own. The Visual check will show whether the rest of it is.
503 high demandand then504timeouts, and its provider's breaker opened, so the review ran on Qwen 397B, Qwen 27B and onenorth-mini-code. Failover worked and no request was lost. One Hetzner request again took 257 s (5,521 tokens out), as in finding 14.Run 171 finished: 104.7 minutes, nobody needed, PR merged, change archived. 38 steps succeeded and 28 were skipped as not needed. No fix round ran.
beam-freeQwen3.8-27BQwen3.5-397Bnorth-mini-code508 requests, none refused, no parks, one compaction, no stalls. 27 failed legs all failed over, mostly open breakers.
Not tested by this run: Kilo running out (260, under its count of about 300), and #368's put-back (every spec-test step passed first time). I expected Kilo to run out and it didn't. Those two rest on their sims and replays.
The game falls short of the brief, and the Visual check passed it anyway. Verdict
pass: 13 pass, 0 fail, 3 unclear. From its screenshots and the code:dealer restriction false · first lead leftOfDealer). The felt table and your hand need a scroll.Illegal playstays on the end screen. The check noted it and still passed that criterion. The PR lists it under Not done, as it does the three unclear items, which is honest.window.__apphook (93 evaluate calls, 5 clicks), bidding 0 every round. Two people in two browsers is covered only by the server's tests.verdict; the retry fixed the file in 3.5 minutes.So Free is now fast and steady enough to finish a whole project alone. What it builds in 5.7 minutes needs checking harder than this run checked it.
Run 172: the same Scratch brief on the local profile (vLLM 0.33.0, Qwen3.8-27B W8A16, MTP k=4, TP4 on 4x RTX 5000), thorough, auto-approve. Succeeded 2026-11-04 15:49: 65/65 steps, 0 failures, 2 attempts, 503 min wall against run 171's 105 min. Repo
cmoriarty/scratch-vllm-mtp.What the 4.8x is. Volume, not speed: the per-message rate is the same, and the 27B model did 7x the generation, 4x the tool calls and 35 browser subagents, delivering 3.4x the source. Steps:
openspec.apply[all]169 min (171: 5.7), visual check 55 + 74 + 58 min (171: 3.7, one round, braid#379), fixes 33 + 16, spec tests 18 + 9 + 8 (braid#376).Compactions. Five alarms at 197-202k of 226k, all in forked sessions: four respawns from Braid's note, one real compaction (197k -> 9k with 28 tasks left). The pauses cost 3.8 min in all; the re-orientation cost more:
readcalls in the apply step went from 3-6 per 15 min before each alarm to 42-75 after, about 10-15 min per event, 40-60 min over the run (8-12 % of wall). braid#378 (alarm wording) and braid#379 (the 64k judge brief that forces the respawn every round) are the two causes filed.Quality of the app itself: not yet judged by the owner.
Closing this log. Runs 170, 171 and 172 are over, and every finding that needed a fix has its own issue.
exportended the runNever filed, small, and still only here:
provider.unknownline (3);Findings 17 to 20 describe the generated game and how the check played it. The owner's judgement of the app's quality was never recorded here.