A Free request can think for minutes and end with nothing #366

Open
opened 2026-11-03 09:41:29 -05:00 by cmoriarty · 2 comments
Owner

What happens
A Free request can run for minutes and end with nothing. On the Scratch run (#365), Cloudflare's gemma-4-26b-a4b-it thought for 300 s, sending about 50 KB of reasoning and no tool call or answer, then the stream was cut. That was about 5 of the review step's 6.5 minutes. Braid retried on Qwen and benched the model for 30 minutes, but the time was already gone. Nothing ends a request short of the router's 3,600 s cap.

Research first

  • How often are requests cut, on which models, and how long had they run?
  • Is 300 s Cloudflare's limit, or does it vary?
  • How long do healthy requests take, and how long does a model normally think before its first tool call or answer? Could a limit tell the two apart?
  • Can the router limit this, or does it have to be in Braid's stream handling?

Fix
Decide from the research. Candidates: limit requests that only think, or move models that get cut down their chains.

Done when
A sim shows a request that only thinks is given up on by the chosen limit, a healthy slow request isn't, and a Free replay of a step that was cut finishes sooner.

**What happens** A Free request can run for minutes and end with nothing. On the Scratch run (#365), Cloudflare's `gemma-4-26b-a4b-it` thought for 300 s, sending about 50 KB of reasoning and no tool call or answer, then the stream was cut. That was about 5 of the review step's 6.5 minutes. Braid retried on Qwen and benched the model for 30 minutes, but the time was already gone. Nothing ends a request short of the router's 3,600 s cap. **Research first** - How often are requests cut, on which models, and how long had they run? - Is 300 s Cloudflare's limit, or does it vary? - How long do healthy requests take, and how long does a model normally think before its first tool call or answer? Could a limit tell the two apart? - Can the router limit this, or does it have to be in Braid's stream handling? **Fix** Decide from the research. Candidates: limit requests that only think, or move models that get cut down their chains. **Done when** A sim shows a request that only thinks is given up on by the chosen limit, a healthy slow request isn't, and a Free replay of a step that was cut finishes sooner.
Author
Owner

Research, first pass. One cut so far, so the numbers below are thin. They grow as the Scratch run goes on.

Braid's own 300 s timer didn't cause it. STREAM_IDLE_S is also 300, which looked suspicious, but GoModel 0.1.103 (internal/llmclient/stream_idle.go):

  • resets that timer on any bytes, so reasoning counts as data
  • reports a firing as provider stream stalled: no data for …

The router's log for this request says provider stream ended before completion: unexpected EOF, so the connection closed from the far side. That points at Cloudflare or the path to it. Cloudflare's Workers AI limits page gives no time limit, so 300 s is still unconfirmed.

The model was stuck, not slow. The lost reasoning is 47,796 characters (7,314 words), and no tool call came. It ends in a loop: "I'll proceed. Wait, I just realized I should check…", with the same review question repeated 9 times.

A limit looks separable here, on thin data. All requests on this run so far (137):

Model Requests Median Slowest healthy
kilo/reflection/beam-free 100 2 s 15 s
cloudflare/…/gemma-4-26b-a4b-it 22 7 s about 11 s (p90)
ovh/Qwen3.5-397B-A17B 14 8 s 41 s

Only the cut request passed 120 s. None of this covers the build step's big writes yet, or earlier runs, whose events are gone. #195 records GLM-5.3 streams still sending at 600 s, so a limit on total time would be wrong. It has to be on thinking with no tool call or answer.

Where a limit could live is the hard part.

  • The router's timer can't do it: it only sees bytes, and it can't tell thinking from output.
  • Braid's log has no events between the request's start and the cut, and the reasoning arrived whole at the end. So Braid may not see thinking as it happens. Check whether opencode's live stream gives deltas.
  • A lower output cap per request might stop a runaway too, but it could also cut a big file write.

Still open

  1. Do cuts on other Cloudflare models and other providers also land near 300 s?
  2. How long does thinking run before the first tool call on the build step?
  3. Does opencode expose reasoning as it streams?
  4. Should repetition count as no progress, rather than time alone?

Sources: Cloudflare Workers AI limits, a forum report of Workers AI timeouts on a Gemma model.

**Research, first pass.** One cut so far, so the numbers below are thin. They grow as the Scratch run goes on. **Braid's own 300 s timer didn't cause it.** `STREAM_IDLE_S` is also 300, which looked suspicious, but GoModel 0.1.103 (`internal/llmclient/stream_idle.go`): - resets that timer on any bytes, so reasoning counts as data - reports a firing as `provider stream stalled: no data for …` The router's log for this request says `provider stream ended before completion: unexpected EOF`, so the connection closed from the far side. That points at Cloudflare or the path to it. Cloudflare's Workers AI limits page gives no time limit, so 300 s is still unconfirmed. **The model was stuck, not slow.** The lost reasoning is 47,796 characters (7,314 words), and no tool call came. It ends in a loop: "I'll proceed. Wait, I just realized I should check…", with the same review question repeated 9 times. **A limit looks separable here, on thin data.** All requests on this run so far (137): | Model | Requests | Median | Slowest healthy | |---|---|---|---| | `kilo/reflection/beam-free` | 100 | 2 s | 15 s | | `cloudflare/…/gemma-4-26b-a4b-it` | 22 | 7 s | about 11 s (p90) | | `ovh/Qwen3.5-397B-A17B` | 14 | 8 s | 41 s | Only the cut request passed 120 s. None of this covers the build step's big writes yet, or earlier runs, whose events are gone. #195 records GLM-5.3 streams still sending at 600 s, so a limit on total time would be wrong. It has to be on thinking with no tool call or answer. **Where a limit could live is the hard part.** - The router's timer can't do it: it only sees bytes, and it can't tell thinking from output. - Braid's log has no events between the request's start and the cut, and the reasoning arrived whole at the end. So Braid may not see thinking as it happens. Check whether opencode's live stream gives deltas. - A lower output cap per request might stop a runaway too, but it could also cut a big file write. **Still open** 1. Do cuts on other Cloudflare models and other providers also land near 300 s? 2. How long does thinking run before the first tool call on the build step? 3. Does opencode expose reasoning as it streams? 4. Should repetition count as no progress, rather than time alone? Sources: [Cloudflare Workers AI limits](https://developers.cloudflare.com/workers-ai/platform/limits/), [a forum report of Workers AI timeouts on a Gemma model](https://community.cloudflare.com/t/ai-gateway-request-timeout/830726).
Author
Owner

Numbers from run 170's build step (36.5 min): no cuts. 26 Gemma requests totalled 199 s, 16 on Qwen 397B totalled 116 s, 61 on beam-free totalled 159 s. The 184 on Zen's DeepSeek have no recorded duration, so they can't help with this. The run failed before its longer steps (#368), so the review's cut is still the only one, and question 2 (thinking time before the first tool call) is still open.

Numbers from run 170's build step (36.5 min): no cuts. 26 Gemma requests totalled 199 s, 16 on Qwen 397B totalled 116 s, 61 on `beam-free` totalled 159 s. The 184 on Zen's DeepSeek have no recorded duration, so they can't help with this. The run failed before its longer steps (#368), so the review's cut is still the only one, and question 2 (thinking time before the first tool call) is still open.
Sign in to join this conversation.
No labels
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
cmoriarty/braid#366
No description provided.