Kilo refuses after 100 requests where its docs and our catalog say 200 an hour #367

Closed
opened 2026-11-03 09:46:33 -05:00 by cmoriarty · 4 comments
Owner

What happens
On the Scratch run (#365), Kilo's reflection/beam-free served 100 requests in 12.2 minutes, then refused every one with a 429: "Sign up for free to continue and explore 500 other models. Takes 2 minutes, no credit card required. Or come back later."

Our catalog, dated 2026-10-17, says 200 requests an hour per IP, and Kilo's usage docs still say that. They also give a different over-limit message: "Rate limit exceeded for free models. Please try again later." So Kilo's behaviour no longer matches its docs, which may mean a policy change. Nothing confirms that yet.

This run was the only Free traffic in the event log, and my laptop replays were over an hour earlier. Whether they count depends on sharing an IP, but they fall outside an hourly window either way.

Why it matters
Kilo is the first model tried in the chain, and the run lost it about 12 minutes after its first Kilo request. The quota count still treats it as having room, so later requests keep trying it first.

Research first

  • When does Kilo answer again? That shows the real window. Try it once the run is over, so nothing competes with it.
  • Is 100 the limit, or is it a burst limit (100 in 12 minutes)?
  • Does a free Kilo key raise it? The refusal offers sign-up, and KILO_API_KEY is already reserved as an optional key. That key would be the owner's to make.

Fix
Decide from the research: correct the catalog's limit, and have a provider's 429 mark it spent in Braid's count, so the top bar and the chain order stop assuming it has room.

Done when
A sim shows a provider refusing below its catalog limit moves behind the others for its window, and the quota says so.

**What happens** On the Scratch run (#365), Kilo's `reflection/beam-free` served 100 requests in 12.2 minutes, then refused every one with a 429: "Sign up for free to continue and explore 500 other models. Takes 2 minutes, no credit card required. Or come back later." Our catalog, dated 2026-10-17, says 200 requests an hour per IP, and Kilo's [usage docs](https://kilo.ai/docs/gateway/usage-and-billing) still say that. They also give a different over-limit message: "Rate limit exceeded for free models. Please try again later." So Kilo's behaviour no longer matches its docs, which may mean a policy change. Nothing confirms that yet. This run was the only Free traffic in the event log, and my laptop replays were over an hour earlier. Whether they count depends on sharing an IP, but they fall outside an hourly window either way. **Why it matters** Kilo is the first model tried in the chain, and the run lost it about 12 minutes after its first Kilo request. The quota count still treats it as having room, so later requests keep trying it first. **Research first** - When does Kilo answer again? That shows the real window. Try it once the run is over, so nothing competes with it. - Is 100 the limit, or is it a burst limit (100 in 12 minutes)? - Does a free Kilo key raise it? The refusal offers sign-up, and `KILO_API_KEY` is already reserved as an optional key. That key would be the owner's to make. **Fix** Decide from the research: correct the catalog's limit, and have a provider's 429 mark it spent in Braid's count, so the top bar and the chain order stop assuming it has room. **Done when** A sim shows a provider refusing below its catalog limit moves behind the others for its window, and the quota says so.
Author
Owner

More evidence from run 170: this looks like 100 an hour, not a burst. Counted from the run's first Kilo request:

Minutes Kilo
0-12 100 served
12-60 None served; refused whenever tried
60-70 Served again, 17 requests

No test of Kilo is needed for this part, but one run is one sample. It doesn't show whether the window is fixed or rolling, and the docs still say 200.

Losing Kilo also cost speed in one step. In agent.write-spec-tests[game-rules], 70 requests got a bare 503 from Kilo's breaker with no failover to another model, each parked as free quota spent, where the build step's requests did fail over. Find out why this step's requests stopped at Kilo. The answer belongs in the fix.

**More evidence from run 170: this looks like 100 an hour, not a burst.** Counted from the run's first Kilo request: | Minutes | Kilo | |---|---| | 0-12 | 100 served | | 12-60 | None served; refused whenever tried | | 60-70 | Served again, 17 requests | No test of Kilo is needed for this part, but one run is one sample. It doesn't show whether the window is fixed or rolling, and the docs still say 200. **Losing Kilo also cost speed in one step.** In `agent.write-spec-tests[game-rules]`, 70 requests got a bare 503 from Kilo's breaker with no failover to another model, each parked as `free quota spent`, where the build step's requests did fail over. Find out why this step's requests stopped at Kilo. The answer belongs in the fix.
Author
Owner

Correction to my last comment: the slow step almost certainly wasn't a failover bug. The step served about 2 requests a minute, exactly OVH's limit, and by then every other strong model was out: Kilo refused, Cloudflare spent, Zen blocked at minute 45, and Gemini Flash-Lite held for vision and judge steps by design. Those 70 requests had nowhere else to go. Not proven request by request, but the numbers line up.

So this ticket's fix is smaller: correct the limit, count a 429 as spent, and fix the label. The capacity itself is #369.

**Correction to my last comment: the slow step almost certainly wasn't a failover bug.** The step served about 2 requests a minute, exactly OVH's limit, and by then every other strong model was out: Kilo refused, Cloudflare spent, Zen blocked at minute 45, and Gemini Flash-Lite held for vision and judge steps by design. Those 70 requests had nowhere else to go. Not proven request by request, but the numbers line up. So this ticket's fix is smaller: correct the limit, count a 429 as spent, and fix the label. The capacity itself is #369.
Author
Owner

Measured with a Kilo account's key (2026-11-03). 299 tiny requests to beam-free were served in 3.1 minutes, and request 300 got a 429: "Free model usage limit reached. Please try again later, or switch to a paid model or your own API key." There's no Retry-After. Five requests with no key straight after were all served, so the two counts are separate, and Kilo's docs saying the limit is per IP for both isn't what happens.

Served before refusing Words
No key (run 170) 100 "Sign up for free to continue…"
With a key 299 "Free model usage limit reached…"

When the key's count comes back isn't known yet; a check is running. The plan is in the OpenSpec change free-refusals-count: read both refusals as quota, spend the model for 10 minutes and try again, and send the key when it's set.

**Measured with a Kilo account's key (2026-11-03).** 299 tiny requests to `beam-free` were served in 3.1 minutes, and request 300 got a 429: "Free model usage limit reached. Please try again later, or switch to a paid model or your own API key." There's no `Retry-After`. Five requests with no key straight after were all served, so the two counts are separate, and Kilo's docs saying the limit is per IP for both isn't what happens. | | Served before refusing | Words | |---|---|---| | No key (run 170) | 100 | "Sign up for free to continue…" | | With a key | 299 | "Free model usage limit reached…" | When the key's count comes back isn't known yet; a check is running. The plan is in the OpenSpec change `free-refusals-count`: read both refusals as quota, spend the model for 10 minutes and try again, and send the key when it's set.
Author
Owner

Shipped in PR 371.

  • Kilo's refusals read as quota. Each one spends it for 10 minutes and teaches no limit, because its count isn't one Braid can keep.
  • KILO_API_KEY is sent when it's set, and is now set on production.
  • A wait no quota explains now says "free models aren't answering", not that quota is spent.

The key's limit, measured:

Served before refusing Came back
No key (run 170) 100 About 60 min after the first request
With a key 299 About 50 min after refusing: refused at 17:43 UTC, still refused at 18:28, served at 18:33

Evidence:

  • sim/free-kilo-refuses-early failed without the fix and passes with it.
  • Through the real router with the key: 110 of 110 served in 73 s, past the 100 a caller without a key gets.
  • Run 170's spec-test step, replayed with Kilo leading: Kilo served all 22 requests.
**Shipped in [PR 371](https://forgejo.underthere.xyz/cmoriarty/braid/pulls/371).** - Kilo's refusals read as quota. Each one spends it for 10 minutes and teaches no limit, because its count isn't one Braid can keep. - `KILO_API_KEY` is sent when it's set, and is now set on production. - A wait no quota explains now says "free models aren't answering", not that quota is spent. **The key's limit, measured:** | | Served before refusing | Came back | |---|---|---| | No key (run 170) | 100 | About 60 min after the first request | | With a key | 299 | About 50 min after refusing: refused at 17:43 UTC, still refused at 18:28, served at 18:33 | **Evidence:** - `sim/free-kilo-refuses-early` failed without the fix and passes with it. - Through the real router with the key: 110 of 110 served in 73 s, past the 100 a caller without a key gets. - Run 170's spec-test step, replayed with Kilo leading: Kilo served all 22 requests.
Sign in to join this conversation.
No labels
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
cmoriarty/braid#367
No description provided.