Kilo refuses after 100 requests where its docs and our catalog say 200 an hour #367
Loading…
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
What happens
On the Scratch run (#365), Kilo's
reflection/beam-freeserved 100 requests in 12.2 minutes, then refused every one with a 429: "Sign up for free to continue and explore 500 other models. Takes 2 minutes, no credit card required. Or come back later."Our catalog, dated 2026-10-17, says 200 requests an hour per IP, and Kilo's usage docs still say that. They also give a different over-limit message: "Rate limit exceeded for free models. Please try again later." So Kilo's behaviour no longer matches its docs, which may mean a policy change. Nothing confirms that yet.
This run was the only Free traffic in the event log, and my laptop replays were over an hour earlier. Whether they count depends on sharing an IP, but they fall outside an hourly window either way.
Why it matters
Kilo is the first model tried in the chain, and the run lost it about 12 minutes after its first Kilo request. The quota count still treats it as having room, so later requests keep trying it first.
Research first
KILO_API_KEYis already reserved as an optional key. That key would be the owner's to make.Fix
Decide from the research: correct the catalog's limit, and have a provider's 429 mark it spent in Braid's count, so the top bar and the chain order stop assuming it has room.
Done when
A sim shows a provider refusing below its catalog limit moves behind the others for its window, and the quota says so.
More evidence from run 170: this looks like 100 an hour, not a burst. Counted from the run's first Kilo request:
No test of Kilo is needed for this part, but one run is one sample. It doesn't show whether the window is fixed or rolling, and the docs still say 200.
Losing Kilo also cost speed in one step. In
agent.write-spec-tests[game-rules], 70 requests got a bare 503 from Kilo's breaker with no failover to another model, each parked asfree quota spent, where the build step's requests did fail over. Find out why this step's requests stopped at Kilo. The answer belongs in the fix.Correction to my last comment: the slow step almost certainly wasn't a failover bug. The step served about 2 requests a minute, exactly OVH's limit, and by then every other strong model was out: Kilo refused, Cloudflare spent, Zen blocked at minute 45, and Gemini Flash-Lite held for vision and judge steps by design. Those 70 requests had nowhere else to go. Not proven request by request, but the numbers line up.
So this ticket's fix is smaller: correct the limit, count a 429 as spent, and fix the label. The capacity itself is #369.
Measured with a Kilo account's key (2026-11-03). 299 tiny requests to
beam-freewere served in 3.1 minutes, and request 300 got a 429: "Free model usage limit reached. Please try again later, or switch to a paid model or your own API key." There's noRetry-After. Five requests with no key straight after were all served, so the two counts are separate, and Kilo's docs saying the limit is per IP for both isn't what happens.When the key's count comes back isn't known yet; a check is running. The plan is in the OpenSpec change
free-refusals-count: read both refusals as quota, spend the model for 10 minutes and try again, and send the key when it's set.Shipped in PR 371.
KILO_API_KEYis sent when it's set, and is now set on production.The key's limit, measured:
Evidence:
sim/free-kilo-refuses-earlyfailed without the fix and passes with it.