A task is charged what the record says a task costs, model by model (#533) #577

Merged
cmoriarty merged 4 commits from claude/estimate-533 into main 2026-11-20 18:26:06 -05:00
Owner

Closes the estimate half of braid#533: a task is charged what the record says a task costs, model by model. The owner chose the estimate alone on 2026-11-20; nothing acts on the verdict yet (the split step is in no pipeline and no brief carries the estimate), so a run plans and runs as before. What changes is what budget.json, the proposal gate's Size check and the pull request's estimate line say.

Why. The estimate charged every task 1,200 tokens. The ledger's 35 apply attempts grew 23,177 tokens a task beyond the estimate's other terms (median), so every group read as fitting one step, the 11 that were compacted included.

What changes (openspec/changes/estimate-per-task-533):

  • The calibration fits the per-task term from the record: the median of (peak - other terms) / tasks, shrunk toward 1,200 like the centre, held to 1,200 to 60,000.
  • The centre and spread are measured against each attempt's estimate re-made from its own terms with that term, not against the p50 of the day, which already carried the centre of its day.
  • The estimate's record (osf.budget.estimated) names the group's n_tasks and the step's model (vLLM qwen3.8-27b, or Free <tier>). Records from before give their task count as their tasks term over 1,200, and their model from their run's settings copy.
  • A run on a model with five or more records of its own is fitted from them, starting from what the other models' records give; with fewer, from every model's records. budget.json names the term and the model.
  • No schema change: the ledger rows read their own estimate record for the terms.

Production's ledger, run through the new code read-only (all 35 records on vLLM qwen3.8-27b):

Per task Centre Spread
Before 1,200 (fixed) 1.71 1.65
After 18,293 (23,177 measured) 1.13 1.53

With other terms of 37,000, a group of up to 4 tasks fits, 5 is flagged and 6 or more reads "split". Estimated again, 8 of the 11 compacted attempts read "split", and 2 of the 24 that were not. This figure is in-sample: the next standard run is the check that is not.

Tests (tests/engine/test_estimate_per_task.py, all small):

  • production's 35 records give the figures above and flag 8 of 11 compactions;
  • two models: a run on B is fitted from B's records starting from A's; a new model gets the fit over every model;
  • the ledger reads each record's tasks, terms and model, including records from before;
  • osfd's fit reaches budget.json through OSF_CALIBRATION_JSON; four attempts keep the bootstrap;
  • an apply attempt records 3 tasks and vLLM qwen3.8-27b, and an older record of its run counts under that model.

On main they fail (the code is not there). ./tools/test.sh passed in 54 s (backend 924, UI 212, browser 64); ./tools/sim.sh --smoke 7 of 7; a kept thorough-request-changes sim recorded n_tasks and the model on its apply attempt, and its budget.json named the per-task term.

Merging deploys osfd.

🤖 Generated with Claude Code

Closes the estimate half of braid#533: a task is charged what the record says a task costs, model by model. The owner chose the estimate alone on 2026-11-20; nothing acts on the verdict yet (the split step is in no pipeline and no brief carries the estimate), so a run plans and runs as before. What changes is what `budget.json`, the proposal gate's Size check and the pull request's estimate line say. **Why.** The estimate charged every task 1,200 tokens. The ledger's 35 apply attempts grew 23,177 tokens a task beyond the estimate's other terms (median), so every group read as fitting one step, the 11 that were compacted included. **What changes** (`openspec/changes/estimate-per-task-533`): - The calibration fits the per-task term from the record: the median of `(peak - other terms) / tasks`, shrunk toward 1,200 like the centre, held to 1,200 to 60,000. - The centre and spread are measured against each attempt's estimate re-made from its own terms with that term, not against the p50 of the day, which already carried the centre of its day. - The estimate's record (`osf.budget.estimated`) names the group's `n_tasks` and the step's `model` (`vLLM qwen3.8-27b`, or `Free <tier>`). Records from before give their task count as their tasks term over 1,200, and their model from their run's settings copy. - A run on a model with five or more records of its own is fitted from them, starting from what the other models' records give; with fewer, from every model's records. `budget.json` names the term and the model. - No schema change: the ledger rows read their own estimate record for the terms. **Production's ledger, run through the new code read-only** (all 35 records on vLLM qwen3.8-27b): | | Per task | Centre | Spread | |---|---|---|---| | Before | 1,200 (fixed) | 1.71 | 1.65 | | After | 18,293 (23,177 measured) | 1.13 | 1.53 | With other terms of 37,000, a group of up to 4 tasks fits, 5 is flagged and 6 or more reads "split". Estimated again, 8 of the 11 compacted attempts read "split", and 2 of the 24 that were not. This figure is in-sample: the next standard run is the check that is not. **Tests** (`tests/engine/test_estimate_per_task.py`, all small): - production's 35 records give the figures above and flag 8 of 11 compactions; - two models: a run on B is fitted from B's records starting from A's; a new model gets the fit over every model; - the ledger reads each record's tasks, terms and model, including records from before; - osfd's fit reaches `budget.json` through `OSF_CALIBRATION_JSON`; four attempts keep the bootstrap; - an apply attempt records 3 tasks and `vLLM qwen3.8-27b`, and an older record of its run counts under that model. On main they fail (the code is not there). `./tools/test.sh` passed in 54 s (backend 924, UI 212, browser 64); `./tools/sim.sh --smoke` 7 of 7; a kept `thorough-request-changes` sim recorded `n_tasks` and the model on its apply attempt, and its `budget.json` named the per-task term. Merging deploys osfd. 🤖 Generated with [Claude Code](https://claude.com/claude-code)
The owner chose the estimate alone on 2026-11-20: fit the per-task term from the budget ledger
instead of the constant 1,200, measure the centre and spread against each attempt's estimate
re-made from its terms, and name the group's task count in the estimate's record. No schema,
brief or gate change.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The owner asked on 2026-11-20 for each model to learn its own: the record names the step's model,
and a run's calibration moves from the fit over every model toward its own model's. Records from
before count under the model their run's settings name.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
feat(estimator): a task is charged what the record says a task costs, model by model (#533)
All checks were successful
test / suite (pull_request) Successful in 1m29s
158439cbce
The estimate charged every task 1,200 tokens while the ledger's 35 apply attempts grew about
23,000 a task beyond the other terms, so every group read as fitting one step, the 11 that were
compacted included. The calibration now fits the per-task term from the record (the median growth
per task, shrunk toward 1,200), and measures the centre and spread against each attempt's estimate
re-made from its own terms rather than against the p50 of the day.

What a task costs is the model's own doing, so the estimate's record names the step's model and
the group's task count, and a run on a model with five or more records of its own is fitted from
them, starting from what the other models' records give. Records from before count under the
model their run's settings name. No schema, brief or gate change.

On production's ledger of 2026-11-20 (all 35 records on vLLM qwen3.8-27b): 18,293 tokens a task,
centre 1.13, spread 1.53; 8 of the 11 compacted groups read "split", 2 of the other 24 do.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Merge branch 'main' into claude/estimate-533
All checks were successful
test / suite (pull_request) Successful in 1m30s
58b83238ea
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign in to join this conversation.
No reviewers
No labels
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
cmoriarty/braid!577
No description provided.