Three of nine groups reached the context alarm: compare the estimate with what filled them #533

Closed
opened 2026-11-13 15:07:32 -05:00 by cmoriarty · 2 comments
Owner

What happens

In run 195 three of nine apply groups compacted at 200k to 213k tokens, each while looking at its pages; in run 194, two of ten (braid#505, finding 6). Each group was estimated before it started and marked as fitting: run 193's group 3 was estimated at 58k to 81k and peaked at 133k. The estimate has no term for the looking phase.

Fix (owner, 2026-11-12: "I originally envisioned that our task estimation would improve over time because we would analyze our results")

  • Break down what filled the contexts of runs 194 and 195 by kind: brief, files read, test output, page inspections, reasoning. Do it for the groups that compacted and for those that did not.
  • Compare with each group's estimate.
  • Check whether a step that compacted is recorded in the calibration at its true size or at the size where it was cut off.
  • Propose a change to the estimate, with the numbers, before changing it.

Done when

The numbers and a proposal are on this issue, and the owner has chosen.

The other session's group-graph functions are in src/osf/engine/budget.py since braid#190; this work is in the estimate's terms, in the same file.

**What happens** In run 195 three of nine apply groups compacted at 200k to 213k tokens, each while looking at its pages; in run 194, two of ten (braid#505, finding 6). Each group was estimated before it started and marked as fitting: run 193's group 3 was estimated at 58k to 81k and peaked at 133k. The estimate has no term for the looking phase. **Fix** (owner, 2026-11-12: "I originally envisioned that our task estimation would improve over time because we would analyze our results") - Break down what filled the contexts of runs 194 and 195 by kind: brief, files read, test output, page inspections, reasoning. Do it for the groups that compacted and for those that did not. - Compare with each group's estimate. - Check whether a step that compacted is recorded in the calibration at its true size or at the size where it was cut off. - Propose a change to the estimate, with the numbers, before changing it. **Done when** The numbers and a proposal are on this issue, and the owner has chosen. The other session's group-graph functions are in `src/osf/engine/budget.py` since braid#190; this work is in the estimate's terms, in the same file.
Author
Owner

Numbers, from run 202 (runs 194 and 195 are deleted from production, so run 202's nine apply groups stand in; three compacted, groups 4, 6 and 8).

The estimate does not see the work. Every group was estimated at 41k to 45k tokens before calibration, 59k to 65k after (bias 1.43 from 17 ledger rows). The peaks were 79k to 202k. All 51 files the groups named did not exist yet, so the files term was 0, and a task is charged 1,200 tokens. Measured from the first turn to the peak, an uncompacted group grew 16k to 45k tokens per task (median 38k).

What filled the contexts (characters before each group's peak, first attempts):

Kind Share
Reasoning carried from earlier turns 56%
Files read 20%
Edits (code written) 13%
Other commands 6%
Test runs 3%
Page inspections, helpers, own words 4%

Compactions are recorded where they were cut off. 11 of the ledger's 35 apply rows are compactions, at 200,322 to 208,652 tokens. calibrate keeps them as measured, so the p80 (3.01 × the estimate) is a floor, and the applied multiplier (1.48) is too low.

Proposal:

  1. Charge a task what a task costs: fit the per-task term from the ledger's uncompacted rows, (peak − floor − docs) / tasks, about 38k today, instead of the constant 1,200. A 4-task group would then estimate near 170k, and a 6-task group would be split before it starts. (Recommended.)
  2. Count a compaction as "at least" its peak in calibrate, so the spread is not set by rows that were cut off.
  3. Separately, not the estimate: stop sending earlier turns' reasoning back to the model. It is over half of every context here. That needs a replay to show the work does not suffer first.

1 and 2 together are one small change in estimator.py. Which do you want?

**Numbers, from run 202** (runs 194 and 195 are deleted from production, so run 202's nine apply groups stand in; three compacted, groups 4, 6 and 8). **The estimate does not see the work.** Every group was estimated at 41k to 45k tokens before calibration, 59k to 65k after (bias 1.43 from 17 ledger rows). The peaks were 79k to 202k. All 51 files the groups named did not exist yet, so the files term was 0, and a task is charged 1,200 tokens. Measured from the first turn to the peak, an uncompacted group grew **16k to 45k tokens per task** (median 38k). **What filled the contexts** (characters before each group's peak, first attempts): | Kind | Share | |---|---| | Reasoning carried from earlier turns | 56% | | Files read | 20% | | Edits (code written) | 13% | | Other commands | 6% | | Test runs | 3% | | Page inspections, helpers, own words | 4% | **Compactions are recorded where they were cut off.** 11 of the ledger's 35 apply rows are compactions, at 200,322 to 208,652 tokens. `calibrate` keeps them as measured, so the p80 (3.01 × the estimate) is a floor, and the applied multiplier (1.48) is too low. **Proposal:** 1. **Charge a task what a task costs:** fit the per-task term from the ledger's uncompacted rows, `(peak − floor − docs) / tasks`, about 38k today, instead of the constant 1,200. A 4-task group would then estimate near 170k, and a 6-task group would be split before it starts. *(Recommended.)* 2. **Count a compaction as "at least" its peak** in `calibrate`, so the spread is not set by rows that were cut off. 3. **Separately, not the estimate:** stop sending earlier turns' reasoning back to the model. It is over half of every context here. That needs a replay to show the work does not suffer first. 1 and 2 together are one small change in `estimator.py`. Which do you want?
Author
Owner

Shipped in #577 (production 2a45c11, 11-20), archived in #580. The owner chose the estimate alone, learned per model.

  • A task is charged the median growth per task that the ledger's apply attempts show beyond the estimate's other terms, shrunk toward 1,200 as the centre is. The centre and spread are measured against each attempt's estimate re-made with that charge.
  • Each estimate's record names the group's tasks and the step's model. A model with 5 or more records is fitted from its own, starting from the other models' fit. Older records take their model from their run's settings.
  • Nothing acts on the verdict yet: it shows at the proposal gate's Size check and on the pull request's estimate line.

Evidence

  • tests/engine/test_estimate_per_task.py (5 tests) fails on main and passes with the fix. On production's 35 records, 8 of the 11 compacted attempts now read "split"; main's estimate called 10 of them fitting. Suite green, smoke sims 7 of 7, and a kept sim's apply attempt recorded its tasks and model.
  • Production, read-only, with the deployed code: all 35 records count under vLLM qwen3.8-27b; 18,293 a task (23,177 measured), centre 1.13 (was 1.71), spread 1.53. Up to 4 tasks fit, 5 is flagged, 6 or more read "split".
  • No step replay: the change is in the estimate made before the apply step, from production's ledger, which a local replay does not have.

Not done: counting a compaction as "at least" its peak, and acting on the verdict (a planner limit or a revise round). The next standard run checks the flags out of sample.

**Shipped** in #577 (production 2a45c11, 11-20), archived in #580. The owner chose the estimate alone, learned per model. - A task is charged the median growth per task that the ledger's apply attempts show beyond the estimate's other terms, shrunk toward 1,200 as the centre is. The centre and spread are measured against each attempt's estimate re-made with that charge. - Each estimate's record names the group's tasks and the step's model. A model with 5 or more records is fitted from its own, starting from the other models' fit. Older records take their model from their run's settings. - Nothing acts on the verdict yet: it shows at the proposal gate's Size check and on the pull request's estimate line. **Evidence** - `tests/engine/test_estimate_per_task.py` (5 tests) fails on main and passes with the fix. On production's 35 records, 8 of the 11 compacted attempts now read "split"; main's estimate called 10 of them fitting. Suite green, smoke sims 7 of 7, and a kept sim's apply attempt recorded its tasks and model. - Production, read-only, with the deployed code: all 35 records count under vLLM qwen3.8-27b; 18,293 a task (23,177 measured), centre 1.13 (was 1.71), spread 1.53. Up to 4 tasks fit, 5 is flagged, 6 or more read "split". - No step replay: the change is in the estimate made before the apply step, from production's ledger, which a local replay does not have. **Not done:** counting a compaction as "at least" its peak, and acting on the verdict (a planner limit or a revise round). The next standard run checks the flags out of sample.
Sign in to join this conversation.
No labels
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
cmoriarty/braid#533
No description provided.