Three of nine groups reached the context alarm: compare the estimate with what filled them #533
Loading…
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
What happens
In run 195 three of nine apply groups compacted at 200k to 213k tokens, each while looking at its pages; in run 194, two of ten (braid#505, finding 6). Each group was estimated before it started and marked as fitting: run 193's group 3 was estimated at 58k to 81k and peaked at 133k. The estimate has no term for the looking phase.
Fix (owner, 2026-11-12: "I originally envisioned that our task estimation would improve over time because we would analyze our results")
Done when
The numbers and a proposal are on this issue, and the owner has chosen.
The other session's group-graph functions are in
src/osf/engine/budget.pysince braid#190; this work is in the estimate's terms, in the same file.Numbers, from run 202 (runs 194 and 195 are deleted from production, so run 202's nine apply groups stand in; three compacted, groups 4, 6 and 8).
The estimate does not see the work. Every group was estimated at 41k to 45k tokens before calibration, 59k to 65k after (bias 1.43 from 17 ledger rows). The peaks were 79k to 202k. All 51 files the groups named did not exist yet, so the files term was 0, and a task is charged 1,200 tokens. Measured from the first turn to the peak, an uncompacted group grew 16k to 45k tokens per task (median 38k).
What filled the contexts (characters before each group's peak, first attempts):
Compactions are recorded where they were cut off. 11 of the ledger's 35 apply rows are compactions, at 200,322 to 208,652 tokens.
calibratekeeps them as measured, so the p80 (3.01 × the estimate) is a floor, and the applied multiplier (1.48) is too low.Proposal:
(peak − floor − docs) / tasks, about 38k today, instead of the constant 1,200. A 4-task group would then estimate near 170k, and a 6-task group would be split before it starts. (Recommended.)calibrate, so the spread is not set by rows that were cut off.1 and 2 together are one small change in
estimator.py. Which do you want?Shipped in #577 (production
2a45c11, 11-20), archived in #580. The owner chose the estimate alone, learned per model.Evidence
tests/engine/test_estimate_per_task.py(5 tests) fails on main and passes with the fix. On production's 35 records, 8 of the 11 compacted attempts now read "split"; main's estimate called 10 of them fitting. Suite green, smoke sims 7 of 7, and a kept sim's apply attempt recorded its tasks and model.Not done: counting a compaction as "at least" its peak, and acting on the verdict (a planner limit or a revise round). The next standard run checks the flags out of sample.