Reasoning effort is Braid's high/medium/low, not the model's: list the backend's own levels and save one per profile #349

Closed
opened 2026-10-30 19:49:42 -04:00 by cmoriarty · 1 comment
Owner

What happens

  • The effort menu offers Braid's own high, medium and low for every backend. "High" sends nothing, so it means whatever the server defaults to, and the menu can't show what that is.
  • Only the vLLM, SGLang and llama.cpp kinds send an effort, and only medium or low. fastllm and tabbyAPI never get one: run 160's profile said high, yet every request ran at the template's medium.
  • Levels differ by model. Qwen3.8 takes xhigh, medium and low; GLM-5.3 takes none, low, high and max.

What we want

  • Settings lists the levels the profile's model accepts, plus "Server default" (send nothing), and saves the choice with the profile.
  • Braid sends the saved level, in the model's own words, on every kind. All four self-hosted kinds accept chat_template_kwargs.
  • The levels come from a per-model list. Where the backend renders prompts (vLLM /tokenize, llama.cpp and tabbyAPI /apply-template), Connect checks each level and drops any the template rejects or ignores.
  • Auto keeps its step map, with each tier mapped to one of the profile's levels.

Done when

  • Each self-hosted profile offers its model's levels, and the server's rendered prompt shows the saved one.
  • "Server default" sends nothing.

Where to look: src/osf/settings.py (REASONING_EFFORTS), src/osf/agent/server.py (CHAT_TEMPLATE_KINDS, SENT_REASONING_EFFORTS, reasoning_options()), src/osf/models.py, the settings UI.

Found during the 2026-10-30 engine refresh. It also skews #342: on vLLM the high-tier steps thought at xhigh, while run 160 on fastllm thought at medium throughout.

**What happens** - The effort menu offers Braid's own high, medium and low for every backend. "High" sends nothing, so it means whatever the server defaults to, and the menu can't show what that is. - Only the vLLM, SGLang and llama.cpp kinds send an effort, and only medium or low. fastllm and tabbyAPI never get one: run 160's profile said high, yet every request ran at the template's medium. - Levels differ by model. Qwen3.8 takes xhigh, medium and low; GLM-5.3 takes none, low, high and max. **What we want** - Settings lists the levels the profile's model accepts, plus "Server default" (send nothing), and saves the choice with the profile. - Braid sends the saved level, in the model's own words, on every kind. All four self-hosted kinds accept `chat_template_kwargs`. - The levels come from a per-model list. Where the backend renders prompts (vLLM `/tokenize`, llama.cpp and tabbyAPI `/apply-template`), Connect checks each level and drops any the template rejects or ignores. - Auto keeps its step map, with each tier mapped to one of the profile's levels. **Done when** - Each self-hosted profile offers its model's levels, and the server's rendered prompt shows the saved one. - "Server default" sends nothing. **Where to look:** `src/osf/settings.py` (`REASONING_EFFORTS`), `src/osf/agent/server.py` (`CHAT_TEMPLATE_KINDS`, `SENT_REASONING_EFFORTS`, `reasoning_options()`), `src/osf/models.py`, the settings UI. *Found during the 2026-10-30 engine refresh. It also skews #342: on vLLM the high-tier steps thought at xhigh, while run 160 on fastllm thought at medium throughout.*
Author
Owner

Done in batch B, merged as #355 (91f4b85f), and live on production: Braid reports commit c00c62f, which includes it. It was checked with the repo's tests and sims before the merge. A live check on vLLM is still to come: trogdor is off the network. Closed at the owner's request.

Done in batch B, merged as #355 (`91f4b85f`), and live on production: Braid reports commit `c00c62f`, which includes it. It was checked with the repo's tests and sims before the merge. A live check on vLLM is still to come: trogdor is off the network. Closed at the owner's request.
Sign in to join this conversation.
No labels
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
cmoriarty/braid#349
No description provided.