osfd serves no Prometheus metrics #10

Closed
opened 2026-09-13 16:14:16 -04:00 by cmoriarty · 1 comment
Owner

Reported from the running system. The operator watches a vLLM Grafana dashboard. vLLM only sees
requests in flight: it cannot see a step running tests rather than calling the model, or a run waiting
on a human for hours. Those numbers live only in osfd, and GET /metrics today falls through to the
single-page-app route and answers text/html, so a Grafana scrape silently parses a web page.

What it should do

Serve the Prometheus text exposition format (text/plain; version=0.0.4) at GET /metrics, with
gauges read off the existing projections:

  • osf_steps{status=…,kind=…} — step counts, so "running but doing local work" is visible
  • osf_runs{status=…} — run counts
  • osf_gates_open — gates waiting on a human
  • osf_step_attempts_max — the highest attempt count of any live step, so a retry storm is a number

The scrape reads the WAL snapshot and never queues behind the writer (it is scraped every 15 s,
forever). A handler reads projections and never writes.

Acceptance

tests/unit/test_metrics.py (4 tests, currently failing, including a latency test) is the
specification; do not weaken it.


Dogfood #1's brief, filed as an issue. Source: docs/issues/06-osfd-has-no-prometheus-metrics.md.

**Reported from the running system.** The operator watches a vLLM Grafana dashboard. vLLM only sees requests in flight: it cannot see a step running tests rather than calling the model, or a run waiting on a human for hours. Those numbers live only in osfd, and `GET /metrics` today falls through to the single-page-app route and answers `text/html`, so a Grafana scrape silently parses a web page. ## What it should do Serve the Prometheus text exposition format (`text/plain; version=0.0.4`) at `GET /metrics`, with gauges read off the existing projections: - `osf_steps{status=…,kind=…}` — step counts, so "running but doing local work" is visible - `osf_runs{status=…}` — run counts - `osf_gates_open` — gates waiting on a human - `osf_step_attempts_max` — the highest attempt count of any live step, so a retry storm is a number The scrape reads the WAL snapshot and never queues behind the writer (it is scraped every 15 s, forever). A handler reads projections and never writes. ## Acceptance `tests/unit/test_metrics.py` (4 tests, currently failing, including a latency test) is the specification; do not weaken it. --- *Dogfood #1's brief, filed as an issue. Source: `docs/issues/06-osfd-has-no-prometheus-metrics.md`.*
Author
Owner

Shipped in #301 (5355515, deployed), archived as metrics-and-tidy-ups in #302; the capability is prometheus-metrics.

GET /metrics now adds osfd's own counts to the throughput it already served, read on a read connection so a scrape never waits on the writer. On production right after the deploy: content type text/plain; version=0.0.4, and gauges such as osf_steps{status="succeeded",kind="agent"} 628, osf_runs{status="succeeded"} 37, osf_gates_open 0, osf_step_attempts_max 0. A test seeds a run waiting at a gate and a step on its third attempt, and reads both back.

Shipped in #301 (5355515, deployed), archived as `metrics-and-tidy-ups` in #302; the capability is `prometheus-metrics`. `GET /metrics` now adds osfd's own counts to the throughput it already served, read on a read connection so a scrape never waits on the writer. On production right after the deploy: content type `text/plain; version=0.0.4`, and gauges such as `osf_steps{status="succeeded",kind="agent"} 628`, `osf_runs{status="succeeded"} 37`, `osf_gates_open 0`, `osf_step_attempts_max 0`. A test seeds a run waiting at a gate and a step on its third attempt, and reads both back.
Sign in to join this conversation.
No labels
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
cmoriarty/braid#10
No description provided.