cmoriarty/braid
1
0
Fork 0
Code Issues 15 Pull requests Projects Releases Packages 1 Wiki Activity Actions

osfd serves no Prometheus metrics #10

New issue
Open
opened 2026-09-13 16:14:16 -04:00 by cmoriarty · 0 comments
cmoriarty commented 2026-09-13 16:14:16 -04:00
Owner
Copy link

Reported from the running system. The operator watches a vLLM Grafana dashboard. vLLM only sees
requests in flight: it cannot see a step running tests rather than calling the model, or a run waiting
on a human for hours. Those numbers live only in osfd, and GET /metrics today falls through to the
single-page-app route and answers text/html, so a Grafana scrape silently parses a web page.

What it should do

Serve the Prometheus text exposition format (text/plain; version=0.0.4) at GET /metrics, with
gauges read off the existing projections:

  • osf_steps{status=…,kind=…} — step counts, so "running but doing local work" is visible
  • osf_runs{status=…} — run counts
  • osf_gates_open — gates waiting on a human
  • osf_step_attempts_max — the highest attempt count of any live step, so a retry storm is a number

The scrape reads the WAL snapshot and never queues behind the writer (it is scraped every 15 s,
forever). A handler reads projections and never writes.

Acceptance

tests/unit/test_metrics.py (4 tests, currently failing, including a latency test) is the
specification; do not weaken it.

Dogfood #1's brief, filed as an issue. Source: docs/issues/06-osfd-has-no-prometheus-metrics.md.

**Reported from the running system.** The operator watches a vLLM Grafana dashboard. vLLM only sees requests in flight: it cannot see a step running tests rather than calling the model, or a run waiting on a human for hours. Those numbers live only in osfd, and `GET /metrics` today falls through to the single-page-app route and answers `text/html`, so a Grafana scrape silently parses a web page. ## What it should do Serve the Prometheus text exposition format (`text/plain; version=0.0.4`) at `GET /metrics`, with gauges read off the existing projections: - `osf_steps{status=…,kind=…}` — step counts, so "running but doing local work" is visible - `osf_runs{status=…}` — run counts - `osf_gates_open` — gates waiting on a human - `osf_step_attempts_max` — the highest attempt count of any live step, so a retry storm is a number The scrape reads the WAL snapshot and never queues behind the writer (it is scraped every 15 s, forever). A handler reads projections and never writes. ## Acceptance `tests/unit/test_metrics.py` (4 tests, currently failing, including a latency test) is the specification; do not weaken it. --- *Dogfood #1's brief, filed as an issue. Source: `docs/issues/06-osfd-has-no-prometheus-metrics.md`.*
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
archive/prefect-kestra-experiments
Labels
Clear labels
No items
No labels
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
cmoriarty/braid#10
No description provided.