A test suite that needs a service runs in the project's CI, and its failures come back to the fix loop #163

Closed
opened 2026-10-09 02:37:44 -04:00 by cmoriarty · 5 comments
Owner

A soundcheck run on Claude Sonnet 5.5 (2026-10-09) could not have finished: soundcheck's e2e suite needs Postgres, which its CI provides as a service (postgres:17-alpine, reached through E2E_PG_HOST/E2E_PG_PORT), and Braid's run environment has none. The apply agent wrote the e2e test and declared it as a lane (.osf/test/e2e.command); the fix loop would have run it, watched it fail at the web server's first query, and tried to "fix" code that was not broken.

Baking each service into Braid (Postgres today, MariaDB, Redis, a queue tomorrow) duplicates what the project already maintains and keeps working: its CI. So the split is: Braid verifies what needs no service; the project's CI verifies the rest, and Braid listens to it.

Already shipped for the first half (3a78aff): task lists no longer hold tasks an apply agent cannot finish (run the whole suite, run e2e, preview, check on a device), and an agent declares no local lane for a suite that needs a service its CI provides.

What it needs

  • Run the service-dependent suite where it can run. When the run's change has tests a local lane cannot run, push the branch so the project's CI runs them. soundcheck's CI runs its full job (Postgres, Playwright e2e) only on a ready, non-draft pull request and on main; fast runs on every push. So Braid opens its pull request as ready (or marks it ready) before the fix loop's last check, rather than only at the end.
  • Read CI's result as a lane. Wait for the commit's statuses on Forgejo (Braid already polls them, and waits on required checks at merge), and record each failed job as a failed lane: its name, and the failing step's log, bounded.
  • Feed it to the fix loop. A failed CI lane goes to the fix round as a failed local lane does: the job's log is the evidence. The fix pushes again; CI runs again.
  • Say it plainly. Run 38 failed at merge on required checks, and Braid reported a different reason. A CI failure names the job and step.
  • The browser check. soundcheck's app does not start without its database either, so the judge cannot open it locally. For such an app the judge reads CI's e2e evidence (its report and screenshots, when the job uploads them) instead of failing to serve the app.

Open questions for the explore step

  • How Braid tells which of the project's suites need a service it lacks: the CI job's services:, a lane that cannot start, or the project's own split (soundcheck's test:fast vs test:e2e).
  • Bounds: how long to wait for CI, and how many CI rounds a fix loop may spend.
  • A project with no CI: it gets local lanes only, and the run says its e2e went unverified.

Done when

A soundcheck run whose change adds an e2e test finishes clean: the fast lane passes locally, the e2e suite passes in CI and the run records it as a lane, and a CI failure planted in a test comes back to a fix round with the job's log and is fixed.

A soundcheck run on Claude Sonnet 5.5 (2026-10-09) could not have finished: soundcheck's e2e suite needs Postgres, which its CI provides as a service (`postgres:17-alpine`, reached through `E2E_PG_HOST`/`E2E_PG_PORT`), and Braid's run environment has none. The apply agent wrote the e2e test and declared it as a lane (`.osf/test/e2e.command`); the fix loop would have run it, watched it fail at the web server's first query, and tried to "fix" code that was not broken. Baking each service into Braid (Postgres today, MariaDB, Redis, a queue tomorrow) duplicates what the project already maintains and keeps working: its CI. So the split is: **Braid verifies what needs no service; the project's CI verifies the rest, and Braid listens to it.** Already shipped for the first half (3a78aff): task lists no longer hold tasks an apply agent cannot finish (run the whole suite, run e2e, preview, check on a device), and an agent declares no local lane for a suite that needs a service its CI provides. ## What it needs - **Run the service-dependent suite where it can run.** When the run's change has tests a local lane cannot run, push the branch so the project's CI runs them. soundcheck's CI runs its `full` job (Postgres, Playwright e2e) only on a ready, non-draft pull request and on main; `fast` runs on every push. So Braid opens its pull request as ready (or marks it ready) before the fix loop's last check, rather than only at the end. - **Read CI's result as a lane.** Wait for the commit's statuses on Forgejo (Braid already polls them, and waits on required checks at merge), and record each failed job as a failed lane: its name, and the failing step's log, bounded. - **Feed it to the fix loop.** A failed CI lane goes to the fix round as a failed local lane does: the job's log is the evidence. The fix pushes again; CI runs again. - **Say it plainly.** Run 38 failed at merge on required checks, and Braid reported a different reason. A CI failure names the job and step. - **The browser check.** soundcheck's app does not start without its database either, so the judge cannot open it locally. For such an app the judge reads CI's e2e evidence (its report and screenshots, when the job uploads them) instead of failing to serve the app. ## Open questions for the explore step - How Braid tells which of the project's suites need a service it lacks: the CI job's `services:`, a lane that cannot start, or the project's own split (soundcheck's `test:fast` vs `test:e2e`). - Bounds: how long to wait for CI, and how many CI rounds a fix loop may spend. - A project with no CI: it gets local lanes only, and the run says its e2e went unverified. ## Done when A soundcheck run whose change adds an e2e test finishes clean: the fast lane passes locally, the e2e suite passes in CI and the run records it as a lane, and a CI failure planted in a test comes back to a fix round with the job's log and is fixed.
Author
Owner

Phase 2: a services command per repository, for a local e2e and a judge that can see the app

Decided 2026-10-09, after weighing alternatives.

Not Docker. An agent pulling and running containers needs the host's Docker socket in the run environment, which is root on the VM for every agent: Braid's database, the pasted API keys and every other run's files with it. Sandboxed runtimes (rootless Podman, a Docker-in-Docker sidecar per run) are possible, but too fiddly and risky to hand to agents. Not act / forgejo-runner exec inside Braid either: it needs the same runtime, and it diverges from real CI at the edges. Phase 1's CI push already gets the same result on the runner host, which has Docker.

Instead, services as ordinary unprivileged processes. Most common services run without root from a downloaded binary inside the run's own directory:

  • Postgres: the embedded-postgres npm package, or prebuilt binaries (initdb, pg_ctl);
  • MariaDB/MySQL: the official Linux tarball (mysqld --no-defaults --datadir=…);
  • MongoDB: mongodb-memory-server;
  • Redis: redis-memory-server, or a static redis-server.

Set once per repository, not improvised per run. Braid already keeps per-repository test choices (unit, browser and serve commands). It gains one more: start the services. For soundcheck that would be an embedded-postgres start plus E2E_PG_HOST/E2E_PG_PORT. Braid runs it unprivileged, in the run's sandbox:

  • before the lanes and the judge in every run of that repository, since an agent's own processes are gone by the time Braid's lanes and judge run;
  • and stops it when the run ends.

An agent may propose the command once, guided by a short brief of the recipes above, and the operator approves it in the repository's settings.

Order. Phase 1 (CI as the environment for service-dependent suites) first: it covers every project and every service with no setup. Phase 2 for repositories where a fast local e2e and a judge that can open the running app are worth one command of setup; soundcheck is the first candidate.

Already shipped toward this issue:

  • 3a78aff: tasks an apply agent cannot finish are kept out of the task list, and no local lane is declared for a suite that needs a service;
  • 6fe3e8e: the judge may say unverifiable and name the missing service. The loop honours it only when the project's CI declares a service, the judge names it, and no criterion failed, so an app that needs CI's Postgres no longer sends a run into fix rounds that cannot help.
## Phase 2: a services command per repository, for a local e2e and a judge that can see the app Decided 2026-10-09, after weighing alternatives. **Not Docker.** An agent pulling and running containers needs the host's Docker socket in the run environment, which is root on the VM for every agent: Braid's database, the pasted API keys and every other run's files with it. Sandboxed runtimes (rootless Podman, a Docker-in-Docker sidecar per run) are possible, but too fiddly and risky to hand to agents. **Not act / `forgejo-runner exec`** inside Braid either: it needs the same runtime, and it diverges from real CI at the edges. Phase 1's CI push already gets the same result on the runner host, which has Docker. **Instead, services as ordinary unprivileged processes.** Most common services run without root from a downloaded binary inside the run's own directory: - Postgres: the `embedded-postgres` npm package, or prebuilt binaries (`initdb`, `pg_ctl`); - MariaDB/MySQL: the official Linux tarball (`mysqld --no-defaults --datadir=…`); - MongoDB: `mongodb-memory-server`; - Redis: `redis-memory-server`, or a static `redis-server`. **Set once per repository, not improvised per run.** Braid already keeps per-repository test choices (unit, browser and serve commands). It gains one more: *start the services*. For soundcheck that would be an `embedded-postgres` start plus `E2E_PG_HOST`/`E2E_PG_PORT`. Braid runs it unprivileged, in the run's sandbox: - before the lanes and the judge in every run of that repository, since an agent's own processes are gone by the time Braid's lanes and judge run; - and stops it when the run ends. An agent may propose the command once, guided by a short brief of the recipes above, and the operator approves it in the repository's settings. **Order.** Phase 1 (CI as the environment for service-dependent suites) first: it covers every project and every service with no setup. Phase 2 for repositories where a fast local e2e and a judge that can open the running app are worth one command of setup; soundcheck is the first candidate. **Already shipped toward this issue:** - 3a78aff: tasks an apply agent cannot finish are kept out of the task list, and no local lane is declared for a suite that needs a service; - 6fe3e8e: the judge may say `unverifiable` and name the missing service. The loop honours it only when the project's CI declares a service, the judge names it, and no criterion failed, so an app that needs CI's Postgres no longer sends a run into fix rounds that cannot help.
Author
Owner

Phase 1 progress (prod 9d4c23f)

Shipped:

  • Briefs (3a78aff): no task an apply agent can't finish (whole-suite, e2e, preview or by-hand checks). A suite that needs a service its CI provides gets its tests written but no local lane.
  • Judge (6fe3e8e): a new unverifiable verdict with a missing_service. It counts as no failure only when no criterion failed and the project's CI workflows declare services:; otherwise it's a failure. So an app that needs Postgres goes to CI rather than into fix rounds.
  • Merge (9d4c23f): when git.pr.merge fails on a CI check, the step now leads with Forgejo's refusal and the failing check. Before, it read "no non-empty file matched: .osf/merged.json".

Tested on a Sonnet 5.5 thorough run on soundcheck#71. The judge's second round returned unverifiable for the Postgres-backed e2e and the run went on to the PR. The PR's CI full then failed on something outside the change: npm audit --audit-level=high hit GHSA-68fv-2mgg-jv7q (source-map-js), published during the run. That fails every soundcheck branch until main takes the fix (a lockfile-only npm audit fix).

Not done yet: a CI failure doesn't come back to the run as a fix round. The run fails at merge with the reason.

## Phase 1 progress (prod 9d4c23f) Shipped: - **Briefs** (3a78aff): no task an apply agent can't finish (whole-suite, e2e, preview or by-hand checks). A suite that needs a service its CI provides gets its tests written but no local lane. - **Judge** (6fe3e8e): a new `unverifiable` verdict with a `missing_service`. It counts as no failure only when no criterion failed and the project's CI workflows declare `services:`; otherwise it's a failure. So an app that needs Postgres goes to CI rather than into fix rounds. - **Merge** (9d4c23f): when `git.pr.merge` fails on a CI check, the step now leads with Forgejo's refusal and the failing check. Before, it read "no non-empty file matched: .osf/merged.json". Tested on a Sonnet 5.5 thorough run on soundcheck#71. The judge's second round returned `unverifiable` for the Postgres-backed e2e and the run went on to the PR. The PR's CI `full` then failed on something outside the change: `npm audit --audit-level=high` hit GHSA-68fv-2mgg-jv7q (source-map-js), published during the run. That fails every soundcheck branch until main takes the fix (a lockfile-only `npm audit fix`). Not done yet: a CI failure doesn't come back to the run as a fix round. The run fails at merge with the reason.
Author
Owner

Phase 1 and Phase 2 shipped tonight, as project-ci-is-a-lane and services-for-the-lanes.

Phase 1: the project's CI is a lane. Once a round's local lanes, the test lint and the README check pass, and the repository has a workflow that runs on a branch push or a pull request, script.run-tests gives the round's code to CI:

  • It commits the worktree as a check commit (with a temporary index, so the run's own history does not move), pushes it to the run's branch, and opens the run's pull request early, ready and never a draft, so a CI that runs its slow jobs only on a ready PR runs them.
  • It reads the commit's statuses every 15 s. No status within 90 s means no CI ran: the round is decided locally and the record says what went unverified. Jobs nobody picks up for 30 minutes, a cancelled job, an error, or the step's time running out is an unfinished CI, which stops the loop for a person.
  • Each failed job comes back as a red lane, ci: <workflow> / <job> (<event>), with the step it failed at, its link and the end of its log, and goes to the fix round like any red lane. The bound and the no-progress rule count CI rounds too. git.pr.open later finds the early PR and replaces its title and description.

Phase 2: a services command per run. New Run's Tests step has a Services field, remembered per repository. Each check starts it unprivileged before the lanes, in its own process group. It is ready once it writes KEY=VALUE settings to $BRAID_SERVICES_ENV, within 180 s. Every lane and the app the judge opens get those settings, and the services stop with the run. A start that fails is one red services lane with its log tail. The guide has recipes: embedded-postgres, the MariaDB tarball, mongodb-memory-server, redis-memory-server.

Verified. The fast and full lanes passed on the integrated batch (3,388 backend tests, 681 vitest, 249 browser specs with the slow and live ones, 29 of 29 simulated runs), and it is live on production as 57971c2 (deploy run 138, self-check braid-selfcheck: ok). Archived in a6ae0dc. New simulated runs: ci-red-then-fix, ci-private-log and services-for-the-lanes.

Not done, and why:

  • The judge gets CI's job results but not its e2e report or screenshots: Forgejo 15.0.4 has no artifact download API (measured; the artifact routes answer 404).
  • A private repository's job log cannot be read with a token (every log route answers 404). Such a lane gives the job's link and its steps from the workflow file, and says why. soundcheck is private.
  • "An agent may propose the services command once" is not built. The operator enters it in New Run, with the recipes in the guide.
  • The Done-when run is still owed: a soundcheck run whose e2e passes in CI, then a planted CI failure fixed by a fix round. It needs a real-model production run, and production's auto-approve merges into soundcheck, so it waits for you. #139's baseline run, started tonight on cmoriarty/scratch-139, will exercise the CI lane, since that repository has a workflow that runs on push.
Phase 1 and Phase 2 shipped tonight, as `project-ci-is-a-lane` and `services-for-the-lanes`. **Phase 1: the project's CI is a lane.** Once a round's local lanes, the test lint and the README check pass, and the repository has a workflow that runs on a branch push or a pull request, `script.run-tests` gives the round's code to CI: - It commits the worktree as a *check commit* (with a temporary index, so the run's own history does not move), pushes it to the run's branch, and opens the run's pull request early, ready and never a draft, so a CI that runs its slow jobs only on a ready PR runs them. - It reads the commit's statuses every 15 s. No status within 90 s means no CI ran: the round is decided locally and the record says what went unverified. Jobs nobody picks up for 30 minutes, a cancelled job, an error, or the step's time running out is an unfinished CI, which stops the loop for a person. - Each failed job comes back as a red lane, `ci: <workflow> / <job> (<event>)`, with the step it failed at, its link and the end of its log, and goes to the fix round like any red lane. The bound and the no-progress rule count CI rounds too. `git.pr.open` later finds the early PR and replaces its title and description. **Phase 2: a services command per run.** New Run's Tests step has a Services field, remembered per repository. Each check starts it unprivileged before the lanes, in its own process group. It is ready once it writes `KEY=VALUE` settings to `$BRAID_SERVICES_ENV`, within 180 s. Every lane and the app the judge opens get those settings, and the services stop with the run. A start that fails is one red `services` lane with its log tail. The guide has recipes: embedded-postgres, the MariaDB tarball, mongodb-memory-server, redis-memory-server. **Verified.** The fast and full lanes passed on the integrated batch (3,388 backend tests, 681 vitest, 249 browser specs with the slow and live ones, 29 of 29 simulated runs), and it is live on production as 57971c2 (deploy run 138, self-check `braid-selfcheck: ok`). Archived in a6ae0dc. New simulated runs: `ci-red-then-fix`, `ci-private-log` and `services-for-the-lanes`. **Not done, and why:** - The judge gets CI's job results but not its e2e report or screenshots: Forgejo 15.0.4 has no artifact download API (measured; the artifact routes answer 404). - A private repository's job log cannot be read with a token (every log route answers 404). Such a lane gives the job's link and its steps from the workflow file, and says why. soundcheck is private. - "An agent may propose the services command once" is not built. The operator enters it in New Run, with the recipes in the guide. - The Done-when run is still owed: a soundcheck run whose e2e passes in CI, then a planted CI failure fixed by a fix round. It needs a real-model production run, and production's auto-approve merges into soundcheck, so it waits for you. #139's baseline run, started tonight on cmoriarty/scratch-139, will exercise the CI lane, since that repository has a workflow that runs on push.
Author
Owner

The CI lane on a real run (Phase 1): #139's baseline run 64 (git-flow, cmoriarty/scratch-139, real model) used it in both of its rounds.

  • In each round, once the local lane passed, script.run-tests committed the worktree as a check commit and pushed it. It opened the run's pull request early, then waited for scratch-139's CI.
  • The 4 jobs, Front end and Back end on push and on the pull request, passed on 393cf8bdfc in round 1 and on cf817983ed in round 2, and were recorded as the round's ci lane.
  • Each round's whole check took about 3.5 minutes.

That shows the push, the early PR and the wait on a real forge. A CI failure fed back to a fix round, and the service-backed suite on soundcheck, are still only covered by the simulated runs. So the Done-when soundcheck run is still owed.

**The CI lane on a real run** (Phase 1): #139's baseline run 64 (git-flow, cmoriarty/scratch-139, real model) used it in both of its rounds. - In each round, once the local lane passed, `script.run-tests` committed the worktree as a check commit and pushed it. It opened the run's pull request early, then waited for scratch-139's CI. - The 4 jobs, Front end and Back end on push and on the pull request, passed on 393cf8bdfc in round 1 and on cf817983ed in round 2, and were recorded as the round's `ci` lane. - Each round's whole check took about 3.5 minutes. That shows the push, the early PR and the wait on a real forge. A CI failure fed back to a fix round, and the service-backed suite on soundcheck, are still only covered by the simulated runs. So the Done-when soundcheck run is still owed.
Author
Owner

Verified on a real run: run 66

Run 66 (run_01M52JT93Q9X1VH3GD05D2HMZY) ran on production (57f9a21) on trogdor, with the thorough pipeline and a new public repository, cmoriarty/braid-combined-test. The repository was seeded with a CI workflow and a Postgres for the tests, and nothing else:

  • CI: unit tests and Playwright tests against a postgres service container, plus a release-notes check that only CI runs.
  • Services: node scripts/services.mjs (the guide's embedded-postgres recipe), entered as the run's Services command.

The brief asked for a small link shortener on Postgres. The run succeeded and merged braid-combined-test#1.

Phase 2, services:

  • Round 1 started the command before the lanes. Its log reads "the services are up after 3.5 s" and "their settings, given to every lane: DATABASE_URL".
  • The unit lane (58 tests) and the e2e lane (17 Playwright tests) ran against that Postgres.
  • Rounds 2 and 3 found the services still running and reused them. The fix round loaded $TMPDIR/braid-services.env as the brief tells it to.

Phase 1, CI's red path, on a real failure rather than a planted one:

  • Round 2's local lanes were all green, and its code went to CI as a check commit, with the run's early pull request.
  • CI failed Unit tests on push and on the pull request. Each failure came back as a red lane naming the job, the step it failed at and its link.
  • The cause was something no local lane could see: a test that starts a second embedded Postgres fails as root, and the CI container runs as root.
  • The fix round (agent.fix.r2, 3.2 min) read the failure and made the test create Postgres's system user only when it runs as root.
  • Round 3's CI passed all 6 jobs, including the Playwright suite against the service container.

The release-notes check meant to fail on purpose never did: the agents wrote a CHANGELOG.md naming the change on their own. The real failure covered the same path better.

This meets the Done-when (e2e green in CI, and a CI failure fixed by a fix round), on a fresh repository rather than soundcheck. The limits listed earlier still stand: no CI artifact download in Forgejo 15, no job logs from private repositories by token, and the operator enters the services command. Closing.

## Verified on a real run: run 66 Run 66 (`run_01M52JT93Q9X1VH3GD05D2HMZY`) ran on production (57f9a21) on trogdor, with the `thorough` pipeline and a new public repository, cmoriarty/braid-combined-test. The repository was seeded with a CI workflow and a Postgres for the tests, and nothing else: - **CI:** unit tests and Playwright tests against a `postgres` service container, plus a release-notes check that only CI runs. - **Services:** `node scripts/services.mjs` (the guide's embedded-postgres recipe), entered as the run's Services command. The brief asked for a small link shortener on Postgres. The run succeeded and merged braid-combined-test#1. **Phase 2, services:** - Round 1 started the command before the lanes. Its log reads "the services are up after 3.5 s" and "their settings, given to every lane: DATABASE_URL". - The unit lane (58 tests) and the e2e lane (17 Playwright tests) ran against that Postgres. - Rounds 2 and 3 found the services still running and reused them. The fix round loaded `$TMPDIR/braid-services.env` as the brief tells it to. **Phase 1, CI's red path, on a real failure rather than a planted one:** - Round 2's local lanes were all green, and its code went to CI as a check commit, with the run's early pull request. - CI failed `Unit tests` on push and on the pull request. Each failure came back as a red lane naming the job, the step it failed at and its link. - The cause was something no local lane could see: a test that starts a second embedded Postgres fails as root, and the CI container runs as root. - The fix round (`agent.fix.r2`, 3.2 min) read the failure and made the test create Postgres's system user only when it runs as root. - Round 3's CI passed all 6 jobs, including the Playwright suite against the service container. The release-notes check meant to fail on purpose never did: the agents wrote a `CHANGELOG.md` naming the change on their own. The real failure covered the same path better. This meets the Done-when (e2e green in CI, and a CI failure fixed by a fix round), on a fresh repository rather than soundcheck. The limits listed earlier still stand: no CI artifact download in Forgejo 15, no job logs from private repositories by token, and the operator enters the services command. Closing.
Sign in to join this conversation.
No labels
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
cmoriarty/braid#163
No description provided.