A test suite that needs a service runs in the project's CI, and its failures come back to the fix loop #163
Loading…
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
A soundcheck run on Claude Sonnet 5.5 (2026-10-09) could not have finished: soundcheck's e2e suite needs Postgres, which its CI provides as a service (
postgres:17-alpine, reached throughE2E_PG_HOST/E2E_PG_PORT), and Braid's run environment has none. The apply agent wrote the e2e test and declared it as a lane (.osf/test/e2e.command); the fix loop would have run it, watched it fail at the web server's first query, and tried to "fix" code that was not broken.Baking each service into Braid (Postgres today, MariaDB, Redis, a queue tomorrow) duplicates what the project already maintains and keeps working: its CI. So the split is: Braid verifies what needs no service; the project's CI verifies the rest, and Braid listens to it.
Already shipped for the first half (
3a78aff): task lists no longer hold tasks an apply agent cannot finish (run the whole suite, run e2e, preview, check on a device), and an agent declares no local lane for a suite that needs a service its CI provides.What it needs
fulljob (Postgres, Playwright e2e) only on a ready, non-draft pull request and on main;fastruns on every push. So Braid opens its pull request as ready (or marks it ready) before the fix loop's last check, rather than only at the end.Open questions for the explore step
services:, a lane that cannot start, or the project's own split (soundcheck'stest:fastvstest:e2e).Done when
A soundcheck run whose change adds an e2e test finishes clean: the fast lane passes locally, the e2e suite passes in CI and the run records it as a lane, and a CI failure planted in a test comes back to a fix round with the job's log and is fixed.
Phase 2: a services command per repository, for a local e2e and a judge that can see the app
Decided 2026-10-09, after weighing alternatives.
Not Docker. An agent pulling and running containers needs the host's Docker socket in the run environment, which is root on the VM for every agent: Braid's database, the pasted API keys and every other run's files with it. Sandboxed runtimes (rootless Podman, a Docker-in-Docker sidecar per run) are possible, but too fiddly and risky to hand to agents. Not act /
forgejo-runner execinside Braid either: it needs the same runtime, and it diverges from real CI at the edges. Phase 1's CI push already gets the same result on the runner host, which has Docker.Instead, services as ordinary unprivileged processes. Most common services run without root from a downloaded binary inside the run's own directory:
embedded-postgresnpm package, or prebuilt binaries (initdb,pg_ctl);mysqld --no-defaults --datadir=…);mongodb-memory-server;redis-memory-server, or a staticredis-server.Set once per repository, not improvised per run. Braid already keeps per-repository test choices (unit, browser and serve commands). It gains one more: start the services. For soundcheck that would be an
embedded-postgresstart plusE2E_PG_HOST/E2E_PG_PORT. Braid runs it unprivileged, in the run's sandbox:An agent may propose the command once, guided by a short brief of the recipes above, and the operator approves it in the repository's settings.
Order. Phase 1 (CI as the environment for service-dependent suites) first: it covers every project and every service with no setup. Phase 2 for repositories where a fast local e2e and a judge that can open the running app are worth one command of setup; soundcheck is the first candidate.
Already shipped toward this issue:
3a78aff: tasks an apply agent cannot finish are kept out of the task list, and no local lane is declared for a suite that needs a service;6fe3e8e: the judge may sayunverifiableand name the missing service. The loop honours it only when the project's CI declares a service, the judge names it, and no criterion failed, so an app that needs CI's Postgres no longer sends a run into fix rounds that cannot help.Phase 1 progress (prod
9d4c23f)Shipped:
3a78aff): no task an apply agent can't finish (whole-suite, e2e, preview or by-hand checks). A suite that needs a service its CI provides gets its tests written but no local lane.6fe3e8e): a newunverifiableverdict with amissing_service. It counts as no failure only when no criterion failed and the project's CI workflows declareservices:; otherwise it's a failure. So an app that needs Postgres goes to CI rather than into fix rounds.9d4c23f): whengit.pr.mergefails on a CI check, the step now leads with Forgejo's refusal and the failing check. Before, it read "no non-empty file matched: .osf/merged.json".Tested on a Sonnet 5.5 thorough run on soundcheck#71. The judge's second round returned
unverifiablefor the Postgres-backed e2e and the run went on to the PR. The PR's CIfullthen failed on something outside the change:npm audit --audit-level=highhit GHSA-68fv-2mgg-jv7q (source-map-js), published during the run. That fails every soundcheck branch until main takes the fix (a lockfile-onlynpm audit fix).Not done yet: a CI failure doesn't come back to the run as a fix round. The run fails at merge with the reason.
Phase 1 and Phase 2 shipped tonight, as
project-ci-is-a-laneandservices-for-the-lanes.Phase 1: the project's CI is a lane. Once a round's local lanes, the test lint and the README check pass, and the repository has a workflow that runs on a branch push or a pull request,
script.run-testsgives the round's code to CI:ci: <workflow> / <job> (<event>), with the step it failed at, its link and the end of its log, and goes to the fix round like any red lane. The bound and the no-progress rule count CI rounds too.git.pr.openlater finds the early PR and replaces its title and description.Phase 2: a services command per run. New Run's Tests step has a Services field, remembered per repository. Each check starts it unprivileged before the lanes, in its own process group. It is ready once it writes
KEY=VALUEsettings to$BRAID_SERVICES_ENV, within 180 s. Every lane and the app the judge opens get those settings, and the services stop with the run. A start that fails is one redserviceslane with its log tail. The guide has recipes: embedded-postgres, the MariaDB tarball, mongodb-memory-server, redis-memory-server.Verified. The fast and full lanes passed on the integrated batch (3,388 backend tests, 681 vitest, 249 browser specs with the slow and live ones, 29 of 29 simulated runs), and it is live on production as
57971c2(deploy run 138, self-checkbraid-selfcheck: ok). Archived ina6ae0dc. New simulated runs:ci-red-then-fix,ci-private-logandservices-for-the-lanes.Not done, and why:
The CI lane on a real run (Phase 1): #139's baseline run 64 (git-flow, cmoriarty/scratch-139, real model) used it in both of its rounds.
script.run-testscommitted the worktree as a check commit and pushed it. It opened the run's pull request early, then waited for scratch-139's CI.cilane.That shows the push, the early PR and the wait on a real forge. A CI failure fed back to a fix round, and the service-backed suite on soundcheck, are still only covered by the simulated runs. So the Done-when soundcheck run is still owed.
Verified on a real run: run 66
Run 66 (
run_01M52JT93Q9X1VH3GD05D2HMZY) ran on production (57f9a21) on trogdor, with thethoroughpipeline and a new public repository, cmoriarty/braid-combined-test. The repository was seeded with a CI workflow and a Postgres for the tests, and nothing else:postgresservice container, plus a release-notes check that only CI runs.node scripts/services.mjs(the guide's embedded-postgres recipe), entered as the run's Services command.The brief asked for a small link shortener on Postgres. The run succeeded and merged braid-combined-test#1.
Phase 2, services:
$TMPDIR/braid-services.envas the brief tells it to.Phase 1, CI's red path, on a real failure rather than a planted one:
Unit testson push and on the pull request. Each failure came back as a red lane naming the job, the step it failed at and its link.agent.fix.r2, 3.2 min) read the failure and made the test create Postgres's system user only when it runs as root.The release-notes check meant to fail on purpose never did: the agents wrote a
CHANGELOG.mdnaming the change on their own. The real failure covered the same path better.This meets the Done-when (e2e green in CI, and a CI failure fixed by a fix round), on a fresh repository rather than soundcheck. The limits listed earlier still stand: no CI artifact download in Forgejo 15, no job logs from private repositories by token, and the operator enters the services command. Closing.