cmoriarty/braid
1
0
Fork 0
Code Issues 15 Pull requests Projects Releases Packages 1 Wiki Activity Actions

scripted test steps #104

New issue
Closed
opened 2026-09-28 23:45:11 -04:00 by cmoriarty · 1 comment
cmoriarty commented 2026-09-28 23:45:11 -04:00
Owner
Copy link

I watched an agent.test.unit agent run and it really faliled around before it figured out the tests, which leads me to realize it would be possible for the step to miss tests. I think the user should have the ability to switch this step to scripted, and select their test suite and how to run it. agent by default is fine, scripted for more advanced users. This should also apply to agent.test.browser.

I watched an agent.test.unit agent run and it really faliled around before it figured out the tests, which leads me to realize it would be possible for the step to miss tests. I think the user should have the ability to switch this step to scripted, and select their test suite and how to run it. agent by default is fine, scripted for more advanced users. This should also apply to agent.test.browser.
cmoriarty commented 2026-09-29 04:33:18 -04:00
Author
Owner
Copy link

Shipped in 47aace1, as the OpenSpec change scripted-test-steps. It is now archived (9d27890). Its requirements are the new scripted-tests spec, and new-run, browser-check and pipeline-steps are updated.

Chosen in New Run. For a pipeline with test steps, the wizard's Tests step offers Agent (the default) or Script with a command, for the unit tests and for the browser check. It also takes an optional App command, which is how the browser check serves the app.

  • It starts from the repository's last choice, which GET /api/repo-profile reports as tests.
  • Start stays disabled while a Script row has no command.
  • POST /api/runs takes the choice as tests and keeps only what differs from the agents. Anything malformed is refused with 400 and the reason.
  • The run carries the choice in .osf/brief.json.

Scripted steps beside the agent ones. script.test.unit and script.test.browser sit beside agent.test.unit and agent.test.browser in every built-in pipeline that tests. Conditions pick one of each pair; the other is skipped, with the command as the reason. A scripted step:

  • runs the command from the repository root with the run's own environment, including its PREVIEW_PORT;
  • streams the suite's output as the step's output;
  • writes the lane report, .osf/test/unit.json or .osf/test/browser-script.json;
  • passes only when the command exits 0 and the worktree has not changed since.

It judges and does not repair: a red suite fails the step.

Verified on production with run 33 (run_01M3P3GXYAS552B7G7EYPNV8KV), whose unit tests were scripted as cd backend && .venv/bin/python -m pytest -q && cd ../frontend && npm test:

  • agent.test.unit was skipped as not needed.
  • script.test.unit was green in 15.7 s, with the suite's output (the backend's pytest, then 57 vitest tests) as the step's output.

Also checked in a real browser against a local osfd: the Tests step, a started run's brief, and a scripted browser suite whose exit code 1 failed its step.

Shipped in 47aace1, as the OpenSpec change `scripted-test-steps`. It is now archived (9d27890). Its requirements are the new `scripted-tests` spec, and `new-run`, `browser-check` and `pipeline-steps` are updated. **Chosen in New Run.** For a pipeline with test steps, the wizard's Tests step offers **Agent** (the default) or **Script** with a command, for the unit tests and for the browser check. It also takes an optional **App command**, which is how the browser check serves the app. - It starts from the repository's last choice, which `GET /api/repo-profile` reports as `tests`. - Start stays disabled while a Script row has no command. - `POST /api/runs` takes the choice as `tests` and keeps only what differs from the agents. Anything malformed is refused with 400 and the reason. - The run carries the choice in `.osf/brief.json`. **Scripted steps beside the agent ones.** `script.test.unit` and `script.test.browser` sit beside `agent.test.unit` and `agent.test.browser` in every built-in pipeline that tests. Conditions pick one of each pair; the other is skipped, with the command as the reason. A scripted step: - runs the command from the repository root with the run's own environment, including its `PREVIEW_PORT`; - streams the suite's output as the step's output; - writes the lane report, `.osf/test/unit.json` or `.osf/test/browser-script.json`; - passes only when the command exits 0 and the worktree has not changed since. It judges and does not repair: a red suite fails the step. **Verified on production** with run 33 (`run_01M3P3GXYAS552B7G7EYPNV8KV`), whose unit tests were scripted as `cd backend && .venv/bin/python -m pytest -q && cd ../frontend && npm test`: - `agent.test.unit` was skipped as not needed. - `script.test.unit` was green in 15.7 s, with the suite's output (the backend's pytest, then 57 vitest tests) as the step's output. Also checked in a real browser against a local osfd: the Tests step, a started run's brief, and a scripted browser suite whose exit code 1 failed its step.
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
archive/prefect-kestra-experiments
Labels
Clear labels
No items
No labels
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
cmoriarty/braid#104
No description provided.