Sandbox each run's agents with bubblewrap: research, then build #203

Closed
opened 2026-10-18 10:44:06 -04:00 by cmoriarty · 1 comment
Owner

Follow-up to #201. A first layer is deployed (c1228b4, agent-shells-see-only-their-run): agent shells get an environment from an allowlist, and a path guard refuses other runs' folders, osfd's secrets and /proc/*/environ. A text guard can be bypassed by a model that computes a path at run time (command substitution, an interpreter script, a symlink), so it is not the fix. The fix is that an agent's processes cannot reach what isn't theirs.

A draft proposal exists, not approved: openspec/changes/agents-reach-only-their-run/ on main (proposal, design, specs, tasks, and probes/run.sh, which re-runs its measurements on any Docker host). It recommends one bubblewrap sandbox per run:

  • user, mount, PID, IPC, UTS and network namespaces;
  • opencode serve and every project command inside it;
  • only the run's folder (read-write), the shared caches, and the image read-only;
  • on the osfd service: a seccomp profile (Docker's default plus the namespace calls), systempaths=unconfined and no-new-privileges, no added capabilities;
  • a bridge so osfd still reaches opencode, a per-run gate that adds the model keys, an egress proxy that refuses private addresses, and the forge token used only outside the sandbox (a push goes through a repository osfd owns).

It rejects a separate uid (fences by file mode only, leaves the network open), Landlock (missing on the local kernel; no network rules on the VM's) and a container per run (needs the Docker socket).

Tested so far, on Docker Desktop on a Mac only:

  • with Docker's built-in seccomp, unshare -Ur fails;
  • with seccomp unconfined plus systempaths=unconfined, the full sandbox works: other runs, keys/ and the token file do not exist inside it, and osfd's API is unreachable;
  • a uid-1000 process with no capabilities can bridge a port into the sandbox's network namespace.

Research needed before building (the operator's call, 2026-10-18):

  1. The production VM's facts: Docker version, kernel, SELinux labelling, whether unprivileged user namespaces work there under a custom seccomp profile, and whether Portainer reads a seccomp file by path.
  2. Joining a running sandbox's mount and PID namespaces from osfd (for lanes, services, the app), untested.
  3. What breaks inside: Chromium's own sandbox under a user namespace (Playwright, #157), docker compose services if a run uses them, npm/uv with the shared caches (#67), git hooks.
  4. Cost: start-up time per run and per command, memory, and what a simulated run measures.
  5. Whether a smaller step (the gate for model keys, the forge token kept out of agents' reach) is worth shipping first on its own.
  6. Rethink or confirm the draft's recommendation from those findings, then propose the build in tasks that each deploy alone.
Follow-up to #201. A first layer is deployed (c1228b4, `agent-shells-see-only-their-run`): agent shells get an environment from an allowlist, and a path guard refuses other runs' folders, osfd's secrets and `/proc/*/environ`. A text guard can be bypassed by a model that computes a path at run time (command substitution, an interpreter script, a symlink), so it is not the fix. The fix is that an agent's processes cannot reach what isn't theirs. **A draft proposal exists, not approved:** `openspec/changes/agents-reach-only-their-run/` on main (proposal, design, specs, tasks, and `probes/run.sh`, which re-runs its measurements on any Docker host). It recommends one bubblewrap sandbox per run: - user, mount, PID, IPC, UTS and network namespaces; - `opencode serve` and every project command inside it; - only the run's folder (read-write), the shared caches, and the image read-only; - on the `osfd` service: a seccomp profile (Docker's default plus the namespace calls), `systempaths=unconfined` and `no-new-privileges`, no added capabilities; - a bridge so osfd still reaches opencode, a per-run gate that adds the model keys, an egress proxy that refuses private addresses, and the forge token used only outside the sandbox (a push goes through a repository osfd owns). It rejects a separate uid (fences by file mode only, leaves the network open), Landlock (missing on the local kernel; no network rules on the VM's) and a container per run (needs the Docker socket). **Tested so far, on Docker Desktop on a Mac only:** - with Docker's built-in seccomp, `unshare -Ur` fails; - with seccomp unconfined plus `systempaths=unconfined`, the full sandbox works: other runs, `keys/` and the token file do not exist inside it, and osfd's API is unreachable; - a uid-1000 process with no capabilities can bridge a port into the sandbox's network namespace. **Research needed before building** (the operator's call, 2026-10-18): 1. The production VM's facts: Docker version, kernel, SELinux labelling, whether unprivileged user namespaces work there under a custom seccomp profile, and whether Portainer reads a seccomp file by path. 2. Joining a running sandbox's mount and PID namespaces from osfd (for lanes, services, the app), untested. 3. What breaks inside: Chromium's own sandbox under a user namespace (Playwright, #157), `docker compose` services if a run uses them, npm/uv with the shared caches (#67), git hooks. 4. Cost: start-up time per run and per command, memory, and what a simulated run measures. 5. Whether a smaller step (the gate for model keys, the forge token kept out of agents' reach) is worth shipping first on its own. 6. Rethink or confirm the draft's recommendation from those findings, then propose the build in tasks that each deploy alone.
Author
Owner

Shipped in cmoriarty/braid#490 (main 799b2f05, on production since 2026-11-11 22:43 UTC), archived in cmoriarty/braid#491 as agents-reach-only-their-run. The capability is agent-sandbox.

What changed

  • A run's opencode serve starts under a Landlock fence: the run's folder, the image and its own key files, nothing else, and signals only to its own processes. The file tools run inside it.
  • Each worktree in use has a sandbox made with bubblewrap: its own processes, its own localhost, its own /tmp, its worktree and the run's shared folders. Steps side by side each have one.
  • Every shell command an agent runs goes there, because the server's SHELL is a client that hands it to a spawner inside. So do the Playwright server, the app tool, installs, lanes, apply's suites, services and a project's own scripted steps.
  • A sandbox reaches the internet with no proxy. Loopback, link-local, the private ranges and the container's own addresses are refused, except OSF_EGRESS_ALLOW. The guard refuses a webfetch to the same.
  • OSF_SANDBOX=step fails closed: a probe at start, and no run starts while it fails.

Evidence

  • Tests: the spawner and client (output, exit status, signals, a killed client, a background process), the fence's rules and a fenced process on CI's kernel, the network rules, the wiring, the deploy files. Suite: 776 passed, 2 skipped where Linux is needed.
  • Sims: all 89 scenarios pass with sandboxes on. agent-leaves-its-step reaches for the run's folder, the state volume, the forge token, the server's process and osfd's port by computed paths, finds none, and merges. In parallel-groups each group had its own sandbox with its own Playwright server, app and shells.
  • The image's self-check, built from the branch on the production host and run with OSF_SANDBOX=step under the stack's options: the fence line, five sandbox lines and six simulated runs with a real opencode. The deploy ran it again (deploy 487).
  • On production: osfd doctor 0 failed, 0 warned; /api/status lists no problem; the log says "the sandbox works (OSF_SANDBOX=step)".

Not in it, each filed: model keys behind a gate (#492); osfd's own git in a worktree, and the forge token's split (#493); the allowlist in the console, with approved skills, plugins and MCP servers (#494); integrity of the shared caches (#495); CPU and memory limits per step (#496).

The first full run on it is the standard run of braid#489; what it turns up goes on that run's findings issue.

**Shipped in cmoriarty/braid#490** (main `799b2f05`, on production since 2026-11-11 22:43 UTC), archived in cmoriarty/braid#491 as `agents-reach-only-their-run`. The capability is `agent-sandbox`. **What changed** - A run's `opencode serve` starts under a Landlock fence: the run's folder, the image and its own key files, nothing else, and signals only to its own processes. The file tools run inside it. - Each worktree in use has a sandbox made with bubblewrap: its own processes, its own `localhost`, its own `/tmp`, its worktree and the run's shared folders. Steps side by side each have one. - Every shell command an agent runs goes there, because the server's `SHELL` is a client that hands it to a spawner inside. So do the Playwright server, the `app` tool, installs, lanes, apply's suites, services and a project's own scripted steps. - A sandbox reaches the internet with no proxy. Loopback, link-local, the private ranges and the container's own addresses are refused, except `OSF_EGRESS_ALLOW`. The guard refuses a `webfetch` to the same. - `OSF_SANDBOX=step` fails closed: a probe at start, and no run starts while it fails. **Evidence** - Tests: the spawner and client (output, exit status, signals, a killed client, a background process), the fence's rules and a fenced process on CI's kernel, the network rules, the wiring, the deploy files. Suite: 776 passed, 2 skipped where Linux is needed. - Sims: all 89 scenarios pass with sandboxes on. `agent-leaves-its-step` reaches for the run's folder, the state volume, the forge token, the server's process and osfd's port by computed paths, finds none, and merges. In `parallel-groups` each group had its own sandbox with its own Playwright server, app and shells. - The image's self-check, built from the branch on the production host and run with `OSF_SANDBOX=step` under the stack's options: the fence line, five sandbox lines and six simulated runs with a real opencode. The deploy ran it again (deploy 487). - On production: `osfd doctor` 0 failed, 0 warned; `/api/status` lists no problem; the log says "the sandbox works (OSF_SANDBOX=step)". **Not in it**, each filed: model keys behind a gate (#492); osfd's own git in a worktree, and the forge token's split (#493); the allowlist in the console, with approved skills, plugins and MCP servers (#494); integrity of the shared caches (#495); CPU and memory limits per step (#496). The first full run on it is the standard run of braid#489; what it turns up goes on that run's findings issue.
Sign in to join this conversation.
No labels
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
cmoriarty/braid#203
No description provided.