A run's folder is mostly per-run copies of caches, installs and temp trees: share what is safe, drop what nobody needs #67

Open
opened 2026-09-26 22:35:05 -04:00 by cmoriarty · 3 comments
Owner

Problem

A run's folder under /var/lib/openspecflow/runs/<id> is far bigger than the work in it:

  • run_01M3FS9FFXEHTY5WTHX36V4Z5Y (openspec-flow#52, three attempts of fix.implement) reclaimed 3,408,606,573 bytes when it was archived. The archive recorded only the total.
  • run_01M3F8PTF3EWBCB721BP869E9N (soundcheck#32, stopped, still on the ledger) is 698 MB. The work itself is about 20 MB.

Nearly all of it is installed, downloaded or temporary. And because each run is isolated with its own HOME, XDG_CACHE_HOME and TMPDIR (§21, so the operator's opencode config never leaks into a run), each run downloads and keeps its own copy of everything.

Measured (2026-09-26)

soundcheck#32, the stopped run (698 MB):

MB what
work/node_modules 221 the project's npm install
home/.npm 161 npm's download cache
home/.config/opencode/node_modules 62 @opencode-ai/plugin, which opencode installs into every config dir that holds plugins
tmp/pgserver 132 an embedded Postgres the project's tests start: 71 MB data, 61 MB packages
tmp/npm-cache, tmp/node-compile-cache 37
oc.db + WAL 32 opencode's sessions
the repository itself ~20 .git 7.4, build output, test results

The fresh soundcheck#32 run (run_01M3GB3KN38YT9TZJ7KA7RQ5M0) at its first step: 174.5 MB before any code or test existed, 155 MB of it opencode installing its own plugin package (62 MB installed plus 94 MB of npm cache). Every run pays that.

This repository's test suite leaves about 250 MB of temp trees per pytest session (thousands of fixture repos and SQLite databases), and pytest keeps the last three, all inside the run's TMPDIR. The 3.4 GB run ran the full pytest tests/ repeatedly across three attempts (#65), and slow tests start real opencode servers with their own homes and plugin installs. That is the likely bulk of it. Inferred, not measured: the folder was already reclaimed. The fresh run is being sampled every 2 minutes, and its per-step growth will be added here.

What's already economical: runs git clone --local (hardlinked objects), checkpoints use alternates, and Playwright's browsers are shared from the image (/opt/ms-playwright). The state volume is XFS with reflink=1, currently 70 of 498 GB used.

Proposal: share what is safe to share, drop what nobody needs, and keep runs isolated

  1. One npm and one uv download cache for all runs (npm_config_cache, UV_CACHE_DIR, and PIP_CACHE_DIR while we're at it) on the state volume. Both are built for concurrent use: npm's cacache writes content-addressed files atomically, and uv takes file locks. Saves 100–300 MB per run and the download time. Collision to design around: npx unpacks into <cache>/_npx, and two concurrent npx runs of the same package can race there (ENOTEMPTY). Either keep _npx per run (npm_config_npx_cache, or a per-run npm_config_cache with only the tarball store shared), or check the npm version in the image for the fix.
  2. Install opencode's plugin package once per opencode version, into shared/opencode-plugin/<version>/ on the state volume. Seed each run's home/.config/opencode/node_modules from it with cp -r --reflink=always: on XFS that is copy-on-write, so near-zero space and no shared mutable files (a hardlink copy would let one run's writes land in another's). opencode's own install then finds the dependency satisfied. Verify that opencode 1.18.21 skips the install when node_modules already matches, and fall back to (1)'s shared cache if it doesn't. Saves about 155 MB per run.
  3. Keep fewer temp trees. Set PYTEST_ADDOPTS=-o tmp_path_retention_policy=failed (pytest ≥ 7.3) in the run's environment, so passing tests leave nothing behind. And when a run finishes (succeeded, failed or stopped), empty its tmp/. Nothing in a finished run is using it; a resumed run's server re-creates it. Per run, nothing is shared, so nothing can collide.
  4. Slim a finished run on the ledger. Once a run ends, delete what can be regenerated (node_modules, .venv, build output, caches, tmp/) and keep the worktree's source, .git, .osf/, oc.db and the transcripts. A resume re-installs on demand: #60's continue-in-place keeps the edits, which are in the source. A stopped run on the ledger drops from about 700 MB to about 50 MB. Show it in the archive dialog as "slimmed".
  5. Say where the bytes are. GET /api/runs/{id}/footprint and the archive dialog break the total down: dependencies, caches, temp, sessions, work. Record the breakdown on osf.run.archived, so a 3.4 GB archive explains itself afterwards.

Considered, not proposed yet: reflink-seeding a new run's node_modules/.venv from the last run of the same repository with the same lockfile. It's the biggest remaining saving (221 MB here), and on XFS it's safe (copy-on-write). But npm ci deletes node_modules first, so it only pays off if agents use npm install. Revisit with (5)'s numbers.

Acceptance

  • Two runs at once, of the same repository, both install and test green with the shared caches, and neither sees the other's files. Include concurrent npx of the same package.
  • A fresh run's footprint at its first step is under 30 MB (from 174.5 MB).
  • A finished run on the ledger is slimmed. A resumed slimmed run re-installs and continues.
  • The footprint and the archive record carry a breakdown.
## Problem A run's folder under `/var/lib/openspecflow/runs/<id>` is far bigger than the work in it: - `run_01M3FS9FFXEHTY5WTHX36V4Z5Y` (openspec-flow#52, three attempts of `fix.implement`) reclaimed **3,408,606,573 bytes** when it was archived. The archive recorded only the total. - `run_01M3F8PTF3EWBCB721BP869E9N` (soundcheck#32, stopped, still on the ledger) is **698 MB**. The work itself is about 20 MB. Nearly all of it is installed, downloaded or temporary. And because each run is isolated with its own `HOME`, `XDG_CACHE_HOME` and `TMPDIR` (§21, so the operator's opencode config never leaks into a run), each run downloads and keeps its own copy of everything. ## Measured (2026-09-26) **soundcheck#32, the stopped run (698 MB):** | | MB | what | |---|---|---| | `work/node_modules` | 221 | the project's npm install | | `home/.npm` | 161 | npm's download cache | | `home/.config/opencode/node_modules` | 62 | `@opencode-ai/plugin`, which opencode installs into every config dir that holds plugins | | `tmp/pgserver` | 132 | an embedded Postgres the project's tests start: 71 MB data, 61 MB packages | | `tmp/npm-cache`, `tmp/node-compile-cache` | 37 | | | `oc.db` + WAL | 32 | opencode's sessions | | the repository itself | ~20 | `.git` 7.4, build output, test results | **The fresh soundcheck#32 run (`run_01M3GB3KN38YT9TZJ7KA7RQ5M0`) at its first step:** 174.5 MB before any code or test existed, **155 MB of it opencode installing its own plugin package** (62 MB installed plus 94 MB of npm cache). Every run pays that. **This repository's test suite** leaves about **250 MB of temp trees per pytest session** (thousands of fixture repos and SQLite databases), and pytest keeps the last three, all inside the run's `TMPDIR`. The 3.4 GB run ran the full `pytest tests/` repeatedly across three attempts (#65), and slow tests start real opencode servers with their own homes and plugin installs. That is the likely bulk of it. **Inferred, not measured**: the folder was already reclaimed. The fresh run is being sampled every 2 minutes, and its per-step growth will be added here. What's already economical: runs `git clone --local` (hardlinked objects), checkpoints use alternates, and Playwright's browsers are shared from the image (`/opt/ms-playwright`). The state volume is XFS with `reflink=1`, currently 70 of 498 GB used. ## Proposal: share what is safe to share, drop what nobody needs, and keep runs isolated 1. **One npm and one uv download cache for all runs** (`npm_config_cache`, `UV_CACHE_DIR`, and `PIP_CACHE_DIR` while we're at it) on the state volume. Both are built for concurrent use: npm's cacache writes content-addressed files atomically, and uv takes file locks. Saves 100–300 MB per run and the download time. **Collision to design around:** `npx` unpacks into `<cache>/_npx`, and two concurrent `npx` runs of the same package can race there (`ENOTEMPTY`). Either keep `_npx` per run (`npm_config_npx_cache`, or a per-run `npm_config_cache` with only the tarball store shared), or check the npm version in the image for the fix. 2. **Install opencode's plugin package once per opencode version**, into `shared/opencode-plugin/<version>/` on the state volume. Seed each run's `home/.config/opencode/node_modules` from it with `cp -r --reflink=always`: on XFS that is copy-on-write, so near-zero space and no shared mutable files (a hardlink copy would let one run's writes land in another's). opencode's own install then finds the dependency satisfied. **Verify** that opencode 1.18.21 skips the install when `node_modules` already matches, and fall back to (1)'s shared cache if it doesn't. Saves about 155 MB per run. 3. **Keep fewer temp trees.** Set `PYTEST_ADDOPTS=-o tmp_path_retention_policy=failed` (pytest ≥ 7.3) in the run's environment, so passing tests leave nothing behind. And when a run **finishes** (succeeded, failed or stopped), empty its `tmp/`. Nothing in a finished run is using it; a resumed run's server re-creates it. Per run, nothing is shared, so nothing can collide. 4. **Slim a finished run on the ledger.** Once a run ends, delete what can be regenerated (`node_modules`, `.venv`, build output, caches, `tmp/`) and keep the worktree's source, `.git`, `.osf/`, `oc.db` and the transcripts. A resume re-installs on demand: #60's continue-in-place keeps the edits, which are in the source. A stopped run on the ledger drops from about 700 MB to about 50 MB. Show it in the archive dialog as "slimmed". 5. **Say where the bytes are.** `GET /api/runs/{id}/footprint` and the archive dialog break the total down: dependencies, caches, temp, sessions, work. Record the breakdown on `osf.run.archived`, so a 3.4 GB archive explains itself afterwards. **Considered, not proposed yet:** reflink-seeding a new run's `node_modules`/`.venv` from the last run of the same repository with the same lockfile. It's the biggest remaining saving (221 MB here), and on XFS it's safe (copy-on-write). But `npm ci` deletes `node_modules` first, so it only pays off if agents use `npm install`. Revisit with (5)'s numbers. ## Acceptance - Two runs at once, of the same repository, both install and test green with the shared caches, and neither sees the other's files. Include concurrent `npx` of the same package. - A fresh run's footprint at its first step is under 30 MB (from 174.5 MB). - A finished run on the ledger is slimmed. A resumed slimmed run re-installs and continues. - The footprint and the archive record carry a breakdown.
Author
Owner

Verified since filing:

  • opencode's plugin install skips when it is already satisfied. From the 1.18.21 binary (Npm.install → Npm.checkNodeModules → Npm.checkDirty), in the config folder:

    • it returns immediately if the folder isn't writable;
    • it installs if node_modules is missing;
    • otherwise it installs only if a dependency in package.json (plus the @opencode-ai/plugin it adds) is missing from package-lock.json's root.

    So seeding a run's home/.config/opencode/ with package.json, package-lock.json and node_modules from a per-version copy means no install and no npm cache. That removes all of the 155 MB each run pays at start. A reflink copy on the XFS volume keeps each run's files its own.

  • What a run's opencode server actually runs with (run_01M3GB3KN38YT9TZJ7KA7RQ5M0, PID 329): HOME=<run>/home, XDG_CONFIG_HOME=<run>/home/.config, XDG_DATA_HOME=<run>/xdg, XDG_CACHE_HOME=<run>/cache, XDG_STATE_HOME=<run>/state, TMPDIR=<run>/tmp, PLAYWRIGHT_BROWSERS_PATH=/opt/ms-playwright. There's no npm_config_cache, UV_CACHE_DIR or PYTEST_ADDOPTS, so npm caches in <run>/home/.npm, uv in <run>/cache/uv, and pytest's temp trees go under <run>/tmp. osfd's own environment sets UV_LINK_MODE=copy; worth checking whether runs inherit it, because with a shared uv cache on the same XFS volume, UV_LINK_MODE=clone would reflink packages into each .venv instead of copying them.

  • npm is 10.9.9 in the image. Before sharing the npm cache, test concurrent npx of the same package against it (the _npx race), or keep _npx per run.

Still sampling the fresh run's growth per step; the breakdown follows when it ends.

Verified since filing: - **opencode's plugin install skips when it is already satisfied.** From the 1.18.21 binary (`Npm.install` → `Npm.checkNodeModules` → `Npm.checkDirty`), in the config folder: - it returns immediately if the folder isn't writable; - it installs if `node_modules` is missing; - otherwise it installs only if a dependency in `package.json` (plus the `@opencode-ai/plugin` it adds) is missing from `package-lock.json`'s root. So seeding a run's `home/.config/opencode/` with `package.json`, `package-lock.json` and `node_modules` from a per-version copy means no install and no npm cache. That removes all of the 155 MB each run pays at start. A reflink copy on the XFS volume keeps each run's files its own. - **What a run's opencode server actually runs with** (`run_01M3GB3KN38YT9TZJ7KA7RQ5M0`, PID 329): `HOME=<run>/home`, `XDG_CONFIG_HOME=<run>/home/.config`, `XDG_DATA_HOME=<run>/xdg`, `XDG_CACHE_HOME=<run>/cache`, `XDG_STATE_HOME=<run>/state`, `TMPDIR=<run>/tmp`, `PLAYWRIGHT_BROWSERS_PATH=/opt/ms-playwright`. There's no `npm_config_cache`, `UV_CACHE_DIR` or `PYTEST_ADDOPTS`, so npm caches in `<run>/home/.npm`, uv in `<run>/cache/uv`, and pytest's temp trees go under `<run>/tmp`. osfd's own environment sets `UV_LINK_MODE=copy`; worth checking whether runs inherit it, because with a shared uv cache on the same XFS volume, `UV_LINK_MODE=clone` would reflink packages into each `.venv` instead of copying them. - **npm is 10.9.9** in the image. Before sharing the npm cache, test concurrent `npx` of the same package against it (the `_npx` race), or keep `_npx` per run. Still sampling the fresh run's growth per step; the breakdown follows when it ends.
Author
Owner

Measured growth, step by step (run_01M3GB3KN38YT9TZJ7KA7RQ5M0, soundcheck#32, sampled every 2 min)

time step total where
22:29 propose.proposal 175 MB home/.npm 94, opencode plugin node_modules 62, .git 6
22:46 4× propose.spec + propose.design 193 MB sessions growing, nothing installed
22:54 propose.tasks 196 MB
23:02 openspec.apply 718 MB home/.npm 389, work/node_modules 221, plugin 62, .git 7

openspec.apply installed the project's packages: +221 MB of node_modules, and npm's per-run cache grew by another 296 MB, a second copy of the same packages. Over half of a run's size is npm keeping two copies of everything, per run. The sampler keeps going to the end of the run.

Also confirmed: a run's opencode server gets no UV_* variables (osfd's own UV_LINK_MODE=copy is not passed on), so uv runs with its defaults inside the run.

Everything proposed, in one place, with what it saves and why runs cannot collide

# Measure Saves per run Collision risk and how it's handled
1 One shared npm download cache (npm_config_cache=/var/lib/openspecflow/shared/npm) ~390 MB (the 389 MB home/.npm) plus the download time npm's cacache is content-addressed and writes atomically: concurrent npm ci/install across runs is its designed use. Exception: npx unpacks into <cache>/_npx, and two concurrent npx runs of one package can race (ENOTEMPTY). Give each run its own npm_config_npx_cache (or equivalent), or verify on npm 10.9.9 first.
2 One shared uv cache (UV_CACHE_DIR) with UV_LINK_MODE=clone the Python half of the same double copy (100–300 MB for a Python project) uv takes file locks and is built for concurrent use. The default link mode, hardlink, would share the same inodes between the cache and every run's .venv, so an edit in one .venv would change every run's. clone (reflink, XFS reflink=1) gives each run copy-on-write files.
3 Seed opencode's plugin folder from shared/opencode-plugin/<opencode version>/ with cp -r --reflink=always (package.json, package-lock.json, node_modules) 155 MB (62 MB plugin plus 94 MB of its cache) Verified in 1.18.21: opencode installs nothing when node_modules exists and the lockfile matches. Reflink copies are copy-on-write, so no run can write into another's. Never hardlink this.
4 Keep fewer pytest temp trees: PYTEST_ADDOPTS=-o tmp_path_retention_policy=failed in a run's environment up to ~750 MB on this repo (3 × ~250 MB sessions) Per run; nothing shared.
5 Empty tmp/ when a run ends (succeeded, failed or stopped) 100–200 MB (soundcheck's embedded Postgres alone is 132 MB) Only once the run's opencode server and its children have stopped; a resume re-creates tmp/. Per run.
6 Slim a finished run on the ledger: drop node_modules, .venv, build output, caches, tmp/; keep source, .git, .osf/, oc.db, transcripts a stopped run ~700 MB → ~50 MB Per run. A resume re-installs; #60's continue-in-place keeps the edits, which are in the source. Show "slimmed" in the archive dialog.
7 Break the footprint down in GET /api/runs/{id}/footprint, the archive dialog and osf.run.archived nothing directly; it explains the next 3.4 GB none
8 (later) Reflink-seed node_modules / .venv from the previous run of the same repository with the same lockfile ~220 MB more Copy-on-write, so safe. But npm ci deletes node_modules first, so it only pays off with npm install. Revisit with (7)'s numbers.

Already economical and unchanged: git clone --local (hardlinked immutable objects), checkpoints through alternates, Playwright's browsers shared from the image (/opt/ms-playwright).

With 1–3 a fresh run would start at under 30 MB, not 175, and reach apply at about 230 MB (node_modules plus the work), not 718. With 4–6 a finished run on the ledger would be about 50 MB.

Acceptance, restated: two runs of the same repository at once, including concurrent npx of one package, both install and test green on the shared caches, and neither sees the other's files.

## Measured growth, step by step (`run_01M3GB3KN38YT9TZJ7KA7RQ5M0`, soundcheck#32, sampled every 2 min) | time | step | total | where | |---|---|---|---| | 22:29 | `propose.proposal` | 175 MB | `home/.npm` 94, opencode plugin `node_modules` 62, `.git` 6 | | 22:46 | 4× `propose.spec` + `propose.design` | 193 MB | sessions growing, nothing installed | | 22:54 | `propose.tasks` | 196 MB | | | **23:02** | **`openspec.apply`** | **718 MB** | **`home/.npm` 389**, `work/node_modules` 221, plugin 62, `.git` 7 | `openspec.apply` installed the project's packages: **+221 MB of `node_modules`, and npm's per-run cache grew by another 296 MB, a second copy of the same packages.** Over half of a run's size is npm keeping two copies of everything, per run. The sampler keeps going to the end of the run. Also confirmed: a run's opencode server gets no `UV_*` variables (osfd's own `UV_LINK_MODE=copy` is not passed on), so uv runs with its defaults inside the run. ## Everything proposed, in one place, with what it saves and why runs cannot collide | # | Measure | Saves per run | Collision risk and how it's handled | |---|---|---|---| | 1 | **One shared npm download cache** (`npm_config_cache=/var/lib/openspecflow/shared/npm`) | **~390 MB** (the 389 MB `home/.npm`) plus the download time | npm's cacache is content-addressed and writes atomically: concurrent `npm ci`/`install` across runs is its designed use. **Exception:** `npx` unpacks into `<cache>/_npx`, and two concurrent `npx` runs of one package can race (`ENOTEMPTY`). Give each run its own `npm_config_npx_cache` (or equivalent), or verify on npm 10.9.9 first. | | 2 | **One shared uv cache** (`UV_CACHE_DIR`) **with `UV_LINK_MODE=clone`** | the Python half of the same double copy (100–300 MB for a Python project) | uv takes file locks and is built for concurrent use. The default link mode, **hardlink**, would share the *same inodes* between the cache and every run's `.venv`, so an edit in one `.venv` would change every run's. `clone` (reflink, XFS `reflink=1`) gives each run copy-on-write files. | | 3 | **Seed opencode's plugin folder** from `shared/opencode-plugin/<opencode version>/` with `cp -r --reflink=always` (`package.json`, `package-lock.json`, `node_modules`) | **155 MB** (62 MB plugin plus 94 MB of its cache) | Verified in 1.18.21: opencode installs nothing when `node_modules` exists and the lockfile matches. Reflink copies are copy-on-write, so no run can write into another's. Never hardlink this. | | 4 | **Keep fewer pytest temp trees:** `PYTEST_ADDOPTS=-o tmp_path_retention_policy=failed` in a run's environment | up to ~750 MB on this repo (3 × ~250 MB sessions) | Per run; nothing shared. | | 5 | **Empty `tmp/` when a run ends** (succeeded, failed or stopped) | 100–200 MB (soundcheck's embedded Postgres alone is 132 MB) | Only once the run's opencode server and its children have stopped; a resume re-creates `tmp/`. Per run. | | 6 | **Slim a finished run on the ledger:** drop `node_modules`, `.venv`, build output, caches, `tmp/`; keep source, `.git`, `.osf/`, `oc.db`, transcripts | a stopped run ~700 MB → ~50 MB | Per run. A resume re-installs; #60's continue-in-place keeps the edits, which are in the source. Show "slimmed" in the archive dialog. | | 7 | **Break the footprint down** in `GET /api/runs/{id}/footprint`, the archive dialog and `osf.run.archived` | nothing directly; it explains the next 3.4 GB | none | | 8 | *(later)* **Reflink-seed `node_modules` / `.venv`** from the previous run of the same repository with the same lockfile | ~220 MB more | Copy-on-write, so safe. But `npm ci` deletes `node_modules` first, so it only pays off with `npm install`. Revisit with (7)'s numbers. | Already economical and unchanged: `git clone --local` (hardlinked immutable objects), checkpoints through alternates, Playwright's browsers shared from the image (`/opt/ms-playwright`). With 1–3 a fresh run would start at under 30 MB, not 175, and reach `apply` at about 230 MB (`node_modules` plus the work), not 718. With 4–6 a finished run on the ledger would be about 50 MB. Acceptance, restated: two runs of the same repository at once, including concurrent `npx` of one package, both install and test green on the shared caches, and neither sees the other's files.
Author
Owner

Final numbers for run_01M3GB3KN38YT9TZJ7KA7RQ5M0 (91 samples, 22:29 → 01:32; the run ended stopped):

time step total largest parts
22:29 propose.proposal 175 MB home/.npm 94, opencode plugin node_modules 62
22:52 propose.tasks 195 MB
22:58 openspec.apply starts 197 MB
01:32 stopped 982 MB home/.npm 427, work/node_modules 280, tmp/opencode 70, plugin node_modules 62, tmp/pg-5433 46

Planning took 22 MB. Implementation and testing took 785 MB. Almost all of that is regenerable:

  • npm's per-run cache: 427 MB, a second copy of node_modules;
  • the project's node_modules: 280 MB;
  • temp trees: 116 MB (opencode's own and a test Postgres);
  • opencode's plugin install: 62 MB.

The measures in the table above would take this run from 982 MB to about 300 MB while it ran:

  • (1) a shared npm cache: −427 MB;
  • (3) a seeded plugin folder: −62 MB;
  • (5) emptying tmp/ at the end: −116 MB.

(6) slimming would take it to tens of MB once stopped.

Final numbers for `run_01M3GB3KN38YT9TZJ7KA7RQ5M0` (91 samples, 22:29 → 01:32; the run ended stopped): | time | step | total | largest parts | |---|---|---|---| | 22:29 | `propose.proposal` | 175 MB | `home/.npm` 94, opencode plugin `node_modules` 62 | | 22:52 | `propose.tasks` | 195 MB | | | 22:58 | `openspec.apply` starts | 197 MB | | | **01:32** | **stopped** | **982 MB** | **`home/.npm` 427**, `work/node_modules` 280, `tmp/opencode` 70, plugin `node_modules` 62, `tmp/pg-5433` 46 | **Planning took 22 MB. Implementation and testing took 785 MB.** Almost all of that is regenerable: - npm's per-run cache: 427 MB, a second copy of `node_modules`; - the project's `node_modules`: 280 MB; - temp trees: 116 MB (opencode's own and a test Postgres); - opencode's plugin install: 62 MB. The measures in the table above would take this run from 982 MB to about 300 MB while it ran: - (1) a shared npm cache: −427 MB; - (3) a seeded plugin folder: −62 MB; - (5) emptying `tmp/` at the end: −116 MB. (6) slimming would take it to tens of MB once stopped.
Sign in to join this conversation.
No labels
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
cmoriarty/braid#67
No description provided.