A run's folder is mostly per-run copies of caches, installs and temp trees: share what is safe, drop what nobody needs #67
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Problem
A run's folder under
/var/lib/openspecflow/runs/<id>is far bigger than the work in it:run_01M3FS9FFXEHTY5WTHX36V4Z5Y(openspec-flow#52, three attempts offix.implement) reclaimed 3,408,606,573 bytes when it was archived. The archive recorded only the total.run_01M3F8PTF3EWBCB721BP869E9N(soundcheck#32, stopped, still on the ledger) is 698 MB. The work itself is about 20 MB.Nearly all of it is installed, downloaded or temporary. And because each run is isolated with its own
HOME,XDG_CACHE_HOMEandTMPDIR(§21, so the operator's opencode config never leaks into a run), each run downloads and keeps its own copy of everything.Measured (2026-09-26)
soundcheck#32, the stopped run (698 MB):
work/node_moduleshome/.npmhome/.config/opencode/node_modules@opencode-ai/plugin, which opencode installs into every config dir that holds pluginstmp/pgservertmp/npm-cache,tmp/node-compile-cacheoc.db+ WAL.git7.4, build output, test resultsThe fresh soundcheck#32 run (
run_01M3GB3KN38YT9TZJ7KA7RQ5M0) at its first step: 174.5 MB before any code or test existed, 155 MB of it opencode installing its own plugin package (62 MB installed plus 94 MB of npm cache). Every run pays that.This repository's test suite leaves about 250 MB of temp trees per pytest session (thousands of fixture repos and SQLite databases), and pytest keeps the last three, all inside the run's
TMPDIR. The 3.4 GB run ran the fullpytest tests/repeatedly across three attempts (#65), and slow tests start real opencode servers with their own homes and plugin installs. That is the likely bulk of it. Inferred, not measured: the folder was already reclaimed. The fresh run is being sampled every 2 minutes, and its per-step growth will be added here.What's already economical: runs
git clone --local(hardlinked objects), checkpoints use alternates, and Playwright's browsers are shared from the image (/opt/ms-playwright). The state volume is XFS withreflink=1, currently 70 of 498 GB used.Proposal: share what is safe to share, drop what nobody needs, and keep runs isolated
npm_config_cache,UV_CACHE_DIR, andPIP_CACHE_DIRwhile we're at it) on the state volume. Both are built for concurrent use: npm's cacache writes content-addressed files atomically, and uv takes file locks. Saves 100–300 MB per run and the download time. Collision to design around:npxunpacks into<cache>/_npx, and two concurrentnpxruns of the same package can race there (ENOTEMPTY). Either keep_npxper run (npm_config_npx_cache, or a per-runnpm_config_cachewith only the tarball store shared), or check the npm version in the image for the fix.shared/opencode-plugin/<version>/on the state volume. Seed each run'shome/.config/opencode/node_modulesfrom it withcp -r --reflink=always: on XFS that is copy-on-write, so near-zero space and no shared mutable files (a hardlink copy would let one run's writes land in another's). opencode's own install then finds the dependency satisfied. Verify that opencode 1.18.21 skips the install whennode_modulesalready matches, and fall back to (1)'s shared cache if it doesn't. Saves about 155 MB per run.PYTEST_ADDOPTS=-o tmp_path_retention_policy=failed(pytest ≥ 7.3) in the run's environment, so passing tests leave nothing behind. And when a run finishes (succeeded, failed or stopped), empty itstmp/. Nothing in a finished run is using it; a resumed run's server re-creates it. Per run, nothing is shared, so nothing can collide.node_modules,.venv, build output, caches,tmp/) and keep the worktree's source,.git,.osf/,oc.dband the transcripts. A resume re-installs on demand: #60's continue-in-place keeps the edits, which are in the source. A stopped run on the ledger drops from about 700 MB to about 50 MB. Show it in the archive dialog as "slimmed".GET /api/runs/{id}/footprintand the archive dialog break the total down: dependencies, caches, temp, sessions, work. Record the breakdown onosf.run.archived, so a 3.4 GB archive explains itself afterwards.Considered, not proposed yet: reflink-seeding a new run's
node_modules/.venvfrom the last run of the same repository with the same lockfile. It's the biggest remaining saving (221 MB here), and on XFS it's safe (copy-on-write). Butnpm cideletesnode_modulesfirst, so it only pays off if agents usenpm install. Revisit with (5)'s numbers.Acceptance
npxof the same package.Verified since filing:
opencode's plugin install skips when it is already satisfied. From the 1.18.21 binary (
Npm.install→Npm.checkNodeModules→Npm.checkDirty), in the config folder:node_modulesis missing;package.json(plus the@opencode-ai/pluginit adds) is missing frompackage-lock.json's root.So seeding a run's
home/.config/opencode/withpackage.json,package-lock.jsonandnode_modulesfrom a per-version copy means no install and no npm cache. That removes all of the 155 MB each run pays at start. A reflink copy on the XFS volume keeps each run's files its own.What a run's opencode server actually runs with (
run_01M3GB3KN38YT9TZJ7KA7RQ5M0, PID 329):HOME=<run>/home,XDG_CONFIG_HOME=<run>/home/.config,XDG_DATA_HOME=<run>/xdg,XDG_CACHE_HOME=<run>/cache,XDG_STATE_HOME=<run>/state,TMPDIR=<run>/tmp,PLAYWRIGHT_BROWSERS_PATH=/opt/ms-playwright. There's nonpm_config_cache,UV_CACHE_DIRorPYTEST_ADDOPTS, so npm caches in<run>/home/.npm, uv in<run>/cache/uv, and pytest's temp trees go under<run>/tmp. osfd's own environment setsUV_LINK_MODE=copy; worth checking whether runs inherit it, because with a shared uv cache on the same XFS volume,UV_LINK_MODE=clonewould reflink packages into each.venvinstead of copying them.npm is 10.9.9 in the image. Before sharing the npm cache, test concurrent
npxof the same package against it (the_npxrace), or keep_npxper run.Still sampling the fresh run's growth per step; the breakdown follows when it ends.
Measured growth, step by step (
run_01M3GB3KN38YT9TZJ7KA7RQ5M0, soundcheck#32, sampled every 2 min)propose.proposalhome/.npm94, opencode pluginnode_modules62,.git6propose.spec+propose.designpropose.tasksopenspec.applyhome/.npm389,work/node_modules221, plugin 62,.git7openspec.applyinstalled the project's packages: +221 MB ofnode_modules, and npm's per-run cache grew by another 296 MB, a second copy of the same packages. Over half of a run's size is npm keeping two copies of everything, per run. The sampler keeps going to the end of the run.Also confirmed: a run's opencode server gets no
UV_*variables (osfd's ownUV_LINK_MODE=copyis not passed on), so uv runs with its defaults inside the run.Everything proposed, in one place, with what it saves and why runs cannot collide
npm_config_cache=/var/lib/openspecflow/shared/npm)home/.npm) plus the download timenpm ci/installacross runs is its designed use. Exception:npxunpacks into<cache>/_npx, and two concurrentnpxruns of one package can race (ENOTEMPTY). Give each run its ownnpm_config_npx_cache(or equivalent), or verify on npm 10.9.9 first.UV_CACHE_DIR) withUV_LINK_MODE=clone.venv, so an edit in one.venvwould change every run's.clone(reflink, XFSreflink=1) gives each run copy-on-write files.shared/opencode-plugin/<opencode version>/withcp -r --reflink=always(package.json,package-lock.json,node_modules)node_modulesexists and the lockfile matches. Reflink copies are copy-on-write, so no run can write into another's. Never hardlink this.PYTEST_ADDOPTS=-o tmp_path_retention_policy=failedin a run's environmenttmp/when a run ends (succeeded, failed or stopped)tmp/. Per run.node_modules,.venv, build output, caches,tmp/; keep source,.git,.osf/,oc.db, transcriptsGET /api/runs/{id}/footprint, the archive dialog andosf.run.archivednode_modules/.venvfrom the previous run of the same repository with the same lockfilenpm cideletesnode_modulesfirst, so it only pays off withnpm install. Revisit with (7)'s numbers.Already economical and unchanged:
git clone --local(hardlinked immutable objects), checkpoints through alternates, Playwright's browsers shared from the image (/opt/ms-playwright).With 1–3 a fresh run would start at under 30 MB, not 175, and reach
applyat about 230 MB (node_modulesplus the work), not 718. With 4–6 a finished run on the ledger would be about 50 MB.Acceptance, restated: two runs of the same repository at once, including concurrent
npxof one package, both install and test green on the shared caches, and neither sees the other's files.Final numbers for
run_01M3GB3KN38YT9TZJ7KA7RQ5M0(91 samples, 22:29 → 01:32; the run ended stopped):propose.proposalhome/.npm94, opencode pluginnode_modules62propose.tasksopenspec.applystartshome/.npm427,work/node_modules280,tmp/opencode70, pluginnode_modules62,tmp/pg-543346Planning took 22 MB. Implementation and testing took 785 MB. Almost all of that is regenerable:
node_modules;node_modules: 280 MB;The measures in the table above would take this run from 982 MB to about 300 MB while it ran:
tmp/at the end: −116 MB.(6) slimming would take it to tens of MB once stopped.