CI runner disk: small and synced writes stall concurrent jobs (4×HDD RAID 10 on XCP-ng) #52

Open
opened 2026-10-29 21:14:15 -04:00 by cmoriarty · 2 comments
Owner

Split out of #44. The CI runner VM (Forgejo Runner, 192.168.1.124) runs on XCP-ng, with its storage on a 4×HDD RAID 10. Its disk streams quickly but stalls on many small or synced writes, and the stalls slow and break other jobs.

What was measured (#44, PR #49 profiling)

  • Sequential writes: 485 MB/s.
  • Small synced writes: 4.5 MB/s.
  • The fast lane's cache restore leaves about 750 MB of unwritten data: the downloaded archives, plus about 7,000 files of venv, node_modules and headless Chromium. That data drains at about 30 MB/s.
  • A publish build running at the same time waits 25–49 s on its 2 MB code copy. Alone, the copy takes 3 s (run 281).
  • The in-image e2e flaked in run 277 during one of these stalls ("'Hawks vs. Magic' is not shown on the page"; the feed fetch has a 10 s timeout). Publish is now gated on the e2e, so a flake like that holds a deploy.

Working explanation

A spinning drive pays roughly 12 ms each time it writes to a new spot, for the head seek plus the rotation. That's about 80 writes a second per drive, and about 160 for two mirrored pairs. A long stream pays that cost once, so it runs at platter speed. Thousands of small files, plus their metadata and the filesystem journal, pay it over and over. A synced write also has to reach the platter before it returns.

As far as I know, XCP-ng passes a VM's flushes through to the storage and has no per-disk cache mode. So the drives are probably the limit, not a setting.

To find out

  • RAID: a hardware controller (with a write cache? battery- or flash-backed?) or software RAID in dom0?
  • The SR type (LVM or EXT), and the drive model and rpm.
  • fio inside the runner VM: sequential write, 4k random write, and 4k random write with fsync=1. Run the same tests in dom0 against the SR, to separate the VM's overhead from the drives'.

Options, roughly cheapest first

  1. Don't write the throwaway data to disk at all. Mount a tmpfs into the CI job container: the runner's config.yml container.options --tmpfs, the same flag Braid uses for /run/braid and its sandbox's /tmp. Then point ci.yml's venv, node_modules and Chromium at it. The runner has 62 GB of RAM.
  2. Bake the dependencies into a CI job image, as braid-ci does, so no job unpacks them. The image layers are already on disk and only get read.
  3. Move the runner's /var/lib/docker onto an SSD, as a separate SR or a passed-through disk.
  4. Turn on the controller's write cache, if it has one with battery or flash backup.

Done when

  • A publish build running alongside the fast lane copies its code in under 5 s.
  • The fio numbers are recorded here.
Split out of #44. The CI runner VM (Forgejo Runner, 192.168.1.124) runs on XCP-ng, with its storage on a 4×HDD RAID 10. Its disk streams quickly but stalls on many small or synced writes, and the stalls slow and break other jobs. ## What was measured (#44, PR #49 profiling) - **Sequential writes:** 485 MB/s. - **Small synced writes:** 4.5 MB/s. - **The fast lane's cache restore** leaves about 750 MB of unwritten data: the downloaded archives, plus about 7,000 files of venv, node_modules and headless Chromium. That data drains at about 30 MB/s. - **A publish build running at the same time** waits 25–49 s on its 2 MB code copy. Alone, the copy takes 3 s (run 281). - **The in-image e2e flaked in run 277** during one of these stalls ("'Hawks vs. Magic' is not shown on the page"; the feed fetch has a 10 s timeout). Publish is now gated on the e2e, so a flake like that holds a deploy. ## Working explanation A spinning drive pays roughly 12 ms each time it writes to a new spot, for the head seek plus the rotation. That's about 80 writes a second per drive, and about 160 for two mirrored pairs. A long stream pays that cost once, so it runs at platter speed. Thousands of small files, plus their metadata and the filesystem journal, pay it over and over. A synced write also has to reach the platter before it returns. As far as I know, XCP-ng passes a VM's flushes through to the storage and has no per-disk cache mode. So the drives are probably the limit, not a setting. ## To find out - [x] RAID: a hardware controller (with a write cache? battery- or flash-backed?) or software RAID in dom0? - [x] The SR type (LVM or EXT), and the drive model and rpm. - [ ] `fio` inside the runner VM: sequential write, 4k random write, and 4k random write with `fsync=1`. Run the same tests in dom0 against the SR, to separate the VM's overhead from the drives'. ## Options, roughly cheapest first 1. **Don't write the throwaway data to disk at all.** Mount a tmpfs into the CI job container: the runner's `config.yml` `container.options` `--tmpfs`, the same flag Braid uses for `/run/braid` and its sandbox's `/tmp`. Then point `ci.yml`'s venv, node_modules and Chromium at it. The runner has 62 GB of RAM. 2. **Bake the dependencies into a CI job image**, as `braid-ci` does, so no job unpacks them. The image layers are already on disk and only get read. 3. **Move the runner's `/var/lib/docker` onto an SSD**, as a separate SR or a passed-through disk. 4. **Turn on the controller's write cache**, if it has one with battery or flash backup. ## Done when - A publish build running alongside the fast lane copies its code in under 5 s. - The `fio` numbers are recorded here.
Author
Owner

What the host and the runner VM show (2026-10-29)

Everything below was read-only: the kernel's disk counters inside the runner VM, and xe, lsblk, lspci, dmesg, sysfs and smartctl -i in dom0. Nothing was changed on either.

The host's storage

RAID Hardware: a Dell PERC H730 (LSI MegaRAID SAS-3 3108) with a 1 GB cache. No software RAID in dom0.
Cache mode Write through on all four of the controller's virtual disks (/sys/class/scsi_disk/*/cache_type).
Drives 8 × 16 TB, 7200 rpm SATA: 4 × Toshiba MG08ACA16TEY, 4 × Seagate Exos X16 ST16000NM005G. No SSD in the host.
The runner's SR "Local storage", type EXT (VHD files on a filesystem in dom0), on the virtual disk sda. 326 VDIs; 3.7 of 5.3 TB used.

The four virtual disks are sda (5.4 TB), sdb (107 GB, dom0's boot disk), sdc (26.5 TB) and sdd (32.0 TB). The sizes add up to two RAID 10 groups of four drives, 32 TB each: sdd fills one, and sda, sdb and sdc share the other. iDRAC would confirm that.

Inside the runner VM

/proc/diskstats for the VM's disk, in 10 s intervals while three CI jobs ran:

Interval Writes/s MB/s ms per write Flushes/s ms per flush
1 375 9.7 15.6 12 39.5
2 647 15.9 5.9 16 14.5
3 1,374 36.8 4.3 89 1.8
4 406 10.8 16.4 17 16.8
5 1,726 26.4 2.8 455 0.5
6 63 0.9 0.2 22 0.1

Since the VM booted: 758,000 flushes at 2.1 ms on average, and writes at 7.0 ms on average. Up to 1.2 GB of unwritten data sat in the VM's memory across two samples. The VM's disk is a tapdisk (vbd3) device that offers write barriers rather than separate cache flushes.

What it means

  • The drives are the limit because the controller's cache isn't used for writes. In write-through mode the controller reports a write done only once the drives have it. A scattered write on a 7200 rpm drive costs a seek plus half a rotation, about 12 ms, in line with the slow intervals' 15–16 ms per write. So the working explanation above is right about the drives, but a setting is involved.
  • The flush isn't the slow part. Flushes are slow exactly when writes are, and nearly free otherwise. Each write is already on the drives when it completes, so a flush only waits for the writes queued ahead of it. That's how a small synced write ends up waiting behind another job's backlog.
  • With write-back on, a small synced write would complete in the controller's cache in well under a millisecond while the cache has room, and the controller could reorder scattered writes before they reach the drives. The write policy is set per virtual disk, so it could be turned on for sda alone. This is option 4.

Still open

  • Why the cache is write-through: configured that way, or forced because its battery has failed or is charging. iDRAC's storage pages show each virtual disk's write policy and the controller's battery. If the battery is bad, it needs replacing before anything else changes. Never use Force Write Back.
  • Whether the drives' own write caches are on. dom0 sends these volumes no cache flushes, since they report write-through, so if the drive caches are on, writes already reported done could be lost in a power cut. iDRAC shows this as each virtual disk's disk cache policy.
  • fio, inside the VM and in dom0, as listed above.
## What the host and the runner VM show (2026-10-29) Everything below was read-only: the kernel's disk counters inside the runner VM, and `xe`, `lsblk`, `lspci`, `dmesg`, sysfs and `smartctl -i` in dom0. Nothing was changed on either. ### The host's storage | | | |---|---| | **RAID** | Hardware: a Dell PERC H730 (LSI MegaRAID SAS-3 3108) with a 1 GB cache. No software RAID in dom0. | | **Cache mode** | **Write through** on all four of the controller's virtual disks (`/sys/class/scsi_disk/*/cache_type`). | | **Drives** | 8 × 16 TB, 7200 rpm SATA: 4 × Toshiba MG08ACA16TEY, 4 × Seagate Exos X16 ST16000NM005G. No SSD in the host. | | **The runner's SR** | "Local storage", type **EXT** (VHD files on a filesystem in dom0), on the virtual disk `sda`. 326 VDIs; 3.7 of 5.3 TB used. | The four virtual disks are `sda` (5.4 TB), `sdb` (107 GB, dom0's boot disk), `sdc` (26.5 TB) and `sdd` (32.0 TB). The sizes add up to two RAID 10 groups of four drives, 32 TB each: `sdd` fills one, and `sda`, `sdb` and `sdc` share the other. iDRAC would confirm that. ### Inside the runner VM `/proc/diskstats` for the VM's disk, in 10 s intervals while three CI jobs ran: | Interval | Writes/s | MB/s | ms per write | Flushes/s | ms per flush | |---|---|---|---|---|---| | 1 | 375 | 9.7 | 15.6 | 12 | 39.5 | | 2 | 647 | 15.9 | 5.9 | 16 | 14.5 | | 3 | 1,374 | 36.8 | 4.3 | 89 | 1.8 | | 4 | 406 | 10.8 | 16.4 | 17 | 16.8 | | 5 | 1,726 | 26.4 | 2.8 | 455 | 0.5 | | 6 | 63 | 0.9 | 0.2 | 22 | 0.1 | Since the VM booted: 758,000 flushes at 2.1 ms on average, and writes at 7.0 ms on average. Up to 1.2 GB of unwritten data sat in the VM's memory across two samples. The VM's disk is a tapdisk (`vbd3`) device that offers write barriers rather than separate cache flushes. ### What it means - **The drives are the limit because the controller's cache isn't used for writes.** In write-through mode the controller reports a write done only once the drives have it. A scattered write on a 7200 rpm drive costs a seek plus half a rotation, about 12 ms, in line with the slow intervals' 15–16 ms per write. So the working explanation above is right about the drives, but a setting is involved. - **The flush isn't the slow part.** Flushes are slow exactly when writes are, and nearly free otherwise. Each write is already on the drives when it completes, so a flush only waits for the writes queued ahead of it. That's how a small synced write ends up waiting behind another job's backlog. - **With write-back on,** a small synced write would complete in the controller's cache in well under a millisecond while the cache has room, and the controller could reorder scattered writes before they reach the drives. The write policy is set per virtual disk, so it could be turned on for `sda` alone. This is option 4. ### Still open - **Why the cache is write-through:** configured that way, or forced because its battery has failed or is charging. iDRAC's storage pages show each virtual disk's write policy and the controller's battery. If the battery is bad, it needs replacing before anything else changes. Never use Force Write Back. - **Whether the drives' own write caches are on.** dom0 sends these volumes no cache flushes, since they report write-through, so if the drive caches are on, writes already reported done could be lost in a power cut. iDRAC shows this as each virtual disk's disk cache policy. - **`fio`**, inside the VM and in dom0, as listed above.
Author
Owner

Correction (2026-10-30): the controller's write cache is on

iDRAC shows all four virtual disks set to Write Back, with the controller's battery Ready:

Virtual disk Size Read policy Write policy
boot 100 GB Read Ahead Write Back
data 24,703 GB Read Ahead Write Back
data2 29,803 GB Read Ahead Write Back
vm 5,000 GB Adaptive Read Ahead Write Back

All four are RAID 10 with a 64K stripe. The "write through" in the comment above came from dom0's /sys/class/scsi_disk/*/cache_type, which doesn't show the controller's write policy. What that changes:

  • The cache is taking writes. That fits the quietest interval above, where writes took 0.2 ms; no spinning drive does that on its own.
  • Option 4 is already in place. There's nothing to turn on.
  • The slow intervals are most likely the cache full. When scattered writes arrive faster than the drives can take them, the 1 GB cache, which all four virtual disks share, fills up, and writes go at the drives' pace: about 15 ms each. A flush then waits behind them. So the working explanation stands: under sustained scattered writes, the drives are the limit.
  • The VM storage shares its drives with data. boot + data + vm = 29,803 GB, exactly data2's size. So there are two RAID 10 groups of four drives: data2 has one to itself, and the VM storage shares the other with the 24.7 TB data volume.

Still open:

  • The drives' own write caches. The vm volume's disk cache policy is Default, which leaves each drive's cache at its factory setting. smartctl -g wcache on each drive would show whether they're on. That matters for safety more than speed, because the controller's battery doesn't protect the drives' caches.
  • fio, as listed above.
## Correction (2026-10-30): the controller's write cache is on iDRAC shows all four virtual disks set to **Write Back**, with the controller's battery **Ready**: | Virtual disk | Size | Read policy | Write policy | |---|---|---|---| | boot | 100 GB | Read Ahead | Write Back | | data | 24,703 GB | Read Ahead | Write Back | | data2 | 29,803 GB | Read Ahead | Write Back | | vm | 5,000 GB | Adaptive Read Ahead | Write Back | All four are RAID 10 with a 64K stripe. The "write through" in the comment above came from dom0's `/sys/class/scsi_disk/*/cache_type`, which doesn't show the controller's write policy. What that changes: - **The cache is taking writes.** That fits the quietest interval above, where writes took 0.2 ms; no spinning drive does that on its own. - **Option 4 is already in place.** There's nothing to turn on. - **The slow intervals are most likely the cache full.** When scattered writes arrive faster than the drives can take them, the 1 GB cache, which all four virtual disks share, fills up, and writes go at the drives' pace: about 15 ms each. A flush then waits behind them. So the working explanation stands: under sustained scattered writes, the drives are the limit. - **The VM storage shares its drives with `data`.** boot + data + vm = 29,803 GB, exactly data2's size. So there are two RAID 10 groups of four drives: data2 has one to itself, and the VM storage shares the other with the 24.7 TB data volume. Still open: - **The drives' own write caches.** The vm volume's disk cache policy is Default, which leaves each drive's cache at its factory setting. `smartctl -g wcache` on each drive would show whether they're on. That matters for safety more than speed, because the controller's battery doesn't protect the drives' caches. - **`fio`**, as listed above.
Sign in to join this conversation.
No labels
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
cmoriarty/sports-breaker#52
No description provided.