CI runner disk: small and synced writes stall concurrent jobs (4×HDD RAID 10 on XCP-ng) #52
Loading…
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Split out of #44. The CI runner VM (Forgejo Runner, 192.168.1.124) runs on XCP-ng, with its storage on a 4×HDD RAID 10. Its disk streams quickly but stalls on many small or synced writes, and the stalls slow and break other jobs.
What was measured (#44, PR #49 profiling)
Working explanation
A spinning drive pays roughly 12 ms each time it writes to a new spot, for the head seek plus the rotation. That's about 80 writes a second per drive, and about 160 for two mirrored pairs. A long stream pays that cost once, so it runs at platter speed. Thousands of small files, plus their metadata and the filesystem journal, pay it over and over. A synced write also has to reach the platter before it returns.
As far as I know, XCP-ng passes a VM's flushes through to the storage and has no per-disk cache mode. So the drives are probably the limit, not a setting.
To find out
fioinside the runner VM: sequential write, 4k random write, and 4k random write withfsync=1. Run the same tests in dom0 against the SR, to separate the VM's overhead from the drives'.Options, roughly cheapest first
config.ymlcontainer.options--tmpfs, the same flag Braid uses for/run/braidand its sandbox's/tmp. Then pointci.yml's venv, node_modules and Chromium at it. The runner has 62 GB of RAM.braid-cidoes, so no job unpacks them. The image layers are already on disk and only get read./var/lib/dockeronto an SSD, as a separate SR or a passed-through disk.Done when
fionumbers are recorded here.What the host and the runner VM show (2026-10-29)
Everything below was read-only: the kernel's disk counters inside the runner VM, and
xe,lsblk,lspci,dmesg, sysfs andsmartctl -iin dom0. Nothing was changed on either.The host's storage
/sys/class/scsi_disk/*/cache_type).sda. 326 VDIs; 3.7 of 5.3 TB used.The four virtual disks are
sda(5.4 TB),sdb(107 GB, dom0's boot disk),sdc(26.5 TB) andsdd(32.0 TB). The sizes add up to two RAID 10 groups of four drives, 32 TB each:sddfills one, andsda,sdbandsdcshare the other. iDRAC would confirm that.Inside the runner VM
/proc/diskstatsfor the VM's disk, in 10 s intervals while three CI jobs ran:Since the VM booted: 758,000 flushes at 2.1 ms on average, and writes at 7.0 ms on average. Up to 1.2 GB of unwritten data sat in the VM's memory across two samples. The VM's disk is a tapdisk (
vbd3) device that offers write barriers rather than separate cache flushes.What it means
sdaalone. This is option 4.Still open
fio, inside the VM and in dom0, as listed above.Correction (2026-10-30): the controller's write cache is on
iDRAC shows all four virtual disks set to Write Back, with the controller's battery Ready:
All four are RAID 10 with a 64K stripe. The "write through" in the comment above came from dom0's
/sys/class/scsi_disk/*/cache_type, which doesn't show the controller's write policy. What that changes:data. boot + data + vm = 29,803 GB, exactly data2's size. So there are two RAID 10 groups of four drives: data2 has one to itself, and the VM storage shares the other with the 24.7 TB data volume.Still open:
smartctl -g wcacheon each drive would show whether they're on. That matters for safety more than speed, because the controller's battery doesn't protect the drives' caches.fio, as listed above.