Finding what is hammering etcd's disk on Talos

After a reboot Flux reported etcd timeouts and etcd warned about a slow disk. Talos has no per-process I/O view, but /proc/<pid>/io can be read through talosctl; here it pointed at a Longhorn snapshot purge.

After rebooting every node of a three-node Talos cluster, Flux kept reporting etcdserver: request timed out, and etcd logged “leader … likely from slow disk”. All three VMs shared one NVMe through an LVM-thin pool on the Proxmox host, so any bulk write on any of them shows up as etcd fsync latency. Host I/O pressure was between 15 and 27 percent, at only about 25 MB/s. That’s not much traffic, but etcd cares about flush latency, and it doesn’t take much to hurt it.

Per-process I/O without a shell

talosctl has no per-process I/O view, and the nodes have no shell. What does work is reading /proc/<pid>/io through talosctl read. I took the PIDs from talosctl processes, sampled each file twice per node, and ran the reads in parallel (xargs -P 16), because one process at a time is slow. The difference between the two samples is the rate.

The busiest process on the replica nodes was a Longhorn sync-agent, doing roughly 6 MB/s of reads and writes together on each node holding a replica.

It was a snapshot purge

Longhorn was purging snapshots on an 80 GiB Prometheus volume after a replica rebuild. The engine objects say so:

kubectl -n longhorn-system get engines.longhorn.io -o json

Look at status.purgeStatus and status.rebuildStatus. The purge was at 46 percent and progressing at about 0.7 percent a minute, so roughly an hour to go. I found no clean way to cancel it, and it ended on its own. Slow-disk warnings from etcd fell from around 700 a minute per node to 110 to 140 while I was watching, and I counted them with talosctl logs etcd --tail 400 | grep -c "took too long".

The lesson isn’t about Longhorn. When etcd is slow after a disruption, find out who else is writing to the same device before touching etcd itself.