Part 1 of 2 in Three etcd members, one NVMe

Three etcd members, one consumer NVMe

A scale-down that never happened led me to leader elections lost on every node, and from there to a single drive holding every etcd member and every storage replica. Three replicas on one disk mostly just triple the writes.

It started with a command that did nothing. I scaled Home Assistant to zero so I could work on its database, and the pod just kept running. kubectl exec sessions dropped halfway through. Nothing reported an error anywhere.

The first real clue turned up in the Deployment itself:

kubectl -n <ns> get deploy <app> \
  -o jsonpath='{.metadata.generation} {.status.observedGeneration}{"\n"}'

Generation 10, observed generation 9. So the API server had accepted my change, but no controller had acted on it. When observedGeneration lags behind generation, the controller manager isn’t doing its job, and the pod you think you stopped is still running. Here that meant a supposedly stopped Home Assistant still had its SQLite database open while a maintenance pod opened the same file.

These days I check observedGeneration before I assume a scale or rollout actually happened. It’s the cheapest evidence you’ll get that the control plane is sick.

Following it down

Every layer I looked at pointed to the one underneath it. kubectl get --raw='/readyz/etcd' came back with context deadline exceeded. The etcd logs had hundreds of apply request took too long and slow fdatasync lines, plus one that spelled it out: leader failed to send out heartbeat on time; took too long, leader is overloaded likely from slow disk. The raft term was over a thousand, which means elections were happening constantly.

One level up, kube-controller-manager and kube-scheduler on all three nodes were logging leaderelection lost, with 20 to 26 restarts each over two days. That explained the scale-down nobody acted on. And every guest showed I/O pressure, with /proc/pressure/io “some” sitting at 51–87 %.

Then I found the number that gave it away. Each VM’s etcd disk was doing almost nothing, 0.3 MB/s and 57 writes a second, but at 30 ms per write and 35 ms per flush. All three VMs showed the same figures. Three independent disks don’t produce identical latency. One shared device does.

One drive under everything

On the Proxmox host, everything the cluster writes ends up on one consumer NVMe: all three etcd members (they live on the VMs’ OS disks) and all three Longhorn data disks, as logical volumes in the same thin pool. The drive was doing about 20 MB/s at 80 % utilisation, with 22.6 ms per write and 22.9 ms per flush. SMART was clean: 4 % wear, no media errors. So the drive wasn’t failing. It was just slow.

etcd wants fsync latency well under 10 ms. It commits every write with an fsync, which makes it the most latency-sensitive tenant on any disk it shares. On a consumer NVMe without power-loss protection, every flush is a real cache flush, and that’s where the 20 ms comes from. Enterprise drives with PLP acknowledge a flush from protected cache in well under a millisecond. That’s the property etcd needs. Throughput isn’t.

Two workloads pushed it over the edge. Longhorn was rebuilding an 80 GiB Prometheus replica, and Home Assistant was purging millions of rows from a 9 GB SQLite database, which was the reason I’d tried to scale it down in the first place.

Three replicas on one disk

On paper the setup looked redundant: three etcd members, three Longhorn replicas per volume, three nodes. But three VMs on one physical disk only protect you against losing a VM. They do nothing if the disk goes. What the replicas did reliably was triple every volume write onto the device etcd was waiting for.

A replica count is a promise that the copies sit in independent failure domains. When they don’t, the extra replicas are pure cost.

Measuring it without adding load

I got all of the numbers above by reading counters. Running a benchmark on a disk that was already saturated would only have made things worse. Two samples of /proc/diskstats taken ten seconds apart give you average latency per operation type as Δ(time spent) / Δ(operations completed):

cat /proc/diskstats; sleep 10; cat /proc/diskstats
# awk columns, after major, minor and device name:
# writes:  $8 (completed) and $11 (ms spent)
# flushes: $19 (completed) and $20 (ms spent)

On Talos, where there’s no shell on the node, talosctl read /proc/diskstats does the same job.

What the disk trouble did to Home Assistant

The database purge that added to the load had a cause of its own, and it’s a common one. Home Assistant’s daily automatic backup includes the database, so each backup came to about 4 GB, and it kept three of them on the same 20 GiB volume as the 9 GB database. The next backup filled the volume and died at 2.7 GB, and the recorder stopped with database or disk is full.

A few things I took from cleaning that up. If an off-box copy of the backup exists, keep just one local copy; it’s the N local copies on the data volume that fill it up. Don’t delete millions of rows in a single transaction on a nearly full volume, either. Every page you touch goes through the WAL first, so the WAL grows by the size of all of them. Making a filtered copy is faster and needs less headroom. And rather than running VACUUM in place, VACUUM INTO a new file and swap it in. That needs free space roughly the size of the new database, but you don’t get WAL growth on top of it.

What I’m doing about it

Roughly in order of cost: drop Longhorn’s default replica count to one or two, which cuts volume writes by up to two thirds; reduce Prometheus retention and scrape frequency; and turn on discard and ssd for the VM disks, so blocks the guests delete get trimmed and the drive has room for garbage collection. The real fix is hardware. That means an NVMe with power-loss protection, or at least a separate device for etcd and the OS disks so the storage layer can’t starve the control plane.

The finding I’d put first, though, is the one that nearly went wrong later. A comment in the infrastructure code said the Longhorn disks lived on the NAS. The hypervisor said they were on local storage. An apply driven by the comment’s value would have proposed replacing those disks, and with them every volume in the cluster. I now treat the hypervisor’s own config as the source of truth. A comment next to a variable is only somebody’s claim about it.