Part 2 of 2 in Three etcd members, one NVMe

After the NVMe: what the fixes bought, and what the rolling restart cost

A follow-up on the cluster whose etcd shared one consumer drive with every storage replica. Flush latency fell from 23 ms to 3 ms, a filtered copy shrank a 9.9 GB SQLite database to 3.8 GB in ten minutes, and turning on discard cost more in replica rebuilds than in reboots.

Two days ago I traced a sick control plane to three etcd members and every storage replica sharing one consumer NVMe. That post ended with a list of fixes. Since then I’ve put them in place, and this is what they did and what they cost me along the way.

Latency: 23 ms to 3 ms

The drive had been taking 22.6 ms per write and 22.9 ms per flush. I measured it the same way today, with two /proc/diskstats samples thirty seconds apart, and got 3.8 ms per write and 2.8 ms per flush. etcd’s health checks haven’t failed since, and the controller manager and scheduler have stopped losing their leases.

I can’t pin that on any single change. Between the two measurements the storage replica count went from three to two, I stopped recording the noisiest Home Assistant sensors, the big database purge ended, and a large replica rebuild finished. All four took writes off the same device. Whichever of them did most of the work, the conclusion from the first post still stands: the control plane was never short of throughput. It was stuck in a queue behind everyone else’s writes.

The database: a filtered copy instead of a delete

In the first post I suggested copying the rows you want to keep rather than deleting the ones you don’t. Now I have numbers for it. The Home Assistant recorder database was 9.89 GB and held 43.6 million state rows. Leaving out the sensors I’d stopped recording left 17.6 million rows, 40 %.

I stopped Home Assistant, made sure its pod was really gone, and ran the copy in a maintenance pod on the same volume:

1. Create the tables in a new file and attach it with journal_mode=OFF, because a half-written copy is simply deleted.
2. Copy every table. Filter states by metadata ID, and keep only the state_attributes rows that surviving states still reference.
3. Only then create the indexes, ANALYZE, and run PRAGMA quick_check plus a row-count comparison per table.
4. Swap the files: delete the old database with its -wal and -shm, and rename the new one.

It took 570 seconds and produced a 3.76 GB file. Two days earlier the row-by-row delete had been managing about 200,000 rows every 3 to 25 minutes. At that pace the 26 million rows would have taken days, and every one of those days would have been spent writing WAL to the drive etcd was waiting on.

Before you run a copy like this, count the rows you’re keeping so you can estimate the size of the result. The new file has to fit next to the old one. And don’t be alarmed when Home Assistant warns on its next start that it “could not validate that the sqlite3 database … was shutdown cleanly”. After a swap like this that warning is harmless.

Turning on discard: the reboots were cheap, the rebuilds were not

For the guests’ deletes to reach the thin pool, the VM disks need discard and ssd enabled. The hypervisor only picks that up on a cold start, so every Kubernetes node needed a full stop and start. I scripted it as a rolling restart, one VM at a time. Before moving on to the next, the script waited until the node was Ready, etcd was healthy and every attached storage volume was healthy.

Each node and its etcd member were back within about 30 seconds, so the reboots themselves were never the slow part. The storage replica rebuilds were. The second node took 23 minutes to count as healthy, because a volume replica on it had to resync. That’s why the script waits for storage and doesn’t settle for the node being Ready. With two replicas, taking down a second node in the middle of a rebuild leaves a volume without a single healthy copy.

Anything pinned to one node is down for that node’s entire cycle. Home Assistant has to run where its Zigbee USB stick is plugged in, so it was gone for about 13 minutes: the drain, the shutdown, the boot, and then several minutes of CSINode … does not contain driver driver.longhorn.io until the storage driver had registered again on the rebooted node. As long as it’s pinned, I don’t see a way around that.

The last surprise came from the hypervisor, which refused to start a VM that had only just stopped. The final qm start failed with timeout waiting on systemd, because the previous instance’s systemd scope was still being torn down. Retrying a minute later worked. A restart script should wait for the scope to go away, or retry, instead of aborting and leaving a node down.

Before any of that, the job had also aborted twice without touching anything. The cause turned out to be that a systemd service has no $HOME, and my health check had been written in a way that couldn’t tell me so. That one got its own page.

Still open

The trim itself hasn’t run yet. The restart job stopped before it reached the trim phase, so the thin pool is still at 84 %. Once every volume is healthy again, the trim runs on its own. The hardware fix from the first post, an NVMe with power-loss protection or at least a separate device for etcd, is still the real fix. Everything above only buys time.