Tagged storage

3 posts.

• One corrupt WAL segment, and Prometheus quietly deleted its own history26 Sept 2026

  A Prometheus volume looked almost full while retentionSize was set. The real story: a single corrupt WAL segment had stopped every checkpoint for weeks, the WAL grew to 48.7 GiB, and the size cap spent the difference by deleting old blocks.

• After the NVMe: what the fixes bought, and what the rolling restart cost25 Sept 2026

  A follow-up on the cluster whose etcd shared one consumer drive with every storage replica. Flush latency fell from 23 ms to 3 ms, a filtered copy shrank a 9.9 GB SQLite database to 3.8 GB in ten minutes, and turning on discard cost more in replica rebuilds than in reboots.

• Three etcd members, one consumer NVMe23 Sept 2026

  A scale-down that never happened led me to leader elections lost on every node, and from there to a single drive holding every etcd member and every storage replica. Three replicas on one disk mostly just triple the writes.