One corrupt WAL segment, and Prometheus quietly deleted its own history

A Prometheus volume looked almost full while retentionSize was set. The real story: a single corrupt WAL segment had stopped every checkpoint for weeks, the WAL grew to 48.7 GiB, and the size cap spent the difference by deleting old blocks.

The storage dashboard said the Prometheus volume was at 75.6 of 80 GB. That surprised me, because I’d set retentionSize: 60GiB precisely so this couldn’t happen. Either the cap wasn’t working or the number was wrong. It turned out to be a bit of both, and the actual fault was one I’d never have guessed from the dashboard.

The number on the dashboard was the wrong number

The 75.6 GB came from Longhorn’s actualSize. That’s the blocks the volume has allocated on the host, and it never goes down on its own. Delete a file inside the filesystem and Longhorn still counts its blocks until something trims them. So it tells you what the volume costs the host, not how full it is.

How full a PVC really is comes from the kubelet:

kubelet_volume_stats_used_bytes{persistentvolumeclaim="<prometheus-pvc>"}

That said 59.6 GiB, which is the retention cap. So the cap was doing its job. The question became what it was spending 60 GiB on.

Where the space was going

Prometheus reports its own storage in two parts:

prometheus_tsdb_wal_storage_size_bytes   # 48.7 GiB
prometheus_tsdb_storage_blocks_bytes     # 10.8 GiB

A healthy write-ahead log is a few GiB at most, because every compaction writes a checkpoint and truncates everything older than it. Mine had grown to 48.7 GiB and was still growing. The logs said why, once per compaction:

WAL truncation in Compact: create checkpoint: read segments:
corruption in segment /prometheus/wal/00000562 ... unexpected full record

prometheus_tsdb_wal_truncations_total was 0. One damaged segment meant no checkpoint could ever be written, so the WAL was never truncated. It doesn’t heal itself either. Every compaction hits the same segment and fails the same way.

This is the part I find nasty: retentionSize counts the WAL. With truncation broken, the WAL just kept growing, and to stay under the cap Prometheus deleted the oldest blocks, which is to say my history. The cap protected the disk and sacrificed the data, and nothing looked broken from the outside. Graphs still rendered. They just reached back less far than they should have.

There was an alert for it. PrometheusTSDBCompactionsFailing had been firing the whole time, and my Alertmanager routed everything to a null receiver, because I’d never set up mail. That’s now fixed. Alerts go out through a small SMTP relay in the cluster, which had a shared-hosting TLS trap of its own.

The fix: throw away the WAL, keep the blocks

The blocks were fine. Only the WAL and the in-memory head chunks were affected, so the repair is to stop Prometheus and delete exactly those two directories:

1. Suspend Flux for the monitoring stack, so it doesn’t undo the next step (flux suspend kustomization … and flux suspend helmrelease …).
2. Scale the Prometheus custom resource to spec.replicas: 0.
3. Start a throwaway pod with the same UID/GID as Prometheus that mounts the PVC, and remove wal/ and chunks_head/.
4. Scale back to 1 and resume Flux.

One detail to know: kube-prometheus-stack mounts the volume with subPath: prometheus-db, so on the raw volume the directories are under prometheus-db/, not at the root.

I lost an hour or two of head data that hadn’t been compacted yet. All ten blocks survived. Used space dropped to 10.8 GiB, and WAL replay on startup went from 7.5 minutes to 2 ms.

The two numbers worth watching from now on are prometheus_tsdb_wal_storage_size_bytes, which should stay in the low single-digit GiB, and prometheus_tsdb_checkpoint_creations_failed_total, which should stay at zero.

Meanwhile, the storage rebuild took out the control plane

Halfway through this, the cluster’s control plane started falling over. etcd logged “leader is overloaded likely from slow disk”, and the scheduler and controller manager were crash-looping.

It was the same Prometheus volume. It had lost a replica, and Longhorn was rebuilding it by copying an 80.7 GB snapshot, on the same consumer NVMe that also holds every etcd member. I’ve written about that disk before. What I didn’t know then is that Longhorn’s v1 engine has no way to throttle a replica rebuild. The replica-rebuilding-bandwidth-limit setting only applies to v2 volumes. A big enough rebuild on storage shared with etcd can take the control plane down, and there’s no knob to prevent it.

What does stop a rebuild is detaching the volume. Scaling Prometheus to zero for the WAL fix did exactly that, and the control plane pods stayed up from then on.

Getting the 80 GB back

After the fix the filesystem held under 11 GiB, but actualSize stayed around 80 GB. That’s because the rebuild snapshot was a full copy, and deleting files later doesn’t shrink a snapshot. Only a filesystem trim that is allowed to reach into snapshots marked as removed gives the space back.

Longhorn has a global setting for that (removeSnapshotsDuringFilesystemTrim), but I didn’t want it. It marks the latest snapshot and its ancestors as removed on every volume, and one of my volumes has daily snapshots that are its only point-in-time copies, because there’s no backup target yet (I’m working on that). The per-volume option unmapMarkSnapChainRemoved: enabled scopes it to the volume you choose, and the trim runs while the volume stays attached:

kubectl -n longhorn-system patch volume <volume> --type merge \
  -p '{"spec":{"unmapMarkSnapChainRemoved":"enabled"}}'
echo '{}' | kubectl create --raw \
  "/api/v1/namespaces/longhorn-system/services/longhorn-backend:9500/proxy/v1/volumes/<volume>?action=trimFilesystem" -f -

actualSize went from 80.8 GB to 13.7 GB in a few seconds, and Prometheus didn’t notice.

What’s still open is the underlying disk. etcd reads still take 100 ms to a second at times, with the guest disks nearly idle. That’s the single-NVMe problem again, and the WAL had nothing to do with it.