A thin pool only gets space back when the guest trims its own filesystem

An LVM thin pool on Proxmox was 85 % allocated while the data inside used a third of that. Longhorn's volume trim moved nothing. fstrim on each node's filesystem took the pool to 25 % in five minutes.

I needed to grow the OS disks of three Kubernetes VMs, and first checked the Proxmox host’s local-lvm thin pool. It held 815 GiB and was 85.5 % allocated, about 118 GiB free. Growing three disks by 30 GiB each looked tight.

But Longhorn, which lives on a 250 GiB data disk in each VM, reported only about 223 GiB used across all three. The pool thought 640 GiB of those data disks were in use.

Data% counts blocks ever written

A thin LV’s Data% in lvs isn’t how full the guest’s filesystem is. It’s the share of blocks that have ever been written. When the guest deletes a file, the filesystem marks the blocks free, but the thin pool doesn’t hear about it unless a discard comes down the whole stack. Without that, freed space is never returned to the pool. Months of Longhorn replica rebuilds and snapshot merges had written and freed a lot.

lvs -o lv_name,lv_size,data_percent pve

Discard was enabled on the virtual disks. What was missing was anyone sending it.

Two layers, and only one of them mattered

Longhorn has a trim operation per volume, trimFilesystem, that trims the filesystem inside a volume. I ran it on all 19 attached volumes through Longhorn’s API. Each returned HTTP 200 in under a second, and the pool stayed at exactly 85.46 %.

The big reclaim was one layer down. Longhorn stores its replicas as files on the node’s own filesystem, on the data disk. Deleted replica data frees blocks in that filesystem, and only an fstrim there passes the discards on to the virtual disk, and from there to the thin pool. On Talos, which has no shell, I ran it from a privileged pod per node with the host’s /var mounted:

fstrim -v /host/var/mnt/storage
fstrim -v /host/var

One node at a time, the pool went from 85.46 % to 68.57, 46.02 and finally 24.84 %. About 613 GiB free instead of 118, in five minutes, and nothing noticed. One Longhorn volume showed degraded for under a minute and recovered by itself.

Both layers still matter. The volume trim releases blocks inside a volume’s filesystem into the replica files, and the node trim passes them down to the pool. On my nodes, the second step had simply never run.

Two numbers that mislead

fstrim -v reported exactly 250 GiB trimmed on every node’s data disk. That’s the size of the free extents it submitted, not what the pool actually released. Read lvs before and after to see the real effect.

And a trim on the host doesn’t give a node more room for its kubelet. Free space inside the guests was the same before and after. The trim made room in the pool so that the disks could be grown, and growing them was the actual fix for the evictions that made me do this.

Before you overcommit

Even after the trim, the virtual sizes on that pool added up to nearly its full size, and the bigger OS disks pushed them over. Thin provisioning allows that. It’s only safe while real usage stays well below the pool size, because a full thin pool freezes every VM on it, and in my case that includes all three etcd members. A trim job that actually runs, inside the guests, is part of the price of overcommitting.