A backup stored on the volume it backs up is just snapshot churn

A 20 GiB Longhorn volume took 36.7 GiB per replica. Home Assistant wrote its daily 1.6 GB backup to the same volume, and seven daily snapshots each kept one. Sending backups off the volume fixed the growth.

A health check flagged Home Assistant’s config volume: 20 GiB in size, 36.7 GiB actually used per replica. Inside the pod, df said 5.5 GiB of the filesystem was in use. Where did the other 31 come from?

The snapshot chain

Longhorn’s actualSize is everything the volume’s replicas hold, and that includes the chain of snapshots. This volume had a recurring job taking a snapshot every day and keeping seven. Listing them showed the pattern:

kubectl -n longhorn-system get snapshots.longhorn.io \
  -l longhornvolume=<volume> \
  -o custom-columns=NAME:.metadata.name,CREATED:.status.creationTime,SIZE:.status.size

The oldest snapshot was 19.96 GB, which is roughly every block ever written to the volume. After that came about 2 GB per day, and one 8.8 GB day.

The 8.8 GB day fits Home Assistant’s monthly database repack, which rewrites the 3.8 GB SQLite file. The steady 2 GB a day was the interesting part.

The backup was the churn

Home Assistant makes a backup every night: a 1.6 GB tar file that includes the database. It was configured to store that in two places, an SFTP share on the NAS and locally, under /config/backups, with a local retention of one.

/config is the Longhorn volume. So every night Home Assistant wrote a new 1.6 GB file, all new blocks, and deleted yesterday’s. The deletion frees space inside the filesystem, but yesterday’s tar is still in yesterday’s snapshot. With seven snapshots kept, the volume carried about seven backup files’ worth of blocks per replica, and none of it was the useful kind. A backup inside the volume can’t survive the loss of the volume.

The fix was to stop the local copy and keep only the SFTP location. In the UI that’s the backup settings. I confirmed it from the WebSocket API’s backup/config/info, which listed only the SFTP agent afterwards. The next nightly backup went to the NAS only.

I removed the leftover local file with a plain rm in the pod, and /config went from 5.5 to 4.0 GiB used. The snapshot chain should shrink over the following week as the old snapshots roll off. I haven’t watched that happen yet.

Why the oldest snapshot doesn’t shrink

When the oldest snapshot is removed, Longhorn merges it into the next one. The blocks don’t go away unless something tells the filesystem layer that they’re free, so the base stays about the size of everything ever written. Trimming the volume’s filesystem is what makes that smaller. While I was looking, I found that 15 volumes on my cluster were outside the group my weekly trim job targets, this one included.

Snapshots aren’t backups either

The other thing this turned up: my cluster has no Longhorn backup target configured (the NAS one needs NFSv4 and is still parked), so the weekly backup job did nothing at all. The snapshots live on the same disks as the volume. The only real backup of Home Assistant is the one it makes itself, to the NAS, which is exactly the copy that was already working.