The old Docker host is empty, and the nodes finally have room
Progress on the homelab cluster over the last ten days: InfluxDB and the document archive moved in, the old Compose host runs nothing, the node disks grew from 20 to 50 GiB the hard way, and a few things moved out again.
The project page for my cluster still describes the storage layer and the Home Assistant move as the current work. Both are done, and the last ten days went into something else: emptying the old Docker Compose host that ran everything before the cluster existed.
Two databases and a document archive moved in
InfluxDB went first, because Home Assistant wrote to it and the old host’s address had to change before anything else could move. The copy itself was straightforward. Closing the gap between an 18-minute backup and the switch of the writer wasn’t, and I wrote up how I moved it without losing a point.
The document archive (Paperless-ngx with Postgres, Redis and two helper
services) was the biggest app left. Two of the things that broke along the
way have their own notes: a Compose command: is a Kubernetes
args:, and a
backup pipeline that can save a truncated dump as a good
one. The scanners send to
a fixed address over FTP and SMB, so the ingest pod took over the old host’s
address with a second interface on the
LAN, after the host itself had
moved to a new one. All 603 documents arrived, and a restore test in a scratch
namespace brought back the same 603.
Holding that ingest pod at zero replicas until the address was free taught me
that flux suspend doesn’t hold a Kustomization that another one
applies, which
corrects one of my older notes.
The offsite sync of the archive to MEGA moved to a Raspberry Pi, where it runs natively instead of in a container. That move had more surprises than I expected. The old host now manages no services, and its plaintext passwords are in the vault.
Bigger node disks, the hard way
Merging the document archive put every node under DiskPressure, because they had 19 GB system disks and its image pulls didn’t fit. I grew the disks to 50 GiB, which needed a trim inside the guests to make room in the thin pool first, and then a rolling reboot that went badly in its own ways. Both are in the post about the rollout. Every node now has about 30 GB free instead of 3 to 4.5.
Moving things out again
Not everything belongs in the cluster. The AI agent I’d deployed there earlier now runs natively on a Raspberry Pi of its own. Its dashboard keeps the old hostname and certificate, because the cluster’s ingress proxies to a host outside the cluster.
Housekeeping
A health check after the disk work turned up a few smaller things. Rolling reboots leave Deployment pods behind that nothing cleans up, so a small CronJob does it now. The chart’s empty etcd Service was a dead duplicate, but the obvious way to remove it would have removed the etcd alerts too. Home Assistant’s volume was nearly twice its size because its nightly backup lived on the volume it backed up. And the start page showed no tiles at all because of one dead Docker host.
Still open
I’ve built a health monitor that runs outside the cluster and reports to a chat bot, starting at Flux and drilling down only where something is wrong. Its tests pass and its read-only role is merged, but it isn’t deployed yet, and it still needs a real fault-injection test. One design lesson came early: Flux’s Ready condition can’t tell you how long something has been broken.
Beyond that: the old shared SSH password is vaulted but not yet rotated, the stale data directories on the NAS are waiting out their two-week grace period, and Longhorn still has no backup target.