The node that came back on the wrong address

Applying a new machine config to a three-node cluster rebooted each node in turn. One came back on a different IP, because its DHCP reservation had been wrong for months and the node had never rebooted to notice.

I had been working towards this apply for a while: the first time my Talos Ansible collection would touch the live cluster instead of the test one. The plan was boring on purpose. Take an etcd snapshot, copy the local Talos config directory somewhere safe, check Longhorn, Flux and pod health, run the whole thing in check mode once more, and only then apply, one node at a time.

The apply finished with a non-zero exit code. That alone didn’t worry me much, because the last step is a cluster health gate and it had run while the final node was still rebooting. What did worry me was that the first node wasn’t where I’d left it. It had come back on a different address, and it was the address of another service in the cluster, the Pi-hole load balancer.

A reservation nobody had tested

The nodes get their addresses from DHCP reservations on the router. I looked up the first node’s client record, and its fixed IP was the load balancer’s address, not the one I’d intended. The other two nodes had the right ones.

It had been wrong for a long time without anyone noticing. A node keeps its lease until it’s rebooted, and that node had held the correct address for 85 days. The reservation only mattered on the day it restarted, and that day was the day I was changing the machine config.

The wrong address was easy to see once I looked. It shows up in the etcd peer URL (talosctl etcd members) and in kubectl get nodes -o wide. I fixed the reservation by writing the whole client object back through the UniFi controller’s API, which I did only because I’d been told explicitly that I could, and then rebooted the node. It took the intended address and rejoined etcd. Talos had updated the peer URL by itself when the node came back on its old address, so there was no member surgery to do.

What I’d check next time

Before rebooting any node in a cluster that gets its addresses from DHCP, I’ll compare the reservations with the addresses I think the nodes have. If a node hasn’t rebooted in months, a stale lease can hide a wrong reservation, and the config change is a poor moment to find out.

Two smaller things from the same afternoon. Pods that were running on a node when it shut down gracefully stay behind in Error with “Pod was terminated in response to imminent node shutdown”. They’re harmless leftovers. And Flux reported etcdserver: request timed out for a while after the reboots; that turned out to be a different problem, and I wrote it up as finding what is hammering etcd’s disk. The cluster is part of the homelab project, and the collection’s wider story is in the post about moving the cluster onto it.