Giving a pod the LAN address another host still holds

MetalLB only hands out addresses from its own pool, so a pod that has to own a LAN address (a scanner target, say) needs a macvlan interface. The hard part is the handover: two hosts answering ARP for one address break both.

My document scanner uploads to a fixed address on the home LAN, over FTP and SMB. That address belonged to the old server. When I moved the document system into the cluster, the new ingest pod had to answer on exactly that address, because the scanner’s setting isn’t something I wanted to touch.

Why not a LoadBalancer

A MetalLB pool can only contain addresses MetalLB can announce on the nodes’ own subnet. Mine is the cluster VLAN, so a Service of type LoadBalancer can never be handed an address on the household LAN. A pod that must own a LAN address needs a second interface instead. I used a Multus macvlan attachment with static IPAM and no gateway, which I describe in the note on putting a pod on the LAN. FTP and SMB can share one pod, and so one address.

The pod ships switched off

Two machines answering ARP for one address break both of them, for a while and unpredictably. So the Deployment goes into Git with replicas: 0, and scales to one only after the old host has given the address up.

I wanted the scale-up to stay under my control, and Flux didn’t help. A manually suspended child Kustomization gets un-suspended by the parent within minutes, which re-applies replicas: 0 or whatever Git says. What held was the annotation kustomize.toolkit.fluxcd.io/reconcile=disabled on the single Deployment, until the pull request with the new state was merged.

The handover order

The old host’s address changes first, and it has to be possible to undo each step:

1. Reserve a new address for the old host in the router or DHCP server, and check that it is free.
2. Add the new address next to the old one (nmcli con mod <con> +ipv4.addresses <new>/22) and confirm SSH works on the new address.
3. From that new session, remove the old address. That ends the old session, so don’t do it from the old one.
4. Check from the workstation that the old address no longer answers.
5. Only now scale the pod to one.

Test from the workstation, not from the node that runs the pod. A macvlan pod’s own node can’t reach its address, which looks exactly like a broken setup.

Things that broke on the way

The FTP image I used (kibatic/proftpd) crashed in a fresh container. Its entrypoint runs useradd --gid 33 and then chown paperless:paperless, but no group of that name exists, so it exits with status 1. The old server only worked because its long-lived container had been set up earlier. Creating the group first, with groupadd -o -g 33 paperless, before the entrypoint runs, fixed it.

For testing, curl smb:// can’t talk to a Samba with SMB1 disabled and just says it could not connect. smbclient works, and the Samba image has it. And a fresh volume’s root directory is mode 0755, which the application’s init script recursively chowns but doesn’t loosen, so the FTP user (a different uid) was refused until an init container ran chmod 0777.