Active/passive failover on a single VIP, which I already had

Kubernetes has no notion of an active/passive Service. MetalLB in L2 mode with externalTrafficPolicy: Local is one, and I had been running it with a single replica the whole time.

Pi-hole is the DNS resolver the whole household actually uses. When it stops answering, nobody in the house can load a website, and I hear about it within minutes. It ran as a single Deployment with replicas: 1 and strategy: Recreate. In other words, it had no redundancy at all.

I started with a narrow question. Would replicas: 2 give me failover? And if it did, would the load balancer then spread queries across both pods? I explicitly didn’t want that, because two Pi-hole instances serving at the same time means two FTL processes writing the same sqlite databases.

The answer ended up being more interesting than either of those outcomes.

Why replicas: 2 doesn’t work

There are two separate problems. The first is storage. The volume is a ReadWriteOnce Longhorn volume, and in Longhorn RWO is scoped to a node. If the second pod lands on another node it doesn’t degrade gracefully. It fails with an explicit Multi-Attach error for volume ... Volume is already exclusively attached to one node.

The second problem is that even if both pods could mount the volume, I wouldn’t want them to. Two FTL processes working on one sqlite file will corrupt it sooner or later, probably at some quiet moment when nobody is watching.

So a second replica needs its own volume. That means a StatefulSet, which is routine work.

Where I had it wrong

I assumed the hard part would be traffic. With two pods, I expected Kubernetes to round-robin DNS queries between them, and I expected that preventing this would take something ugly.

It doesn’t. Two settings were already in place, and I had simply never thought about them together. The Service is exposed by MetalLB in L2 mode, and it carries externalTrafficPolicy: Local.

In L2 mode, MetalLB elects exactly one node to answer ARP for the virtual IP. Local makes that node forward only to a pod running on itself, never to another node. Combine the two and every query reaches exactly one pod. Nothing has to enforce that; it just follows from how the pieces fit. On top of that, MetalLB only considers nodes with a ready local endpoint as announcer candidates. When the active pod dies, the announcement moves by gratuitous ARP to the node that still has one.

That’s active/passive. Kubernetes has no first-class concept for it (there’s no “standby” field on a Service), yet this combination gives you exactly that as a side effect. I’d been running the mechanism correctly for months, with only one pod plugged into it.

So I didn’t have to build failover. I only had to add the second replica the design had been waiting for.

The numbers

After the cutover I killed the active pod and read the MetalLB speaker logs:

Failover took about 4 seconds and failback about 19. The third node, which runs no Pi-hole pod, never announced at any point. That’s the guarantee doing its job.

The VIP itself never changes. Only the EndpointSlice does, so every client keeps the DNS server address it was configured with. I had wondered whether the IP would switch during failover, and it turns out there’s nothing to pay for either way.

Things that don’t behave the way they look

I tried to use the same trick to pin the web UI to whichever pod was serving DNS. That can’t work, because externalTrafficPolicy: Local has no effect on cluster-internal traffic. Requests coming from the ingress controller go through ordinary kube-proxy routing, and externalTrafficPolicy only applies to traffic entering through a LoadBalancer or NodePort. I pinned the UI to a fixed pod instead.

It also turns out that MetalLB elects an announcer for each Service independently. Two Services that select the same pods don’t have to agree on which node is active, and there’s no selector for “the pod MetalLB is currently announcing from”. So I can’t express “service B follows service A”. Day to day this is mildly confusing: the dashboard is pinned to the first pod, while the second pod is often the one actually answering queries, so the statistics page can look suspiciously quiet.

Finally, none of this holds if you run MetalLB in BGP mode. BGP gives you ECMP, which load balances across all endpoints. That’s exactly the behaviour the L2 setup avoids, and the same manifests would quietly do the opposite.

Probing with dig exit codes

The obvious readiness probe for Pi-hole is an HTTP request to the admin page. It’s the wrong signal. FTL and the web server are separate processes, so a wedged resolver sitting behind a perfectly healthy lighttpd stays Ready, and MetalLB keeps announcing from a node that can’t resolve anything.

Running dig against the pod’s own resolver is a much more honest test, and its exit codes happen to fit the job:

• 0 whenever the server replies at all, including NXDOMAIN and SERVFAIL.
• 9 when it doesn’t reply.

That distinction matters more than it first seems. If the probe queried a real external name, it would go red during any upstream outage. Every replica would turn NotReady at the same moment, and the whole design would collapse at exactly the time I need it. Querying the local resolver checks whether the process works, which is the only thing the probe should care about.

An outage halfway through

In the middle of all this, household DNS went down. It wasn’t anything I was doing. I only noticed because a kubectl exec failed with container not found, and then I found the pod in ImagePullBackOff. It had been down for about ten minutes.

An earlier commit, cheerfully titled “updated pihole to current latest version”, had set an image tag that doesn’t exist. The newest tag in the registry was 2026.09.0; the commit said 2026.09.2. Pi-hole’s patch numbers skip values, so bumping one by hand is guesswork that only looks like arithmetic.

What turned a typo into an outage was strategy: Recreate. It destroys the running pod before pulling the replacement, so there’s no healthy pod left to keep serving while the new one fails. With a RollingUpdate, the old pod would have stayed up and the bad image would have sat in ImagePullBackOff forever. That would have been something to get alerted about, and nobody in the house would have noticed.

I can’t just blame Recreate, though. It’s forced there, because a single RWO volume can’t be attached to the old and new pods at once. For me that’s another argument for the multi-replica StatefulSet. I don’t really need two pods serving, but with one replica on an RWO volume, every image bump is an unguarded deploy.

Testing DNS from the wrong machine

One last trap, and this one cost me real time. My workstation shell is WSL2, and there both UDP and TCP port 53 are transparently intercepted. Queries to addresses in the documentation ranges, and to unassigned addresses on my own LAN, all came back with forged answers from a spoofed source. TCP connects to :53 on those same addresses succeed as well.

So anything that shell told me about DNS working was worthless, and for a few minutes it hid whether Pi-hole was serving at all. It’s cheap to detect: query a TEST-NET address that nothing should answer for, and see whether you get an answer anyway.

These days I test from inside the cluster.